A Language-Guided Multi-Attribute Quality Assessment Method for Medical Images
By constructing an image quality assessment network with visual and text branches, the problem that existing medical image quality assessment methods are difficult to adapt to complex distortions and multi-attribute modeling is solved, achieving high accuracy and stable image quality assessment and improving clinical usability.
Patent Information
- Application Number
- CN202611342680.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-01
- Publication Date
- 2026-11-17
AI Technical Summary
Existing medical image quality assessment methods are ill-suited to complex distortions in images like colonoscopy videos, lack multi-attribute joint modeling, and the assessment results deviate from clinical usability.
A language-guided multi-attribute quality assessment method for medical images is adopted. By constructing an image quality assessment network including visual and text branches, the visual branch is used to extract spatiotemporal features and the text branch is combined to provide semantic supervision, thereby achieving multi-attribute quality assessment.
It improves the accuracy and stability of medical image quality assessment, enhances the clinical reliability of model predictions, and can effectively capture image quality in complex distortion scenarios to meet the needs of clinical decision support.
Smart Images

Figure CN123118545A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, specifically to a language-guided method for evaluating the quality of multiple attributes in medical images. Background Technology
[0002] Medical imaging is a crucial information carrier in clinical diagnosis, disease screening, and treatment evaluation. Its image quality directly affects doctors' observation and judgment of lesion areas, tissue boundaries, and abnormal morphologies. However, in actual image acquisition, it is easily affected by factors such as uneven lighting, excessively low or high brightness, specular reflection, field of view obstruction, blurring, and unstable lens movement, leading to a decline in image quality and consequently affecting the clarity and reliability of clinical observation.
[0003] Existing methods for assessing medical image quality mainly include subjective human evaluation, traditional indicator evaluation, and automated evaluation methods based on deep learning. Subjective human evaluation relies on the experience of physicians or professionals. While it can reflect certain clinical needs, it suffers from problems such as being time-consuming, costly, and highly subjective, making it difficult to meet the needs of large-scale data processing and real-time quality monitoring. Traditional indicator evaluation methods are typically based on manual features such as brightness, contrast, sharpness, noise, and structural similarity. However, these methods are mostly geared towards natural or static images and are ill-suited to the complex medical imaging features and various distortion factors present in images like colonoscopy videos. In recent years, while deep learning methods can automatically extract quality-related features and improve the objectivity and accuracy of evaluations, existing methods lack joint modeling of multiple attributes. Furthermore, most methods primarily rely on visual features and do not fully incorporate semantic information about quality, potentially leading to discrepancies between the final evaluation results and clinical usability, thus limiting their application in actual clinical decision support. Summary of the Invention
[0004] The purpose of this invention is to provide a language-guided multi-attribute quality assessment method for medical images, in order to solve the problems of existing medical image quality assessment methods, such as difficulty in adapting to complex distortions in videos like colonoscopy, lack of multi-attribute joint modeling, and inability of assessment results to meet actual clinical needs.
[0005] The objective of this invention is achieved as follows: The present invention provides a language-guided method for multi-attribute quality assessment of medical images, comprising the following steps: S1. Obtain the medical image dataset and perform preprocessing; S2. Construct an image quality assessment network that includes visual and textual branches: The visual branch includes an image encoder module, a video spatiotemporal encoding module, a feature fusion module, a graph attention inference module, and a DSMF module; the text branch includes image and text encoding modules. After medical image videos are processed by the image encoder module, video spatiotemporal coding module, feature fusion module, graph attention inference module and DSMF module of the visual branch, different dimensions and overall quality scores are output. S3. The image quality assessment network is trained using the training set. During the training phase, the frame-level visual embeddings extracted by the image encoding module of the text branch and the text embeddings of the image quality semantics extracted by the text encoding module are mapped to the same shared feature space to complete similarity modeling. The Fidelity loss is used to constrain the training of the visual branch. After training, the visual branch serves as the main image quality assessment model, and the text branch assists in supervising the visual branch. S4. Input the medical image video to be tested into the image quality evaluation model, and calculate and output the quality scores of different dimensions and the overall quality.
[0006] In step S1, the medical image dataset used is the MAC-VQA dataset. Each medical image video in the MAC-VQA dataset is labeled with five distortion attributes: specular reflection, brightness, utility, spatiotemporal visibility, and stability. The dataset also provides the average opinion score for the overall quality and the average opinion score for each distortion attribute.
[0007] In step S2, the specific method for outputting the overall quality score of the medical image video through the visual branch is as follows: The image encoder module is used to extract the static spatial features of a single frame, and the video spatiotemporal coding module is used to extract spatiotemporal features. The spatial features and spatiotemporal features are fused by the feature fusion module and then generated into multiple sets of attribute-aware features through multiple parallel MLP paths. The graph attention inference module employs a multi-head graph attention mechanism to process multiple sets of attribute-aware features, outputting multiple sets of attribute-enhanced features. Each set of attribute-enhanced features is input into a regression head, outputting the sub-item quality score of the corresponding distorted attribute. Simultaneously, the multiple sets of attribute-enhanced features are first divided into dynamic and static perception streams. The attribute-enhanced features within each perception stream are then fused to obtain dynamic and static features. Finally, the dynamic and static features are subjected to bidirectional cross-attention and weighted fusion with the MoE module to output the overall quality score.
[0008] The visual branch includes a visual feature extraction unit and a cross-feature relationship reasoning unit. The image encoder module, video spatiotemporal coding module, and feature fusion module are integrated in the visual feature extraction unit; the graph attention inference module and DSMF module are integrated in the cross-feature relationship inference unit.
[0009] The medical image video needs to be processed before being input into the image encoder module. The specific method is as follows: each medical image video is sampled uniformly in the time dimension for several frames, and each frame image is scaled to a preset fixed pixel size before being input into the image encoder module.
[0010] Among them, medical image videos need to be processed before being input into the video spatiotemporal coding module. The specific method is as follows: First, each frame of the video is divided into several non-overlapping spatial grids. Then, a single grid is randomly selected from all the spatial grids. A fixed pixel image block is randomly cropped out from the selected grid to obtain the spatial image block corresponding to a single frame. The video is evenly divided into a preset number of time segments along the time axis, and multiple consecutive frames of images are sampled from each time segment; The spatial image blocks corresponding to the temporally continuous images within each temporal segment are aligned according to frame temporal sequence to generate compact spatiotemporal segments that preserve the continuity of local motion; all compact spatiotemporal segments are spliced together in the spatial dimension, and the spatiotemporal tensor obtained after splicing is input into the video spatiotemporal coding module.
[0011] The specific method for fusing spatial and spatiotemporal features through the feature fusion module is as follows: The spatial features and spatiotemporal features have different channel dimensions; channel dimension alignment is performed through the feature fusion module to map features with high channel dimensions to low channel dimensions; the two types of features after channel alignment are spliced and fused to obtain spliced and fused features; the spliced and fused features are then reshaped in dimension.
[0012] The total loss function used for iterative training of the visual branch of the medical image quality evaluation network includes an overall quality loss term, an attribute quality loss term, a ranking consistency loss term, and a load balancing loss term, which are weighted together.
[0013] The total loss function is: in, and These represent the subjective score and predicted value of the overall quality, respectively. and This represents the subjective scores and predicted values for each quality attribute. Mean squared error is used to constrain the difference between the prediction results and the subjective scores. and It is a trade-off parameter used to balance the sorting loss and the load balancing regularization term, respectively.
[0014] The image encoder module uses a frozen image encoder module, and the text branch uses a frozen image and text encoding module.
[0015] The beneficial effects of this invention are as follows: The visual branch of this invention integrates a visual feature extraction unit and a cross-feature relation inference unit, enabling the model to capture spatiotemporal representations and dependencies between attributes. It balances global spatial details with local motion temporal information, achieving a balance between computational efficiency and power efficiency. Furthermore, this application implements a dual-branch collaborative network structure of visual and text branches. The independent text branch, as a cross-modal semantic auxiliary supervision branch, provides linguistic semantic constraints on static distortion attributes, allowing the model to learn the linguistic logic of human doctors judging image quality. This enhances the stability of model predictions and addresses the shortcomings of insufficient interpretability in pure visual model predictions and clinical subjective MOS scores.
[0016] Furthermore, this invention sets up parallel single-frame image encoder and video spatiotemporal coding dual extraction branches. The image encoder relies on CLIP pre-trained ResNet50 to extract pure static spatial features frame by frame, while the video spatiotemporal coding module extracts spatiotemporal features based on a segmented and block-based sampling strategy. This can efficiently capture dynamic distortions such as lens shake, motion blur, and long-term brightness drift, effectively improving the model evaluation accuracy in complex composite distortion scenarios. Attached Figure Description
[0017] Figure 1 This is a network framework diagram of the present invention; Figure 2 This is a schematic diagram of the structure of the Graphical Attention Reasoning (GAR) module of the present invention; Figure 3 This is a schematic diagram of the structure of the dynamic and static hybrid expert system fusion module (DSMF) of the present invention; Figure 4 This is a schematic diagram of the text branching architecture of the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] like Figure 1 As shown, a language-guided multi-attribute quality assessment method for medical images according to the present invention includes the following steps: S1. Obtain the medical image dataset and perform preprocessing.
[0020] In step S1, the medical image dataset used is the MAC-VQA dataset. Each medical image video is labeled with five distortion attributes: specular reflection, brightness, utility, spatiotemporal visibility, and stability. The dataset also provides the mean opinion score for overall quality and the mean opinion score for each distortion attribute. Specifically, the MAC-VQA dataset includes 1000 realistically distorted colonoscopy videos. These 1000 colonoscopy videos were extracted from 40 long videos taken by 40 patients (24 men and 16 women, aged 20-69 years). Each video is 9-11 seconds long, with a resolution of 1920×1080 and a frame rate of 24. These videos were subjectively annotated by 15 participants, providing the mean opinion score (MOS ranging from 1 to 5) for overall quality and the five distortion attributes: specular reflection, brightness, utility, spatiotemporal visibility, and stability (MOS ranging from 0 to 3).
[0021] Preprocessing of medical image datasets includes dividing the dataset into training and testing sets.
[0022] S2. Construct an image quality evaluation network that includes visual and text branches, where the visual and text branches run in parallel.
[0023] The visual branch includes an image encoder module, a video spatiotemporal encoding module, a feature fusion module, a graph attention inference module, and a DSMF module; the text branch includes image and text encoding modules.
[0024] The image encoder module, video spatiotemporal coding module, and feature fusion module are integrated into the visual feature extraction unit; the graph attention inference module and DSMF module are integrated into the cross-feature relation inference unit (CFRR). In other words, the visual branch integrates the visual feature extraction unit (VFE) and the cross-feature relation inference unit (CFRR), enabling the model to capture spatiotemporal representations and dependencies between attributes, and to resolve complex interactions between various distorted features. This redefines the quality assessment of medical imaging examinations as a multi-attribute inference problem, rather than a simple feature regression task.
[0025] like Figure 1 , Figure 4 As shown, the text branch provides supervision by using text templates to describe frame-level distortion attributes and quality levels, providing high-level semantic cues that contribute to more accurate and robust quality predictions.
[0026] In this embodiment, the image encoder module adopts CLIP-ResNet50 (i.e., CLIP-based ResNet50 network), the video spatiotemporal coding module adopts Video-Swin-T (i.e., lightweight Video Swin Transformer), and the text branch adopts CLIP-ResNet50, which consists of a ResNet-50-based visual encoder and a Transformer-based text encoder.
[0027] Medical image videos are processed by the image encoder module, video spatiotemporal coding module, feature fusion module, graph attention inference module, and DSMF module of the aforementioned visual branch, outputting different dimensions and overall quality scores. The specific process is as follows: In this embodiment, the specific method for outputting different dimensions and overall quality scores of medical image videos through the visual branch is as follows: The image encoder module is used to extract static spatial features from a single frame, and the video spatiotemporal coding module is used to extract spatiotemporal features. The spatial and spatiotemporal features are fused by the feature fusion module and then generated into multiple sets of attribute-aware features through parallel MLP paths. The specific method for fusing spatial and spatiotemporal features through the feature fusion module is as follows: Spatial features and spatiotemporal features have different channel dimensions. Since the features generated by the two paths have different channel dimensions, we use a fusion module to align and integrate them. The feature fusion module performs channel dimension alignment, mapping features with high channel dimensions to low channel dimensions. The two types of features after channel alignment are then concatenated and fused to obtain concatenated and fused features. Finally, the dimensions of these concatenated and fused features are reshaped. Specifically, Projecting from 2048 dimensions to 768 dimensions in the channel dimension, and then... The features are concatenated to form a 1536-dimensional feature representation; finally, the fused features are reshaped into... ( The feature is further processed through five parallel MLP paths to generate five sets of attribute-aware features. This is used for subsequent quality score prediction for various distortion attributes.
[0028] The graph attention inference module employs a multi-head graph attention mechanism to process multiple sets of attribute-aware features, outputting multiple sets of attribute-enhanced features. Each set of attribute-enhanced features is input into a regression head, outputting the sub-item quality score of the corresponding distorted attribute. Simultaneously, the multiple sets of attribute-enhanced features are first divided into dynamic and static perception streams. The attribute-enhanced features within each perception stream are then fused to obtain dynamic and static features. Finally, the dynamic and static features are subjected to bidirectional cross-attention and weighted fusion with the MoE module to output the overall quality score.
[0029] S3. The image quality assessment network is trained using the training set. During the training phase, frame-level visual embeddings are extracted by the text branch image encoding module (CLIP's ResNet50) and mapped to the same shared feature space with the text embeddings of image quality semantics extracted by the text encoding module (Transformer) to complete similarity modeling. The Fidelity loss is used to constrain the training of the visual branch. After training, the visual branch serves as the main image quality assessment model, and the text branch assists in supervising the visual branch.
[0030] Medical video images need to be processed before being input into the image encoder module. The specific method is as follows: each medical video image is sampled uniformly in the time dimension for several frames, and each frame image is scaled to a preset fixed pixel size before being input into the image encoder module.
[0031] Before being input into the video spatiotemporal coding module, medical imaging videos need to be processed. The specific method is as follows: First, each frame of the video is divided into several non-overlapping spatial grids. A single grid is randomly selected from all the spatial grids, and a fixed pixel image block is randomly cropped from the selected grid to obtain the spatial image block corresponding to a single frame. The video is evenly divided into a preset number of temporal segments along the time axis, and multiple consecutive frames are sampled from each temporal segment. The spatial image blocks corresponding to the temporally consecutive images in each temporal segment are aligned according to the frame time sequence to generate compact spatiotemporal segments that retain the continuity of local motion. All compact spatiotemporal segments are stitched together in the spatial dimension, and the spatiotemporal tensor obtained after stitching is input into the video spatiotemporal coding module.
[0032] The role of text branches: Considering that static distortion attributes (such as brightness, specular reflection, and usability) directly affect the visual interpretability and clinical usability of colonoscopy videos, and are more consistent with human perception than with temporal changes, enhancing the modeling of static attributes is particularly important for reliable quality assessment. To strengthen static attribute modeling, this study introduces a CLIP-based language-guided quality assessment unit, achieving semantic-perceptual prediction of overall video quality through cross-modal alignment. For example... Figure 4 As shown, this module uses a frozen image encoding module (CLIP-ResNet50) and a text encoding module (Transformer) to extract frame-level visual features and text embeddings of predefined quality semantic descriptions, respectively, and performs similarity modeling in the same feature space.
[0033] like Figure 1 , Figure 4 As shown, specifically, given a video input The image encoder module extracts frame-level features. Simultaneously, a discrete quality text set ranging from "bad" to "perfect" is constructed, and semantic embeddings are obtained through a text encoding module. By calculating the normalized cosine similarity between frame-level visual features and text embeddings, and averaging over the time dimension, matching scores for different quality levels are obtained, thus forming a video-level quality distribution. Subsequently, the discrete quality probability distribution is obtained through Softmax normalization, and then further mapped to a continuous semantic quality estimate in expectation form: In the formula, k represents the numerical index of the k-th quality level, corresponding to five semantic quality levels, such as bad, poor, fair, good, and perfect. This represents the probability that, given an input x, the model predicts it belongs to the k-th semantic quality level.
[0034] Then, the mean of the static distortion attribute quality score is used as supervision, and Fidelity loss is introduced to constrain the semantic prediction distribution to enhance its consistency and stability with the real quality distribution.
[0035] In this invention, the text branch does not improve visual representation by modifying the pre-trained CLIP encoder; instead, it incorporates high-quality semantic information into the training of the learnable visual path through joint auxiliary prediction, thereby encouraging greater consistency between the visual representation of the task and the semantic quality structure conveyed by the language cues. In this way, the text branch provides semantic-level auxiliary supervision. This auxiliary task is not directly used for the final score, but rather guides the main task to learn more consistent and discriminative feature representations by providing high-level semantic priors, thus improving multi-dimensional prediction performance.
[0036] S4. Input the medical image video to be tested into the image quality evaluation model, and calculate and output the quality scores of different dimensions and the overall quality.
[0037] The concept of this invention is as follows: For input medical image videos, we first extract multi-scale visual representations through visual branching to capture spatiotemporal distortion patterns, represented as follows: .like Figure 1 , Figure 2 As shown, the obtained features are then projected onto a five-attribute perceptual representation. In this process, the Graph Attention Inference (GAR) module is used to further model the correlations between attributes, thereby generating attribute-enhanced features. Next, we will... Input the regression header to obtain the quality score of the i-th dimension. .at the same time, It is decomposed into static and dynamic features, and adaptively fused through a hybrid expert system module (MoE) to infer the overall quality score. Given that static distortion attributes such as specular reflection, brightness, and usability characterize the overall perceived quality of medical images, these attributes are crucial for improving the reliability of fine-grained quality assessment. To ensure accurate modeling, we introduce a text branch as an auxiliary static quality supervision component. This branch transforms frame-level quality descriptions into text templates and calculates the correspondence between text embeddings and visual embeddings using cosine similarity. The resulting quality level probabilities are then converted into static quality scores through weighted summation, supervised by the average MOS of static attributes using Fidelity loss. In this way, the text branch provides additional quality level constraints for static distortion assessment, enhancing the reliability of the final quality evaluation.
[0038] In this embodiment, the Visual Feature Extraction (VFE) unit adopts a dual-path architecture. The upper-layer path extracts frame-level spatial features through CLIP's ResNet50, and considering the temporal differences of the input medical images, it uniformly samples T frames in the time dimension. Each frame of the image is scaled up to Each pixel is independently input into a CLIP ResNet50. The resulting frame-by-frame features are stacked along the temporal dimension to form spatial features. ,in This represents the spatial resolution of the feature map.
[0039] The lower-level path aims to capture spatiotemporal distortion characteristics while maintaining computational efficiency. Instead of directly processing image segments, we employ the segment sampling strategy used in FASTER-VQA: first, the video is divided into... A non-overlapping spatial grid is used, and randomly cropped from it. The video is divided into 8 segments along the time axis, and 4 consecutive frames are sampled from each segment. The spatial image blocks are aligned with the time frames to form compact spatiotemporal segments that preserve the continuity of local motion. Finally, these segments are spatially stitched together and fed into the Swin-T (Video Swin TransformerTiny) backbone network to construct spatiotemporal features. .
[0040] In this embodiment, the specific calculation steps in the CFRR unit are as follows: The CFRR unit is used to simultaneously capture the relationships between attributes and the spatiotemporal features of heterogeneity. This unit includes a graph attention inference module (i.e., a GAR module) and a DSMF module. In quality assessment tasks, different distorted attributes often have inherent dependencies, which prompts us to construct cross-attribute interaction models to achieve more discriminative representation learning. Therefore, the GAR module can be used to explicitly capture the relationships between multiple quality-related attributes. Figure 2 As shown, the GAR module models the five distortion attributes as graph nodes and learns the dependencies between them through a graph attention layer; this module employs a multi-head attention mechanism with K=8 heads to improve representation learning performance. Specifically, the input is features. The output is a representation of enhanced attribute correlation. These representations are then fed into a regression head to predict the quality score of the i-th distortion attribute. .
[0041] For the k-th attention head, the normalized attention weight between attribute node i and attribute node j is calculated using the following formula: in This indicates the transpose operation. This represents the learnable linear projection matrix of the k-th head. Let || be the corresponding learnable attention vector, and || denote feature concatenation. Since the attribute graph is fully connected, the Softmax normalization operation will be applied to all five attribute features (including the features themselves).
[0042] based on The attribute features are updated by aggregating information from all attribute nodes. The outputs of multiple attention heads are then concatenated and linearly projected to generate the final enhanced attribute-relation representation. .
[0043] in Represents a non-linear activation function. This is a linear projection function. Finally, Input a lightweight regression head (consisting of fully connected layers and linear layers containing Gelu activation functions) to estimate the quality score for each distortion attribute. .
[0044] like Figure 3 As shown, The data is further divided into two complementary streams: a dynamic sensing stream for capturing temporal variations and a static sensing stream for characterizing spatial distortion. Instead of directly aggregating features from all attributes, we first perform specific stream fusion within the DSMF module before aggregation. Specifically, for the dynamic sensing stream, the two temporal attribute features... and (All are) (Dimension) First, sum by element-wise The sequences are then fused; subsequently, the fused sequences are projected and optimized using a multi-head self-attention mechanism to obtain... ,in This represents a learnable linear projection matrix used for feature mapping of the fused dynamic features before entering Multi-Head Attention; this is then followed by residual connection layers and normalization operations. Finally, time-average pooling is applied to obtain a compact dynamic representation: in express The time step in the middle.
[0045] Simultaneously, the static perceptual flow aggregates spatial attributes including lighting, geometric distortion, and content visibility. It combines three feature vectors... Overlay In, an attention-weighted module is used to calculate... Then the static representation of the fusion is obtained as follows: Building upon the dynamic and static flow modeling, bidirectional cross-attention is introduced to achieve cross-information exchange. Specifically, dynamic features... As a query ( ), while static features As a keyword ( ) and value ( Then, multi-head attention is applied to obtain dynamic perceptual representations. This representation contains complementary static information. Conversely, static features... regarded as dynamic features Used as and To obtain a static perceptual representation that enriches the timeline. Finally, by symmetrically merging these two representations, a unified feature is obtained: To further enhance adaptability to different distortions, we introduce a [missing information - likely a parameter or feature]. ( ) experts The MoE module (MoE (Hybrid Expert) is the last core submodule within the DSMF module, responsible for performing the final "adaptive weighted fusion" task). Each expert is defined as a two-layer MLP. The final fusion representation is calculated using the following formula: (1) in Indicates the first An expert. Unlike token-based routing, gating functions... Based on set features, expert weights are generated: in This represents the normalized weights for all experts.
[0046] Ultimately, the characteristics of fusion The video is projected and reshaped into a spatiotemporal representation, which is used as input to a subsequent regression head to obtain the overall video quality score. .
[0047] The total loss function used for iterative training of the visual branch of the medical image quality evaluation network includes an overall quality loss term, an attribute quality loss term, a ranking consistency loss term, and a load balancing loss term, which are weighted together.
[0048] The total loss function is: in, and These represent the subjective score and predicted value of the overall quality, respectively. and This represents the subjective scores and predicted values for each quality attribute. Mean squared error is used to constrain the difference between the prediction results and the subjective scores. and It is a trade-off parameter used to balance the sorting loss and the load balancing regularization term, respectively.
[0049] The text branch obtains the predicted static attribute quality score Ys by matching visual features with text templates of different quality levels. To further constrain the ranking consistency between this prediction result and the subjective static quality assessment, a Fidelity-based ranking loss is introduced. Its specific implementation is as follows: in, This represents the set of sample pairs in a mini-batch. This represents the true preference label between sample i and sample j. This represents the predicted preference probability between sample i and sample j. This loss improves the consistency of the model's quality prediction ranking by constraining the predicted quality ranking relationship between samples, ensuring that the predicted quality score aligns with the ranking order reflected in subjective quality evaluations.
[0050] Introducing a load balancing loss function: in, For the number of experts, The proportion of experts selected. This represents the average route probability. (Parameter) and Used to adjust the weights of each loss. In this embodiment, MoE selects the weights of different experts through routing, and the number of experts is set to 5, L. lb It is the loss function of MoE.
[0051] The image encoder module uses a frozen image encoder module, and the text encoding module uses a frozen text encoding module, which can alleviate the problem of insufficient dataset size and prevent overfitting.
[0052] The network model of this invention is implemented in the PyTorch deep learning framework and runs on the Ubuntu 20.04 operating system. The network model uses the AdamW optimizer, with the backbone network trained at a learning rate of 1×10⁻⁵ and the remaining layers optimized at a higher learning rate of 1×10⁻⁴. Throughout the training process, both learning rates are scheduled using cosine annealing. The batch size of the training data is 8, and the number of training epochs is set to 70.
[0053] This invention uses the Spearman Rank Correlation Coefficient (SRCC), a widely accepted evaluation metric in the field of medical image quality assessment, to measure network performance. Generally, a higher SRCC indicates better performance. Experimental results show that the network model of this invention achieves an SRCC of 0.936 on the MAC-VQA dataset.
[0054] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A language-guided method for evaluating the quality of multiple attributes in medical images, characterized in that, Includes the following steps: S1. Obtain the medical image dataset and perform preprocessing; S2. Construct an image quality assessment network that includes visual and textual branches: The visual branch includes an image encoder module, a video spatiotemporal encoding module, a feature fusion module, a graph attention inference module, and a DSMF module; the text branch includes image and text encoding modules. After medical image videos are processed by the image encoder module, video spatiotemporal coding module, feature fusion module, graph attention inference module and DSMF module of the visual branch, different dimensions and overall quality scores are output. S3. The image quality assessment network is trained using the training set. During the training phase, the frame-level visual embeddings extracted by the image encoding module of the text branch and the text embeddings of the image quality semantics extracted by the text encoding module are mapped to the same shared feature space to complete similarity modeling. The Fidelity loss is used to constrain the training of the visual branch. After training, the visual branch serves as the main image quality assessment model, and the text branch assists in supervising the visual branch. S4. Input the medical image video to be tested into the image quality evaluation model, and calculate and output the quality scores of different dimensions and the overall quality.
2. The language-guided multi-attribute quality assessment method for medical images according to claim 1, characterized in that, In step S1, the medical image dataset selected is the MAC-VQA dataset. Each medical image video in the MAC-VQA dataset is labeled with five distortion attributes: specular reflection, brightness, utility, spatiotemporal visibility, and stability. The dataset also provides the average opinion score for the overall quality and the average opinion score for each distortion attribute.
3. The language-guided multi-attribute quality assessment method for medical images according to claim 1, characterized in that, In step S2, the specific method for outputting different attributes and overall quality scores from the medical image video through the visual branch is as follows: The image encoder module is used to extract the static spatial features of a single frame, and the video spatiotemporal coding module is used to extract spatiotemporal features. The spatial features and spatiotemporal features are fused by the feature fusion module and then generated into multiple sets of attribute-aware features through multiple parallel MLP paths. The graph attention inference module employs a multi-head graph attention mechanism to process multiple sets of attribute-aware features, outputting multiple sets of attribute-enhanced features. Each set of attribute-enhanced features is input into a regression head, outputting the sub-item quality score of the corresponding distorted attribute. Simultaneously, the multiple sets of attribute-enhanced features are first divided into dynamic and static perception streams. The attribute-enhanced features within each perception stream are then fused to obtain dynamic and static features. Finally, the dynamic and static features are subjected to bidirectional cross-attention and weighted fusion with the MoE module to output the overall quality score.
4. The language-guided multi-attribute quality assessment method for medical images according to claim 1, characterized in that, The visual branch includes a visual feature extraction unit and a cross-feature relationship reasoning unit; The image encoder module, video spatiotemporal coding module, and feature fusion module are integrated in the visual feature extraction unit; the graph attention inference module and DSMF module are integrated in the cross-feature relationship inference unit.
5. The language-guided multi-attribute quality assessment method for medical images according to claim 3, characterized in that, Medical video images need to be processed before being input into the image encoder module. The specific method is as follows: each medical video image is sampled uniformly in the time dimension for several frames, and each frame image is scaled to a preset fixed pixel size before being input into the image encoder module.
6. The language-guided multi-attribute quality assessment method for medical images according to claim 3, characterized in that, Medical video images need to be processed before being input into the video spatiotemporal coding module. The specific method is as follows: First, each frame of the video is divided into several non-overlapping spatial grids. Then, a single grid is randomly selected from all the spatial grids. A fixed pixel image block is randomly cropped out from the selected grid to obtain the spatial image block corresponding to a single frame. The video is evenly divided into a preset number of time segments along the time axis, and multiple consecutive frames of images are sampled from each time segment; The spatial image blocks corresponding to the temporally continuous images within each temporal segment are aligned according to frame temporal sequence to generate compact spatiotemporal segments that preserve the continuity of local motion; all compact spatiotemporal segments are spliced together in the spatial dimension, and the spatiotemporal tensor obtained after splicing is input into the video spatiotemporal coding module.
7. The language-guided multi-attribute quality assessment method for medical images according to claim 3, characterized in that, The specific method for fusing spatial and spatiotemporal features through the feature fusion module is as follows: Spatial features and spatiotemporal features have different channel dimensions; channel dimension alignment is performed through the feature fusion module to map features with high channel dimensions to low channel dimensions; the two types of features after channel alignment are spliced and fused to obtain spliced and fused features; The splicing and fusion features are dimensionally reshaped.
8. The language-guided multi-attribute quality assessment method for medical images according to claim 1, characterized in that, The total loss function used for iterative training of the visual branch of the medical image quality assessment network includes an overall quality loss term, an attribute quality loss term, a ranking consistency loss term, and a load balancing loss term, which are weighted together.
9. The language-guided multi-attribute quality assessment method for medical images according to claim 8, characterized in that, The total loss function is: in, and These represent the subjective score and predicted value of the overall quality, respectively. and This represents the subjective scores and predicted values for each quality attribute. Mean squared error is used to constrain the difference between the prediction results and the subjective scores. and It is a trade-off parameter used to balance the sorting loss and the load balancing regularization term, respectively.
10. The language-guided multi-attribute quality assessment method for medical images according to claim 1, characterized in that, The image encoding module uses a frozen image encoder module, and the text branch uses a frozen image and text encoding module.
