A slit lamp image quality assessment method based on feature cross transformer
By combining the feature cross transformer and the attention mechanism, the problem of insufficient utilization of local features in the existing technology for slit lamp image quality assessment of the eye is solved, and more accurate image quality assessment is achieved, which is suitable for small sample data, assists doctors in screening unqualified images, and improves diagnostic efficiency.
Patent Information
- Application Number
- CN202310875426.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-17
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2043-07-17
AI Technical Summary
Existing deep learning methods fail to effectively utilize local features in slit lamp image quality assessment, resulting in inaccurate assessment results. In particular, the importance of lesion quality is not fully considered, and image quality assessment technology is not adaptable enough to small sample data.
A slit lamp image quality assessment algorithm was designed using a feature cross transformer-based method combined with multi-scale feature extraction and attention mechanism. The correlation between lesion quality and other dimensional features was extracted through the self-attention module and the feature cross transformer module. The algorithm was trained and evaluated using multi-dimensional labeled data such as lesion quality, overall clarity, brightness, and disease type.
It improves the accuracy and robustness of image quality assessment, can better reflect the impact of lesion quality on diagnosis, reduces the interference of other factors on the assessment results, improves the assessment effect under small sample data, and meets the actual screening standards of doctors.
Smart Images
Figure CN116883367B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image quality assessment, and in particular to an image quality assessment method for eye slit lamp images based on a feature cross transformer. Background Art
[0002] Ocular slit lamp images are an important tool widely used in ophthalmology clinics for examining and evaluating anterior segment diseases such as keratitis, cataracts, and pterygium. Slit lamp image quality assessment is critical in ensuring accurate and reliable clinical diagnosis. However, due to various factors, such as differences in image acquisition equipment, operator technique, and patient characteristics, the quality of slit lamp images can vary greatly. A good assessment of slit lamp image quality is crucial for correctly interpreting and analyzing the structural details in the images. High-quality images can provide clear and accurate image information, helping doctors diagnose and monitor ocular lesions. Conversely, low-quality images may have problems such as blurring, noise, insufficient lighting, or overexposure, which may lead to misdiagnosis, missed diagnosis, or inaccurate assessment results.
[0003] Image quality assessment is a key research topic in the field of image quality recognition. Image quality assessment technology can be used to assess the quality of medical images, providing doctors with more accurate and rapid image quality assessments and reducing misdiagnosis and missed diagnosis rates. Image quality assessment technology does not rely on raw image information, but instead uses algorithms to extract image features and information for evaluation, demonstrating its practicality and adaptability.
[0004] With the rapid development of deep learning in computer vision, applying deep learning techniques to clinical medical images has become a research hotspot. Automatic classification and segmentation techniques for ophthalmic images have matured, and there have been numerous research results using deep learning techniques for quality assessment of fundus images. However, significant research gaps remain in image quality assessment techniques for slit-lamp anterior segment images.
[0005] Several existing deep learning-based medical image quality assessment technologies mostly focus on the image quality of the entire medical image. Research [1] extracted global features through convolutional neural networks for learning and training; Research [2] proposed a medical image quality assessment method for chest X-ray images by adding a residual module to the basic convolutional neural network; Research [3] proposed using Swin Transformer to solve the image quality assessment task of chest X-ray images. There are few studies on image quality assessment of ocular medical images. Due to the particularity of ocular medical images, the importance of lesion quality is higher than that of other areas. Most studies have not utilized this local feature and only focused on global features.
[0006] The application of attention in computer vision is a mechanism that simulates the human visual system and learns the importance of regions. Applying attention to image quality assessment can reflect the impact of different factors on image quality, such as lesion quality, image exposure, and disease type, thereby improving the accuracy and robustness of the assessment.
[0007] References
[0008] [1]Zhang S,Wang Y,Jiang J,et al.CNN-based medical ultrasound image quality assessment[J].Complexity,2021,2021:1-9.
[0009] [2] Cui
[0010] [3] C,Güler A,Cansever AT,et al.Shifted Windows Transformers for Medical Image Quality Assessment[C] / / Machine Learning in Medical Imaging:13thInternational Workshop,MLMI 2022,Held in Conjunction with MICCAI 2022,Singapore,September 18,2022,Proceedings.Cham:Springer Nature Switzerland,2022:425-435. Summary of the Invention
[0011] In view of the shortcomings of the existing technology, the present invention proposes a slit lamp image quality assessment method based on feature cross transformer.
[0012] A first aspect of the present invention provides a slit lamp image quality assessment method based on a feature cross transformer, the method comprising the following steps:
[0013] 1) The image quality of the collected slit lamp images was annotated in four dimensions: lesion quality, overall clarity, brightness, and disease type, and two batches of data were collected;
[0014] The first batch of data only annotated the lesion quality, and all the annotations were excellent; the second batch of data annotated all four dimensions;
[0015] 2) Perform data amplification on the first batch of data in step 1); then label the amplified data with new labels based on the degree of impact of artificial distortion on image quality, and use them as the training set x for the pre-training model. A ;
[0016] 3) Scale all samples from steps 1) and 2) to the same resolution, and then divide the second batch of data into training, validation, and test sets;
[0017] 4) Using the residual network F θ and training set x A Train to get model M:
[0018] 5) Using the pre-trained model M obtained in step 4) as the initial model weight, the second batch of data is input as training samples into the constructed quality assessment network for training to obtain a slit lamp image quality assessment model;
[0019] 6) The slit lamp image to be tested is input into the slit lamp image quality assessment model to obtain the quality assessment result of the image to be predicted.
[0020] The second aspect of the present invention provides a slit lamp image quality assessment device based on a feature cross transformer, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the slit lamp image quality assessment method based on the feature cross transformer is implemented.
[0021] A third aspect of the present invention provides a computer-readable storage medium storing a computer program for executing the above-mentioned slit lamp image quality assessment method based on feature cross transformer.
[0022] Beneficial effects of the present invention:
[0023] The present invention introduces an attention mechanism and multi-scale feature extraction. On the basis of being able to better extract global features and local features of lesion quality, in order to better eliminate the influence of low-priority factors such as exposure and semantic features on the quality assessment results, a feature cross-transformer module is designed to ignore unimportant interfering features. This method effectively solves the problem of small data volume and large number of features caused by small samples, greatly improving the accuracy of the image quality assessment results of computer-generated slit lamp image data of the eye, and better meeting the judgment criteria of doctors in the actual screening process.
[0024] This paper combines deep learning methods with an attention mechanism to propose a comprehensive algorithmic workflow for assessing the image quality of complete slit-lamp images. This approach not only assesses the image quality of the lesion area, but also comprehensively considers the relationship between other factors and the image quality of the lesion area, assists in learning features, and finally integrates the learned features to provide medical experts with a means of automatically screening out substandard images, improving diagnostic efficiency and possessing significant practical significance. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is the overall flow chart of the present invention;
[0026] Figure 2 This is the quality assessment network structure diagram proposed by the present invention;
[0027] Figure 3 This is a flow chart of the characteristic cross-transformer module proposed by the present invention;
[0028] Figure 4 Figure 1. Device diagram for slit lamp image quality assessment based on feature cross-transformer. DETAILED DESCRIPTION
[0029] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0030] like Figure 1 As shown, a slit lamp image quality assessment method based on a feature cross transformer provided by an embodiment of the present invention includes the following steps:
[0031] 1) Slit-lamp images taken by patients during medical consultations were collected as a data source. Medical experts annotated the image quality of these slit-lamp images based on four dimensions: lesion quality, overall clarity, brightness, and disease type (semantic). Each dimension, except for the semantic label, was categorized as excellent, good, or poor. Semantic labels included keratitis, pterygium, and cataract. The slit-lamp images and labels were combined to form the true distorted data. Two batches of data were collected: the first batch only annotated lesion quality, all labeled as excellent; the second batch annotated all four dimensions.
[0032] In one embodiment, the dataset includes two batches:
[0033] The first batch included 192 slit-lamp images of cataract patients, 281 slit-lamp images of pterygium patients, and 324 slit-lamp images of keratitis patients, with a resolution of 2576 x 1934 pixels. All four evaluation dimensions in this batch were labeled as excellent, with no good or poor classification.
[0034] The second batch includes 300 slit lamp images of cataract patients, 543 slit lamp images of pterygium patients and 512 slit lamp images of keratitis patients, with the same resolution of 2576*1934 pixels. Each evaluation dimension in this batch contains three categories of labels: excellent, good and poor. The annotation method and slit lamp samples are as follows: Figure 1 shown.
[0035] 2) Perform data augmentation on the first batch of data from step 1). Using data augmentation algorithms, artificial distortions are added, such as changing exposure, adjusting color temperature, using Gaussian blur, and using motion blur, to simulate realistic distorted scenes and thus increase the sample size of the dataset.
[0036] The amplified data is then labeled with new labels based on the degree of impact of artificial distortion on image quality. The first batch of data and the sample set after data augmentation and their corresponding labels are merged as the training set xA of the pre-training model.
[0037] In one embodiment, the Imgaug library is used to add artificial distortion to the image samples with excellent quality evaluation in each dimension through a data enhancement algorithm to simulate real distortion.
[0038] First, change the color temperature of the samples in the first batch of data sets, select the images with yellowish color temperature and change them to
[0039] x'1=ChangeColorTemperature(x, 8000)
[0040] , get the image x'1 with a bluish color temperature after data enhancement, and then select the image with a bluish color temperature and change it to x'2 = ChangeColorTemperature(x, 4000)
[0041] , we get the image x'2 with yellowish color temperature after data enhancement, merge the original sample x in the first batch of data sets and the amplified x'1 and x'2 into x', and get a new data set with lesion quality labels marked as excellent.
[0042] Secondly, x' is artificially distorted by Gaussian blur and motion blur to simulate the good and poor quality of the lesion.
[0043] x”=MotionBlur(x’, 15,-144)+GaussianBlur(x’, 4.0),
[0044] x''=MotionBlur(x', 20, (-75, 75))+GaussianBlur(x', 15.0), get samples x'' and x''' after adding artificial distortion, mark the lesion quality label of x'' as good, and the lesion quality label of x'' as poor, merge x', x'', x'' and their corresponding labels as the training set x of the pre-training model A .
[0045] 3) Scale all samples in step 1) and step 2) to the same resolution, and then divide the second batch of data into training set, validation set and test set in a ratio of 5:3:2.
[0046] 4) To improve the training effect of the network, use the residual network F θ and training set x A The model M is trained on the ,first by minimizing the loss function The difference between the model output and the true label is measured. The regularization term Ω is used to constrain the model parameters. The model weights are then updated through the stochastic gradient descent optimization algorithm, and the learning rate η and the regularization term weight λ are set. The specific process is as follows:
[0047]
[0048] Among them, θ is the network parameter, F θ (x A ) is the network input data x A The output on y A are the labels in the training set.
[0049] By iteratively calculating the gradients of the loss function and the regularization term, the network parameters are updated until the predetermined number of training iterations or convergence conditions are reached, and the trained model M is obtained.
[0050] 5) After preparing the data set, the pre-trained model M obtained in step 4) is used as the initial model weight, and the second batch of data is input as training samples into the constructed quality assessment network for training. The optimizer uses the stochastic gradient descent algorithm. Each training uses 100 training rounds, and the weight decay rate is 10 -6 , each mini-batch contains 32 samples, and the initial learning rate is 0.001.
[0051] In one embodiment, if Figure 2 As shown in Figure 2, the structure of the quality assessment network is as follows:
[0052] Multi-scale feature extraction module: The feature extraction part of the backbone network ResNet50 is divided into four stages. In each stage, the feature map is scaled to 1 / 2 of the input of that stage. Finally, the feature maps output by each stage are connected to complete feature extraction and preserve spatial information.
[0053] Self-attention module: Each label information of different dimensions corresponds to an independent self-attention module. The features extracted by the feature extraction module are passed through four parallel attention modules to learn the spatial attention of lesion quality, overall quality, exposure, and semantics. Each attention module consists of two 3*3 convolutional layers and two 1*1 convolutional layers.
[0054] Feature cross-transformer module: This module uses feature cross-attention to calculate the correlation between lesion quality features and features of other dimensions, and realizes the learning of lesion quality features assisted by special diagnosis in other dimensions.
[0055] Specifically, assuming that the similarity between lesion quality features and exposure features is calculated, the steps are as follows: Given the output feature vector from the self-attention module: lesion quality feature X A ,Exposure feature X B First, use different weights to project the input feature vector into three vectors Q, K, and V, thereby obtaining Q A , K A 、V A and Q B , K B 、V B Taking the exposure feature as an example, calculate the feature cross attention:
[0056]
[0057] Then the information is projected into a specific space through a multi-layer perceptron, and a nonlinear mapping is performed to enhance the feature representation, such as Figure 3 As shown in Figure 2, residual connections and layer normalization are used after the feature cross attention and multi-layer perceptron to preserve information and stabilize training. Finally, the similarity between the exposure features and the lesion quality features is output.
[0058] Further:
[0059] First, the image (with dimensions [W, H, 3]) is passed to the multi-scale feature extraction module, which extracts multi-scale features of the image through a series of convolutional and residual block layers. These feature maps contain abstract representations of the image at different levels and scales, namely [W / 4, H / 4, 256], [W / 8, H / 8.512], [W / 16, H / 16, 1024], and [W / 32, H / 32, 2048]. Then, for each image, in the final stage of feature extraction, a global average pooling operation is applied to average the channel dimensions of each feature map to obtain a fixed-length feature vector of size [512, W / 32, H / 32].
[0060] After feature extraction, the self-attention mechanism is applied to each of the four labels. The self-attention mechanism focuses on and weights different channels in the feature vector to better capture the characteristic information associated with a specific label. First, the attention weight is calculated for each channel in the feature vector corresponding to each label. The attention weight is then multiplied by each channel of the feature vector to aggregate the features in a weighted manner. This allows the network to focus more on the characteristic information associated with a specific label, thereby improving its ability to model and understand that label. Ultimately, for each label, a weighted, aggregated feature representation is obtained using the self-attention mechanism.
[0061] Then, the output of the self-attention mechanism is multiplied by the weight as the K and V inputs of the feature cross transformer corresponding to each label. Then, the self-attention output of the lesion quality is multiplied by the weight as the Q input of the feature cross transformer corresponding to the overall clarity, exposure, and semantic labels. In this way, the similarity between the lesion quality feature and the semantic feature, the overall quality feature, and the exposure feature can be calculated. Then, the three feature vectors are fused, and the fusion result is multiplied by the weight as the input Q of the label cross transformer corresponding to the lesion quality label. The lesion quality feature X A Multiply the weights as K and V, calculate the feature cross attention, and get the predicted quality feature vector through the feature cross transformer module.
[0062] Finally, the predicted image quality category is obtained through the fully connected layer.
[0063] 6) Using the slit lamp image to be tested as input, run the slit lamp image quality assessment model trained in the above steps to obtain the quality assessment results of the image to be predicted. The effectiveness of this method is tested using samples in the test set. The test results are shown in the following table:
[0064] Loss ACC 0.5333 0.8672
[0065] The embodiment of the present invention also discloses a slit lamp image quality assessment device based on a feature cross transformer, see Figure 4 , including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned slit lamp image quality assessment method based on feature cross transformer is implemented.
[0066] Embodiments of the present invention further disclose a computer-readable storage medium storing a computer program for executing the aforementioned slit lamp image quality assessment method based on a feature cross-transformer. Persons skilled in the art will appreciate that all or part of the processes in the aforementioned method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the aforementioned method embodiments.
[0067] In summary, the attention mechanism in the present invention fully considers the impact of lesion quality and other local factors on image quality, effectively improving the reliability and accuracy of the evaluation results. However, different factors have different effects on the doctor's diagnosis results. Compared with lesion quality, the overall image quality, exposure, disease type and other factors have relatively small effects. In order to achieve a large gap between other factors and lesion quality attention, reduce the impact of other factors on the final result, by adding a feature cross transformer module, the overfitting problem caused by small samples and multiple features is effectively solved, so that the model can better assist doctors in achieving image quality screening. Whether compared with traditional image quality assessment methods or with basic deep learning methods, the present invention has a significant improvement in the image quality assessment of slit lamp images of the eye.
[0068] The embodiments of the present invention described above do not limit the scope of protection of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the claims of the present invention.
Claims
1. A slit lamp image quality assessment method based on feature cross transformer, characterized in that: The method comprises the following steps: 1) The image quality of the collected slit lamp images was annotated in four dimensions: lesion quality, overall clarity, brightness, and disease type, and two batches of data were collected; The first batch of data only annotated the lesion quality, and all the annotations were excellent; the second batch of data annotated all four dimensions; 2) Perform data amplification on the first batch of data in step 1); then label the amplified data with new labels based on the degree of impact of artificial distortion on image quality, and use them as the training set x for the pre-training model. A ; 3) Scale all samples from steps 1) and 2) to the same resolution, and then divide the second batch of data into training, validation, and test sets; 4) Using the residual network F θ and training set x A Perform training to obtain model M; 5) Using the pre-trained model M obtained in step 4) as the initial model weight, the second batch of data is input as training samples into the constructed quality assessment network for training to obtain a slit lamp image quality assessment model; 6) The slit lamp image to be tested is input into the slit lamp image quality assessment model to obtain the quality assessment result of the image to be predicted; In step 5), the quality assessment network includes a multi-scale feature extraction module, four self-attention modules and four feature cross transformer modules, each self-attention module corresponds to a feature cross transformer module; Multi-scale feature extraction module: The feature extraction part of the backbone network ResNet50 is divided into four stages. In each stage, the feature map is scaled to 1 / 2 of the input of that stage. The feature maps output by each stage are connected to complete feature extraction and preserve spatial information. Self-attention module: The features extracted by the feature extraction module are passed through four parallel attention modules to learn the spatial attention corresponding to lesion quality features, overall clarity features, brightness features, and disease type features; The feature cross-transformer module uses feature cross-attention to calculate the correlation between lesion quality features and features of other dimensions, thereby enabling special diagnosis in other dimensions to assist in learning lesion quality features. The Feature Crossover Transformer module performs the following operations: The outputs of the three attention modules corresponding to the three dimensional features of overall clarity, brightness, and disease type are multiplied by weights as the K input and V input of the feature cross transformer module corresponding to each of the three dimensional features above; The output of the self-attention module corresponding to the lesion quality feature is multiplied by the weight as the Q input of the feature cross transformer corresponding to the overall clarity feature, brightness feature and disease type feature; Calculate the similarity between lesion quality features and disease type features, overall clarity features, and brightness features; The three feature vectors output by the three feature cross transformers corresponding to the overall clarity feature, brightness feature, and disease type feature are fused, and the fusion result is multiplied by the weight as the Q input of the feature cross transformer corresponding to the lesion quality feature; The output of the attention module corresponding to the lesion quality feature is multiplied by the weight as the K input and V input of the feature cross transformer module corresponding to the lesion quality feature, the feature cross attention is calculated, and the predicted quality feature vector is obtained through the feature cross transformer module corresponding to the lesion quality feature.
2. The slit lamp image quality assessment method based on feature cross transformer according to claim 1 is characterized in that: The data augmentation described in step 2) is to add artificial distortion through the data enhancement algorithm to simulate the real distortion scene and thus increase the sample size of the data set.
3. The slit lamp image quality assessment method based on feature cross transformer according to claim 2 is characterized in that: The data amplification is specifically: Change the color temperature of the samples in the first batch of data sets, and select the images with yellowish color temperature to x'1=ChangeColorTemperature(x,8000) After data enhancement, we get the image x'1 with a bluish color temperature, and then select the image with a bluish color temperature and change it to x'2=ChangeColorTemperature(x,4000) Get the image x'2 with yellowish color temperature after data enhancement; The original sample x in the first batch of data sets and the amplified x'1 and x'2 are merged into x' to obtain a new data set with lesion quality labels marked as excellent; Secondly, x' is artificially distorted by Gaussian blur and motion blur to simulate the good and poor quality of the lesion. x"=MotionBlur(x',15,-144)+GaussianBlur(x',4.0), x''=Motion.Blur(x', 20, (-75, 75)) + GaussianBlur(x', 15.0) to obtain the artificially distorted samples x' and x''. The lesion quality label of the x'' sample is good, and the lesion quality label of the x'' sample is poor. Combine x', x", x"' and their corresponding labels as the training set x for the pre-trained model A .
4. The slit lamp image quality assessment method based on feature cross transformer according to claim 1, characterized in that: Step 4) is specifically: First, the difference between the model output and the true label is measured by minimizing the loss function L, and the regularization term Ω is used to constrain the model parameters; Then update the model weights through the stochastic gradient descent optimization algorithm, and set the learning rate η and the regularization term weight λ; By iteratively calculating the gradients of the loss function and the regularization term, the network parameters are updated until the predetermined number of training iterations or convergence conditions are reached, and the trained model M is obtained.
5. The slit lamp image quality assessment method based on feature cross transformer according to claim 1, characterized in that: Each of the attention modules consists of two 3*3 convolutional layers and two 1*1 convolutional layers.
6. A slit lamp image quality assessment device based on a feature cross transformer, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the slit lamp image quality assessment method based on feature cross transformer described in any one of claims 1 to 5 is implemented.
7. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the slit lamp image quality assessment method based on feature cross transformer according to any one of claims 1 to 5.