Training method of keratitis grading model fusing focus and complication image blocks

Abstract

The invention provides a training method of a keratitis grading model fusing focus and complication image blocks, and relates to the field of artificial intelligence, and the method comprises the steps: collecting a plurality of slit lamp image samples; for each slit lamp image sample, identifying a cornea region and a conjunctiva region in the slit lamp image sample by using an image positioning algorithm; randomly cutting at least one first image block in each of upper, lower, left and right boundary areas of the conjunctiva area; randomly cutting at least one second image block in the cornea area; inputting the first image blocks and the second image blocks into a DenseNet121 model in sequence to obtain image features of the image blocks; cascading the image features to obtain a first cascading feature; performing dimension reduction on the first cascade feature by using a PCA algorithm to obtain a first dimension reduction feature; and performing iterative training on the cornea classifier based on the first dimension reduction feature until the cornea classifier converges. According to the method, the performance of the keratitis grading model and the keratitis grading accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method for a keratitis grading model that integrates lesion and complication image blocks. Background Art

[0002] Keratitis is one of the leading causes of visual impairment and blindness worldwide, posing a serious threat to human vision health. As the symptoms of keratitis are complex and diverse, accurate grading of keratitis is crucial to prevent worsening of the disease and improve patient prognosis.

[0003] With the rapid development of deep learning algorithms and ophthalmic imaging data, AI-based slit lamp and fundus image processing has gradually become a reality in the intelligent identification of eye diseases. Deep learning models can automatically extract features from a large number of ophthalmic images to achieve intelligent identification of eye diseases. In related technologies, many researchers have proposed various keratitis grading methods based on deep learning and slit lamp images.

[0004] Prior art 1 is a keratitis intelligent grading method based on the entire slit lamp image. This type of method directly processes the entire slit lamp image, uses a machine learning algorithm or a deep learning algorithm to extract high-level features from the slit lamp image and classify keratitis.

[0005] Prior art 2 is an automatic positioning and intelligent grading of keratitis lesions based on target positioning or segmentation algorithms. This method uses target positioning or segmentation algorithms to crop the corneal or corneal conjunctival region of interest from the slit lamp image. Since the original slit lamp image contains the whole picture of the eye, including non-lesion areas such as eyelashes and eyelids, these irrelevant areas are bound to interfere with the deep learning model and affect the accuracy of keratitis grading.

[0006] Prior art 3 is a keratitis intelligent screening method based on deep meta-learning. This method collects and annotates slit lamp images, builds an auxiliary intelligent screening model with the deep learning classification algorithm ConvNeXt, inputs the slit lamp images into the model to extract keratitis features and train the model. Combining the Canton slit lamp and corneal images taken by mobile devices, the meta-learning strategy is used to extract the features of the slit lamp images, and the model is continuously trained and updated until convergence.

[0007] However, the above methods all directly process the entire slit lamp image, or operate on the cropped cornea or corneal conjunctiva. Although these methods achieve intelligent grading of keratitis, the trained models generally have low accuracy and poor universality because the lesions of keratitis are small, featureless, irregularly distributed, and often accompanied by conjunctival congestion-like complications. Specifically, the lesions of keratitis usually manifest as small, inconspicuous, and irregularly distributed lesions such as corneal clouding and macules. It is difficult for conventional deep learning models to capture the features of these lesions from the entire slit lamp image or a local area, and they are easily disturbed by noise factors such as eyelashes and eyelids around the cornea, which further reduces the accuracy and reliability of the grading system. In addition to corneal lesions, the conjunctival area is often accompanied by congestion-like complications. The existing technology does not pay enough attention to this complication, which is strong evidence for grading the severity of keratitis. These factors make it difficult for existing intelligent grading methods to focus on and complementarily integrate small lesions and complications of keratitis, and are unable to effectively capture these detailed features, resulting in insufficient performance of the keratitis grading model and an inability to accurately grade keratitis.

[0008] Therefore, how to improve the performance of the keratitis grading model and thus improve the accuracy of keratitis grading is a problem that needs to be solved urgently. Summary of the invention

[0009] In view of the problems existing in the prior art, an embodiment of the present invention provides a training method for a keratitis grading model that integrates lesion and complication image blocks.

[0010] In a first aspect, the present invention provides a training method for a keratitis grading model that fuses lesion and complication image blocks, wherein the keratitis grading model includes a densely connected network (Densely Connected Convolutional Networks 121, DenseNet121) model and a cornea classifier, and the method includes:

[0011] Step A: Collect multiple slit lamp image samples, obtain labels for each of the slit lamp image samples, wherein the labels include at least one of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis and viral keratitis; for each of the slit lamp image samples, perform steps BG respectively:

[0012] Step B: using an image positioning algorithm to identify the cornea region and the conjunctiva region in the slit lamp image sample; wherein the lesion region associated with keratitis is located in the cornea region, and the complication region associated with keratitis is located in the conjunctiva region;

[0013] Step C: randomly cropping at least one first image block in the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctival area, respectively, wherein each of the first image blocks does not include eyelashes and eyelids; and randomly cropping at least one second image block in the cornea area; wherein the labels of each of the first image blocks and each of the second image blocks are the same as the labels of the corresponding slit lamp image samples;

[0014] Step D: inputting each of the first image blocks and each of the second image blocks into the DenseNet121 model in sequence, and obtaining image features of each of the first image blocks and image features of each of the second image blocks output by the DenseNet121 model; the DenseNet121 model is obtained by training an initial DenseNet121 model based on each of the first image blocks and each of the second image blocks;

[0015] Step E: cascading the image features of each of the first image blocks and the image features of each of the second image blocks to obtain a first cascade feature;

[0016] Step F: using a principal component analysis (PCA) algorithm to reduce the dimension of the first cascade feature to obtain a first reduced dimension feature of the first cascade feature;

[0017] Step G: Based on the first dimension reduction feature, iteratively train the cornea classifier until the cornea classifier converges.

[0018] Optionally, the using an image positioning algorithm to identify the cornea area and the conjunctiva area in the slit lamp image sample comprises:

[0019] Inputting the slit lamp image sample into the YOLOv8 model, obtaining the boundary coordinates of the cornea area and the boundary coordinates of the conjunctiva area output by the YOLOv8 model;

[0020] Determining the corneal region based on the boundary coordinates of the corneal region;

[0021] The conjunctival region is determined based on the boundary coordinates of the conjunctival region.

[0022] Optionally, the shape of the conjunctiva region is a rectangle; and randomly cropping at least one first image block in an upper boundary region, a lower boundary region, a left boundary region, and a right boundary region of the conjunctiva region respectively comprises:

[0023] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the upper boundary area are calculated based on the following formula (1): min ,y min ):

[0024]

[0025] Among them, d x1 Indicates the random offset of the horizontal axis during sampling;

[0026] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the lower boundary area are calculated based on the following formula (2): min ,y min ):

[0027]

[0028] Among them, d x2 Indicates the random offset of the horizontal axis during sampling;

[0029] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the left boundary area are calculated based on the following formula (3): min ,y min ):

[0030]

[0031] Among them, d y1 Indicates the random offset of the ordinate during sampling;

[0032] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the right boundary area are calculated based on the following formula (4): min ,y min ):

[0033]

[0034] Among them, d y2 Indicates the random offset of the ordinate during sampling; (x cm ,y cm ) represents the coordinates of the upper left vertex of the conjunctival region, W cm represents the horizontal width of the conjunctival area, H cm represents the height of the conjunctival area in the vertical direction; r x and r y represent the width and height of the first image block respectively; x and z y They respectively represent the width and height of at least one image shielding area in the conjunctiva area, and the image shielding area includes the eyelid area and eyelashes in the conjunctiva area.

[0035] Optionally, before randomly cropping at least one first image block from the upper boundary region, the lower boundary region, the left boundary region, and the right boundary region of the conjunctiva region, respectively, the method further includes:

[0036] Based on a target detection algorithm, identifying boundary coordinates of at least one of the eyelid regions and eyelashes in the conjunctival region;

[0037] For each eyelid region and eyelash, determining the boundary coordinates of the image shielding region based on the boundary coordinates of the eyelid region and the eyelash;

[0038] The width and height of each image masking area are determined based on the boundary coordinates of any image masking area.

[0039] Optionally, randomly cropping at least one second image block in the cornea region includes:

[0040] For any second image block, the coordinates (x') of the upper left vertex of the second image block are calculated based on the following formula (5): min , y' min ):

[0041]

[0042] Among them, (x km ,y km ) represents the coordinates of the upper left vertex of the corneal area, W km represents the width of the corneal area, H km represents the height of the corneal area, d x and d y Respectively represent the random offset of the horizontal and vertical coordinates during sampling, r' x and r' y Respectively represent the width and height of the second image block.

[0043] Optionally, the DenseNet121 model includes a classifier;

[0044] The DenseNet121 model is trained in the following way:

[0045] Inputting each of the first image blocks and each of the second image blocks into the initial DenseNet121 model in sequence for iterative training until the target loss function of the classifier reaches a preset threshold;

[0046] The classifier is removed to obtain the trained DenseNet121 model.

[0047] Optionally, the objective loss function is expressed by the following formula (6):

[0048]

[0049] Wherein, L represents the target loss function, N represents the total number of the first image blocks and the second image blocks, and pi represents the probability that the classifier predicts the i-th sample as the true label, w i is the weight of sample i corresponding to the true label.

[0050] In a second aspect, the present invention further provides a keratitis grading method that fuses lesion and complication image blocks, the method being applied to a keratitis grading model, the keratitis grading model comprising a DenseNet121 model and a cornea classifier, the method comprising:

[0051] Collect slit lamp images to be graded;

[0052] Using an image positioning algorithm, identifying a corneal region and a conjunctival region in the slit lamp image to be graded; wherein a lesion region associated with keratitis is located in the corneal region, and a complication region associated with keratitis is located in the conjunctival region;

[0053] At least one third image block is randomly cropped from the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctiva area, respectively, wherein each of the third image blocks does not include eyelashes and eyelids; at least one fourth image block is randomly cropped from the cornea area;

[0054] Inputting each of the third image blocks and each of the fourth image blocks into the DenseNet121 model in sequence, and obtaining image features of each of the third image blocks and image features of each of the fourth image blocks output by the DenseNet121 model;

[0055] Cascading the image features of each of the third image blocks and the image features of each of the fourth image blocks to obtain a second cascade feature;

[0056] Performing dimensionality reduction on the second cascade feature using a principal component analysis (PCA) algorithm to obtain a second dimensionality reduction feature of the second cascade feature;

[0057] Inputting the second dimension reduction feature into the cornea classifier to obtain a classification result of the slit lamp image to be classified output by the cornea classifier;

[0058] The keratitis grading model is trained using the training method for the keratitis grading model of fused lesion and complication image blocks described in any one of claims 1 to 7.

[0059] In a third aspect, the present invention further provides a training device for a keratitis grading model that integrates lesion and complication image blocks, wherein the keratitis grading model includes a DenseNet121 model and a cornea classifier, and the device includes:

[0060] The first acquisition module is used to acquire multiple slit lamp image samples and obtain labels of each of the slit lamp image samples, wherein the labels include at least one of normal cornea, amoebic keratitis, bacterial keratitis, fungal keratitis and viral keratitis; for each of the slit lamp image samples, the following modules are called in sequence:

[0061] A first recognition module is used to use an image positioning algorithm to identify a corneal region and a conjunctival region in the slit lamp image sample; wherein a lesion region associated with keratitis is located in the corneal region, and a complication region associated with keratitis is located in the conjunctival region;

[0062] A first cropping module is used to randomly crop at least one first image block in the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctival area, respectively, wherein each of the first image blocks does not include eyelashes and eyelids; and to randomly crop at least one second image block in the cornea area; wherein the labels of each of the first image blocks and each of the second image blocks are the same as the labels of the corresponding slit lamp image samples;

[0063] A first image feature extraction module is used to input each of the first image blocks and each of the second image blocks into the DenseNet121 model in sequence to obtain image features of each of the first image blocks and image features of each of the second image blocks output by the DenseNet121 model; the DenseNet121 model is obtained by training an initial DenseNet121 model based on each of the first image blocks and each of the second image blocks;

[0064] A first feature cascading module, configured to cascade the image features of each of the first image blocks and the image features of each of the second image blocks to obtain a first cascade feature;

[0065] A first feature dimension reduction module, used for reducing the dimension of the first cascade feature by using a principal component analysis (PCA) algorithm to obtain a first dimension reduction feature of the first cascade feature;

[0066] A training module is used to iteratively train the cornea classifier based on the first dimensionality reduction feature until the cornea classifier converges.

[0067] In a fourth aspect, the present invention further provides a keratitis grading device that fuses lesion and complication image blocks, the device comprising:

[0068] A second acquisition module is used to acquire slit lamp images to be graded;

[0069] A second recognition module is used to use an image positioning algorithm to identify the cornea area and the conjunctiva area in the slit lamp image to be classified; wherein the lesion area related to keratitis is located in the cornea area, and the complication area related to keratitis is located in the conjunctiva area;

[0070] A second cropping module is used to randomly crop at least one third image block in the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctiva area, respectively, wherein each of the third image blocks does not include eyelashes and eyelids; and randomly crop at least one fourth image block in the cornea area;

[0071] A second image feature extraction module is used to input each of the third image blocks and each of the fourth image blocks into a dense connection network DenseNet121 model in sequence to obtain image features of each of the third image blocks and image features of each of the fourth image blocks output by the DenseNet121 model; the keratitis grading model includes the DenseNet121 model and the cornea classifier;

[0072] A second feature cascading module, used for cascading the image features of each of the third image blocks and the image features of each of the fourth image blocks to obtain a second cascade feature;

[0073] A second feature dimension reduction module, used for reducing the dimension of the second cascade feature by using a principal component analysis (PCA) algorithm to obtain a second reduced dimension feature of the second cascade feature;

[0074] A grading module, used for inputting the second dimension reduction feature into the cornea classifier to obtain a grading result of the slit lamp image to be graded output by the cornea classifier;

[0075] The keratitis grading model is trained using the training method for the keratitis grading model that fuses lesion and complication image blocks described in the first aspect.

[0076] In a fifth aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a training method for a keratitis grading model that fuses lesion and complication image blocks as described in the first aspect above, or implements a keratitis grading method that fuses lesion and complication image blocks as described in the second aspect above.

[0077] In a sixth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements a training method for a keratitis grading model that fuses lesion and complication image blocks as described in the first aspect above, or implements a keratitis grading method that fuses lesion and complication image blocks as described in the second aspect above.

[0078] In the seventh aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the training method of the keratitis grading model that fuses lesion and complication image blocks as described in the first aspect above, or implements the keratitis grading method that fuses lesion and complication image blocks as described in the second aspect above.

[0079] The training method of the keratitis grading model integrating lesion and complication image blocks provided by the present invention comprises the following steps: collecting a plurality of slit lamp image samples, obtaining labels of each slit lamp image sample, wherein the labels include at least one of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis and viral keratitis; for each slit lamp image sample, using an image positioning algorithm to identify the cornea area and conjunctiva area in the slit lamp image sample, and randomly cropping at least one first image block in the upper boundary area, the lower boundary area, the left boundary area and the right boundary area of ​​the conjunctiva area; randomly cropping at least one second image block in the cornea area; at least part of the image blocks in each of the above-mentioned first image blocks include complications in the conjunctiva area, and at least part of the image blocks in each of the second image blocks include lesions in the cornea area; it should be noted that the influence of noises such as eyelashes and eyelids on the subsequent cornea classifier training is eliminated in each of the first image blocks, and at the same time, the plurality of randomly cropped first image blocks and second image blocks are conducive to the cornea classifier to extract small, insignificant and irregularly distributed keratitis features, thereby improving the overall performance of the cornea classifier.

[0080] Then, each first image block and each second image block are sequentially input into the DenseNet121 model to obtain the image features of each first image block and the image features of each second image block output by the DenseNet121 model; the image features of each first image block and the image features of each second image block are cascaded to obtain the first cascade feature that integrates the conjunctival region complication features of each first image block and the corneal region lesion features of each second image block; the first cascade feature is reduced in dimension using the PCA algorithm to obtain the first dimensionality reduction feature of the first cascade feature, while retaining the main component of the first cascade feature, the redundant information is removed, so that the first dimensionality reduction feature is more suitable for the training of the corneal classifier. Finally, the corneal classifier is iteratively trained based on the first dimensionality reduction feature until the target loss function of the corneal classifier reaches a preset threshold. Through the above training method, the overall performance of the keratitis grading model is effectively improved, thereby improving the accuracy of keratitis grading. The present invention is suitable for intelligent grading of keratitis in real clinical scenarios, especially in areas with limited medical conditions, and can enhance the recognition ability and robustness of the keratitis grading model for complex corneal lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0082] Figure 1 This is one of the flow charts of the training method of the keratitis grading model integrating the lesion and complication image blocks provided by the present invention;

[0083] Figure 2 It is one of the schematic diagrams of slit lamp image samples provided by the present invention;

[0084] Figure 3 This is the second schematic diagram of the slit lamp image sample provided by the present invention;

[0085] Figure 4 It is a training logic diagram of the DenseNet121 model provided by the present invention;

[0086] Figure 5 It is a logical framework schematic diagram of a training method for a keratitis grading model that fuses lesion and complication image blocks provided by the present invention;

[0087] Figure 6 This is the second flow chart of the training method of the keratitis grading model integrating lesion and complication image blocks provided by the present invention;

[0088] Figure 7 It is a schematic flow chart of a keratitis grading method by fusing lesion and complication image blocks provided by the present invention;

[0089] Figure 8 It is a schematic diagram of the structure of a training device for a keratitis grading model that fuses lesion and complication image blocks provided by the present invention;

[0090] Fig. 9 It is a schematic diagram of the structure of a keratitis grading device for fusing lesion and complication image blocks provided by the present invention;

[0091] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] In order to facilitate a clearer understanding of the embodiments of the present application, some relevant knowledge is first introduced as follows.

[0094] Keratitis is one of the leading causes of visual impairment and blindness worldwide, posing a serious threat to human vision health. Keratitis that is not identified in time can easily lead to corneal ulcers and scars, which in turn cause irreversible visual impairment or even blindness. Due to its complex and diverse symptoms, accurate grading and identification of keratitis is crucial to prevent the disease from worsening and improve patient prognosis.

[0095] At present, keratitis identification mainly relies on ophthalmologists to manually analyze the patient's eye images using a slit lamp microscope. However, this manual identification method has the following shortcomings: first, the identification process is time-consuming and labor-intensive, which is not conducive to the rapid screening of a large number of patients; second, the identification results are greatly affected by the doctor's subjective judgment and there are large individual differences; third, in remote areas with scarce medical resources, the shortage of professional ophthalmologists limits the popularity and accessibility of keratitis grading and identification. With the rapid development of deep learning algorithms and ophthalmic imaging data, the application of slit lamp and fundus image processing based on artificial intelligence in the intelligent identification of eye diseases has gradually become a reality. Deep learning models can automatically extract features from a large number of ophthalmic images to achieve rapid intelligent grading of eye diseases.

[0096] At present, the intelligent grading methods for keratitis still have the problem of insufficient image processing in practical applications, that is, they cannot effectively extract and fuse corneal lesions and conjunctival complications in slit lamp images, and are affected by noise interference and background complexity. Existing methods mainly focus on the entire image or cropped corneal and corneal conjunctival areas, but ignore the characteristics of keratitis lesions that are small, have insignificant features, are irregularly distributed, and are often accompanied by congestion-like complications. This makes it difficult for conventional methods to extract significant and discriminative lesion features from the entire image or a local area. The model often shows low accuracy and poor universality under complex background and noise conditions.

[0097] Since conjunctival complications are often accompanied by congestion-like complications when keratitis occurs, there is a certain correlation between complications and lesions, which is a strong evidence in the process of diagnosing keratitis. In clinical identification, ophthalmologists should not only observe the lesions in the corneal area, but also analyze the distribution of congestion-like complications in the conjunctival area, and conduct a comprehensive analysis to determine whether keratitis occurs and its severity. Existing methods mainly focus on keratitis lesions, but do not pay enough attention to conjunctival congestion-like complications, lacking comprehensive analysis. In addition, keratitis lesions and complications are usually small, insignificant, and irregularly distributed. Conventional deep learning models are difficult to capture the characteristics of these lesions and complications from the entire slit lamp image or a local area, and are easily disturbed by noise factors such as eyelashes and eyelids around the cornea, which further reduces the accuracy and reliability of the recognition system. Finally, the similarity characteristics of different keratitis subtypes and the difference characteristics of the same subtype further increase the difficulty of intelligent diagnosis of keratitis.

[0098] These factors make it difficult for existing intelligent grading methods to focus on and complementarily integrate small lesions and complications of keratitis, and are unable to effectively capture these detailed features, resulting in insufficient grading and recognition performance. Especially in remote areas where there is a lack of professional ophthalmologists, the quality and complexity of slit lamp images further exacerbate the difficulty of recognition. The performance of existing intelligent grading methods in these scenarios is limited, affecting their scalability and reliability in clinical applications.

[0099] Based on the above existing technical problems, since keratitis not only causes lesions in the corneal area, but also is accompanied by congestion-like complications in the conjunctival area, the embodiment of the present invention proposes a training method for a keratitis grading model that fuses lesion and complication image blocks by simulating the experience and path of ophthalmologists in seeing patients, which mainly includes the following parts: automatic positioning of corneal and conjunctival regions of interest, random cropping of lesion and complication image blocks, extraction and fusion of image block features, and automatic keratitis grading. The above method can improve the performance of the keratitis grading model, thereby improving the accuracy of keratitis grading.

[0100] Combine the following Figure 1-Figure 6 The present invention describes a method for training a keratitis grading model that fuses lesion and complication image patches.

[0101] It should be noted that the keratitis grading model includes the DenseNet121 model and the corneal classifier. The corneal classifier includes but is not limited to the support vector machine (SVM) classifier, random forest, Softmax and other classifiers; the corneal classifier is used to automatically grade keratitis, and realizes the intelligent recognition of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis, viral keratitis and other corneal abnormalities.

[0102] Figure 1 This is one of the flow charts of the training method of the keratitis grading model integrating the lesion and complication image blocks provided by the present invention, which specifically includes the following steps:

[0103] Step 101: Collect a plurality of slit lamp image samples and obtain labels of the slit lamp image samples, wherein the labels include at least one of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis and viral keratitis.

[0104] For each slit lamp image sample, steps 102 - 107 are performed respectively.

[0105] Step 102: Use an image positioning algorithm to identify the corneal region and the conjunctival region in the slit lamp image sample; wherein the lesion region associated with keratitis is in the corneal region, and the complication region associated with keratitis is in the conjunctival region.

[0106] In the embodiment of the present invention, an image positioning algorithm is used to accurately position the corneal area and conjunctival area, retaining the lesions and complications related to keratitis to the greatest extent, and providing an effective optional area for random cropping of subsequent image blocks.

[0107] In practical applications, image positioning algorithms include YOLO model, Faster RCNN model, edge detection algorithm, etc.

[0108] Step 103: randomly cropping at least one first image block in the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctival area, respectively, wherein each first image block does not include eyelashes and eyelids; and randomly cropping at least one second image block in the cornea area; wherein the labels of each first image block and each second image block are the same as the labels of the corresponding slit lamp image samples.

[0109] Exemplarily, four first image blocks and four second image blocks are cropped in the cornea region and the conjunctiva region, respectively, for a total of eight image blocks, and the size of each image block is 512 pixels×512 pixels.

[0110] Since complications of keratitis are distributed in the upper, lower, left and right four boundary areas of the conjunctival region, and there are eyelid, eyelash and other noises at the four corners of the boundary, this noise will affect the training accuracy of the subsequent DenseNet121 model. Therefore, at least one first image block that does not contain eyelashes and eyelids is randomly cropped in the upper, lower, left and right boundary areas of the conjunctival region. Since there is no eyelid and eyelash noise in the corneal region, and in order to obtain the corneal lesion characteristics to a greater extent, this embodiment randomly crops four second image blocks in the entire corneal area. This operation can provide a high-quality training data set for the subsequent training of the DenseNet121 model.

[0111] Step 104: input each of the first image blocks and each of the second image blocks into the DenseNet121 model in sequence to obtain the image features of each first image block and the image features of each second image block output by the DenseNet121 model; the DenseNet121 model is obtained after training the initial DenseNet121 model based on each of the first image blocks and each of the second image blocks.

[0112] Step 105: cascading the image features of each of the first image blocks and the image features of each of the second image blocks to obtain a first cascade feature.

[0113] Specifically, after obtaining the image features of each first image block and the image features of each second image block, the above image features need to be fused. In the fusion process, a cascade fusion strategy needs to be used to complement the use of corneal lesions and conjunctival complications, and the image features are cascaded and fused in the order of the corneal area first and the conjunctival area, and the conjunctival area is in the order of left, top, right, and bottom to obtain the first cascade feature.

[0114] Step 106: Use a principal component analysis (PCA) algorithm to reduce the dimension of the first cascade feature to obtain a first reduced dimension feature of the first cascade feature.

[0115] In the embodiment of the present invention, after obtaining the first cascade feature, since the first cascade feature has a high dimension and a lot of redundant information, it is not suitable for training the keratitis grading model. Therefore, it is necessary to use the PCA algorithm to reduce the dimension of the first cascade feature, for example, reduce the features of the above 8 image blocks from high dimension to low dimension, and remove redundant information while retaining the main component of the feature.

[0116] Step 107: Iteratively train the cornea classifier based on the first dimension reduction feature until the cornea classifier converges.

[0117] In an embodiment of the present invention, SVM is selected as the cornea classifier, the first dimensionality reduction feature is input into the SVM, and the maximum margin hyperplane in the feature space is found to realize automatic classification of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis, viral keratitis and other corneal abnormalities.

[0118] Through automatic positioning of corneal and conjunctival regions, random cropping of lesion and complication image blocks, extraction of image block features, and identification of corneal classifiers, the intelligent identification of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis, viral keratitis, and other corneal abnormalities is achieved, significantly improving the model's ability to identify complex keratitis lesions. This method integrates and complements the features of lesion and complication image blocks, not only capturing the features of small, inconspicuous, and irregularly distributed corneal lesions, but also taking into account the features of conjunctival congestion complications, providing a feasible solution for the accurate diagnosis of complex lesions of different subtypes of keratitis in clinical practice.

[0119] The training method of the keratitis grading model integrating lesion and complication image blocks provided by the present invention comprises the following steps: collecting a plurality of slit lamp image samples, obtaining labels of each slit lamp image sample, wherein the labels include at least one of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis and viral keratitis; for each slit lamp image sample, using an image positioning algorithm to identify the cornea area and conjunctiva area in the slit lamp image sample, and randomly cropping at least one first image block in the upper boundary area, the lower boundary area, the left boundary area and the right boundary area of ​​the conjunctiva area; randomly cropping at least one second image block in the cornea area; at least part of the image blocks in each of the above-mentioned first image blocks include complications in the conjunctiva area, and at least part of the image blocks in each of the second image blocks include lesions in the cornea area; it should be noted that the influence of noises such as eyelashes and eyelids on the subsequent cornea classifier training is eliminated in each of the first image blocks, and at the same time, the plurality of randomly cropped first image blocks and second image blocks are conducive to the cornea classifier to extract small, insignificant and irregularly distributed keratitis features, thereby improving the overall performance of the cornea classifier.

[0120] Then, each first image block and each second image block are sequentially input into the DenseNet121 model to obtain the image features of each first image block and the image features of each second image block output by the DenseNet121 model; the image features of each first image block and the image features of each second image block are cascaded to obtain the first cascade feature that integrates the conjunctival region complication features of each first image block and the corneal region lesion features of each second image block; the first cascade feature is reduced in dimension using the PCA algorithm to obtain the first dimensionality reduction feature of the first cascade feature, while retaining the main component of the first cascade feature, the redundant information is removed, so that the first dimensionality reduction feature is more suitable for the training of the corneal classifier. Finally, the corneal classifier is iteratively trained based on the first dimensionality reduction feature until the target loss function of the corneal classifier reaches a preset threshold. Through the above training method, the overall performance of the keratitis grading model is effectively improved, thereby improving the accuracy of keratitis grading. The present invention is suitable for intelligent grading of keratitis in real clinical scenarios, especially in areas with limited medical conditions, and can enhance the recognition ability and robustness of the keratitis grading model for complex corneal lesions.

[0121] Optionally, an image positioning algorithm is used to identify the cornea region and the conjunctiva region in the slit lamp image sample, which is specifically achieved by the following steps:

[0122] Step 1), input the slit lamp image sample into the YOLOv8 model to obtain the boundary coordinates of the corneal area and the boundary coordinates of the conjunctival area output by the YOLOv8 model;

[0123] Step 2), determining the corneal region based on the boundary coordinates of the corneal region;

[0124] The corneal region determined in step 2) refers to the corneal region identified from the slit lamp image sample using the YOLOv8 model. Figure 5 As shown, the cornea area identified by the YOLOv8 model is a rectangle.

[0125] Step 3) Determine the conjunctival region based on the boundary coordinates of the conjunctival region.

[0126] The conjunctival region determined in step 3) refers to the conjunctival region identified from the slit lamp image sample using the YOLOv8 model. Figure 5 As shown, the conjunctival area identified by the YOLOv8 model is a rectangle.

[0127] In an embodiment of the present invention, before identifying the cornea region and the conjunctiva region in the slit lamp image sample, it is necessary to obtain a training sample set for training the YOLOv8 model, and mark the cornea and conjunctiva boundaries in the training sample set. The YOLOv8 model is trained using the training sample set until convergence. Then, the boundary coordinates of the cornea region and the conjunctiva region in the slit lamp image sample are automatically located using the trained YOLOv8 model, providing an effective region of interest for subsequent image block cropping.

[0128] It should be noted that, relative to the corneal area, the conjunctival area is distributed around the corneal area, and complications often occur in the four boundary areas of the upper, lower, left and right corresponding to the center of the conjunctival area. Therefore, the above four boundary areas are cropped using a random perturbation algorithm to increase the diversity of the first image block while obtaining complications to the greatest extent, thereby improving the robustness of the keratitis grading model. The random perturbation algorithm specifically includes the following formulas (1)-(4).

[0129] Optionally, the shape of the conjunctiva region is a rectangle; at least one first image block is randomly cropped in an upper boundary region, a lower boundary region, a left boundary region, and a right boundary region of the conjunctiva region, respectively, which is specifically implemented in the following manner:

[0130] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the upper boundary area are calculated based on the following formula (1):min ,y min ):

[0131]

[0132] Among them, d x1 Indicates the random offset of the horizontal axis during sampling;

[0133] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the lower boundary area are calculated based on the following formula (2): min ,y min ):

[0134]

[0135] Among them, d x2 Indicates the random offset of the horizontal axis during sampling;

[0136] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the left boundary area are calculated based on the following formula (3): min ,y min ):

[0137]

[0138] Among them, d y1 Indicates the random offset of the ordinate during sampling;

[0139] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the right boundary area are calculated based on the following formula (4): min ,y min ):

[0140]

[0141] Among them, d y2 Indicates the random offset of the ordinate during sampling; (x cm ,y cm ) represents the coordinates of the upper left vertex of the conjunctival region, W cm Indicates the horizontal width of the conjunctival area, H cm Indicates the vertical height of the conjunctival area; r x and r y Respectively represent the width and height of the first image block; z x and z y They respectively represent the width and height of at least one image masking area in the conjunctiva area, and the image masking area includes the eyelid area and eyelashes in the conjunctiva area.

[0142] Since the cornea area is located in the middle of the slit lamp image sample, is continuous and small, an unrestricted random cropping method is used on the entire cornea area to crop at least one second image block. The random cropping algorithm is shown in the following formula (5).

[0143] That is, randomly cropping at least one second image block in the cornea region is achieved by the following steps:

[0144] For any second image block, the coordinates (x') of the upper left vertex of the second image block are calculated based on the following formula (5): min , y' min ):

[0145]

[0146] Among them, (x km ,y km ) represents the coordinates of the upper left vertex of the corneal area, W km Indicates the width of the corneal area, H km represents the height of the corneal area, d x and d y Respectively represent the random offset of the horizontal and vertical coordinates during sampling, r' x and r' y Respectively represent the width and height of the second image block.

[0147] For example, Figure 2 This is one of the schematic diagrams of slit lamp image samples provided by the present invention. Figure 2 The eyelid area and eyelashes, corneal area, conjunctival area, corneal leukoplakia, corneal opacity, and conjunctival hyperemia are shown in FIG. Specifically, the area surrounded by the inner solid ellipse is the actual corneal area of ​​the patient's eye, the area surrounded by the dotted ellipse is the actual conjunctival area of ​​the patient's eye, and the area surrounded by the outer solid ellipse is the boundary of the eye in the slit lamp image sample. Analysis Figure 2 Slit lamp image samples of patients with keratitis show corneal clouding and corneal macules in the corneal area, and complications such as congestion in the conjunctiva. Since lesions and complications are usually small, insignificant, and irregularly distributed, traditional deep learning models that use the entire image or a local area cannot accurately capture the characteristics of lesions and complications of keratitis, and often show poor classification performance and weak generalization ability.

[0148] In the example of the present invention, in order to effectively extract the features of lesions and complications related to keratitis, and thus improve the performance of intelligent graded diagnosis of keratitis, the embodiment of the present invention provides a random cropping algorithm for multiple image blocks of lesions and complications. Figure 3 As shown, Figure 3 This is the second schematic diagram of the slit lamp image sample provided by the present invention.

[0149] Specifically, Figure 3 As shown, each first image block in the conjunctival region and each second image block in the corneal region after random cropping are shown. The area surrounded by the inner solid ellipse is the corneal region, and the area surrounded by the dotted ellipse is the conjunctival region. After the corneal region and the conjunctival region are located, 4 image blocks of 512 pixels × 512 pixels are cropped in the corneal region and the conjunctival region, respectively, for a total of 8 image blocks: first image blocks 1, 2, 3, 4 and second image blocks 5, 6, 7, 8.

[0150] exist Figure 3 In the slit lamp image sample shown, since the upper left, upper right, lower left and lower right corners of the conjunctival region are usually composed of noise such as eyelids and eyelashes, these areas become image shielding areas (i.e. Figure 3 In order to filter out the noises irrelevant to keratitis as much as possible, these four areas are avoided when cropping each first image block, so that each first image block does not include eyelashes and eyelids.

[0151] It should be emphasized that in order to enhance the robustness of the keratitis grading model, in addition to removing the four image shielding areas, it is necessary to use a random perturbation algorithm to crop the first image block in the upper, lower, left and right boundary areas corresponding to the center of the conjunctival area, and randomly crop the second image block in the corneal area to increase the diversity of lesion and complication image block selection. The reason is that since the corneal area is in the middle of the image and continuous, while the conjunctiva is distributed around the cornea, the image block cropping of the conjunctival area is different from that of the corneal area. The conjunctival area needs to be cropped in four different directions under the random perturbation algorithm.

[0152] Optionally, before randomly cropping at least one first image block in the upper boundary region, the lower boundary region, the left boundary region, and the right boundary region of the conjunctival region, the width z of each image shielding region needs to be determined. x and height z y , so that the masked areas of each image can be removed. This is achieved by the following steps:

[0153] Step 1) Based on the target detection algorithm, the boundary coordinates of at least one eyelid region and eyelashes in the conjunctiva region are identified.

[0154] The target detection algorithm includes but is not limited to using the YOLO model, Single Shot Multibox Detector (SSD) and other algorithms to identify the boundary coordinates of each eyelid area and eyelashes.

[0155] Step 2), for each eyelid region and eyelash, based on the boundary coordinates of the eyelid region and the eyelash, determine the boundary coordinates of the image shielding region;

[0156] Optionally, the areas of the image shielding regions may be the same.

[0157] Step 3) Based on the boundary coordinates of any image shielding area, determine the width and height of each image shielding area.

[0158] Optionally, a classifier is included in the DenseNet121 model;

[0159] The DenseNet121 model is trained in the following way:

[0160] Step 1), input each first image block and each second image block into the initial DenseNet121 model in sequence for iterative training until the target loss function of the classifier reaches a preset threshold;

[0161] Step 2) Remove the classifier and obtain the trained DenseNet121 model.

[0162] In the embodiment of the present invention, the DenseNet121 model is a backbone network for feature extraction, which is used to extract image features of each first image block and each second image block. Figure 4 It is a training logic diagram of the DenseNet121 model provided by the present invention, such as Figure 4 As shown, the DenseNet121 model consists of 4 groups of dense blocks. The DenseNet121 model is trained based on the automatically cropped first image block and the second image block. The labels of each first image block and each second image block are the same as the labels of the corresponding slit lamp image samples.

[0163] Each layer of the Dense Block has a direct connection channel with all the previous layers. This dense connection structure ensures the full transmission of information in the network, alleviates the gradient vanishing problem, and is conducive to extracting features that can distinguish small keratitis lesions and complications. The part between different Dense Blocks is the convolution pooling layer, also known as the Transition (Trans. in the figure) layer, which is designed to reduce the dimension and size of the features output by the Dense Block.

[0164] The black solid line represents the forward propagation path of the DenseNet121 model, and the black dotted line represents the error back propagation process of the DenseNet121 model. The bold black line connected to the last global average pooling layer (Pool 7×7) and pointing to the first cascade feature of the 8 image blocks represents the path of the image blocks directly output from the pooling layer through forward propagation after the model training is completed. This channel is used to extract the features of the 8 image blocks.

[0165] Exemplarily, in the process of training the DenseNet121 model, the DenseNet121 model is first trained based on 8 image blocks (i.e., 4 first image blocks and 4 second image blocks). After the DenseNet121 model training is completed, the fully connected layer as the classifier is removed, and the 7×7×1024 feature map output by the last Dense Block is passed into the 7×7 global average pooling layer to reduce the feature map size of each image block to 1×1×1024. After this operation, features with a length of 1024 can be extracted from each image block, and the 8 image blocks have a total of 8×1024-dimensional deep feature sets. Finally, the features of the 8 image blocks are cascaded and fused in the order of the cornea area first and the conjunctiva area second, and the conjunctiva area is in the order of left, top, right, and bottom to obtain the first cascade feature.

[0166] Optionally, the present invention also considers the problem of unbalanced data between different subtypes of keratitis in clinical practice and the problem of difficulty in distinguishing individual categories. Figure 4 As shown, when the DenseNet121 model is trained based on the first image block and the second image block, the cost-sensitive method is embedded in the cross-entropy loss function, so that the trained DenseNet121 model pays more attention to the amebic keratitis subtype with a smaller number of samples. At the same time, the second largest cost-sensitive weight is set for the more difficult to distinguish bacterial and fungal keratitis subtypes, which helps the trained DenseNet121 model to extract features of subtypes with fewer categories and difficult to distinguish.

[0167] The objective loss function is expressed by the following formula (6):

[0168]

[0169] Where L represents the target loss function, N represents the total number of the first image block and the second image block, and p i represents the probability that the classifier predicts the true label for the i-th sample, w i is the weight of sample i corresponding to the true label.

[0170] The embodiment of the present invention combines the above method with the category weight optimization strategy to alleviate the impact of data category imbalance and difficult subtypes on model training, and enhance the model's ability to recognize keratitis subtypes with fewer samples and difficult to distinguish.

[0171] After obtaining the first cascade features, the first cascade features also need to be reduced in dimension.

[0172] Optionally, the PCA algorithm is used to reduce the dimension of the first cascade feature to obtain a first reduced dimension feature of the first cascade feature, which is specifically achieved by the following steps:

[0173] Step 1), using the PCA algorithm, linearly project multiple original feature points of the first cascade feature in the first dimensional space to the second dimensional space to obtain multiple target feature points of the first cascade feature in the second dimensional space; the dimension of the first dimensional space is higher than that of the second dimensional space;

[0174] Step 2), sort the target feature points from large to small according to their variance;

[0175] Step 3) Select a preset number of target feature points as the first dimensionality reduction feature corresponding to the first cascade feature.

[0176] In an embodiment of the present invention, since the first cascade feature has a high dimension and a lot of redundant information, it is not suitable for the training of the cornea classifier, and the first cascade feature needs to be further reduced in dimension and fused. The embodiment of the present invention selects the PCA algorithm, and uses the projection matrix to linearly project the original feature points in the high-dimensional first dimensional space to the low-dimensional second dimensional space to reduce the dimension of the feature. The PCA method sorts the projected target feature points according to the variance to form the principal component. For example, the top 1024 target feature points are taken as the first dimensionality reduction feature after dimensionality reduction. This feature dimension is consistent with the feature length extracted by the DenseNet121 model for a single image.

[0177] Figure 5 Schematic diagram of the logic framework of the training method of the keratitis classification model provided by the present invention by fusing the lesion and complication image blocks. Figure 5 , the training method of the keratitis grading model that fuses lesion and complication image blocks of the present invention is further explained.

[0178] like Figure 5 As shown in part (a), firstly, the image blocks need to be randomly cropped based on the image positioning algorithm and the random cropping algorithm. Specifically:

[0179] First, any slit lamp image sample among the multiple slit lamp image samples collected is input into the YOLOv8 model so that the YOLOv8 model can identify the cornea area and conjunctiva area in the slit lamp image sample. Then, the cornea area and conjunctiva area are randomly cropped based on the random cropping algorithm to obtain complication image blocks and lesion image blocks, namely the first image block and the second image block mentioned above.

[0180] It should be noted that each first image block is obtained by cropping the conjunctival region based on a random perturbation algorithm. Figure 5In the part shown in (a), the scheme of the present invention first automatically locates the corneal area and conjunctival area based on the YOLOv8 algorithm to filter out the noise of the eyelids and eyelashes around the cornea, and retains the lesions and complications related to keratitis; then, 4 image blocks are randomly cropped in the corneal area, and at the same time, 4 image blocks are cropped using a random perturbation algorithm in the upper, lower, left and right boundary areas corresponding to the center of the conjunctival area. The size of each image block is 512 pixels × 512 pixels, so that the image blocks are focused on the lesions and complications related to keratitis, and the randomly cropped image blocks can enhance the robustness of the algorithm.

[0181] like Figure 5 As shown in (b), after obtaining the complication image blocks and lesion image blocks in the slit lamp image sample, it is necessary to cascade fuse the complication image blocks and the lesion image blocks and perform keratitis grading. Specifically: First, the complication image blocks and the lesion image blocks are sequentially input into the feature extraction network (i.e., the DenseNet121 model mentioned above), and the DenseNet121 model is trained until the target loss function of the classifier in the DenseNet121 model reaches a preset threshold; then the classifier is removed to obtain a trained DenseNet121 model.

[0182] Then, the complication image blocks and lesion image blocks are sequentially input into the trained DenseNet121 model for feature extraction to obtain the image block features of each image block (i.e. Figure 5 The above image block features are cascaded and fused to obtain the first cascade feature. The first cascade feature is reduced in dimension using the PCA algorithm to obtain the first reduced dimension feature f, which is input into the cornea classifier (i.e. Figure 5 The trained cornea classifier can realize the intelligent recognition of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis, viral keratitis and other corneal abnormalities.

[0183] Figure 6 This is the second flow chart of the training method of the keratitis grading model integrating the lesion and complication image blocks provided by the present invention. The method is applied to the keratitis grading model, which includes a DenseNet121 model and a cornea classifier, and specifically includes the following steps:

[0184] Step 601: Collect multiple slit lamp image samples and obtain labels for each slit lamp image sample, where the labels include at least one of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis and viral keratitis.

[0185] Step 602: for each slit lamp image sample, input the slit lamp image sample into the YOLOv8 model to obtain the boundary coordinates of the corneal area and the boundary coordinates of the conjunctival area in the slit lamp image sample output by the YOLOv8 model; determine the corneal area based on the boundary coordinates of the corneal area; and determine the conjunctival area based on the boundary coordinates of the conjunctival area.

[0186] Step 603: randomly cropping at least one first image block in the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctival area, respectively, wherein each first image block does not include eyelashes and eyelids; and randomly cropping at least one second image block in the cornea area; wherein the labels of each first image block and each second image block are the same as the labels of the corresponding slit lamp image samples.

[0187] Step 604: input each first image block and each second image block into the DenseNet121 model in sequence to obtain the image features of each first image block and the image features of each second image block output by the DenseNet121 model.

[0188] Step 605: cascade the image features of each first image block and the image features of each second image block to obtain a first cascade feature.

[0189] Step 606: Use the PCA algorithm to linearly project multiple original feature points of the first cascade feature in the first dimensional space into the second dimensional space to obtain multiple target feature points of the first cascade feature in the second dimensional space; the dimension of the first dimensional space is higher than that of the second dimensional space.

[0190] Step 607: Sort the target feature points from large to small according to their variances.

[0191] Step 608: Select a preset number of target feature points as the first dimension reduction feature corresponding to the first cascade feature.

[0192] Step 609: Iteratively train the cornea classifier based on the first dimensionality reduction feature until the cornea classifier converges.

[0193] Figure 7 : is a flow chart of a keratitis grading method for fusing lesion and complication image blocks provided by the present invention. The keratitis grading model includes a DenseNet121 model and a cornea classifier. The method specifically includes the following steps:

[0194] Step 701, collecting slit lamp images to be classified;

[0195] Step 702: using an image positioning algorithm to identify the cornea region and the conjunctiva region in the slit lamp image to be classified; wherein the lesion region associated with keratitis is located in the cornea region, and the complication region associated with keratitis is located in the conjunctiva region;

[0196] Step 703, randomly cropping at least one third image block in the upper boundary region, the lower boundary region, the left boundary region, and the right boundary region of the conjunctiva region, respectively, wherein each of the third image blocks does not include eyelashes and eyelids; and randomly cropping at least one fourth image block in the cornea region;

[0197] Step 704: input each of the third image blocks and each of the fourth image blocks into the DenseNet121 model in sequence, and obtain image features of each of the third image blocks and image features of each of the fourth image blocks output by the DenseNet121 model;

[0198] Step 705: cascade the image features of each of the third image blocks and the image features of each of the fourth image blocks to obtain a second cascade feature;

[0199] Step 706: Use a principal component analysis (PCA) algorithm to reduce the dimension of the second cascade feature to obtain a second reduced dimension feature of the second cascade feature;

[0200] Step 707: input the second dimension reduction feature into the cornea classifier to obtain a classification result of the slit lamp image to be classified output by the cornea classifier.

[0201] Among them, the keratitis grading model is based on Figures 1 to 6 The keratitis grading model is trained by a training method that fuses lesion and complication image patches.

[0202] The keratitis grading method for fusing lesion and complication image blocks provided by the present invention comprises the following steps: collecting a slit lamp image to be graded, using an image positioning algorithm to identify a corneal region and a conjunctival region in the slit lamp image to be graded, and randomly cropping at least one third image block in the upper boundary region, the lower boundary region, the left boundary region, and the right boundary region of the conjunctival region; and randomly cropping at least one fourth image block in the corneal region; then, inputting each third image block and each fourth image block into a DenseNet121 model in sequence to obtain image features of each third image block and image features of each fourth image block output by the DenseNet121 model; cascading the image features of each third image block and the image features of each fourth image block to obtain a second cascade feature that fuses the complication features of the conjunctival region of each third image block and the lesion features of the corneal region of each fourth image block; and using a PCA algorithm to reduce the dimension of the second cascade feature to obtain a second reduced dimension feature of the second cascade feature, removing redundant information while retaining the principal component of the second cascade feature, so that the second reduced dimension feature is more suitable for a corneal classifier to perform identification and grading. Through the above classification method, the influence of lesions and complications on keratitis classification is comprehensively considered, and the accuracy of keratitis classification is effectively improved. The present invention is suitable for intelligent keratitis classification in real clinical scenarios, especially in areas with limited medical conditions, and can enhance the recognition ability and robustness of keratitis classification models for complex corneal lesions.

[0203] The training device for the keratitis grading model that fuses lesion and complication image blocks provided by the present invention is described below. The training device for the keratitis grading model that fuses lesion and complication image blocks described below and the training method for the keratitis grading model that fuses lesion and complication image blocks described above can be referenced to each other.

[0204] Figure 8 : is a schematic diagram of the structure of a training device for a keratitis grading model integrating lesion and complication image blocks provided by the present invention, the training device comprises the following modules:

[0205] The first acquisition module 801 is used to acquire multiple slit lamp image samples and obtain labels of each slit lamp image sample, wherein the labels include at least one of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis and viral keratitis; for each slit lamp image sample, the following modules are called in sequence: the first recognition module 802, the first cropping module 803, the first image feature extraction module 804, the first feature cascade module 805, the first feature dimension reduction module 806 and the training module 807. For example, for the first slit lamp image sample, the first recognition module 802, the first cropping module 803, the first image feature extraction module 804, the first feature cascade module 805, the first feature dimension reduction module 806 and the training module 807 are called in sequence; for the second slit lamp image sample, the first recognition module 802, the first cropping module 803, the first image feature extraction module 804, the first feature cascade module 805, the first feature dimension reduction module 806 and the training module 807 are called in sequence.

[0206] The first recognition module 802 is used to use an image positioning algorithm to identify the cornea region and the conjunctiva region in the slit lamp image sample; wherein the lesion region related to keratitis is in the cornea region, and the complication region related to keratitis is in the conjunctiva region;

[0207] The first cropping module 803 is used to randomly crop at least one first image block in the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctiva area, respectively, wherein each first image block does not include eyelashes and eyelids; and randomly crop at least one second image block in the cornea area; wherein the labels of each first image block and each second image block are the same as the labels of the corresponding slit lamp image samples;

[0208] The first image feature extraction module 804 is used to input each first image block and each second image block into the DenseNet121 model in sequence to obtain the image features of each first image block and the image features of each second image block output by the DenseNet121 model; the DenseNet121 model is obtained by training the initial DenseNet121 model based on each first image block and each second image block;

[0209] A first feature cascade module 805, configured to cascade the image features of each first image block and the image features of each second image block to obtain a first cascade feature;

[0210] A first feature dimension reduction module 806 is used to reduce the dimension of the first cascade feature by using a principal component analysis (PCA) algorithm to obtain a first dimension reduction feature of the first cascade feature;

[0211] The training module 807 is used to iteratively train the cornea classifier based on the first dimension reduction feature until the cornea classifier converges.

[0212] Optionally, the first identification module 802 is further configured to:

[0213] The slit lamp image sample is input into the YOLOv8 model to obtain the boundary coordinates of the cornea area and the boundary coordinates of the conjunctiva area output by the YOLOv8 model;

[0214] determining a corneal region based on boundary coordinates of the corneal region;

[0215] Based on the boundary coordinates of the conjunctival region, the conjunctival region is determined.

[0216] Optionally, the conjunctival region is rectangular in shape;

[0217] The first cutting module 803 is further used for:

[0218] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the upper boundary area are calculated based on the following formula (1): min ,y min ):

[0219]

[0220] Among them, d x1 Indicates the random offset of the horizontal axis during sampling;

[0221] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the lower boundary area are calculated based on the following formula (2): min ,y min ):

[0222]

[0223] Among them, d x2 Indicates the random offset of the horizontal axis during sampling;

[0224] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the left boundary area are calculated based on the following formula (3): min ,y min ):

[0225]

[0226] Among them, d y1 Indicates the random offset of the ordinate during sampling;

[0227] The coordinates (x) of the upper left vertex of the first image block randomly cropped in the right boundary area are calculated based on the following formula (4): min ,y min ):

[0228]

[0229] Among them, d y2 Indicates the random offset of the ordinate during sampling; (x cm ,y cm ) represents the coordinates of the upper left vertex of the conjunctival region, W cm Indicates the horizontal width of the conjunctival area, H cm Indicates the vertical height of the conjunctival area; r x and r y Respectively represent the width and height of the first image block; z x and z y They respectively represent the width and height of at least one image masking area in the conjunctiva area, and the image masking area includes the eyelid area and eyelashes in the conjunctiva area.

[0230] Optionally, the device further comprises:

[0231] A third recognition module, for recognizing boundary coordinates of at least one eyelid region and eyelashes in the conjunctiva region based on a target detection algorithm;

[0232] A first determination module is used to determine the boundary coordinates of the image shielding area based on the boundary coordinates of the eyelid area and the eyelash for each eyelid area and eyelash;

[0233] The second determination module is used to determine the width and height of each image shielding area based on the boundary coordinates of any image shielding area.

[0234] Optionally, the first cropping module 803 is further configured to:

[0235] For any second image block, the coordinates (x') of the upper left vertex of the second image block are calculated based on the following formula (5): min , y' min ):

[0236]

[0237] Among them, (x km ,y km ) represents the coordinates of the upper left vertex of the corneal area, W km Indicates the width of the corneal area, H km represents the height of the corneal area, d x and d y Respectively represent the random offset of the horizontal and vertical coordinates during sampling, r' x and r' y Respectively represent the width and height of the second image block.

[0238] Optionally, a classifier is included in the DenseNet121 model;

[0239] The DenseNet121 model is trained in the following way:

[0240] Input each first image block and each second image block into the initial DenseNet121 model in sequence for iterative training until the target loss function of the classifier reaches a preset threshold;

[0241] Remove the classifier and get the trained DenseNet121 model.

[0242] Optionally, the objective loss function is expressed by the following formula (6):

[0243]

[0244] Where L represents the target loss function, N represents the total number of the first image block and the second image block, and p i represents the probability that the classifier predicts the true label for the i-th sample, w i is the weight of sample i corresponding to the true label.

[0245] The keratitis grading device for fusing lesion and complication image blocks provided by the present invention is described below. The keratitis grading device for fusing lesion and complication image blocks described below and the keratitis grading method for fusing lesion and complication image blocks described above can be referenced to each other.

[0246] Fig. 9 : is a schematic diagram of the structure of a keratitis grading device for fusing lesion and complication image blocks provided by the present invention. The keratitis grading device for fusing lesion and complication image blocks comprises the following modules:

[0247] The second acquisition module 901 is used to acquire slit lamp images to be classified;

[0248] The second recognition module 902 is used to use an image positioning algorithm to identify the cornea region and the conjunctiva region in the slit lamp image to be classified; wherein the lesion region related to keratitis is in the cornea region, and the complication region related to keratitis is in the conjunctiva region;

[0249] The second cropping module 903 is used to randomly crop at least one third image block in the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctiva area, respectively, wherein each third image block does not include eyelashes and eyelids; and randomly crop at least one fourth image block in the cornea area;

[0250] The second image feature extraction module 904 is used to input each third image block and each fourth image block into the dense connection network DenseNet121 model in sequence to obtain the image features of each third image block and the image features of each fourth image block output by the DenseNet121 model; the keratitis grading model includes the DenseNet121 model and the cornea classifier;

[0251] A second feature cascade module 905, configured to cascade the image features of each third image block and the image features of each fourth image block to obtain a second cascade feature;

[0252] A second feature dimension reduction module 906, configured to reduce the dimension of the second cascade feature by using a principal component analysis (PCA) algorithm to obtain a second dimension reduction feature of the second cascade feature;

[0253] A classification module 907, used for inputting the second dimension reduction feature into a cornea classifier to obtain a classification result of the slit lamp image to be classified output by the cornea classifier;

[0254] Among them, the keratitis grading model is based on Figures 1 to 6 The keratitis grading model is trained by a training method that fuses lesion and complication image patches.

[0255] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the training method of the keratitis grading model by fusing the lesion and complication image blocks, or execute the keratitis grading method by fusing the lesion and complication image blocks.

[0256] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0257] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can perform the above Figures 1 to 6 The method for training the keratitis grading model by fusing the lesion and complication image patches provided, or performing the above Figure 7 A method for grading keratitis using fused lesion and complication image patches is provided.

[0258] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is executed by a processor to perform the above Figures 1 to 6 The method for training the keratitis grading model by fusing the lesion and complication image patches provided, or performing the above Figure 7 A method for grading keratitis using fused lesion and complication image patches is provided.

[0259] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0260] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0261] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a keratitis grading model that integrates lesion and complication image blocks, characterized in that: The keratitis grading model includes a densely connected network DenseNet121 model and a cornea classifier, and the method includes: Step A: Collect multiple slit lamp image samples, obtain labels for each of the slit lamp image samples, wherein the labels include at least one of normal cornea, amebic keratitis, bacterial keratitis, fungal keratitis and viral keratitis; for each of the slit lamp image samples, perform steps BG respectively: Step B: using an image positioning algorithm to identify the cornea region and the conjunctiva region in the slit lamp image sample; wherein the lesion region associated with keratitis is located in the cornea region, and the complication region associated with keratitis is located in the conjunctiva region; Step C: randomly cropping at least one first image block in the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctival area, respectively, wherein each of the first image blocks does not include eyelashes and eyelids; and randomly cropping at least one second image block in the cornea area; wherein the labels of each of the first image blocks and each of the second image blocks are the same as the labels of the corresponding slit lamp image samples; Step D: inputting each of the first image blocks and each of the second image blocks into the DenseNet121 model in sequence, and obtaining image features of each of the first image blocks and image features of each of the second image blocks output by the DenseNet121 model; the DenseNet121 model is obtained by training an initial DenseNet121 model based on each of the first image blocks and each of the second image blocks; Step E: cascading the image features of each of the first image blocks and the image features of each of the second image blocks to obtain a first cascade feature; Step F: using a principal component analysis (PCA) algorithm to reduce the dimension of the first cascade feature to obtain a first reduced dimension feature of the first cascade feature; Step G: Based on the first dimension reduction feature, iteratively train the cornea classifier until the cornea classifier converges.

2. The method for training a keratitis grading model by fusing lesion and complication image blocks according to claim 1, characterized in that: The method of using an image positioning algorithm to identify a cornea region and a conjunctiva region in the slit lamp image sample comprises: Inputting the slit lamp image sample into the YOLOv8 model, obtaining the boundary coordinates of the cornea area and the boundary coordinates of the conjunctiva area output by the YOLOv8 model; Determining the corneal region based on the boundary coordinates of the corneal region; The conjunctival region is determined based on the boundary coordinates of the conjunctival region.

3. The training method of the keratitis grading model by fusing lesion and complication image blocks according to claim 1, characterized in that: The shape of the conjunctiva region is a rectangle; at least one first image block is randomly cropped in the upper boundary region, the lower boundary region, the left boundary region, and the right boundary region of the conjunctiva region, respectively, including: The coordinates (x) of the upper left vertex of the first image block randomly cropped in the upper boundary area are calculated based on the following formula (1): min ,y min ): Among them, d x1 Indicates the random offset of the horizontal axis during sampling; The coordinates (x) of the upper left vertex of the first image block randomly cropped in the lower boundary area are calculated based on the following formula (2): min ,y min ): Among them, d x2 Indicates the random offset of the horizontal axis during sampling; The coordinates (x) of the upper left vertex of the first image block randomly cropped in the left boundary area are calculated based on the following formula (3): min ,y min ): Among them, d y1 Indicates the random offset of the ordinate during sampling; The coordinates (x) of the upper left vertex of the first image block randomly cropped in the right boundary area are calculated based on the following formula (4): min ,y min ): Among them, d y2 Indicates the random offset of the ordinate during sampling; (x cm ,y cm ) represents the coordinates of the upper left vertex of the conjunctival region, W cm represents the horizontal width of the conjunctival area, H cm represents the height of the conjunctival area in the vertical direction; r x and r y represent the width and height of the first image block respectively; x and z y They respectively represent the width and height of at least one image shielding area in the conjunctiva area, and the image shielding area includes the eyelid area and eyelashes in the conjunctiva area.

4. The method for training a keratitis grading model by fusing lesion and complication image blocks according to claim 3, characterized in that: Before randomly cropping at least one first image block in the upper boundary region, the lower boundary region, the left boundary region, and the right boundary region of the conjunctival region, respectively, the method further includes: Based on a target detection algorithm, identifying boundary coordinates of at least one of the eyelid regions and eyelashes in the conjunctival region; For each eyelid region and eyelash, determining the boundary coordinates of the image shielding region based on the boundary coordinates of the eyelid region and the eyelash; The width and height of each image masking area are determined based on the boundary coordinates of any image masking area.

5. The method for training a keratitis grading model by fusing lesion and complication image blocks according to claim 1, characterized in that: The randomly cutting at least one second image block in the corneal region comprises: For any second image block, the coordinates (x') of the upper left vertex of the second image block are calculated based on the following formula (5): min , y' min ): Among them, (x km ,y km ) represents the coordinates of the upper left vertex of the corneal area, W km represents the width of the corneal area, H km represents the height of the corneal area, d x and d y Respectively represent the random offset of the horizontal and vertical coordinates during sampling, r′ x and r′ y Respectively represent the width and height of the second image block.

6. The method for training a keratitis grading model by fusing lesion and complication image blocks according to any one of claims 1 to 5, characterized in that: The DenseNet121 model includes a classifier; The DenseNet121 model is trained in the following way: Inputting each of the first image blocks and each of the second image blocks into the initial DenseNet121 model in sequence for iterative training until the target loss function of the classifier reaches a preset threshold; The classifier is removed to obtain the trained DenseNet121 model.

7. The method for training a keratitis grading model by fusing lesion and complication image blocks according to claim 6, characterized in that: The objective loss function is expressed by the following formula (6): Wherein, L represents the target loss function, N represents the total number of the first image blocks and the second image blocks, and p i represents the probability that the classifier predicts the i-th sample as the true label, w i is the weight of sample i corresponding to the true label.

8. A keratitis grading method by fusing lesion and complication image blocks, characterized in that: The method is applied to a keratitis grading model, wherein the keratitis grading model includes a dense connection network DenseNet121 model and a cornea classifier, and the method includes: Collect slit lamp images to be graded; Using an image positioning algorithm, identifying a corneal region and a conjunctival region in the slit lamp image to be graded; wherein a lesion region associated with keratitis is located in the corneal region, and a complication region associated with keratitis is located in the conjunctival region; At least one third image block is randomly cropped from the upper boundary area, the lower boundary area, the left boundary area, and the right boundary area of ​​the conjunctiva area, respectively, wherein each of the third image blocks does not include eyelashes and eyelids; at least one fourth image block is randomly cropped from the cornea area; Inputting each of the third image blocks and each of the fourth image blocks into the DenseNet121 model in sequence, and obtaining image features of each of the third image blocks and image features of each of the fourth image blocks output by the DenseNet121 model; Cascading the image features of each of the third image blocks and the image features of each of the fourth image blocks to obtain a second cascade feature; Performing dimensionality reduction on the second cascade feature using a principal component analysis (PCA) algorithm to obtain a second dimensionality reduction feature of the second cascade feature; Inputting the second dimension reduction feature into the cornea classifier to obtain a classification result of the slit lamp image to be classified output by the cornea classifier; The keratitis grading model is trained using the training method for the keratitis grading model of fused lesion and complication image blocks described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the training method of the keratitis grading model by fusing lesion and complication image blocks as described in any one of claims 1 to 7, or implements the keratitis grading method by fusing lesion and complication image blocks as described in claim 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the keratitis grading model by fusing lesion and complication image blocks as described in any one of claims 1 to 7, or implements the keratitis grading method by fusing lesion and complication image blocks as described in claim 8.

Images