Retina image hierarchical model training method and device, equipment and medium

Abstract

The invention provides a retinal image hierarchical model training method and device, equipment and a medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining an initial retinal image and at least one corresponding lesion label, inputting the initial retinal image and at least one lesion label into a multi-label division model, and obtaining an initial training set and a verification set; inputting the initial training set into a preprocessing module, and carrying out augmentation processing to obtain a training set; the preprocessing module comprises a plurality of types of preprocessing units which are connected in series, and each preprocessing unit carries out augmentation processing on an input image according to a preset probability; training the initial retina image layering model based on the training set and the verification set to obtain a retina image layering model; the retinal image layering model comprises a plurality of coding layers and a plurality of decoding layers; the decoding layer comprises a deep convolution layer, a point-by-point convolution layer and a self-attention module. According to the technical scheme, the training effect of the retinal image layering model can be improved, and the calculation amount of the model is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a retinal image layering model training method, device, equipment and medium. Background Art

[0002] In ophthalmic medical imaging diagnosis, Optical Coherence Tomography (OCT) technology is widely used due to its non-invasive and high-resolution imaging characteristics. OCT can provide detailed three-dimensional images of eye structures such as the retina, which is crucial for the diagnosis and treatment of ophthalmic diseases.

[0003] In clinical practice, doctors need to manually identify and segment different retinal layers in OCT images, a process that is not only time-consuming but also susceptible to subjective judgment, resulting in limited consistency and repeatability of the results. In addition, OCT images from different patients vary in quality, such as signal attenuation, noise interference, and contrast changes, which further increase the difficulty of manual segmentation.

[0004] Traditional image processing techniques, such as threshold segmentation, edge detection, and region growing, often require complex parameter adjustments and are highly dependent on image quality, making it difficult to adapt to OCT images acquired from different patients and different devices. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a retinal image hierarchical model training method, device, equipment and medium, use an improved deep learning model to train images collected by OCT equipment, automatically learn to capture image features, and augment the training set through various technical means, so that the model can obtain more effective structural segmentation results for complex lesion retinal OCT images, ensure that reliable and high-precision structural information is given in retinal morphology analysis, thereby improving diagnostic efficiency and accuracy.

[0006] In a first aspect, an embodiment of the present invention provides a retinal image hierarchical model training method, comprising:

[0007] Acquire an initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set;

[0008] Inputting the initial retinal image set and each lesion label into the multi-label partitioning model, partitioning the initial retinal image set to obtain an initial training set and a validation set;

[0009] Inputting the initial training set into the preprocessing module, performing augmentation processing on each initial retinal image in the initial training set to obtain a training set; the preprocessing module includes a plurality of types of preprocessing units connected in series, each of which performs augmentation processing on the input image according to a preset probability; the training set includes the initial training set and the augmented retinal image output by the preprocessing module;

[0010] The initial retinal image hierarchical model is trained based on the training set and the validation set to obtain a retinal image hierarchical model; the retinal image hierarchical model includes multiple encoding layers and multiple decoding layers; the decoding layer includes a deep convolution layer, a point-by-point convolution layer and a self-attention module.

[0011] In a preferred embodiment of the present invention, the above-mentioned obtaining the initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set includes:

[0012] Obtaining an initial set of retinal images collected by an OCT device;

[0013] Performing grayscale mapping on each initial retinal image in the initial retinal image set to obtain a grayscale image corresponding to each initial retinal image;

[0014] The grayscale images corresponding to each initial retinal image are labeled with disease types to obtain a layered result map corresponding to each initial retinal image; the layered result map includes at least one lesion label.

[0015] In a preferred embodiment of the present invention, the types of the above-mentioned pre-processing units include size scaling, normalization, horizontal flipping, brightness and contrast adjustment, Gaussian noise, translation and rotation, and blur processing.

[0016] In a preferred embodiment of the present invention, the initial retinal image hierarchical model is trained based on the training set and the validation set to obtain the retinal image hierarchical model, including:

[0017] Based on the training set, the initial retinal image hierarchical model is trained to obtain a trained initial retinal image hierarchical model;

[0018] Based on the validation set, the performance of the trained initial retinal image hierarchical model is evaluated to obtain the evaluation results;

[0019] When the evaluation result is greater than or equal to the evaluation threshold, it is determined that the training of the initial retinal image hierarchical model is completed, and the trained initial retinal image hierarchical model is used as the retinal image hierarchical model.

[0020] In a preferred embodiment of the present invention, the performance evaluation of the trained initial retinal image hierarchical model is performed based on the validation set to obtain the evaluation results, including:

[0021] For each verification image in the verification set, the verification image is input into the trained initial retinal image stratification model to obtain a predicted stratification result map corresponding to the verification image; the verification image refers to the initial retinal image divided into the verification set;

[0022] Get the hierarchical result graph corresponding to the verification image;

[0023] According to the stratification result map and the predicted stratification result map, the segmentation area difference and the stratification line difference are determined;

[0024] The segmentation area difference and the stratification line difference are weighted averaged to obtain the evaluation result.

[0025] In a preferred embodiment of the present invention, after the initial retinal image hierarchical model is trained based on the training set and the validation set to obtain the retinal image hierarchical model, the following steps are included:

[0026] Inputting the retinal image into the retinal image layering model to obtain an initial retinal layering image;

[0027] The initial retinal layered image is input into the post-processing model, and the initial retinal layered image is post-processed to obtain a retinal layered result image.

[0028] In a preferred embodiment of the present invention, the post-processing model comprises a maximum connected area calculation unit, a filling unit, a boundary extraction unit and a smoothing unit connected in series.

[0029] In a second aspect, an embodiment of the present invention further provides a retinal image hierarchical model training device, comprising:

[0030] An image set acquisition module, used to acquire an initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set;

[0031] An image set partitioning module is used to input the initial retinal image set and each lesion label into a multi-label partitioning model, and partition the initial retinal image set to obtain an initial training set and a validation set;

[0032] A preprocessing module is used to input the initial training set into the preprocessing module, and perform augmentation processing on each initial retinal image in the initial training set to obtain a training set; the preprocessing module includes a plurality of types of preprocessing units connected in series, and each preprocessing unit performs augmentation processing on the input image according to a preset probability; the training set includes the initial training set and the augmented retinal image output by the preprocessing module;

[0033] The training module is used to train the initial retinal image hierarchical model based on the training set and the validation set to obtain the retinal image hierarchical model; the retinal image hierarchical model includes multiple encoding layers and multiple decoding layers; the decoding layer includes a deep convolution layer, a point-by-point convolution layer and a self-attention module.

[0034] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the retinal image hierarchical model training method of the first aspect.

[0035] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the retinal image layered model training method of the first aspect.

[0036] The embodiments of the present invention bring the following beneficial effects:

[0037] The embodiment of the present invention provides a retinal image hierarchical model training method, which uses a multi-label partitioning model to partition the initial retinal image set, ensures that the distribution of the training set and the verification set on the lesion label is relatively balanced, and improves the accuracy and generalization ability of the retinal image hierarchical model training results. The preprocessing module has multiple types of preprocessing units connected in series, and performs augmentation processing on the input image according to the preset probability. This random and diverse processing method can effectively increase the richness of the training data, so that the retinal image hierarchical model can see more diverse inputs during the training process, thereby enhancing the model's robustness to factors such as noise, illumination changes, and image distortion, so that the retinal image hierarchical model can obtain more effective structural segmentation results for complex lesion retinal OCT images, ensuring that reliable high-precision structural information is given in retinal morphological analysis, thereby improving diagnostic efficiency and accuracy. The retinal image hierarchical model uses an improved deep learning model, and uses deep convolution layers, point-by-point convolution layers, and self-attention modules in the decoding layer to replace traditional convolution operations, which enhances the model's ability to understand the relationship between spatial features and reduces the model's parameters and calculation amount.

[0038] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1a A flowchart of a retinal image hierarchical model training method provided by an embodiment of the present invention;

[0042] Figure 1b A schematic diagram of the structure of a preprocessing module provided in an embodiment of the present invention;

[0043] Figure 2a A flowchart of another retinal image hierarchical model training method provided by an embodiment of the present invention;

[0044] Figure 2b A schematic diagram of a grayscale image corresponding to an initial retinal image provided by an embodiment of the present invention;

[0045] Figure 2c A schematic diagram of a hierarchical result graph provided by an embodiment of the present invention;

[0046] Figure 3 A flowchart of another retinal image hierarchical model training method provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of a retinal image layered model training device provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] With the aging of the global population and changes in lifestyle, the incidence of ophthalmic diseases such as macular degeneration, glaucoma and diabetic retinopathy has increased year by year, posing a serious threat to human visual health. As an advanced non-invasive imaging technology, ophthalmic OCT technology can provide high-resolution cross-sectional images of eye structures, namely B-scan images, without the need for tissue sections. These images are important tools for ophthalmologists to diagnose and monitor eye diseases.

[0051] However, the analysis and interpretation of OCT images requires a high degree of professional skills and a lot of time investment. In clinical practice, doctors need to manually identify and segment different retinal layers in OCT images, such as the inner layer membrane (ILM), retinal pigment epithelium (RPE), etc. This process is not only time-consuming, but also susceptible to subjective judgment, resulting in limited consistency and repeatability of the results. In addition, there are differences in the quality of OCT images from different patients, such as signal attenuation, noise interference, and contrast changes, which further increase the difficulty of manual segmentation.

[0052] In order to solve these problems, automated image processing technology came into being. Traditional image processing technology mainly relies on image processing technology, such as threshold segmentation, edge detection, etc. These methods often have high requirements for image quality, and performance cannot be effectively guaranteed when the image quality is not stable enough. In actual clinical scenarios, complex or pathological eye structures need to be processed frequently, and the B-scan images generated by ophthalmic OCT devices will also show corresponding complexity and variability. When dealing with various pathological conditions, traditional image processing technology is difficult to effectively capture key features and correctly analyze the image structure.

[0053] Based on this, a retinal image hierarchical model training method provided in an embodiment of the present invention can use an improved deep learning model to train images collected by the OCT device, automatically learn to capture image features, and augment the training set through a variety of technical means, so that the model can obtain more effective structural segmentation results for retinal OCT images with complex lesions, ensuring that reliable and high-precision structural information is provided in retinal morphology analysis, thereby improving diagnostic efficiency and accuracy.

[0054] To facilitate understanding of this embodiment, a retinal image hierarchical model training method disclosed in an embodiment of the present invention is first introduced in detail.

[0055] Example 1

[0056] The embodiment of the present invention provides a retinal image layered model training method, Figure 1a A flowchart of a retinal image layered model training method provided by an embodiment of the present invention. Figure 1aAs shown, the retinal image layered model training method may include the following steps:

[0057] Step S101 : acquiring an initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set.

[0058] The initial retinal image refers to the retinal image collected by the OCT device for patients with various lesions. The initial retinal image set refers to an image set composed of multiple initial retinal images. The lesion label is used to describe whether there is a certain or certain specific retinal lesions in the initial retinal image. According to the type of lesion, the lesion label can be set according to different levels. For example, the lesion label can be a label for various common fundus lesions such as macular disease, optic nerve disease, glaucoma disease and cataract disease. When the lesion label is a macular disease, it can further include macular edema, macular degeneration, macular hole, macular hemorrhage, etc.

[0059] Specifically, multiple initial retinal images may be obtained to form an initial retinal image set, and each initial retinal image may be annotated to obtain at least one lesion label corresponding to each initial retinal image. When annotating the initial retinal images, professional doctors may use annotation software to annotate, or may use annotation software to automatically annotate.

[0060] Step S102, inputting the initial retinal image set and each lesion label into a multi-label partitioning model, partitioning the initial retinal image set to obtain an initial training set and a validation set.

[0061] The multi-label partitioning model can be a model constructed using iterative stratification, a partitioning algorithm for multi-label data. In the process of dividing the initial retinal image set into an initial training set and a validation set, the multi-label partitioning model can ensure the consistency of the proportion of each lesion label in the initial training set and the validation set, thereby effectively preventing the problem of insufficient generalization ability of the model due to overfitting of some diseases. Exemplarily, the lesion labels include macular disease and optic nerve disease. After division by the multi-label partitioning model, in the initial training set, the number of initial retinal images with the lesion label of macular disease accounts for 80% of the number of initial retinal images in the initial retinal image set, and the number of initial retinal images with the lesion label of optic nerve disease accounts for 80% of the number of initial retinal images in the initial retinal image set; in the validation set, the number of initial retinal images with the lesion label of macular disease accounts for 20% of the number of initial retinal images in the initial retinal image set, and the number of initial retinal images with the lesion label of optic nerve disease accounts for 20% of the number of initial retinal images in the initial retinal image set; at the same time, the number of initial retinal images in the initial training set accounts for 80% of the number of initial retinal images in the initial retinal image set, and the number of initial retinal images in the validation set accounts for 20% of the number of initial retinal images in the initial retinal image set.

[0062] Specifically, before the initial retinal image set and each lesion label are input into the multi-label partitioning model, the lesion labels can be sorted and classified to determine the type of lesion labels. For each type, the corresponding initial retinal image is determined, so as to matrix the lesion labels to obtain a lesion label matrix. It can be understood that in the lesion label matrix, each row corresponds to an initial retinal image, each column corresponds to a lesion label type, and the elements in the lesion label matrix are used to describe the lesion label corresponding to the initial retinal image. Exemplarily, the lesion labels include macular disease, optic nerve disease, glaucoma disease and cataract disease. The lesion labels corresponding to the initial retinal image 1 include glaucoma disease and cataract disease. In the lesion label matrix A, macular disease corresponds to the first column, optic nerve disease corresponds to the second column, glaucoma disease corresponds to the third column, cataract disease corresponds to the fourth column, and the initial retinal image 1 corresponds to the first row. Therefore, element a 11 is 0, element a 12 is 0, element a 13 is 1, element a 14 is 1, indicating that the lesion labels corresponding to the initial retinal image 1 include glaucoma disease and cataract disease. The lesion label matrix is ​​input into the multi-label partitioning model, and the initial retinal image set is partitioned to obtain the initial training set and the validation set.

[0063] Step S103, the initial training set is input into a preprocessing module, and each initial retinal image in the initial training set is augmented to obtain a training set.

[0064] The preprocessing module includes multiple types of preprocessing units connected in series, and each preprocessing unit performs augmentation processing on the input image according to a preset probability; the training set includes the initial training set and the augmented retinal image output by the preprocessing module; the types of preprocessing units include size scaling, normalization, horizontal flipping, brightness and contrast adjustment, Gaussian noise, translation and rotation, and blur processing.

[0065] In the preprocessing module, each preprocessing unit performs augmentation processing on the input image according to a preset probability, which is equivalent to that for each preprocessing unit, the probability that the input image is different from the output image is the preset probability. It can be understood that the input image input to the preprocessing unit can be directly output without any processing, or the input image can be processed before output, wherein the probability of the input image being processed before output is the preset probability.

[0066] In the preprocessing module, each preprocessing unit has a different type. Based on the type of the preprocessing unit, the preprocessing unit may include a scaling unit, a normalization unit, a flipping unit, a correction unit, a noise unit, a movement unit, and a blurring unit. Figure 1b A schematic diagram of the structure of a preprocessing module provided by an embodiment of the present invention. Figure 1b As shown, the type of the scaling unit is size scaling, which is used to enlarge or reduce the size of the image input to the scaling unit, and the size of the output image output by the scaling unit is 512*512, and the preset probability is 1. The type of the normalization unit is normalization, which is used to normalize the values ​​of each pixel in the image input to the normalization unit, and the preset probability is 1. The type of the flip unit is horizontal flipping, which is used to flip the image input to the flipping unit to obtain the flipped image, and the preset probability is 0.5. The type of the correction unit is brightness and contrast adjustment, which is used to adjust the brightness and / or contrast of the image input to the correction unit, wherein the degree of brightness adjustment is random, and the degree of contrast adjustment is random, and the preset probability is 0.3. The type of the noise unit is Gaussian noise, which is used to add Gaussian noise to the image input to the noise unit, and the preset probability is 0.3. The type of the moving unit is translation and rotation, which is used to translate and / or rotate the image input to the moving unit, the translation distance is random, and the rotation angle is a randomly selected angle within the range of ±15°, and the preset probability is 0.3 or 0.5. The type of the blur unit is blur processing, which is used to blur the image input to the blur unit to simulate the effect of image blur caused by shaking during the photo shooting process. The preset probability is 0.1.

[0067] Specifically, the initial training set is input into the preprocessing module, and in the preprocessing module, each initial retinal image in the initial training set is augmented to obtain an augmented retinal image output by the preprocessing module, and the augmented retinal image is added to the initial training set to obtain a training set. For each initial retinal image, an augmented retinal image is obtained after the augmentation process of the preprocessing module. Since each preprocessing unit in the preprocessing module randomly augments the input image according to a preset probability, when the same initial retinal image is input into the preprocessing module multiple times, the augmented retinal images obtained may be different. In the preprocessing module, the order in which each preprocessing unit is serially connected can be set according to actual conditions.

[0068] Step S104, training the initial retinal image hierarchical model based on the training set and the validation set to obtain a retinal image hierarchical model.

[0069] The retinal image layering model is used to analyze the input retinal image, layer the retinal image based on the layered structure of the retina, mark each layer of the retinal structure from the retinal image, and calibrate at least one lesion label corresponding to the retinal image. The final output of the retinal image layering model is a layered result map of 512*512*N number of segmented regions. For each pixel position in the layered result map, there are 1xN values ​​as the probability value for predicting that the current pixel belongs to different levels in the layered structure of the retina. An additionally designed CUDA operator is used to perform a one hot operation, that is, to obtain the maximum value of the 1xN values ​​as the level to which the current pixel belongs in the layered structure of the retina. The above operation is performed on all pixels in the layered result map to obtain the final layered result map.

[0070] The retinal image hierarchical model includes multiple encoding layers and multiple decoding layers; the decoding layer includes a deep convolutional layer, a point-by-point convolutional layer, and a self-attention module.

[0071] The initial retinal image hierarchical model can be a Unet model, which is widely used because of its generalization. As an encoder-decoder U-shaped model, it can effectively learn the context information and detail features of the image. In an embodiment of the present invention, an improvement is made on the basis of the Unet model. For each decoding layer, the traditional convolution operation is replaced by deep convolution and point-by-point convolution calculation, so that the number of parameters and the amount of calculation are effectively reduced. At the same time, the self-attention module design is introduced. For the query Q (Query), the key K (Key), and the value V (Value) are obtained by linear transformation of the feature channel value, that is, the output result of the point-by-point convolution is normalized. The attention score is calculated using the calculation method of the Q and K dot product, and the softmax is used to perform a normalization operation. Finally, the normalized attention score is multiplied with V, and the multiplication result is summed with the attention score as the output of the decoding layer. Due to the different scales of different decoding layers, the self-attention module operation performed at different scales enhances the understanding ability of the retinal image hierarchical model in the spatial feature relationship, thereby enhancing the generalization ability of the retinal image hierarchical model for OCT images.

[0072] Specifically, based on the training set, the initial retinal image hierarchical model is trained to obtain the trained initial retinal image hierarchical model, and based on the validation set, the trained initial retinal image hierarchical model is validated, and when the validation result meets the training completion condition, the trained initial retinal image hierarchical model is determined as the retinal image hierarchical model. The training completion condition can be set according to actual conditions.

[0073] The embodiment of the present invention provides a retinal image hierarchical model training method, which uses a multi-label partitioning model to partition the initial retinal image set, ensures that the distribution of the training set and the verification set on the lesion label is relatively balanced, and improves the accuracy and generalization ability of the retinal image hierarchical model training results. The preprocessing module has multiple types of preprocessing units connected in series, and performs augmentation processing on the input image according to the preset probability. This random and diverse processing method can effectively increase the richness of the training data, so that the retinal image hierarchical model can see more diverse inputs during the training process, thereby enhancing the model's robustness to factors such as noise, illumination changes, and image distortion, so that the retinal image hierarchical model can obtain more effective structural segmentation results for complex lesion retinal OCT images, ensuring that reliable high-precision structural information is given in retinal morphological analysis, thereby improving diagnostic efficiency and accuracy. The retinal image hierarchical model uses an improved deep learning model, and uses deep convolution layers, point-by-point convolution layers, and self-attention modules in the decoding layer to replace traditional convolution operations, which enhances the model's ability to understand the relationship between spatial features and reduces the model's parameters and calculation amount.

[0074] Example 2

[0075] An embodiment of the present invention also provides another retinal image hierarchical model training method; this method is implemented on the basis of the method of the above embodiment; this method focuses on obtaining an initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set, and training the initial retinal image hierarchical model based on a training set and a validation set to obtain a specific implementation method of the retinal image hierarchical model.

[0076] Figure 2a A flowchart of another retinal image hierarchical model training method provided by an embodiment of the present invention, such as Figure 2a As shown, the retinal image layered model training method may include the following steps:

[0077] Step S201, obtaining an initial retinal image set collected by an OCT device.

[0078] Specifically, an initial retinal image set is obtained from an OCT device.

[0079] Step S202, performing grayscale mapping on each initial retinal image in the initial retinal image set to obtain a grayscale image corresponding to each initial retinal image.

[0080] The grayscale image refers to the image of the initial retina in the grayscale space. For each initial retinal image in the initial retinal image set, the initial retinal image is mapped to the grayscale color space to obtain a 1024*512 grayscale image.

[0081] Step S203 , annotating the grayscale images corresponding to the initial retinal images with the disease types, and obtaining the layered result images corresponding to the initial retinal images.

[0082] Specifically, professional doctors can be used to annotate the grayscale image corresponding to the initial retinal image with the disease type. During the annotation process, the quality of the initial retinal image is first screened, unqualified initial retinal images are eliminated, and qualified initial retinal images are annotated with the disease type. Disease type annotation is divided into data structure annotation and lesion label annotation. Data structure annotation refers to dividing the structure of each layer of the retinal structure in the grayscale image corresponding to the initial retinal image based on the hierarchical structure of the retina to obtain a hierarchical result map. For example, Figure 2b A schematic diagram of a grayscale image corresponding to an initial retinal image provided by an embodiment of the present invention. Figure 2cA schematic diagram of a layered result map provided by an embodiment of the present invention. Lesion labeling refers to determining at least one lesion label corresponding to the initial retinal image, that is, at least one lesion label included in the layered result map. After obtaining the layered result map, the layered result map is evaluated again to determine whether the labeling of the layered result map is accurate. If it is determined that the labeling of the layered result map is accurate, the layered result map is used as the labeling result.

[0083] In the process of disease type labeling, the stratified result map corresponding to the initial retinal image is finally determined through two screenings, which can improve the accuracy of the stratified result map corresponding to the initial retinal image and provide a data basis for the subsequent training of the initial retinal image stratification model.

[0084] Step S204, inputting the initial retinal image set and each lesion label into a multi-label partitioning model, partitioning the initial retinal image set to obtain an initial training set and a validation set.

[0085] Step S205 , the initial training set is input into a preprocessing module, and each initial retinal image in the initial training set is augmented to obtain a training set.

[0086] Step S206: training the initial retinal image hierarchical model based on the training set to obtain a trained initial retinal image hierarchical model.

[0087] Based on the training set, the training set is input into the initial retinal image hierarchical model, and the initial retinal image hierarchical model is trained. After multiple iterations, the trained initial retinal image hierarchical model is obtained.

[0088] Step S207, based on the validation set, the performance of the trained initial retinal image hierarchical model is evaluated to obtain an evaluation result.

[0089] The evaluation result is used to determine the error between the output result of the initial retinal image stratification model and the annotation result. The annotation result refers to the stratification result map obtained in step S203. Specifically, the verification set is input into the trained initial retinal image stratification model to obtain the output result of each initial retinal image in the verification set. Based on the stratification result map corresponding to each initial retinal image in the initial retinal image set obtained in step S203, the stratification result map corresponding to each initial retinal image in the verification set is screened out as the annotation result corresponding to each initial retinal image in the verification set. For each initial retinal image in the verification set, the difference between the annotation result and the output result is calculated to obtain the evaluation result. The difference between the annotation result and the output result can be the mean of the difference between each pixel in the image.

[0090] Specifically, the evaluation result can be determined through steps A1 to A4.

[0091] Step A1: for each verification image in the verification set, input the verification image into the trained initial retinal image hierarchical model to obtain a predicted hierarchical result map corresponding to the verification image.

[0092] Among them, the validation image refers to the initial retinal image divided into the validation set;

[0093] Step A2, obtaining a hierarchical result map corresponding to the verification image.

[0094] Specifically, the stratified result maps corresponding to the initial retinal images are screened, and the stratified result maps corresponding to the initial retinal images divided into the verification set are used as the stratified result maps corresponding to the verification images.

[0095] Step A3, determining the segmentation area difference and the stratification line difference according to the stratification result map and the predicted stratification result map.

[0096] In the layered result map, different grayscale values ​​are used to describe the different levels corresponding to the layered structure of the retina. The area with the same grayscale value is regarded as a segmentation area, and the edge lines of different segmentation areas are regarded as layering lines. It can be understood that the segmentation areas are distributed layer by layer from top to bottom. The segmentation area difference refers to the difference between the segmentation areas at the same level in the layered result map and the predicted layered result map. The layering line difference refers to the difference between the edge lines of the segmentation areas at the same level in the layered result map and the predicted layered result map. The segmentation area difference is obtained by calculating the intersection and union ratio of the areas of the segmentation areas at the same level in the layered result map and the predicted layered result map. The layering line difference is obtained by calculating the difference between the pixel points on the edge lines of the segmentation areas at the same level in the layered result map and the predicted layered result map.

[0097] Step A4: weighted average the segmentation area difference and the stratification line difference to obtain an evaluation result.

[0098] Among them, the weights of segmentation area difference and stratification line difference can be set according to actual conditions.

[0099] Step S208, when the evaluation result is greater than or equal to the evaluation threshold, it is determined that the training of the initial retinal image hierarchical model is completed, and the trained initial retinal image hierarchical model is used as the retinal image hierarchical model.

[0100] The retinal image layering model training method provided by the embodiment of the present invention can significantly reduce the amount of data processing, reduce the computational complexity, and improve the model training efficiency by converting the initial retinal image into a grayscale image. After the initial retinal image layering training is completed, the validation set is used for verification, which can ensure that the initial retinal image layering model obtained after training can obtain accurate output results. During the evaluation process, the segmentation area difference and the layering line difference are jointly evaluated, which can improve the accuracy of the evaluation results, more comprehensively reflect the performance of the retinal image layering model, and provide guidance for further optimization and improvement of the model.

[0101] Example 3

[0102] The embodiment of the present invention also provides another retinal image hierarchical model training method; this method is implemented on the basis of the above-mentioned embodiment method; this method focuses on describing the specific implementation method after training the initial retinal image hierarchical model based on the training set and the validation set to obtain the retinal image hierarchical model.

[0103] Figure 3 A flowchart of another retinal image hierarchical model training method provided by an embodiment of the present invention, such as Figure 3 As shown, the retinal image layered model training method may include the following steps:

[0104] Step S301: acquiring an initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set.

[0105] Step S302: input the initial retinal image set and each lesion label into a multi-label partitioning model, and partition the initial retinal image set to obtain an initial training set and a validation set.

[0106] Step S303: input the initial training set into a preprocessing module, perform augmentation processing on each initial retinal image in the initial training set, and obtain a training set.

[0107] Step S304: the preprocessing module includes a plurality of types of preprocessing units connected in series, and each preprocessing unit performs augmentation processing on the input image according to a preset probability.

[0108] Step S305 , training the initial retinal image hierarchical model based on the training set and the validation set to obtain a retinal image hierarchical model.

[0109] Step S306, inputting the retinal image into the retinal image layering model to obtain an initial retinal layered image.

[0110] The retinal image refers to an image collected from the patient's retina using an OCT device, and the number of retinal images is multiple. For each retinal image, the layered result map output by the retinal layering model is used as the initial retinal layered image.

[0111] Before inputting the retinal image into the retinal image hierarchical model, the retinal image may be resized to 512*512 to obtain a resized retinal image. Then, the values ​​of each pixel in the resized retinal image are normalized to obtain a normalized retinal image, which is then updated as the retinal image.

[0112] Step S307, inputting the initial retinal layered image into the post-processing model, performing post-processing on the initial retinal layered image, and obtaining a retinal layered result image.

[0113] The post-processing model is used to further process the initial retinal image. The post-processing model includes a maximum connected area calculation unit, a filling unit, a boundary extraction unit and a smoothing unit connected in series. Among them, the maximum connected area calculation unit is used to determine the pixel points with the same and adjacent pixel values ​​in the initial retinal image as a connected area, and for different pixel values, retain the connected area with the largest area. The filling unit is used to fill other connected areas embedded in the connected area, and there are no other embedded connected areas in the filled connected area. The boundary extraction unit is used to extract the boundary line of the connected area. The smoothing unit is used to smooth the boundary line extracted by the boundary extraction unit.

[0114] Specifically, after the initial retinal layered image is input into the post-processing model, it passes through the maximum connected area calculation unit, the filling unit, the boundary extraction unit and the smoothing unit in sequence, and the result output by the smoothing unit is used as the retinal layered result map.

[0115] In an embodiment of the present invention, after the initial retinal image layering model training is completed, the retinal image layering model can be deployed to the system of the application end. In the process of deployment to the application end, the retinal layering model can be read into the application end, and the application end assigns an address to the retinal image layering model. After receiving the retinal image, the system scales and normalizes the retinal image, runs the retinal image layering model in the GPU (Graphic Processing Unit) of the application end, that is, the retinal image after size scaling and normalization is input into the retinal image layering model to obtain an initial retinal layering map, and the initial retinal layering image is input into the post-processing model, and the retinal layering result map is obtained after the maximum connected area calculation unit, the filling unit, the boundary extraction unit and the smoothing unit included in the post-processing model.

[0116] The retinal image stratification model training method provided in the embodiment of the present invention can improve the accuracy and clarity of the retinal stratification result image by post-processing the initial retinal stratification image after obtaining it, which helps to reduce the risk of misdiagnosis and missed diagnosis and improve the diagnostic accuracy of the disease.

[0117] Example 4

[0118] Corresponding to the above method embodiment, the embodiment of the present invention provides a retinal image hierarchical model training device, Figure 4 A schematic diagram of the structure of a retinal image layered model training device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the retinal image layered model training device may include:

[0119] An image set acquisition module 401 is used to acquire an initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set;

[0120] An image set partitioning module 402 is used to input the initial retinal image set and each lesion label into a multi-label partitioning model, and partition the initial retinal image set to obtain an initial training set and a validation set;

[0121] The preprocessing module 403 is used to input the initial training set into the preprocessing module, and perform augmentation processing on each initial retinal image in the initial training set to obtain a training set; the preprocessing module includes a plurality of types of preprocessing units connected in series, and each preprocessing unit performs augmentation processing on the input image according to a preset probability; the training set includes the initial training set and the augmented retinal image output by the preprocessing module;

[0122] The training module 404 is used to train the initial retinal image hierarchical model based on the training set and the validation set to obtain the retinal image hierarchical model; the retinal image hierarchical model includes multiple encoding layers and multiple decoding layers; the decoding layer includes a deep convolution layer, a point-by-point convolution layer and a self-attention module.

[0123] The embodiment of the present invention provides a retinal image hierarchical model training method, which uses a multi-label partitioning model to partition the initial retinal image set, ensures that the distribution of the training set and the verification set on the lesion label is relatively balanced, and improves the accuracy and generalization ability of the retinal image hierarchical model training results. The preprocessing module has multiple types of preprocessing units connected in series, and performs augmentation processing on the input image according to the preset probability. This random and diverse processing method can effectively increase the richness of the training data, so that the retinal image hierarchical model can see more diverse inputs during the training process, thereby enhancing the model's robustness to factors such as noise, illumination changes, and image distortion, so that the retinal image hierarchical model can obtain more effective structural segmentation results for complex lesion retinal OCT images, ensuring that reliable high-precision structural information is given in retinal morphological analysis, thereby improving diagnostic efficiency and accuracy. The retinal image hierarchical model uses an improved deep learning model, and uses deep convolution layers, point-by-point convolution layers, and self-attention modules in the decoding layer to replace traditional convolution operations, which enhances the model's ability to understand the relationship between spatial features and reduces the model's parameters and calculation amount.

[0124] In some embodiments, the image set acquisition module 401 is further configured to:

[0125] Obtaining an initial set of retinal images collected by an OCT device;

[0126] Performing grayscale mapping on each initial retinal image in the initial retinal image set to obtain a grayscale image corresponding to each initial retinal image;

[0127] The grayscale images corresponding to each initial retinal image are labeled with disease types to obtain a layered result map corresponding to each initial retinal image; the layered result map includes at least one lesion label.

[0128] In some embodiments, the types of pre-processing units include resizing, normalization, horizontal flipping, brightness and contrast adjustment, Gaussian noise, translation and rotation, and blurring.

[0129] In some embodiments, the training module 404 is further configured to:

[0130] Based on the training set, the initial retinal image hierarchical model is trained to obtain a trained initial retinal image hierarchical model;

[0131] Based on the validation set, the performance of the trained initial retinal image hierarchical model is evaluated to obtain the evaluation results;

[0132] When the evaluation result is greater than or equal to the evaluation threshold, it is determined that the training of the initial retinal image hierarchical model is completed, and the trained initial retinal image hierarchical model is used as the retinal image hierarchical model.

[0133] In some embodiments, based on the validation set, the performance of the trained initial retinal image hierarchical model is evaluated to obtain an evaluation result, which is also used for:

[0134] For each verification image in the verification set, the verification image is input into the trained initial retinal image stratification model to obtain a predicted stratification result map corresponding to the verification image; the verification image refers to the initial retinal image divided into the verification set;

[0135] Get the hierarchical result graph corresponding to the verification image;

[0136] According to the stratification result map and the predicted stratification result map, the segmentation area difference and the stratification line difference are determined;

[0137] The segmentation area difference and the stratification line difference are weighted averaged to obtain the evaluation result.

[0138] In some embodiments, the apparatus further comprises:

[0139] An image input module, used for inputting the retinal image into the retinal image layering model to obtain an initial retinal layering image;

[0140] The post-processing module is used to input the initial retinal layered image into the post-processing model, post-process the initial retinal layered image, and obtain a retinal layered result image.

[0141] In some embodiments, the post-processing model includes a maximum connected area calculation unit, a filling unit, a boundary extraction unit, and a smoothing unit connected in series.

[0142] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0143] Example 5

[0144] The embodiment of the present invention also provides an electronic device for running the above-mentioned retinal image layered model training method; see Figure 5 A structural schematic diagram of an electronic device is shown, which includes a memory 500 and a processor 501, wherein the memory 500 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 501 to implement the above-mentioned retinal image hierarchical model training method.

[0145] Further, Figure 5 The electronic device shown further includes a bus 502 and a communication interface 503 , and the processor 501 , the communication interface 503 and the memory 500 are connected via the bus 502 .

[0146] The memory 500 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 503 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 502 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 501. The above processor 501 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 500, and the processor 501 reads the information in the memory 500 and completes the steps of the method of the above embodiment in combination with its hardware.

[0148] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned retinal image layered model training method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0149] The computer program product for the retinal image hierarchical model training method provided in the embodiment of the present invention includes a computer-readable storage medium storing a non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0151] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0154] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0155] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A retinal image hierarchical model training method, characterized in that: include: Acquire an initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set; Inputting the initial retinal image set and each lesion label into a multi-label partitioning model, partitioning the initial retinal image set to obtain an initial training set and a validation set; The initial training set is input into a preprocessing module, and each initial retinal image in the initial training set is augmented to obtain a training set; the preprocessing module includes a plurality of types of preprocessing units connected in series, and each preprocessing unit performs augmentation processing on the input image according to a preset probability; the training set includes the initial training set and the augmented retinal image output by the preprocessing module; The initial retinal image hierarchical model is trained based on the training set and the validation set to obtain a retinal image hierarchical model; the retinal image hierarchical model includes multiple encoding layers and multiple decoding layers; the decoding layer includes a deep convolution layer, a point-by-point convolution layer and a self-attention module.

2. The method according to claim 1, characterized in that: The acquiring of the initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set comprises: Obtaining an initial set of retinal images collected by an OCT device; Performing grayscale mapping on each initial retinal image in the initial retinal image set to obtain a grayscale image corresponding to each initial retinal image; The grayscale images corresponding to the initial retinal images are labeled with disease types to obtain a layered result map corresponding to the initial retinal images; the layered result map includes at least one lesion label.

3. The method according to claim 1, characterized in that The types of pre-processing units include resizing, normalization, horizontal flipping, brightness and contrast adjustment, Gaussian noise, translation and rotation, and blur processing.

4. The method according to claim 2, characterized in that: The initial retinal image hierarchical model is trained based on the training set and the validation set to obtain the retinal image hierarchical model, including: Based on the training set, the initial retinal image hierarchical model is trained to obtain a trained initial retinal image hierarchical model; Based on the validation set, performing a performance evaluation on the trained initial retinal image hierarchical model to obtain an evaluation result; When the evaluation result is greater than or equal to the evaluation threshold, it is determined that the training of the initial retinal image hierarchical model is completed, and the trained initial retinal image hierarchical model is used as the retinal image hierarchical model.

5. The method according to claim 4, characterized in that The step of performing a performance evaluation on the trained initial retinal image hierarchical model based on the validation set to obtain an evaluation result includes: For each verification image in the verification set, input the verification image into the trained initial retinal image hierarchical model to obtain a predicted hierarchical result map corresponding to the verification image; the verification image refers to the initial retinal image divided into the verification set; Obtaining a hierarchical result graph corresponding to the verification image; Determine the segmentation area difference and the stratification line difference according to the stratification result map and the predicted stratification result map; The segmented area difference and the layered line difference are weighted averaged to obtain an evaluation result.

6. The method according to claim 1, characterized in that After the initial retinal image hierarchical model is trained based on the training set and the validation set to obtain the retinal image hierarchical model, the method further includes: Inputting the retinal image into the retinal image layering model to obtain an initial retinal layering image; The initial retinal layered image is input into a post-processing model, and the initial retinal layered image is post-processed to obtain a retinal layered result image.

7. The method according to claim 6, characterized in that The post-processing model includes a maximum connected area calculation unit, a filling unit, a boundary extraction unit and a smoothing unit which are connected in series.

8. A retinal image layered model training device, characterized in that: include: An image set acquisition module, used to acquire an initial retinal image set and at least one lesion label corresponding to each initial retinal image in the initial retinal image set; An image set partitioning module, used for inputting the initial retinal image set and each lesion label into a multi-label partitioning model, partitioning the initial retinal image set to obtain an initial training set and a validation set; A preprocessing module, configured to input the initial training set into the preprocessing module, perform augmentation processing on each initial retinal image in the initial training set, and obtain a training set; the preprocessing module comprises a plurality of types of preprocessing units connected in series, each of the preprocessing units performs augmentation processing on the input image according to a preset probability; the training set comprises the initial training set and the augmented retinal image output by the preprocessing module; A training module is used to train the initial retinal image hierarchical model based on the training set and the validation set to obtain a retinal image hierarchical model; the retinal image hierarchical model includes multiple encoding layers and multiple decoding layers; the decoding layer includes a deep convolution layer, a point-by-point convolution layer and a self-attention module.

9. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the retinal image hierarchical model training method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the retinal image layered model training method described in any one of claims 1 to 7.

Images