Astigmatic surgery prediction and planning system based on a generative adversarial network model

Abstract

The disclosure discloses an astigmatism surgery planning method and system based on a generative adversarial network model, wherein the method comprises the following steps: S100, collecting a data set; S300, obtaining input data and real target data; S400, training a generative adversarial network model; and S500, predicting and planning astigmatism surgery by using the trained generative adversarial network model. In this way, a scheme for predicting a postoperative corneal topography from a preoperative corneal topography is realized, thereby facilitating the use of data processing technology to assist AK surgery planning.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the field of medical data processing, and particularly relates to an astigmatism surgery prediction and planning system based on a generative adversarial network model. BACKGROUND

[0002] Astigmatism is a major factor affecting the uncorrected visual acuity and visual quality of patients after refractive cataract surgery. In recent years, the newly developed femtosecond laser cataract surgery (remark: 1 femtosecond is equal to 10 -15 second, the speed of light is 3 x 10 8 , 1 femtosecond, the distance of light transmission in vacuum is 3 x 10 -7 meters, that is, 0.3 microns, that is, the accuracy can reach microns), which can make precise corneal arc incision to correct astigmatism at the same time of cataract surgery, which attracts the attention of people in the field.

[0003] Corneal arc incision, its full English name is Astigmatic Keratotomy, also known as AK surgery. In the practice of AK surgery, it is often inseparable from the device that can obtain the corneal topography (referred to as corneal topography instrument, not limited to various principles to obtain corneal topography device), which is a new type of device that presents the corneal shape through computer assistance, which can more accurately analyze the shape and curvature of the entire corneal surface, making it possible to systematically, objectively and accurately analyze the corneal shape. Figure 1 The picture shown is a picture of such a device when performing combined surgery of AK surgery and cataract surgery, in which the arc incision, the arc incision axis, the cataract main incision, Figure 2-1 The axial curvature map of the corneal topography is shown, Figure 2-2 The corneal thickness map of the corneal topography is shown. In the axial curvature map, the color information represents the refractive power of the corneal part, the warm color has large refractive power, and the cold color has small refractive power. In the thickness map, the warm color represents the relatively thin cornea, and the cold color represents the relatively thick cornea. By the color distribution of the axial curvature map, it can be seen whether the corneal shape is regular, which part of the corneal refractive power is the largest, which part is the smallest, the distribution and "shape" of astigmatism; experienced doctors can also analyze and consider the postoperative incision edema site and edema change through the corneal thickness map, and predict how it affects the change of astigmatism.

[0004] With the development of technology, although the corneal topography device can accurately obtain the corneal topography of the patient before surgery, its image processing system can digitally analyze the corneal shape and display the obtained information in different characteristic color maps. Because it looks like the ups and downs of the terrain surface in geography, the corresponding map is called corneal topography. Such devices are expected to measure and analyze the curvature of any point on the entire corneal surface with very high precision and detect the corneal refractive power.

[0005] Although the corneal topography device is constantly developing, the field is currently in urgent need of a scheme for surgical assistance planning using data processing technology in the field of computer. SUMMARY

[0006] To solve the above problems, the disclosure discloses a method for astigmatism surgery planning based on a generative adversarial network model, comprising the following steps:

[0007] S100, collect data sets:

[0008] Obtain preoperative corneal topography data of at least 100 patients (300 patients are better), and postoperative corneal topography data of the at least 100 patients for at least 1 month (3 months are better), wherein,

[0009] The preoperative corneal topography data and the postoperative 1-month corneal topography data are derived from the corneal topography measuring device Pentacam: the curvature radius of each point of the cornea with the vertex of the cornea as the center and 7mm as the radius, with a unit of 0.1mm, so as to obtain the original data before and after the operation;

[0010] S300, obtain input data and real target data:

[0011] Based on the preoperative corneal topography data and the information about the AK surgery performed by the at least 100 patients, obtain the input data of the generative adversarial network model;

[0012] Based on the postoperative 1-month corneal topography data, obtain the real target data to be compared by the generative adversarial network model;

[0013] S400, train the generative adversarial network model:

[0014] Input the input data of the generative adversarial network model into the generator of the adversarial generative network, and the generator generates virtual target data;

[0015] The discriminator of the generative adversarial network model compares the virtual target data with the real target data to obtain the similarity;

[0016] And in the process of alternately fixing the discriminator or the generator, train the generator or the discriminator until they reach balance;

[0017] S500, predict and plan astigmatism surgery with the trained generative adversarial network model.

[0018] Preferably,

[0019] The original data before and after the operation is stored in the form of a CSV table.

[0020] Preferably,

[0021] Step S300 further comprises:

[0022] The mean value of the refractive power corresponding to all points on each diameter in the 3mm ring is calculated, and the mean values are sorted to find the diameter with the maximum mean value of the refractive power in the ring. The path corresponding to the diameter is the steep axis, and the path corresponding to the diameter perpendicular to the steep axis is the flat axis.

[0023] Preferably,

[0024] After step S100 is executed and before step S300 is executed, the following step is further included:

[0025] S200, for the preoperative original data and postoperative original data, the curvature radius of each point of the cornea is mapped to the refractive power D of each point of the cornea, and the refractive power D of each point of the cornea is displayed in pseudo-color according to the preoperative and postoperative corneal topography with a step length of 0.25D, so as to realize the display of the Pentacam-like axial curvature chart;

[0026] and the flat axis and steep axis of the central 3mm area of the cornea before and after the operation are drawn, and the astigmatism and axis position on and in the 3mm ring under the preoperative and postoperative conditions are calculated;

[0027] The pseudo-color visualization display of the postoperative corneal topography and the astigmatism and axis position on and in the 3mm ring under the postoperative condition are used to compare with the output of the generative adversarial network model in the training process, so as to optimize the training of the generative adversarial network model and assist in planning the astigmatism operation.

[0028] Preferably,

[0029] Step S200 further comprises:

[0030] The original data is read to analyze the curvature radius of the cornea, and then the curvature radius is converted into the refractive power D through the mapping relationship between the curvature radius and the refractive power.

[0031] Preferably,

[0032] Step S400 comprises the following steps:

[0033] The first stage: fixing the discriminator of the generative adversarial network model, training the generator of the generative adversarial network model, saving the virtual target data generated by the generator in the buffer pool, and if the similarity is too small compared with a threshold value, the discriminator feeds back to the generator, so that the generator can continue to train until the similarity is equal to or just greater than the threshold value;

[0034] The second stage: fixing the generator, training the discriminator, and training the discriminator using the real target data and the virtual target data in the buffer pool; and the above steps are repeated until the two reach a balance.

[0035] Preferably,

[0036] In step S400, the loss function is as follows:

[0037]

[0038] Preferably,

[0039] In step S400,

[0040] The input of the discriminator is the real postoperative corneal topography or the corneal topography prediction map, wherein the generative adversarial network model can generate the postoperative corneal topography prediction map.

[0041] Preferably,

[0042] In step S400,

[0043] The generator adopts the Unet network model as the generator of the GAN generative adversarial network model to extract the feature information of the preoperative corneal topography, age and AK arc length. This network extracts abstract features while retaining more image details by using low-level features to enhance the delicacy of the corneal topography prediction map.

[0044] Wherein, for the patient corneal data, in the down-sampling stage, the feature map of the patient corneal data is output through the multi-layer convolutional neural network; and in the up-sampling stage, the image is restored through the transposed convolution (Transposed convolution) to achieve the purpose of generating the image.

[0045] In the up-sampling stage, the skip layer connection is used to splice the low-level features and the high-level features to better restore the image details. The high-level features retain the key features of the patient data, and the low-level features retain more image details.

[0046] Preferably,

[0047] The discriminator of the generative adversarial network model adopts the fully convolutional network model, which is realized by using a multi-layer convolutional network. It breaks the traditional design of the discriminator in the previous generative adversarial network, which only generates true or false scalars, and can easily realize the expansion of the output of the discriminator model from the pixel level to the block and image size.

[0048] That is, due to the innovative work of the present disclosure, the discriminator model is highlighted to be subdivided into three levels of pixels, blocks and images. For the three discriminators:

[0049] The pixel-level discriminator is used to realize the mapping from the input space to the input space pixel by pixel, so that the color space of the image is more delicate and accurate, but the geometric space information of the image is lost.

[0050] The block-level discriminator is used to realize the mapping from a small area (such as 70x70) in the input space to a certain value in the output space, so that the geometric space information of the image is very accurate and not easy to lose, and the color space is also relatively uniform and smooth.

[0051] The image-level discriminator is a traditional discriminator design, and the output is a scalar, and the obtained effect is the worst, and the model parameter quantity is also relatively large, which is not as good as the block-level discriminator.

[0052] In addition, the disclosure also discloses a corneal topography surgery planning system based on a generative adversarial network model, wherein,

[0053] The system comprises a processor, a memory;

[0054] The system is used for executing the method described in any of the preceding.

[0055] Through the above technical solution, the disclosure can realize the scheme of predicting the postoperative corneal topography by the preoperative corneal topography by using computer data processing technology, so as to facilitate the prediction of the postoperative corneal topography by using data processing technology to assist AK surgery planning. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a screenshot of the implementation of the femtosecond laser cataract surgery combined with AK surgery in the prior art;

[0057] Figure 2-1 is an axial curvature map of the corneal topography of a certain patient;

[0058] Figure 2-2 is a thickness map of the corneal topography of a certain patient;

[0059] Figure 3 is a schematic AK arc length and axis position map;

[0060] Figure 4 is a schematic diagram of the image existence form of matrices A, B and C in an embodiment of the disclosure;

[0061] Figure 5 is a schematic diagram of a color band when pseudo-color visualization is performed in an embodiment of the disclosure;

[0062] Figure 6 is a schematic diagram of steep axes and flat axes in an embodiment of the disclosure;

[0063] Figure 7 is a schematic diagram of the structure of a generator in an embodiment of the disclosure;

[0064] Figure 8is a schematic diagram of the structure of a block-level discriminator in one embodiment of the present disclosure;

[0065] Figure 9 is a schematic diagram of the overall structure of a network comprising both a generator and a discriminator in one embodiment of the present disclosure;

[0066] Figure 10 is a schematic diagram of the pseudo-color visualization of corneal topography in one embodiment of the present disclosure; where the left figure is an axial curvature map exported by the Pentacam device; and the right figure is a pseudo-color map synthesized based on the refractive power values of each point of the cornea (same patient);

[0067] Figure 11 is a schematic diagram of the postoperative 1-month corneal topography prediction of a certain patient in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0068] In order for those skilled in the art to understand the technical solutions disclosed in the present disclosure, the following will be combined with the embodiments and related drawings to describe the technical solutions disclosed in the present disclosure. Figures 1 to 11 The technical solutions of various embodiments are described, and the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. The terms "first", "second", etc. used in the present disclosure are used to distinguish different objects, rather than to describe a specific order. In addition, "including" and "having" and any variations thereof are intended to cover and are not exclusive. For example, a process, or method, or system, or product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, system, product or device.

[0069] In this document, the reference to "embodiments" means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present disclosure. The appearance of this phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive or alternative to other embodiments. Those skilled in the art can understand that the embodiments described herein can be combined with other embodiments.

[0070] To solve the above problems, the present disclosure discloses a method for planning astigmatism surgery based on a generative adversarial network model, comprising the following steps:

[0071] S100, collecting a data set:

[0072] Obtaining preoperative corneal topography data of at least 100 patients (300 patients are better), and postoperative corneal topography data of the at least 100 patients for at least 1 month (3 months are better), wherein,

[0073] The preoperative corneal topography data and the postoperative 1-month corneal topography data are derived from a corneal topography measuring device Pentacam: the corneal curvature radius of each point is obtained from the corneal vertex as the center, 7 mm as the radius, and 0.1 mm as the unit, so as to obtain the preoperative and postoperative original data;

[0074] S300, obtaining input data and real target data:

[0075] Based on the preoperative corneal topography data and the information of the AK surgery performed by the at least 100 patients, the input data for generating the generative adversarial network model is obtained;

[0076] Based on the postoperative 1-month corneal topography data, the real target data for comparison by the generative adversarial network model is obtained;

[0077] Exemplarily,

[0078] For the preoperative corneal topography data, the input data for generating the generative adversarial network model is obtained according to the corneal topography axial curvature original data matrix, AK surgery arc length and axis information, and patient age;

[0079] The input of the model includes the corneal topography axial curvature original data matrix, AK arc length and axis (for example, main incision size and axis), and patient age. Since converting the corneal axial curvature matrix into an int8 type image will cause a reduction in data accuracy, the original corneal axial curvature matrix information is normalized as part of the model input, normalized to the interval [a, b], and the formula for normalization to the interval is:

[0080]

[0081] where Normal_Y is the normalized result, Y is the corneal axial curvature matrix data, k is the normalization coefficient, and the calculation formula of k is (b-a) / (Max-Min), Max is the maximum value in the axial curvature matrix, and Min is the minimum value in the axial curvature matrix.

[0082] Exemplarily, the normalized original corneal axial curvature matrix information is expressed as the following matrix A:

[0083]

[0084] On the other hand, the AK arc length and axis information is used to draw an AK arc length and axis diagram (also known as AK position diagram), for example, the arch incision diameter is 8, the first incision axis is 0, the arc length is 55, the second incision axis is 180, and the arc length is 55. The AK arc length and axis diagram is also expressed as an m*n matrix, and an example is the following matrix B:

[0085]

[0086] AK arc length and axis map, see Figure 3 .

[0087] To ensure the dimensional consistency of the input data, one-dimensional information is processed by dimensionality. The age information is normalized to 0 to 1, and then up-sampled to two-dimensional information; for example:

[0088] The patient is 51 years old, and the normalized value is 0.51:

[0089] Up-sampling to two-dimensional information, the example is as follows:

[0090]

[0091] For the above image example, the dimensional consistency refers to: the A matrix is m rows n columns, and the B matrix is also m rows n columns.

[0092] The normalized corneal curvature information, AK arc length and axis map, and dimensionality of the age information are spliced as the final input of the model.

[0093] For example, the splicing refers to the elements of the matrices A, B, and C at corresponding positions, and the D matrix is obtained, as shown in the following example:

[0094]

[0095] Among them, the image form of matrix A, B, and C can be seen from Figure 4 .

[0096] For postoperative corneal topography data 1 month, the output true target data is the normalized value of the original corneal curvature data of postoperative corneal topography 1 month. The reason for normalization is that the value range of the original corneal curvature data of postoperative corneal topography 1 month is relatively large, and normalization is performed to optimize model convergence;

[0097] Among them,

[0098] The normalization method is the same as the input data;

[0099] Further, the original corneal curvature data can be obtained by inverse normalization, and the inverse normalization formula is Y=(Normal_Y-a+k*Min) / k. According to the original data, the flat axis, steep axis, astigmatism and axis position on the 3mm ring and inside the ring can be further calculated, so as to obtain the real target data required by the generative adversarial network model for comparison;

[0100] S400, training a generative adversarial network model:

[0101] The input data of the generative adversarial network model, such as the D matrix described above, is input into the generator of the generative adversarial network, and the generator generates virtual target data;

[0102] The discriminator of the generative adversarial network model compares the virtual target data with the real target data to obtain a similarity;

[0103] And in the process of alternately fixing the discriminator or the generator, the generator or the discriminator is trained until they reach a balance;

[0104] It should be noted that,

[0105] The real target data refers to the normalized value of the original corneal curvature data of the postoperative corneal topography at 1 month (or, more preferably, 3 months) after the operation.

[0106] Training the generator or the discriminator until they reach a balance means that the similarity between the prediction result output by the generator and the real result no longer improves;

[0107] S500, predicting and planning astigmatism surgery with the trained generative adversarial network model.

[0108] It should be noted that, in actual prediction, the input is the information pointed to by the A, B, and C matrices described above, including: corneal topography curvature information, AK arc length and axis map information, and age information, wherein the AK arc length and axis information is given by a doctor;

[0109] The output obtained is the expected postoperative corneal topography at 1 month (or 3 months accordingly), for example:

[0110]

[0111] And the flat axis, steep axis, astigmatism and axis position on and in the 3mm ring calculated accordingly. According to the output, whether to optimize the AK arc length and axis information can be considered based on the principle of the lowest astigmatism.

[0112] In another embodiment,

[0113] After step S100 is performed and before step S300 is performed, the following steps are further included:

[0114] S200, for the preoperative and postoperative original data, the curvature radius of each point of the cornea is mapped to the dioptric power D of each point of the cornea, and the dioptric power D of each point of the cornea is processed according to a step length of 0.25D to perform pseudo-color visualization display of the corresponding preoperative and postoperative corneal topography, so as to realize the display of the Pentacam-like axial curvature map;

[0115] and draw the flat axis and steep axis of the 3mm central corneal region before and after surgery, and calculate the astigmatism and axis of the 3mm ring and the ring in the preoperative and postoperative conditions;

[0116] The pseudo-color visualization of the postoperative corneal topography and the astigmatism and axis of the 3mm ring and the ring in the postoperative condition are used to compare with the output of the generated adversarial network model in the training process to optimize the training of the generated adversarial network model and assist in planning the astigmatism surgery.

[0117] Exemplary,

[0118] The data obtained by the pentacam is expressed in the matrix as follows:

[0119]

[0120] First, according to the original data of the corneal topography, the curvature radius of each point of the cornea is mapped to the refractive power D of each point of the cornea, and the refractive power D of each point of the cornea is displayed in pseudo-color according to a step length of 0.25D to achieve the display of the axial curvature map similar to the Pentacam, and the flat axis and steep axis of the 3mm central corneal region are drawn to calculate the astigmatism and axis of the 3mm ring and the ring;

[0121] Second, as for the above matrix, the specific process is as follows:

[0122] 1. The mapping formula is dpt = ((1.3375-1)*1000) / R, where R is the curvature radius of each point of the cornea in millimeters, and dpt is the refractive power of each point of the cornea.

[0123] 2. The values in the range of 37.5 to 52.5 of the refractive power are pseudo-color visualized according to a step length of 0.25D, and the corresponding 16-bit color code from small to large is:

[0124] ["#a930cd", "#aa28d2", "#972fba", "#9628bd", "#8d28b4", "#7c2da2", "#7828a3", "#6d289d", "#612b8b", "#592891", "#4d2b82", "#3c158c", "#29008c", "#1e0390", "#1300b4", "#1d1ec8", "#2c3ec8", "#2950dc", "#2f64cc", "#1e6ed1", "#1478f0", "#1c89e6", "#0ba0fc", "#05a7fc", "#10b3ee", "#00c5fc", "#14caef", "#00e2fb", "#00f1fc", "#1cd89e", "#13d246", "#37e224", "#63de14", "#74f101", "#93f417", "#c8fd01", "#ffff00", "#f8ee1a", "#fce600", "#ecd700", "#d7c618", "#dcb800", "#d6aa16", "#dca100", "#dc9600", "#d38312", "#dc6e00", "#dc6000", "#cf540e", "#d24100", "#bb360b", "#be1800", "#b40001", "#9c0607", "#a40000", "#a41e1d", "#963735", "#aa4646", "#ac5453", "#be5a5a", "#c86464"], color bands as shown in Figure 5 .

[0125] 3. Traverse all diameters in the 3mm ring, calculate the mean of the refractive power of two points on the circle and the line inside the circle, and sort them, the angle between the diameter with the largest mean and the horizontal axis is the steep axis on the ring and the steep axis inside the ring, the normal vector of the steep axis is the flat axis on the ring and inside the ring, the astigmatism is the absolute value of the interpolation of the mean of the refractive power on the steep axis inside the ring and the mean of the refractive power on the flat axis inside the ring, and the visualization effect is shown in Figure 6 .

[0126] Figure 6 , wherein the axis inclined to the longitudinal direction is the steep axis, the refractive power on the ring is 46.10, and the refractive power inside the ring is 46.50, and the axis inclined to the transverse direction is the flat axis, the refractive power on the steep axis on the ring is 44.73, and the refractive power inside the ring is 44.66.

[0127] In one embodiment,

[0128] The preoperative and postoperative raw data are stored in a CSV table.

[0129] In one embodiment,

[0130] Step S200 further includes:

[0131] The raw data is read to parse the radius of curvature of the cornea, and then the radius of curvature is converted into the dioptric power D through the mapping relationship between the radius of curvature and the dioptric power.

[0132] In one embodiment,

[0133] Step S400 includes the following steps:

[0134] First stage: fixing the discriminator of the generative adversarial network model, training the generator of the generative adversarial network model, saving the virtual target data generated by the generator in the buffer pool, and if the similarity is less than a threshold, the discriminator feeds back to the generator, so that the generator can continue to train until the similarity is equal to or just greater than the threshold;

[0135] Second stage: fixing the generator, training the discriminator, and training the discriminator using the real target data and the virtual target data in the buffer pool; and reciprocating until the two reach equilibrium.

[0136] In one embodiment,

[0137] In step S400, the loss function is as follows:

[0138]

[0139] In one embodiment,

[0140] In step S400,

[0141] The input of the discriminator is: a real postoperative corneal topography or a corneal topography prediction map, wherein the generative adversarial network model can generate a postoperative corneal topography prediction map.

[0142] In one embodiment,

[0143] In step S400,

[0144] The generator uses the Unet network model as the generator of the GAN generative adversarial network model to extract feature information of the preoperative corneal topography, age, AK arc length and other input data. This network extracts abstract features while retaining more image details using low-level features to enhance the delicacy of the corneal topography prediction map.

[0145] In the down-sampling stage, the feature map of the patient corneal data is outputted by the multi-layer convolutional neural network; and in the up-sampling stage, the image is restored by transposed convolution to achieve the purpose of generating the image.

[0146] In the up-sampling stage, the low-level features and high-level features are spliced by using the skip layer connection to restore the image details, the high-level features retain the key features of the patient data, and the low-level features retain more image details.

[0147] It should be noted that, in one embodiment,

[0148] Referring to Figure 7 The neural network of the constructed generator has 30 layers in total, including an input layer, a down-sampling layer, an up-sampling layer, and an output layer, wherein:

[0149] The input layer is the first layer, and the size of the input multi-modal data is required to be 256*256*3;

[0150] The down-sampling layer is composed of 14 layers, and the function is feature extraction. The a-1 and a-2 layers are convolutional layers, the number of convolutional kernels is 64, the size of the convolutional kernel is 3*3, the step is 1, the padding mode is 0 padding, and a relu activation layer is arranged after the convolutional layer to improve the nonlinear fitting ability of the network. The a-3 layer is a maximum pooling layer, the window size is 2*2, the step is 2, and the size of the feature map output by the pooling layer will become smaller. The a-4 and a-5 layers are convolutional layers, in order to ensure that the number of features does not decrease, the number of convolutional kernels is increased to 128. The a-6 layer is a pooling layer, and the parameter configuration is the same as that of the a-3 layer. The a-7 and a-8 layers are convolutional layers, and the number of convolutional kernels is 256. The a-9 layer is a maximum pooling layer, and the parameter configuration is the same as that of the a-3 layer. The a-10 and a-11 layers are convolutional layers, and the number of convolutional kernels is 512. The a-12 layer includes a Dropout layer and a maximum pooling layer, and the dropout rate is 0.5. The a-13 and a-14 layers are convolutional layers, and the number of convolutional kernels is 1024.

[0151] The upsampling layer consists of 13 layers, aiming to restore image details and extract target regions. Layer b-1 is a transposed convolutional layer with a kernel size of 2×2. This layer enlarges the feature map size, and its output is concatenated with the output feature map of the downsampling layer a-11 to form the final output. This concatenation preserves more image details. Layers b-2 and b-3 are convolutional layers. As the feature map size increases, the number of convolutional kernels decreases. These two layers have 512 convolutional kernels with a kernel size of 3×3. Layer b-4 is a transposed convolutional layer with the same parameters as layer b-1 and a kernel size of 2×2. Its output is concatenated with the output feature map of the downsampling layer a-8 to form the final output. Layer b-5... Layer b-6 is a convolutional layer with 256 kernels and a kernel size of 3×3. Layer b-7 is a transposed convolutional layer with the same parameters as layer b-1. Its output is concatenated with the output feature map of downsampled layer a-5 to generate its output. Layers b-8 and b-9 are convolutional layers with 128 kernels and a kernel size of 3×3. Layer b-10 is a transposed convolutional layer with the same parameters as layer b-1. Its output is concatenated with the output feature map of downsampled layer a-2 to generate its output. Layers b-11 and b-12 are convolutional layers with 64 kernels and a kernel size of 3×3. Layer b-13 is a convolutional layer with 64 kernels and a kernel size of 2×2.

[0152] Output layer: consists of 2 layers, the purpose of which is to output the final simulation results. Layer c-1 is a convolutional layer with 1 convolutional kernel and a kernel size of 1×1. Layer c-2 is a sigmoid layer. The sigmoid layer normalizes the output results to obtain the final corneal curvature normalized prediction result. The final corneal curvature data is obtained by inverse normalization.

[0153] In one embodiment,

[0154] The discriminator of the generative adversarial network model adopts a fully convolutional network model, which is implemented using a multi-layer convolutional network. This breaks away from the traditional design of the discriminator in previous generative adversarial networks that only generates true and false scalars. It can be very easy to expand the output perception field of the discriminator model from pixels to blocks and image sizes.

[0155] In other words, due to the innovative work of this disclosure, the discriminator model is subdivided into three levels: pixel, block, and image. For these three discriminators:

[0156] A pixel-level discriminator is used to achieve pixel-by-pixel mapping from input space to input space, making the color space of the image more delicate and accurate, but it will cause the loss of geometric space information of the image.

[0157] The block-level discriminator is used to realize the mapping from a small area (such as 70x70) in the input space to a certain value in the output space, so that the geometric space information of the image is very accurate and not easy to lose, and the color space is also relatively uniform and smooth.

[0158] The image-level discriminator is a traditional discriminator design, and the output is a scalar. The obtained effect is the worst, and the model parameter quantity is also relatively large, which is not as good as the block-level discriminator. Therefore, the block-level discriminator is preferred in the disclosure.

[0159] In one embodiment,

[0160] Referring to Figure 8 which shows the structure of a block-level discriminator, the neural network has 10 layers in total, including an input layer, a convolutional layer and an output layer, wherein:

[0161] The input layer requires the input data to be the result of the generator and the result of the real label data after concatenation in the channel layer, and the size is 256x256;

[0162] The convolutional layer is mainly composed of 8 layers, and the function is to down-sample and extract features. The d-1 layer is a convolutional layer and includes a LeakyReLU activation layer, the activation coefficient is 0.2, the input channel number is 3, the output channel number is 64, the convolution kernel size is 4x4, the step is 2, the padding mode is the default zeros, and the padding size is 1; the d-2 layer is a convolutional layer, the input channel number is 64, the output channel number is 128, and the other parameter configurations are the same as those of the d-1 layer; the d-3 layer is a BatchNorm2d layer, and the input channel number is 128; the d-4 layer is a convolutional layer, the input channel number is 128, the output channel number is 256, and the other parameter configurations are the same as those of the d-1 layer; the d-5 layer is a BatchNorm2d layer, and the input channel number is 256. The d-6 layer is a convolutional layer, the input channel number is 256, the output channel number is 512, the step is 1, and the other parameter configurations are the same as those of the d-1 layer; the d-7 layer is a BatchNorm2d layer, and the input channel number is 512. The d-8 layer is a convolutional layer, the input channel number is 512, the output channel number is 1, the step is 1, and the other parameter configurations are the same as those of the d-1 layer;

[0163] The output layer is mainly the feature map of the last layer of the convolutional layer, and the size is 30x30.

[0164] It should be noted that each pixel on the feature map represents a 70x70 receptive field area on the original image, which can be calculated back by the following formula:

[0165] input_size l-1= (output_size l - 1) * k_stride l + k_size l

[0166] wherein input_size l-1 is the input size of the l-1 layer; output_size l is the output size of the l layer;

[0167] k_stride l is the stride of the convolution kernel of the current layer; and k_size l is the size of the convolution kernel of the current layer.

[0168] Referring to Figure 9 , in one embodiment, the overall structure of the network comprising the generator and the discriminator is illustrated. Through the combination of the generator and the discriminator, the prediction model is established.

[0169] In another embodiment,

[0170] The pre-experiment is completed based on the Ubuntu 18.04 system, the programming language is Python 3.6, the deep learning framework is PyTorch, the hardware environment is that the CPU is i7-8700 and the GPU is GTX-1080Ti.

[0171] In terms of data, the accuracy of the model depends largely on the quality of the training data set. In this project, real medical record images are collected from the clinic, and new data is continuously fed back to the data set in the subsequent clinical application of the system, so that the model is in continuous learning and optimization.

[0172] In addition, the disclosure also discloses a cylindrical surgery planning system based on a generative adversarial network model, wherein

[0173] The system comprises a processor, a memory;

[0174] The system is used to execute the method described in any of the preceding.

[0175] Referring to Figure 10 , in one embodiment, the schematic diagram of the corneal topography pseudo-color visualization for the same patient is illustrated; wherein the left graph is the axial curvature graph of the cornea of the patient derived by the Pentacam device; the right graph is the pseudo-color graph synthesized based on the refractive power values of each point of the cornea of the patient, and the visualization of the steep axis and the flat axis is superimposed, so as to facilitate the doctor to observe and compare with the output of the training or prediction of the generative adversarial network model, thereby assisting the training of the generative adversarial network model or assisting the doctor in planning the surgery.

[0176] Figure 11 is a schematic diagram of the postoperative corneal topography prediction of a patient in one embodiment of the present disclosure. Through the scheme disclosed by the present disclosure, the postoperative situation can be predicted.

[0177] Through the scheme disclosed by the present disclosure, in general,

[0178] The preoperative corneal topography, age, AK arc length and axial map are taken as model inputs, and the corneal topography at time points such as 1 week, 1 month, 3 months after the operation is taken as time sequence labels, the model is trained to enable it to predict the postoperative corneal topography through the preoperative data of the patient. Through the analysis of the postoperative corneal topography obtained by the prediction model, the corneal topography feature information (such as flatness, astigmatism and axis) in the central corneal 3mm region and 5mm region is obtained.

[0179] At present, the postoperative corneal topography prediction model for 1 month has been realized, and the results are as shown in Figure 11 The schematic diagram shows the comparison between the predicted results and the true results of the postoperative corneal topography (axial curvature) of two patients for 1 month, which illustrates the effect of predicting the postoperative corneal topography through the generator.

[0180] The units in the system of the embodiments of the present disclosure can be combined, divided and deleted according to actual needs. It should be noted that, for the foregoing steps, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described action sequence, because according to the present disclosure, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions, modules, units involved are not necessarily necessary for the present disclosure.

[0181] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0182] In several embodiments provided by the present disclosure, it should be understood that the disclosed system, module and unit can be implemented by other manners. For example, the above-described embodiments are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each unit or component can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical or other forms.

[0183] The units described as separate components may or may not be physically separate, and may be located in one place, or distributed over multiple network units. Some or all of the units can be selected as needed to achieve the purposes of the embodiments.

[0184] In addition, each functional unit in various embodiments of the present disclosure can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0185] The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present disclosure, essentially or in part, or all or part of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a smart phone, a personal digital assistant, a wearable device, a notebook computer, a tablet computer, a server, a cluster) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various other media that can store program codes.

[0186] The above-described embodiments are merely used to illustrate the technical solutions of the present disclosure, rather than limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not cause the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for astigmatic surgery prediction and planning based on a generative adversarial network model, comprising the following steps: S100, collecting a data set: obtaining preoperative corneal topography data of at least 100 patients, and postoperative corneal topography data of the at least 100 patients at least one month after surgery, wherein, the preoperative corneal topography data and the postoperative corneal topography data at one month are both derived from a corneal topography measuring device Pentacam: the curvature radius of each point on the cornea with the vertex of the cornea as the center and 7mm as the radius, in units of 0.1mm, so as to obtain the original data before and after surgery; S300, obtaining input data and real target data: obtaining input data of the generative adversarial network model based on the preoperative corneal topography data and information about AK surgery that has been implemented for the at least 100 patients; obtaining real target data to be compared by the generative adversarial network model based on the postoperative corneal topography data at one month; S400, training the generative adversarial network model: inputting the input data of the generative adversarial network model into a generator of the generative adversarial network, and the generator generates virtual target data; the discriminator of the generative adversarial network compares the virtual target data with the real target data to obtain a similarity; and in the process of alternately fixing the discriminator or the generator, the generator or the discriminator is trained until they reach a balance; S500, predicting and planning astigmatic surgery with the trained generative adversarial network model; in step S400, the generator adopts a Unet network model as the generator of the GAN generative adversarial network model to extract feature information of the preoperative corneal topography, age and AK arc length input data; wherein, for the patient corneal data, in the down-sampling stage, a multi-layer convolutional neural network is used to output a feature map of the patient corneal data; and in the up-sampling stage, the image is restored by transposed convolution to achieve the purpose of generating an image; in the up-sampling stage, in order to better restore the image details, a skip connection is used to splice the low-level features and the high-level features, the high-level features retain the key features in the patient data, and the low-level features retain more image details. 2.The method of claim 1, wherein, the original data before and after surgery is stored in the form of a CSV table. 3.The method of claim 1, wherein, step S300 further comprises: calculating the mean value of the refractive power of all points on each diameter within a 3mm ring, and sorting it to find the diameter with the maximum mean value of the refractive power within the ring, and the path corresponding to the diameter is the steep axis, and the path corresponding to the diameter perpendicular to the steep axis is the flat axis.

4. The method of claim 1, wherein, After step S100 is executed and before step S300 is executed, the following step is further included: S200, for the preoperative original data and postoperative original data, the corneal curvature radius of each point is mapped to the corneal power D of each point, and the preoperative and postoperative corneal topography is pseudo-color visualized according to the step length of 0.25D, so as to realize the display of the Pentacam axial curvature chart; And draw the flat axis and steep axis of the 3mm central corneal region before and after the operation, and calculate the astigmatism and axis of the 3mm ring and the ring under the preoperative and postoperative conditions respectively. Wherein, the pseudo-color visual display of the postoperative corneal topography and the astigmatism and axis of the 3mm ring and the ring under the postoperative condition are used to compare with the output of the generative adversarial network model in the training process, so as to optimize the training of the generative adversarial network model and assist in planning the astigmatism operation.

5. The method of claim 4, wherein, Step S200 further comprises: Reading the original data to analyze the curvature radius of the cornea, and then converting the curvature radius into the refractive power D through the mapping relationship between the curvature radius and the refractive power.

6. The method of claim 1, wherein, Step S400 comprises the following steps: The first stage: fixing the discriminator of the generative adversarial network model, training the generator of the generative adversarial network model, saving the virtual target data generated by the generator in the buffer pool, and if the similarity is less than a threshold, the discriminator feeds back to the generator, so that the generator can continue to train until the similarity is equal to or just greater than the threshold; The second stage: fixing the generator, training the discriminator, and training the discriminator using the real target data and the virtual target data in the buffer pool; This way, the two are balanced.

7. The method of claim 1, wherein, In step S400, The input of the discriminator is: the real postoperative corneal topography or the corneal topography prediction map.

8. The method of claim 1, wherein, The discriminator of the generative adversarial network model adopts a full convolution network model.

9. A astigmatism operation planning system based on a generative adversarial network model, wherein, The system comprises a processor, a memory; The system is used to execute the method of any one of claims 1 to 8.

Images