Anterior segment structure prediction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an anterior segment structure prediction method and device, electronic equipment and a storage medium. The method comprises the following steps: encoding an anterior segment structure image and an ICL implantation direction before an ICL operation through an encoder module to obtain anterior segment structure image features; performing iris image generation on the anterior segment structure image features based on a first auxiliary decoder, a second auxiliary decoder and a geometric information sharing module to obtain an iris segmentation image output by the first auxiliary decoder and an iris contour and crystalline lens upper surface image output by the second auxiliary decoder; decoding the anterior segment structure image features based on a main decoder and a first auxiliary decoder to obtain an ICL postoperative anterior segment structure image; and displaying an ICL postoperative anterior segment structure image, an iris segmentation image and an iris contour and crystalline lens upper surface image in the ICL implantation direction. According to the embodiment of the invention, postoperative anterior segment structure prediction can be realized, and the accuracy is high.

Description

Technical Field

[0001] The present application relates to the field of deep learning and image processing technology, and in particular to an anterior segment structure prediction method and device, an electronic device, and a storage medium. Background Art

[0002] Implantable Collamer Lens (ICL) is a soft elastic gel lens made of collagen polymer material. ICL surgery is an ophthalmic surgery that implants ICL into the eye for refractive correction of hyperopia, myopia, etc. When planning ICL surgery, it is necessary to predict the postoperative arch height, anterior chamber angle and other parameters through the anterior segment structure parameters before surgery, so as to plan the ICL size and ICL implantation direction selected during surgery, which is very important for reducing postoperative complications and achieving better postoperative recovery effects.

[0003] Related technologies generally use regression models to predict the postoperative anterior chamber angle or postoperative arch height, such as using parameters such as the preoperative trabecular iris angle and scleral apex angle to predict the postoperative anterior chamber angle and other angle-related parameters. These ICL postoperative prediction technology solutions usually use the anterior chamber structure parameters before ICL surgery to predict a single postoperative parameter (such as anterior chamber angle, arch height, etc.). The above solution is actually a numerical prediction technology. However, the prediction of the anterior chamber structure is relatively complex, and it is impossible to measure the overall anterior chamber structure state after ICL surgery with a single parameter, that is, it is impossible to predict the postoperative anterior chamber structure. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and device for predicting the anterior segment structure, an electronic device and a storage medium, aiming to achieve prediction of the anterior segment structure with high accuracy.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an anterior segment structure image prediction method, the method comprising:

[0006] Obtain anterior segment structural images and ICL implantation direction before ICL surgery;

[0007] The anterior segment structure image before ICL surgery and the ICL implantation direction information are input into a pre-trained anterior segment structure image generation model, wherein the anterior segment structure image generation model comprises an encoder module, a decoder module and a geometric information sharing module, wherein the decoder module comprises a main decoder, a first auxiliary decoder and a second auxiliary decoder;

[0008] The encoder module is used to encode the anterior segment structural image of the eye before ICL surgery and the implantation direction of the ICL to obtain anterior segment structural image features;

[0009] Based on the first auxiliary decoder, the second auxiliary decoder and the geometric information sharing module, an iris image is generated for the anterior segment structure image features to obtain an iris segmentation image output by the first auxiliary decoder and an iris contour and a lens upper surface image output by the second auxiliary decoder;

[0010] Decoding the anterior segment structure image features based on the main decoder and the first auxiliary decoder to obtain an anterior segment structure image after ICL surgery;

[0011] The anterior segment structure image after ICL surgery, the iris segmentation image, and the iris contour and lens upper surface image are displayed in the direction of ICL implantation.

[0012] In some embodiments, the first auxiliary decoder includes a first upsampling layer, a first splicing layer, and a first fusion layer, the second auxiliary decoder includes a second upsampling layer, a second splicing layer, and a second fusion layer, the anterior segment structure image generation model also includes a first output layer and a second output layer, and the iris image generation based on the first auxiliary decoder, the second auxiliary decoder, and the geometric information sharing module is performed on the anterior segment structure image features to obtain the iris segmentation image output by the first auxiliary decoder and the iris contour and lens upper surface image output by the second auxiliary decoder, including:

[0013] Upsampling the anterior segment structure image feature through the first upsampling layer to obtain a first anterior segment structure feature reconstructed image;

[0014] Upsampling the anterior segment structure image feature through the second upsampling layer to obtain a second anterior segment structure feature reconstructed image;

[0015] Extracting geometric information from the first anterior segment structure feature reconstructed image and the second anterior segment structure feature reconstructed image by a geometric information sharing module to obtain a geometric information sharing result;

[0016] Splicing the first anterior segment structure feature reconstructed image and the anterior segment structure image feature through the first splicing layer to obtain a first feature image splicing result;

[0017] Splicing the second anterior segment structure feature reconstructed image and the anterior segment structure image feature through the second splicing layer to obtain a second feature image splicing result;

[0018] The first fusion layer is used to fuse the first feature image stitching result and the geometric information extraction result to obtain a first image fusion result;

[0019] The second feature image stitching result and the geometric information extraction result are fused through the second fusion layer to obtain a second image fusion result;

[0020] Upsampling the first image fusion result through a first output layer to output an iris segmentation image;

[0021] The second image fusion result is upsampled through the second output layer to output the iris contour and the upper surface image of the lens.

[0022] In some embodiments, the geometric information sharing module includes a convolution layer, an average pooling layer, a maximum pooling layer, and an activation layer. The geometric information sharing module is used to extract geometric information from the first anterior segment structural feature reconstructed image and the second anterior segment structural feature reconstructed image to obtain a geometric information sharing result, including:

[0023] splicing the first anterior segment structure feature reconstructed image and the second anterior segment structure feature reconstructed image to obtain a third feature image splicing result;

[0024] Performing feature extraction on the third feature image splicing result based on the convolution layer to obtain a geometric feature extraction result;

[0025] Based on the maximum pooling layer, channel maximum pooling is performed on the third feature image splicing result to obtain a maximum pooling result;

[0026] Based on the average pooling layer, channel average pooling is performed on the third feature image splicing result to obtain an average pooling result;

[0027] splicing the maximum pooling result, the average pooling result and the geometric feature extraction result to obtain a fourth feature image splicing result;

[0028] The fourth feature image splicing result is nonlinearly transformed through the activation layer to obtain the geometric information sharing result.

[0029] In some embodiments, before acquiring the anterior segment structure image before ICL surgery, the method further includes: training the anterior segment structure image generation model, and the training process specifically includes:

[0030] Acquire an image sample set and an ICL implantation direction, wherein the image sample set includes a plurality of anterior segment structure image samples, a reference iris segmentation image, and a reference iris contour and lens image;

[0031] The anterior segment structure image sample is input and the ICL implantation direction is input to the anterior segment structure image generation model for image prediction, so as to obtain an anterior segment structure prediction image, an iris segmentation prediction image and an iris contour and lens prediction image after ICL surgery in the ICL implantation direction;

[0032] Inputting the predicted image of the anterior segment structure after ICL surgery into a preset anterior segment structure prediction discriminator for image discrimination to obtain an image discrimination result, and determining the adversarial loss according to the image discrimination result;

[0033] Determine a first loss according to the iris segmentation prediction image and the reference iris segmentation image, and determine a second loss according to the iris contour and lens prediction image and the reference iris contour and lens image;

[0034] A target loss is determined according to the first loss, the second loss and the adversarial loss, and model parameters of the anterior segment structure image generation model are updated based on the target loss.

[0035] In some embodiments, determining the first loss according to the iris segmentation prediction image and the reference iris segmentation image includes:

[0036] Get the preset iris root position weight map;

[0037] Determining a binary cross entropy loss according to the iris segmentation prediction image and the reference iris segmentation image;

[0038] The binary cross entropy loss is weighted according to the iris root position weight map to obtain the first loss.

[0039] In some embodiments, obtaining an anterior segment structure image before ICL surgery includes:

[0040] Adjust the brightness of the input anterior segment structure image before ICL surgery according to a preset brightness adjustment curve to obtain a brightness adaptation image of the anterior segment structure before ICL surgery, wherein there are multiple brightness adjustment curves;

[0041] Performing image stitching on a plurality of brightness adaptation images of the anterior segment structure of the ICL before surgery with different brightness to obtain a multi-brightness stitching image;

[0042] Based on a preset attention module, feature weights are assigned to the multi-brightness spliced ​​image to obtain the anterior segment structure image before ICL surgery.

[0043] In some embodiments, the anterior segment structure image before ICL surgery also includes a corresponding ICL size, and the method further includes:

[0044] Inputting the anterior segment structure image before ICL surgery into the anterior segment structure image generation model to generate an anterior segment structure image after ICL surgery corresponding to the ICL size;

[0045] Recommending the ICL implantation direction based on the anterior segment structure image after ICL surgery and the corresponding ICL implantation direction;

[0046] The ICL size recommendation is made based on the anterior segment structure image after ICL surgery and the corresponding ICL size.

[0047] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an anterior segment structure image prediction device, which is applied to the method described in the first aspect.

[0048] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0049] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0050] The anterior segment structure image prediction method device, electronic device and storage medium proposed in the present application provide a visual method for predicting the anterior segment structure after ICL surgery. It obtains the anterior segment structure image before ICL surgery and the ICL implantation direction, and predicts the anterior segment structure image, iris segmentation image and iris contour and lens upper surface image after ICL surgery corresponding to the ICL implantation direction based on the pre-trained anterior segment structure image generation model. The input and output of the generation model are both images. Compared with the related art that can only perform numerical prediction on the anterior segment structure parameters and can only predict a single postoperative parameter (such as arch height, anterior chamber angle, etc.), the prediction method provided in the embodiment of the present application can more comprehensively and accurately predict the anterior segment structure after ICL surgery, which is helpful for analyzing complex anterior segment structure states. In addition, the present application obtains the anterior segment structure image features through the encoder module. On the basis of setting the main decoder to decode the anterior segment structure image features and generating the anterior segment structure image after ICL surgery, a first auxiliary decoder and a second auxiliary decoder are also set. The main decoder is combined with the first auxiliary decoder. The first auxiliary decoder can guide the main decoder to focus on important anterior segment structures, such as iris position, to improve the accuracy of postoperative image prediction. The first auxiliary decoder and the second auxiliary decoder can respectively output iris segmentation images, iris contours and lens upper surface images, which are helpful to observe important structures of the anterior segment and ultimately improve the prediction accuracy of the generated model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the anterior segment structure after ICL surgery provided in an embodiment of the present application;

[0052] Figure 2 is a flow chart of the anterior segment structure prediction method provided by an embodiment of the present application;

[0053] Figure 3 is a schematic diagram of an AS-OCT modality image provided in an embodiment of the present application;

[0054] Figure 4 is a schematic diagram of an input image or an output image provided by an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of a multi-brightness adaptation module provided in an embodiment of the present application;

[0056] Figure 6 It is a schematic diagram of the framework of the anterior segment structure image generation model provided in the embodiment of the present application;

[0057] Figure 7 is a schematic diagram of a geometric information sharing module provided in an embodiment of the present application;

[0058] Figure 8 It is a training block diagram of the anterior segment structure image generation model provided in the embodiment of the present application;

[0059] Fig. 9 is a schematic diagram of the structure of an anterior segment structure prediction device provided in an embodiment of the present application;

[0060] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] Before introducing the anterior segment structure prediction method provided by this application, the relevant background of ICL surgery is first explained. Figure 1 , Figure 1 Schematic diagram of the anterior segment structure of the eye after ICL surgery provided in the embodiment of the present application. The anterior segment structure mainly includes ciliary muscle, cornea, iris and lens. ICL includes lens body and ICL antenna. ICL surgery is a vision correction surgery in which an ophthalmologist implants an ICL lens into the eye. During the surgery, the ICL lens is pushed into the posterior chamber. After implantation, the ICL is located in front of the lens and behind the iris.

[0065] Postoperative vault is a crucial parameter in ICL surgical planning. Vault is defined as the maximum distance between the posterior surface of the ICL and the anterior surface of the lens, or by connecting two SS points and taking the median perpendicular line. The median perpendicular line and the posterior surface of the ICL and the anterior surface of the lens each produce an intersection, and the distance between these two intersections is the vault. Too low a vault may cause secondary cataracts, and too high a vault may cause angle stenosis and closed glaucoma. The anterior chamber angle is the angle between the cornea and the iris. A narrow anterior chamber angle may cause poor aqueous humor flow, increased intraocular pressure, and damage to the optic nerve. Therefore, a reasonable choice of ICL size and the correct implant direction can help avoid postoperative complications.

[0066] In the related technologies, some technical solutions use regression models or integrated models to use the anterior segment structural parameters of the ICL before surgery, such as sulcus to sulctus (STS), white to white (WTW), and anterior chamber depth (ACD) to predict the postoperative arch height and recommend the ICL size. Some technical solutions use regression models to predict the postoperative chamber angle state, for example, using parameters such as preoperative trabecular iris angle (TIA) and scleral spur angle (SSA) to predict the postoperative anterior chamber angle (ACA) and other chamber angle-related parameters.

[0067] However, the related technology can only predict a single postoperative parameter based on some preoperative anterior segment structural parameters, such as arch height and anterior chamber angle. This scheme is actually a numerical prediction technology. However, the anterior segment structural state is relatively complex, and it is impossible to measure the overall anterior segment structural state after ICL surgery with a single parameter, resulting in low accuracy of anterior segment structural prediction.

[0068] Based on this, the embodiments of the present application provide an anterior segment structure prediction method and device, an electronic device and a storage medium, aiming to improve the accuracy of anterior segment structure prediction. It should be noted that the anterior segment structure prediction method provided in the embodiments of the present application can accurately predict and display the anterior segment structure image after ICL surgery, and can analyze the anterior segment structure image before surgery and give the corresponding ICL recommended size and ICL recommended implantation direction.

[0069] It is known to those skilled in the art that the ICL size is determined based on the width of the human eye and other related factors. The width of the human eye is basically fixed, and the ICL sizes generally include: 12.1mm, 12.6mm, 13.2mm and 13.7mm. The implantation direction of the ICL includes horizontal implantation and vertical implantation.

[0070] The anterior segment structure prediction method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the anterior segment structure prediction method in the embodiments of the present application is described.

[0071] The anterior segment structure prediction method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the anterior segment structure prediction method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] Figure 2 is an optional flow chart of the anterior segment structure prediction method provided in the embodiment of the present application, Figure 2 The method may include but is not limited to steps 201 to 206.

[0074] Step 201, obtaining an anterior segment structure image and an ICL implantation direction before ICL surgery;

[0075] Step 202, inputting the anterior segment structure image before ICL surgery and the ICL implantation direction into a pre-trained anterior segment structure image generation model;

[0076] Step 203, encoding the anterior segment structural image and the ICL implantation direction before ICL surgery through an encoder module to obtain anterior segment structural image features;

[0077] Step 204, generating an iris image based on the anterior segment structure image features by the first auxiliary decoder, the second auxiliary decoder and the geometric information sharing module, to obtain an iris segmentation image output by the first auxiliary decoder and an iris contour and lens upper surface image output by the second auxiliary decoder;

[0078] Step 205, decoding the anterior segment structure image features based on the main decoder and the first auxiliary decoder to obtain an anterior segment structure image after ICL surgery;

[0079] Step 206, displaying the anterior segment structure image, iris segmentation image, and iris contour and lens upper surface image after ICL surgery in the direction of ICL implantation.

[0080] In step 201 of some embodiments, the anterior segment structure image obtained before ICL surgery can be an image of any modality, which can be an anterior segment-optical coherence tomography (AS-OCT) modality image or an ultrasound biomicroscopy (UBM) modality image, which is not limited here. The present application embodiment is described with an AS-OCT modality image, see Figure 3 , Figure 3 It is a schematic diagram of the AS-OCT modality image provided in the embodiment of the present application. Figure 3 The left middle image is a preoperative anterior segment OCT image before the ICL lens is implanted. The positions of the cornea, iris, and lens are marked in the left image. Figure 3 The middle right image is the AS-OCT modality image of the anterior segment structure after the implantation of the ICL lens. It can be seen that the structure of the anterior segment of the eye has changed accordingly with the implantation of the lens.

[0081] Specifically, the anterior segment structure image before ICL surgery includes the corresponding ICL implantation direction, that is, the anterior segment structure image after ICL surgery under the current ICL implantation direction is predicted. Figure 4 , Figure 4 Schematic diagram of an input image or output image provided in an embodiment of the present application. The input image is an anterior segment structure image before ICL surgery, and the output image is an anterior segment structure image after ICL surgery. The input anterior segment structure image before ICL surgery can be unidirectional or multi-directional, and the predicted anterior segment structure image after ICL surgery can also be unidirectional or multi-directional.

[0082] For example, by inputting a unidirectional ICL preoperative anterior segment structure image, the model can be controlled to output a multi-directional ICL postoperative anterior segment structure image or a unidirectional ICL postoperative anterior segment structure image according to the requirements; by inputting a multi-directional ICL preoperative anterior segment structure image, the model can be controlled to output a multi-directional ICL postoperative anterior segment structure image or a unidirectional ICL postoperative anterior segment structure image according to the requirements.

[0083] Figure 4In the figure, each solid black line represents an anterior segment image corresponding to the angle thereof, where the angle is the angle between the projection of a section parallel to the sagittal axis on the coronal plane and the cross section. A single direction image is a cross-sectional view corresponding to a single angle section. The black lines represent sections, and the sections are planes parallel to the sagittal axis. The cross-sectional view seen by cutting into the section is the ICL preoperative anterior segment structure image used in the embodiment of the present application or the ICL postoperative anterior segment structure image generated by the model. The figure on the left represents an image at a single angle, the image in the middle represents multiple images centered on the insertion direction (here taking the horizontal implantation of the ICL as an example) and equally spaced front and back, and the image on the right represents an image in all directions.

[0084] In some embodiments, obtaining an anterior segment structure image before ICL surgery includes:

[0085] The brightness of the input anterior segment structure image before ICL surgery is adjusted according to a preset brightness adjustment curve to obtain a brightness adaptation image of the anterior segment structure before ICL surgery, and there are multiple brightness adjustment curves;

[0086] Image stitching is performed on multiple brightness adaptation images of the anterior segment structure before ICL surgery with different brightness to obtain a multi-brightness stitching image;

[0087] Based on the preset attention module, feature weights are assigned to the multi-brightness stitching image to obtain the anterior segment structure image before ICL surgery.

[0088] Specifically, see Figure 5 , Figure 5 It is a schematic diagram of a multi-brightness adaptation module provided in an embodiment of the present application. Five brightness adjustment curves are preset, and the brightness of the input ICL preoperative anterior segment structure image is adjusted to obtain the ICL preoperative anterior segment structure brightness adaptation image. The ICL preoperative anterior segment structure brightness adaptation image under different brightness can be spliced ​​to obtain a multi-brightness spliced ​​image. Further, based on the preset attention module, the multi-brightness spliced ​​image is assigned feature weights to obtain the ICL preoperative anterior segment structure image. Among them, the attention module is a convolutional attention module (Convolutional BlockAttention Module, CBAM), including a channel attention module and a spatial attention module. The channel attention module calculates the importance of each channel to distinguish the features between different channels. The spatial attention module calculates the importance of each pixel in space to better capture the spatial structure in the image. The embodiment of the present application is processed by a variety of brightness adjustments and attention modules, so that the anterior segment structure image generation model can adapt to images of different brightness.

[0089] In step 202 of some embodiments, the anterior segment structure image before ICL surgery and the ICL implantation direction information are input into a pre-trained anterior segment structure image generation model. The ICL implantation direction information may be text information, and the input text information may also include ICL size information. Specifically, the pre-trained anterior segment structure image generation model may include an encoder module, a decoder module, and a geometric information sharing module, and the decoder module may include a main decoder, a first auxiliary decoder, and a second auxiliary decoder.

[0090] Please refer to Figure 6 , Figure 6 Schematic diagram of the framework of the anterior segment structure image generation model provided by the embodiment of the present application. Specifically, the anterior segment structure image generation model may include an encoder module and a decoder module, and the composition of the anterior segment structure image generation model will be introduced in detail later.

[0091] In step 203 of some embodiments, the encoder module can be used to encode the anterior segment structural image before ICL surgery, extract image features, and obtain anterior segment structural image features. Specifically, the encoder module may include multiple downsampling modules, three of which are taken as examples here. Each downsampling module may include a convolution layer, an activation function layer, and a normalization (Norm) layer. The image resolution is reduced by downsampling to speed up the calculation process. The anterior segment structural image generation model provided in the embodiment of the present application adopts a skip connection design (Skip Connection), that is, the first downsampling module is not spliced ​​with the decoder module, and the decoding results of the output of the main decoder are respectively spliced ​​(Concatenation) with the features of the corresponding downsampling module output in the encoder module, so as to achieve the effect of fusing multi-scale information and improve the accuracy of anterior segment structural prediction.

[0092] In step 204 of some embodiments, iris image generation is performed based on the first auxiliary decoder, the second auxiliary decoder and the geometric information sharing module for the anterior segment structure image features to obtain an iris segmentation image output by the first auxiliary decoder and an iris contour and lens upper surface image output by the second auxiliary decoder, including but not limited to the following steps:

[0093] Upsampling the anterior segment structure image features through a first upsampling layer to obtain a first anterior segment structure feature reconstructed image;

[0094] Upsampling the anterior segment structure image features through a second upsampling layer to obtain a second anterior segment structure feature reconstructed image;

[0095] Extracting geometric information from the first anterior segment structural feature reconstructed image and the second anterior segment structural feature reconstructed image through a geometric information sharing module to obtain a geometric information sharing result;

[0096] The first anterior segment structure feature reconstructed image and the anterior segment structure image feature are spliced ​​through the first splicing layer to obtain a first feature image splicing result;

[0097] The second anterior segment structure feature reconstructed image and the anterior segment structure image feature are spliced ​​through the second splicing layer to obtain a second feature image splicing result;

[0098] The first feature image splicing result and the geometric information extraction result are fused through the first fusion layer to obtain a first image fusion result;

[0099] The second feature image splicing result and the geometric information extraction result are fused through the second fusion layer to obtain a second image fusion result;

[0100] Up-sampling the first image fusion result through the first output layer, and outputting an iris segmentation image;

[0101] The second image fusion result is upsampled through the second output layer to output the iris contour and the upper surface image of the lens.

[0102] Specifically, continue to refer to Figure 6 , the decoder module may include three branches, a main decoder branch, a first auxiliary decoder branch and a second auxiliary decoder branch (not marked in the figure). The main decoder branch may include multiple main decoders, the first auxiliary decoder branch may include multiple first auxiliary decoders, and the second auxiliary decoder branch may include multiple second auxiliary decoders. The main decoder may include an upsampling module and two splicing layers. The first auxiliary decoder may include a first upsampling layer, a first splicing layer and a first fusion layer; the second auxiliary decoder may include a second upsampling layer, a second splicing layer and a second fusion layer. Among them, the number of upsampling layers, splicing layers and fusion layers is not limited and can be set according to actual needs. The embodiment of the present application takes two main decoders, two first auxiliary decoders and two second auxiliary decoders as an example. The first auxiliary decoder and the second auxiliary decoder can guide the anterior segment structure image generation model to focus on important anterior segment structures, such as the root position of the iris, through different network parameter settings.

[0103] Furthermore, the anterior segment structure image feature can be upsampled by the first upsampling layer of the first auxiliary decoder to obtain a first anterior segment structure feature reconstructed image. The first anterior segment structure feature reconstructed image and the anterior segment structure image feature output by the third downsampling module in the encoder module are feature spliced ​​by the first splicing layer of the first auxiliary decoder to obtain a first feature image splicing result. In addition, the first anterior segment structure feature reconstructed image is used as one of the inputs of the geometric information sharing module.

[0104] Then, the anterior segment structure image features can be upsampled through the second upsampling layer to obtain a second anterior segment structure feature reconstructed image. The second anterior segment structure feature reconstructed image and the anterior segment structure image features output by the third downsampling module in the encoder module are feature spliced ​​through the second splicing layer to obtain a second feature image splicing result. In addition, the second anterior segment structure feature reconstructed image is used as another input of the geometric information sharing module.

[0105] In some embodiments, the geometric information sharing module extracts geometric information from the first anterior segment structural feature reconstructed image and the second anterior segment structural feature reconstructed image to obtain a geometric information sharing result, including but not limited to the following steps:

[0106] splicing the first anterior segment structural feature reconstructed image and the second anterior segment structural feature reconstructed image to obtain a third feature image splicing result;

[0107] Perform feature extraction on the splicing result of the third feature image based on the convolution layer to obtain a geometric feature extraction result;

[0108] Based on the maximum pooling layer, channel maximum pooling is performed on the splicing result of the third feature image to obtain a maximum pooling result;

[0109] Based on the average pooling layer, channel average pooling is performed on the splicing result of the third feature image to obtain an average pooling result;

[0110] The maximum pooling result, the average pooling result and the geometric feature extraction result are spliced ​​to obtain a fourth feature image splicing result;

[0111] The fourth feature image splicing result is nonlinearly transformed through the activation layer to obtain the geometric information sharing result.

[0112] The geometric information sharing module is used to extract geometric information from the first anterior segment structural feature reconstructed image and the second anterior segment structural feature reconstructed image to obtain a geometric information sharing result. The geometric information sharing result can help the generation model pay more attention to image information with geometric characteristics in the image to distinguish different shapes or structures in the image, such as the position of the root of the iris.

[0113] Please refer to Figure 7 , Figure 7: is a schematic diagram of a geometric information sharing module provided in an embodiment of the present application. The geometric information sharing module includes a convolution layer, a maximum pooling layer, an average pooling layer, and an activation layer. The convolution layer includes a 1X3 convolution kernel and a 3X1 convolution kernel. It can be understood that the number of convolution layers, maximum pooling layers, average pooling layers, and activation layers in the geometric information sharing module can be set according to actual needs. For example, the number of activation layers and pooling layers can be four layers, five layers, or seven layers, etc., which are not limited here. Specifically, the first anterior segment structure feature reconstructed image and the second anterior segment structure feature reconstructed image can be spliced ​​to obtain a third feature image splicing result. The third feature image splicing result can be a feature map obtained by splicing the feature map output by the first upsampling layer in the first auxiliary decoder and the feature map output by the second upsampling layer in the second auxiliary decoder. The feature map can be represented by a three-dimensional array, and the three-dimensional array can include three dimensions: height, width, and number of channels. The first auxiliary decoder branch focuses on the features of the iris root endpoint position, and the second auxiliary decoder branch focuses on the features of the iris and lens upper surface contours.

[0114] Furthermore, the splicing result of the third feature image is input into the convolution layer, and the features in the horizontal direction of the spliced ​​feature image are extracted by a 1X3 convolution kernel, and the features in the vertical direction of the spliced ​​feature image are extracted by a 3X1 convolution kernel, and the features in the horizontal direction and the features in the vertical direction are pixel-fused to obtain the geometric feature extraction result. The embodiment of the present application strengthens the feature representation through two asymmetric convolution kernels and fuses features of different shapes, which helps to capture the geometric structure in the feature image.

[0115] The splicing result of the third feature image is input into the maximum pooling layer and the average pooling layer respectively. The maximum pooling layer performs channel maximum pooling on the splicing result of the third feature image to obtain the maximum pooling result. The average pooling layer performs channel average pooling on the splicing result of the third feature image to obtain the average pooling result. This method can help extract important information from different channels in the feature image while reducing the impact of noise.

[0116] Furthermore, the maximum pooling result output by the maximum pooling layer, the average pooling result output by the average pooling layer, and the geometric feature extraction result information output by the convolution layer are spliced ​​to obtain a fourth feature image splicing result. Splicing feature maps extracted from different sources can enhance the model's understanding and representation of images. The feature map obtained in the above steps, that is, the fourth feature image splicing result, is nonlinearly transformed by the activation layer to enhance the geometric information in the feature map or form a new feature map to obtain a geometric information sharing result. Among them, the activation function used by the activation layer is the sigmoid function.

[0117] The geometric information sharing result output by the activation layer is multiplied pixel by pixel with the first anterior segment structural feature reconstructed image to obtain the first anterior segment structural feature updated image, and the geometric information sharing result output by the activation layer is multiplied pixel by pixel with the second anterior segment structural feature reconstructed image to obtain the second anterior segment structural feature updated image. The first anterior segment structural feature updated image is spliced ​​back to the first auxiliary decoder branch, that is, input to the first fusion layer of the first auxiliary decoder, and the second anterior segment structural feature updated image is spliced ​​back to the second auxiliary decoder branch, that is, input to the second fusion layer of the second auxiliary decoder. It can be understood that if there are multiple first auxiliary decoders, second auxiliary decoders and geometric information sharing modules, the above steps can be repeated multiple times.

[0118] In some embodiments, the first anterior segment structure feature reconstructed image and the anterior segment structure image feature are spliced ​​together through a first splicing layer to obtain a first feature image splicing result; the second anterior segment structure feature reconstructed image and the anterior segment structure image feature are spliced ​​together through a second splicing layer to obtain a second feature image splicing result. For details, please refer to Figure 6 , the first anterior segment structure feature update image output by the geometric information sharing module is input to the first fusion layer of the first decoder, and another input of the first fusion layer is the first feature image splicing result. The first fusion layer multiplies the first anterior segment structure feature update image and the first feature image splicing result pixel by pixel to obtain the first image fusion result.

[0119] Furthermore, the second anterior segment structure feature update image output by the geometric information sharing module is input to the second fusion layer of the second decoder, and another input of the second fusion layer is the second feature image splicing result. The second fusion layer multiplies the second anterior segment structure feature update image and the second feature image splicing result pixel by pixel to obtain the second image fusion result. The first image fusion result output by the first fusion layer is input to the second splicing layer of the main decoder, and feature splicing is performed with the feature map obtained by the main decoder branch to ensure that when generating the anterior segment structure image after ICL surgery, the iris root position can be focused on to improve the prediction performance of the model.

[0120] In step 205 of some embodiments, the anterior segment structural image features are decoded based on the main decoder and the first auxiliary decoder to obtain the anterior segment structural image after ICL surgery. The main decoder includes an upsampling layer and two splicing layers. The upsampling module is used to upsample the anterior segment structural image features output by the encoder module, and splice the upsampling result with the anterior segment structural image features output by the third downsampling module in the encoder module based on the splicing layer in the main decoder, and restore the feature information lost by downsampling through feature splicing. Furthermore, the feature splicing result output by the first splicing layer and the first anterior segment structural feature update image output by the geometric information sharing module are input to the second splicing layer of the main decoder, guiding the main decoder to generate an anterior segment structural feature map focusing on the root position of the iris.

[0121] In some embodiments, an iris image is generated based on the anterior segment structural image features of the first auxiliary decoder, the second auxiliary decoder and the geometric information sharing module. Specifically, the anterior segment structural image generation model may also include a first output layer corresponding to the first auxiliary decoder branch, the first output layer is also an upsampling layer, the output of the first auxiliary decoder is upsampled through the first output layer, and a high-resolution iris root position image is output. The anterior segment structural image generation model may also include a second output layer corresponding to the second auxiliary decoder branch, the second output layer is also an upsampling layer, the output of the second auxiliary decoder is upsampled through the second output layer, and a high-resolution iris contour and lens map is output. In addition, the main branch where the main decoder is located may also include a corresponding target output layer, and the target output layer is used to upsample the anterior segment structural feature map output by the main decoder, and output a high-resolution ICL postoperative anterior segment structural image.

[0122] In some embodiments, before obtaining the anterior segment structure image before ICL surgery, the method may further include: training an anterior segment structure image generation model, wherein the training process specifically includes:

[0123] Acquire an image sample set and an ICL implantation direction, the image sample set including a plurality of anterior segment structure image samples;

[0124] The anterior segment structure image samples and the ICL implantation direction are input into the anterior segment structure image generation model for image prediction, and the ICL postoperative anterior segment structure prediction image, iris segmentation prediction image, and iris contour and lens prediction image of the ICL implantation direction are obtained;

[0125] Inputting the predicted anterior segment structure image after ICL surgery into a preset anterior segment structure prediction discriminator for image discrimination, obtaining an image discrimination result, and determining the adversarial loss according to the image discrimination result;

[0126] Determine a first loss according to the iris segmentation prediction image and the reference iris segmentation image, and determine a second loss according to the iris contour and lens prediction image and the reference iris contour and lens image;

[0127] A target loss is determined according to the first loss, the second loss and the adversarial loss, and model parameters of an anterior segment structure image generation model are updated based on the target loss.

[0128] Specifically, see Figure 8 , Figure 8 It is a training block diagram of the anterior segment structure image generation model provided by the embodiment of the present application. An image sample set and an ICL implantation direction are obtained, and the image sample set includes multiple anterior segment structure image samples for training the anterior segment structure image generation model. Among them, the multiple anterior segment structure image samples include paired ICL preoperative anterior segment structure image samples and ICL postoperative anterior segment structure image samples. The ICL preoperative anterior segment structure image sample is input into the anterior segment structure image generation model for image prediction, and the ICL postoperative anterior segment structure prediction image, iris segmentation prediction image and iris contour and lens prediction image under the corresponding ICL implantation direction are obtained. Among them, the iris contour and lens prediction image are the prediction images of the iris contour and the upper surface of the lens. The embodiment of the present application uses a generative adversarial network to predict the anterior segment structure after surgery. The generative adversarial network is composed of a generator-discriminator, wherein the anterior segment structure image generation model is a generator, which is responsible for generating the postoperative anterior segment structure image, and the anterior segment structure prediction discriminator is responsible for judging whether the image is a real image or a generated image. The ICL postoperative anterior segment structure prediction image is input into the preset anterior segment structure prediction discriminator for image discrimination. The anterior segment structure prediction discriminator can compare the ICL postoperative anterior segment structure prediction image generated by the generator with the corresponding ICL postoperative anterior segment structure image sample to obtain the image discrimination result. Then, the adversarial loss can be determined based on the image discrimination result. As the adversarial loss decreases, the generator can generate data that is more similar to the target domain data, and the discriminator can improve the ability to distinguish the generated data from the original data. The adversarial loss is expressed as L adv (G,D), the specific expression is as follows:

[0129] L adv (G,D)=E[log(D(x))]+E[log(1-D(G(x)))]

[0130] Wherein, x represents the input ICL preoperative anterior segment structure image sample, G represents the generator, D represents the discriminator, G(x) represents the ICL postoperative anterior segment structure prediction image generated by the anterior segment structure image generation model according to the input data x, D(x) represents the judgment result of the anterior segment structure prediction discriminator on the input data x, wherein the judgment result of D(x) may include: the input data x is real data, and the input data x is generated data; D(G(x)) represents the judgment result of the anterior segment structure prediction discriminator on the input data G(x); wherein the judgment result of D(G(x)) may include: the input data G(x) is real data, and the input data G(x) is generated data.

[0131] Furthermore, the reference iris segmentation image can be used as an output label of the anterior segment structure image generation model, i.e., an iris segmentation image label, and the first loss can be determined according to the iris segmentation prediction image and the reference iris segmentation image; the reference iris contour and lens image can be used as another output label of the anterior segment structure image generation model, and then the second loss can be determined according to the iris contour and lens prediction image and the reference iris contour and lens image. The iris contour and lens prediction image represent the iris contour and lens upper surface image, and the reference iris contour and lens image represent the reference iris contour and lens upper surface image, i.e., the iris contour and lens upper surface image label.

[0132] In some embodiments, determining the first loss according to the iris segmentation prediction image and the reference iris segmentation image includes:

[0133] Get the preset iris root position weight map;

[0134] Determine a binary cross entropy loss based on the iris segmentation prediction image and the reference iris segmentation image;

[0135] The binary cross entropy loss is weighted according to the iris root position weight map to obtain the first loss.

[0136] Specifically, the iris segmentation prediction image is represented as x I ′ , the reference iris segmentation image is represented as y I , according to x I ′ and I Determine the binary cross entropy loss L BCE (x I ′ ,y I ). Get the preset iris root position weight map W H,The iris root position weight map is a pseudo-color map, which shows the way to apply the iris root position information. The pseudo-color map contains two Gaussian peaks. The center position of the Gaussian peak (displayed as a yellow dot) represents the iris root position. This map is used as a weight map to increase the weight of the iris root position in the first auxiliary decoder branch, which improves the accuracy of iris prediction based on endpoints.

[0137] According to the iris root position weight map W H For the binary cross entropy loss L BCE (x I ′ ,y I ) to adjust the weights and get the first loss in, represents pixel-by-pixel multiplication, and the first loss represents the loss function of the iris region prediction auxiliary branch (first auxiliary decoder branch). Further, according to the iris contour and the lens prediction image x ′ C and the preset reference iris contour and lens image y C Determine the second loss L as input Dice (x ′ C ,y C ), the second loss function is the Dice similarity coefficient loss function, and the second loss represents the loss function of the iris contour branch (the second auxiliary decoder branch).

[0138] According to the above steps, the target loss L of the embodiment of the present application is as follows:

[0139]

[0140] Among them, L adv (G,D) represents adversarial loss, represents the first loss, L Dice (x ′ C ,y C ) represents the second loss, λ I and λ C is the preset weight value. The higher the weight value, the greater the influence of this branch on the network. As the target loss L decreases, the model parameters of the anterior segment structure image generation model are updated based on the target loss L, gradually improving the prediction accuracy of the anterior segment structure prediction model.

[0141] In some embodiments, in addition to inputting the anterior segment structure image and the ICL implantation direction before ICL surgery into the anterior segment structure image generation model, the input information may also include the ICL size. The method provided in the embodiment of the present application may also include:

[0142] The anterior segment structure image before ICL surgery is input into the anterior segment structure image generation model to generate the anterior segment structure image after ICL surgery corresponding to the ICL size;

[0143] Recommend the ICL implantation direction based on the anterior segment structure image after ICL surgery and the corresponding ICL implantation direction;

[0144] ICL size recommendation is made based on the anterior segment structure images after ICL surgery and the corresponding ICL size.

[0145] Specifically, the embodiment of the present application provides three ICL size and ICL implantation direction recommendation schemes. The anterior segment structure prediction model provided in the embodiment of the present application can take two ICL implantation directions, four ICL sizes and ICL preoperative anterior segment structure images as input, and output eight combinations of ICL postoperative anterior segment structure images, wherein the eight combinations are respective combinations of ICL implantation directions and ICL sizes, and according to the ICL postoperative anterior segment structure images under each combination, the ICL postoperative anterior segment structure images can be unidirectional or multi-directional, and can be set by itself, and the final ICL implantation direction and ICL size recommendation scheme are displayed; or the ICL postoperative anterior segment structure images corresponding to the recommended ICL size and the recommended ICL implantation direction are output, and the recommended ICL size and the recommended ICL implantation direction are displayed; or only the text corresponding to the recommended ICL size and the recommended ICL implantation direction is output.

[0146] See also Fig. 9 The present application also provides an anterior segment structure prediction device 900, which can implement the above anterior segment structure prediction method. The device includes:

[0147] The acquisition unit 910 is used to acquire the anterior segment structural image of the ICL before surgery and the ICL implantation direction; the input unit 920 is used to input the anterior segment structural image of the ICL before surgery and the ICL implantation direction into a pre-trained anterior segment structural image generation model, the anterior segment structural image generation model includes an encoder module, a decoder module and a geometric information sharing module, the decoder module includes a main decoder, a first auxiliary decoder and a second auxiliary decoder; the encoding unit 930 is used to encode the anterior segment structural image of the ICL before surgery and the ICL implantation direction through the encoder module to obtain the anterior segment structural image features; the generation unit 940, used to generate an iris image based on the anterior segment structure image features of the eye based on the first auxiliary decoder, the second auxiliary decoder and the geometric information sharing module, and obtain the iris segmentation image output by the first auxiliary decoder and the iris contour and the upper surface image of the lens output by the second auxiliary decoder; the decoding unit 950, used to decode the anterior segment structure image features of the eye based on the main decoder and the first auxiliary decoder, and obtain the anterior segment structure image after ICL surgery; the display unit 960, used to display the anterior segment structure image, iris segmentation image, iris contour and upper surface image of the eye after ICL surgery in the direction of ICL implantation.

[0148] The specific implementation of the anterior segment structure prediction device is substantially the same as the specific implementation of the anterior segment structure prediction method described above, and will not be described in detail herein.

[0149] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned anterior segment structure prediction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0150] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0151] The processor 1001 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0152] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the anterior segment structure prediction method of the embodiment of the present application;

[0153] Input / output interface 1003, used to implement information input and output;

[0154] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0155] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0156] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0157] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned anterior segment structure prediction method is implemented.

[0158] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0160] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0161] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0163] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0164] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0165] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0169] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for predicting anterior segment structure, characterized in that: The method comprises: Obtain anterior segment structural images and ICL implantation direction before ICL surgery; The anterior segment structural image of the eye before ICL surgery and the implantation direction of the ICL are input into a pre-trained anterior segment structural image generation model, wherein the anterior segment structural image generation model comprises an encoder module, a decoder module and a geometric information sharing module, wherein the decoder module comprises a main decoder, a first auxiliary decoder and a second auxiliary decoder; The encoder module is used to encode the anterior segment structural image of the eye before ICL surgery and the implantation direction of the ICL to obtain anterior segment structural image features; Based on the first auxiliary decoder, the second auxiliary decoder and the geometric information sharing module, an iris image is generated for the anterior segment structure image features to obtain an iris segmentation image output by the first auxiliary decoder and an iris contour and a lens upper surface image output by the second auxiliary decoder; Decoding the anterior segment structure image features based on the main decoder and the first auxiliary decoder to obtain an anterior segment structure image after ICL surgery; The anterior segment structure image after ICL surgery, the iris segmentation image, and the iris contour and lens upper surface image are displayed in the direction of ICL implantation.

2. The prediction method according to claim 1, characterized in that: The first auxiliary decoder includes a first upsampling layer, a first splicing layer and a first fusion layer, the second auxiliary decoder includes a second upsampling layer, a second splicing layer and a second fusion layer, the anterior segment structure image generation model also includes a first output layer and a second output layer, and the iris image is generated based on the first auxiliary decoder, the second auxiliary decoder and the geometric information sharing module for the anterior segment structure image features to obtain the iris segmentation image output by the first auxiliary decoder and the iris contour and lens upper surface image output by the second auxiliary decoder, including: Upsampling the anterior segment structure image feature through the first upsampling layer to obtain a first anterior segment structure feature reconstructed image; Upsampling the anterior segment structure image feature through the second upsampling layer to obtain a second anterior segment structure feature reconstructed image; Extracting geometric information from the first anterior segment structure feature reconstructed image and the second anterior segment structure feature reconstructed image by a geometric information sharing module to obtain a geometric information sharing result; Splicing the first anterior segment structure feature reconstructed image and the anterior segment structure image feature through the first splicing layer to obtain a first feature image splicing result; Splicing the second anterior segment structure feature reconstructed image and the anterior segment structure image feature through the second splicing layer to obtain a second feature image splicing result; The first fusion layer is used to fuse the first feature image stitching result and the geometric information extraction result to obtain a first image fusion result; The second feature image stitching result and the geometric information extraction result are fused through the second fusion layer to obtain a second image fusion result; Upsampling the first image fusion result through a first output layer to output an iris segmentation image; The second image fusion result is upsampled through the second output layer to output the iris contour and the upper surface image of the lens.

3. The prediction method according to claim 2, characterized in that: The geometric information sharing module includes a convolution layer, an average pooling layer, a maximum pooling layer, and an activation layer. The geometric information sharing module is used to extract geometric information from the first anterior segment structural feature reconstructed image and the second anterior segment structural feature reconstructed image to obtain a geometric information sharing result, including: splicing the first anterior segment structure feature reconstructed image and the second anterior segment structure feature reconstructed image to obtain a third feature image splicing result; Performing feature extraction on the third feature image splicing result based on the convolution layer to obtain a geometric feature extraction result; Based on the maximum pooling layer, channel maximum pooling is performed on the third feature image splicing result to obtain a maximum pooling result; Based on the average pooling layer, channel average pooling is performed on the third feature image splicing result to obtain an average pooling result; splicing the maximum pooling result, the average pooling result and the geometric feature extraction result to obtain a fourth feature image splicing result; The fourth feature image splicing result is nonlinearly transformed through the activation layer to obtain the geometric information sharing result.

4. The prediction method according to claim 1, characterized in that: Before obtaining the anterior segment structure image before ICL surgery, the method further includes: training the anterior segment structure image generation model, and the training process specifically includes: Acquire an image sample set and an ICL implantation direction, wherein the image sample set includes a plurality of anterior segment structure image samples, a reference iris segmentation image, and a reference iris contour and lens image; Inputting the anterior segment structure image sample and the ICL implantation direction into the anterior segment structure image generation model for image prediction, and obtaining an ICL postoperative anterior segment structure prediction image, an iris segmentation prediction image, and an iris contour and lens prediction image under the ICL implantation direction; Inputting the predicted image of the anterior segment structure after ICL surgery into a preset anterior segment structure prediction discriminator for image discrimination to obtain an image discrimination result, and determining the adversarial loss according to the image discrimination result; Determine a first loss according to the iris segmentation prediction image and the reference iris segmentation image, and determine a second loss according to the iris contour and lens prediction image and the reference iris contour and lens image; A target loss is determined according to the first loss, the second loss and the adversarial loss, and model parameters of the anterior segment structure image generation model are updated based on the target loss.

5. The prediction method according to claim 4, characterized in that: The determining a first loss according to the iris segmentation prediction image and the reference iris segmentation image comprises: Get the preset iris root position weight map; Determining a binary cross entropy loss according to the iris segmentation prediction image and the reference iris segmentation image; The binary cross entropy loss is weighted according to the iris root position weight map to obtain the first loss.

6. The prediction method according to claim 1, characterized in that: The obtaining of anterior segment structural images before ICL surgery includes: Adjust the brightness of the input anterior segment structure image before ICL surgery according to a preset brightness adjustment curve to obtain a brightness adaptation image of the anterior segment structure before ICL surgery, wherein there are multiple brightness adjustment curves; Performing image stitching on a plurality of brightness adaptation images of the anterior segment structure of the ICL before surgery with different brightness to obtain a multi-brightness stitching image; Based on a preset attention module, feature weights are assigned to the multi-brightness spliced ​​image to obtain the anterior segment structure image before ICL surgery.

7. The method according to claim 1, characterized in that The anterior segment structure image before ICL surgery also includes a corresponding ICL size, and the method further includes: Inputting the anterior segment structure image of the eye before ICL surgery into the anterior segment structure image generation model to generate an anterior segment structure image of the eye after ICL surgery corresponding to the ICL size; Recommending the ICL implantation direction based on the anterior segment structure image after ICL surgery and the corresponding ICL implantation direction; The ICL size recommendation is made based on the anterior segment structure image after ICL surgery and the corresponding ICL size.

8. An anterior segment structure prediction device, characterized in that: The device comprises: An acquisition unit, used to acquire an anterior segment structural image and an ICL implantation direction before ICL surgery; An input unit, used to input the pre-ICL anterior segment structural image and the ICL implantation direction into a pre-trained anterior segment structural image generation model, wherein the anterior segment structural image generation model includes an encoder module, a decoder module and a geometric information sharing module, wherein the decoder module includes a main decoder, a first auxiliary decoder and a second auxiliary decoder; An encoding unit, used for encoding the anterior segment structural image of the eye before the ICL operation and the ICL implantation direction through the encoder module to obtain anterior segment structural image features; a generating unit, configured to generate an iris image based on the anterior segment structure image features of the first auxiliary decoder, the second auxiliary decoder and the geometric information sharing module, to obtain an iris segmentation image output by the first auxiliary decoder and an iris contour and a lens upper surface image output by the second auxiliary decoder; A decoding unit, used for decoding the anterior segment structure image features based on the main decoder and the first auxiliary decoder to obtain an anterior segment structure image after ICL surgery; The display unit is used to display the anterior segment structure image after ICL surgery, the iris segmentation image, and the iris contour and lens upper surface image in the direction of ICL implantation.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the anterior segment structure prediction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the anterior segment structure prediction method according to any one of claims 1 to 7 is implemented.

Images