Femtosecond corneal refractive surgery aspiration loss, black spot warning method, device and storage medium

Abstract

The present application relates to a method, device, and storage medium for early warning of aspiration and melasma during femtosecond corneal refractive surgery. This application employs a deep learning model consisting of an encoder and a decoder, and utilizes the trained deep learning model to output a predicted frame at the current moment. By comparing the predicted frame with the actual frame at the current moment, the application determines that the actual frame has deviated from the predicted direction when a deviation occurs between the predicted and actual frames. This application then determines that aspiration or melasma has occurred based on the deviation of the actual frame from the predicted direction. This method enables timely detection of aspiration and early warning before aspiration occurs during all-femtosecond corneal refractive surgery, thereby reducing the incidence of aspiration and improving safety.

Description

Technical Field

[0001] The present application relates to the field of detection and early warning technology, and in particular to a method, device and storage medium for early warning of aspiration loss and dark spots during femtosecond corneal refractive surgery. Background Art

[0002] SMILE is an advanced laser myopia correction procedure performed entirely with a femtosecond laser, eliminating the need for a traditional corneal flap. It is renowned for its minimally invasive, precise, and safe approach. The femtosecond laser scans directly into the cornea, creating a thin layer of lens-like tissue that is then removed through a tiny 2-4 mm incision, reshaping the cornea's curvature and correcting myopia and astigmatism.

[0003] Loss of aspiration during femtosecond laser surgery refers to the failure of the vacuum ring to stabilize the eyeball during surgery, leading to interruption of the laser procedure. In mild cases, this can interrupt the procedure, require repositioning, prolong the procedure, and increase patient anxiety. However, prompt treatment allows the procedure to continue. In severe cases, it can cause cutting errors, leading to undercorrection, overcorrection, or astigmatism, and may also cause corneal damage and postoperative complications.

[0004] Therefore, timely detection of aspiration during all-femtosecond surgery and early warning before aspiration occurs can effectively improve the success rate of the surgery and reduce the surgical risk for patients. Summary of the Invention

[0005] Based on this, a method, device and storage medium for warning of aspiration and black spots in femtosecond corneal refractive surgery are provided to solve the technical problem that the current all-femtosecond surgery process cannot detect aspiration in time and issue a warning before aspiration occurs, resulting in a high incidence of aspiration.

[0006] In one aspect, a method for early warning of aspiration loss and melasma during femtosecond corneal refractive surgery is provided, the method comprising:

[0007] The operation under the surgical microscope is recorded through video stream, and a sequence of image frames is formed in chronological order;

[0008] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence;

[0009] Inputting a sequence of image frames multiple frames before the current moment into the trained deep learning model, the deep learning model outputting a predicted frame at the current moment;

[0010] By comparing the predicted frame at the current moment with the real frame, the distance score is obtained;

[0011] The distance score is compared with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future and an early warning is given.

[0012] In one embodiment, setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes:

[0013] A discriminator is set to verify the output result of the deep learning model, wherein the discriminator obtains the predicted frame output by the decoder at the current moment and the real frame at the current moment in the acquired image frame sequence, compares the predicted frame at the current moment with the real frame, and records the first value if they are the same, and records the second value if they are different;

[0014] Whether the deep learning model has completed training is determined based on the ratio of the first value to the second value.

[0015] In one embodiment, obtaining a distance score by comparing the predicted frame and the real frame at the current moment includes:

[0016] By formula The distance score is calculated, where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0017] In one embodiment, the step of comparing the distance score with the image frame change threshold to determine whether an abnormal change occurs in the real frame at the current moment, and if so, determining that aspiration loss or dark spots is about to occur in the future and issuing a warning includes:

[0018] Setting the image frame change threshold to include a first threshold and a second threshold, wherein the first threshold is greater than the second threshold;

[0019] When the distance score is greater than the first threshold, it is determined that aspiration loss or black spot occurs at the current moment;

[0020] When the distance score is between the first threshold and the second threshold, it is determined that aspiration loss or dark spot is about to occur in the future, and an early warning is given;

[0021] When the distance score is smaller than the second threshold, it is determined that the current moment is normal.

[0022] In one embodiment, setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes:

[0023] Setting the deep learning model to include a forward prediction model and a backward prediction model, wherein both the forward prediction model and the backward prediction model are composed of an encoder and a decoder, the forward prediction model is used to predict future frames from past frames, and the backward prediction model is used to predict past frames from future frames;

[0024] When training the forward prediction model, an image frame sequence from the acquired image frame sequence is input into the forward prediction model, the forward prediction model outputs a predicted image sequence that predicts a number of frames at a subsequent moment, a real image sequence corresponding to the predicted image sequence from the acquired image frame sequence is obtained, a loss of the forward prediction model is calculated using an absolute value loss function, and whether the forward prediction model has completed training is determined based on the value of the absolute value loss function;

[0025] When training the backward prediction model, an image frame sequence in the acquired image frame sequence is input into the backward prediction model, the backward prediction model outputs a predicted image sequence that predicts several frames at the previous moment, and a real image sequence corresponding to the predicted image sequence in the acquired image frame sequence is obtained. The loss of the backward prediction model is calculated using a square loss function, and whether the backward prediction model has completed training is determined based on the value of the square loss function.

[0026] In one embodiment, obtaining a distance score by comparing the predicted frame and the real frame at the current moment includes:

[0027] Inputting a sequence of image frames before the current moment that is collected in real time into the forward prediction model, the forward prediction model outputs a predicted frame at the current moment, and obtaining a real frame at the current moment;

[0028] By formula Calculate the distance score between the predicted frame and the real frame at the current moment, where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0029] In one embodiment, the step of comparing the distance score with the image frame change threshold to determine whether an abnormal change occurs in the real frame at the current moment, and if so, determining that aspiration loss or dark spots is about to occur in the future and issuing a warning includes:

[0030] When the distance score is greater than or equal to the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

[0031] In one embodiment, the method for early warning of aspiration loss and melasma during femtosecond corneal refractive surgery further includes:

[0032] Inputting a real-time acquired image frame sequence into the forward prediction model, the forward prediction model outputting a predicted image sequence of several frames after the current moment, inputting a predicted image sequence of several frames after the current moment into the backward prediction model, the backward prediction model outputting a backward-inferred image sequence corresponding to the real-time acquired image frame sequence, calculating a distance score between the real-time acquired image frame sequence and an image frame at each moment in the backward-inferred image sequence, and obtaining a maximum distance score;

[0033] When the maximum value of the distance score is greater than the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

[0034] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are performed:

[0035] The operation under the surgical microscope is recorded through video stream, and a sequence of image frames is formed in chronological order;

[0036] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence;

[0037] Inputting a sequence of image frames multiple frames before the current moment into the trained deep learning model, the deep learning model outputting a predicted frame at the current moment;

[0038] By comparing the predicted frame at the current moment with the real frame, the distance score is obtained;

[0039] The distance score is compared with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future and an early warning is given.

[0040] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0041] The operation under the surgical microscope is recorded through video stream, and a sequence of image frames is formed in chronological order;

[0042] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence;

[0043] Inputting a sequence of image frames multiple frames before the current moment into the trained deep learning model, the deep learning model outputting a predicted frame at the current moment;

[0044] By comparing the predicted frame at the current moment with the real frame, the distance score is obtained;

[0045] The distance score is compared with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future and an early warning is given.

[0046] The above-mentioned method, device and storage medium for warning of aspiration and black spots in femtosecond corneal refractive surgery, by setting a deep learning model composed of an encoder and a decoder, uses the trained deep learning model to output the predicted frame at the current moment, and by comparing the predicted frame at the current moment with the real frame, it is judged that the real frame has deviated from the predicted direction when there is a deviation between the predicted frame and the real frame, thereby judging that aspiration or black spots have occurred based on the deviation of the real frame from the predicted direction. It can detect aspiration in time during the full femtosecond corneal refractive surgery and issue a warning before aspiration occurs, thereby reducing the incidence of aspiration and improving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 This is a schematic diagram of a camera recording an operation under a surgical microscope through a video stream to form an image frame sequence in a scenario according to an embodiment of the present application;

[0049] Figure 2 This is a schematic diagram of a deep learning model in an embodiment of the present application that is an autoencoder solution;

[0050] Figure 3 This is a schematic diagram of a deep learning model in one embodiment of the present application that is combined with a generative adversarial network solution;

[0051] Figure 4 This is a flow chart of a method for early warning of aspiration loss and dark spots in femtosecond corneal refractive surgery in one embodiment of the present application;

[0052] Figure 5 This is a schematic diagram of a method for determining loss of aspiration using forward and backward bidirectional prediction in one embodiment of the present application;

[0053] Figure 6 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] To solve the above problems, the embodiments of the present invention creatively propose a method for warning of aspiration loss and dark spots in femtosecond corneal refractive surgery, wherein the detection of aspiration loss refers to detecting the occurrence of aspiration loss or dark spots in the current surgery when aspiration loss occurs. When aspiration loss occurs, it will cause lens separation and damage. Lens separation and damage refers to the separation when a lens breakage line appears at the head of the separator, the separator breaks from the bottom of the lens to the top of the lens, and the curvature of the originally arc-shaped lens edge changes significantly. Warning of aspiration loss refers to judging in advance that aspiration loss is about to occur and giving a warning before aspiration loss occurs. Dark spots / black areas will appear before aspiration loss occurs. The dark spot / black area problem is: when there are opaque foreign objects or media that deflect light, such as grease, air bubbles, water droplets, etc. on the surface of the cornea, the photofracture effect cannot occur, the tissue cannot be cut, and a "black" "uncut" area is left on the cornea, which is a dark spot / black area. Therefore, this solution is suitable for identifying black spots / black areas, lens separation and breakage, and abnormalities in other surgical steps during SMILE surgery.

[0056] The detection of aspiration loss refers to the detection of aspiration loss or black spots during the current surgery when aspiration loss occurs.

[0057] The early warning of loss of aspiration refers to judging in advance that loss of aspiration is about to occur and giving a warning before the phenomenon of loss of aspiration occurs.

[0058] like Figure 1 As shown, in the scenario of this application, the camera records the operation under the surgical microscope through the video stream to form an image frame sequence. Let the current time be t, then the image frame sequence is expressed as .

[0059] At time t, the detection of loss of aspiration is to judge Whether loss of aspiration or dark spots occurs at the moment, if loss of aspiration or dark spots occurs, it is recorded as 1, and if it is normal, it is recorded as 0; the warning of loss of aspiration is to judge whether loss of aspiration or dark spots will occur within a period of time, for example, within k frames, that is, whether loss of aspiration or dark spots occurs from t+1 to t+k frames. If loss of aspiration or dark spots occurs in the time period [t+1, t+k], it is recorded as 1, and if it is normal, it is recorded as 0.

[0060] When the sampling frame rate remains stable, the number of frames can be converted into the advance warning time. Therefore, the larger the warning interval k is, the longer the advance warning time is.

[0061] Example 1

[0062] The method for early warning of aspiration loss and dark spots in femtosecond corneal refractive surgery provided in Example 1 of the present application can be applied to Figure 2 、 Figure 3 In the application environment shown, the input is a sequence of n images before the current moment, and the output is the predicted current frame. By comparing the predicted current frame with the actual current frame, we determine whether the current frame is abnormal. During training, we only use negative samples, which has the advantage of easy access to datasets and no need for extensive annotation.

[0063] like Figure 2 、 Figure 3 As shown in the figure, in the prediction of the current frame, the deep learning model can be divided into two schemes, the autoencoder scheme and the scheme combined with the generative adversarial network.

[0064] like Figure 2 As shown, this is a self-encoder solution, the input is , after a deep learning model consisting of an encoder and a decoder, the output is . Figure 1 An optional deep learning model is UNet.

[0065] like Figure 3 As shown in Figure 2 A discriminator is added to improve the quality of the generated image. The difficulty of the discriminator is not determining whether absorptive or dark spots have occurred, but rather determining whether the generated image is authentic. By comparing the predicted image with the real image, it can determine whether the real image has deviated from the predicted direction. If so, absorptive or dark spots are detected.

[0066] In this embodiment, if Figure 4 As shown, a method for early warning of aspiration loss and dark spots in femtosecond corneal refractive surgery is provided, comprising the following steps:

[0067] Step S1, recording the operation under the surgical microscope through a video stream, and forming an image frame sequence in chronological order;

[0068] Step S2, setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence;

[0069] Step S3, inputting a sequence of image frames multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs a predicted frame at the current moment;

[0070] Step S4, obtaining a distance score by comparing the predicted frame and the real frame at the current moment;

[0071] Step S5: comparing the distance score with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future and an early warning is given.

[0072] like Figure 3 As shown, in this embodiment, the setting of a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence include:

[0073] A discriminator is set to verify the output result of the deep learning model, wherein the discriminator obtains the predicted frame output by the decoder at the current moment and the real frame at the current moment in the acquired image frame sequence, compares the predicted frame at the current moment with the real frame, and records the first value if they are the same, and records the second value if they are different;

[0074] Whether the deep learning model has completed training is determined based on the ratio of the first value to the second value.

[0075] It's understandable that the first value represents the fidelity of the generated prediction, determining whether it's realistic. The discriminator looks at the difference between the generated image (the prediction) and the original image (the real image), primarily serving as a support in training.

[0076] In this embodiment, obtaining the distance score by comparing the predicted frame and the real frame at the current moment includes:

[0077] By formula The distance score is calculated, where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0078] The larger the distance score value is, the more dissimilar the predicted frame is to the real frame at the current moment, and the more dissimilar it is, the more likely it is to be warned.

[0079] It should be noted that the formula for calculating the distance score is not unique. Any measurement method that can reflect the difference between the predicted frame and the real frame can be used as an alternative for distance calculation, such as peak signal-to-noise ratio.

[0080] In this embodiment, the comparison between the distance score and the image frame change threshold determines whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future, and an early warning is given, including:

[0081] Setting the image frame change threshold to include a first threshold and a second threshold, wherein the first threshold is greater than the second threshold;

[0082] When the distance score is greater than the first threshold, it is determined that aspiration loss or black spot occurs at the current moment;

[0083] When the distance score is between the first threshold and the second threshold, it is determined that aspiration loss or dark spot is about to occur in the future, and an early warning is given;

[0084] When the distance score is smaller than the second threshold, it is determined that the current moment is normal.

[0085] That is, the first threshold is set to and the second threshold is ,in ;when When , it is determined that loss of aspiration or black spots occurs at the current moment; when When it is determined that aspiration loss or dark spots will occur in the future, a warning is given; when , it is determined that the current moment is normal.

[0086] in The selection can be done by statistically analyzing past cases. The distribution is determined.

[0087] Example 2

[0088] In this embodiment, if Figure 4 As shown, a method for early warning of aspiration loss and dark spots in femtosecond corneal refractive surgery is provided, comprising the following steps:

[0089] Step S1, recording the operation under the surgical microscope through a video stream, and forming an image frame sequence in chronological order;

[0090] Step S2, setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence;

[0091] Step S3, inputting a sequence of image frames multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs a predicted frame at the current moment;

[0092] Step S4, obtaining a distance score by comparing the predicted frame and the real frame at the current moment;

[0093] Step S5: comparing the distance score with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future and an early warning is given.

[0094] In this embodiment, setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes:

[0095] Setting the deep learning model to include a forward prediction model and a backward prediction model, wherein both the forward prediction model and the backward prediction model are composed of an encoder and a decoder, the forward prediction model is used to predict future frames from past frames, and the backward prediction model is used to predict past frames from future frames;

[0096] When training the forward prediction model, an image frame sequence from the acquired image frame sequence is input into the forward prediction model, the forward prediction model outputs a predicted image sequence that predicts a number of frames at a subsequent moment, a real image sequence corresponding to the predicted image sequence from the acquired image frame sequence is obtained, a loss of the forward prediction model is calculated using an absolute value loss function, and whether the forward prediction model has completed training is determined based on the value of the absolute value loss function;

[0097] When training the backward prediction model, an image frame sequence in the acquired image frame sequence is input into the backward prediction model, the backward prediction model outputs a predicted image sequence that predicts several frames at the previous moment, and a real image sequence corresponding to the predicted image sequence in the acquired image frame sequence is obtained. The loss of the backward prediction model is calculated using a square loss function, and whether the backward prediction model has completed training is determined based on the value of the square loss function.

[0098] The absolute value loss function is , y' is the image in the predicted image sequence, y is the image in the real image sequence, i is the pixel subscript in the image, and n is the number of pixels in the image.

[0099] The square loss function is , y' is the image in the predicted image sequence, y is the image in the real image sequence, i is the pixel subscript in the image, and n is the number of pixels in the image.

[0100] In this embodiment, obtaining the distance score by comparing the predicted frame and the real frame at the current moment includes:

[0101] Inputting a sequence of image frames before the current moment that is collected in real time into the forward prediction model, the forward prediction model outputs a predicted frame at the current moment, and obtaining a real frame at the current moment;

[0102] By formula Calculate the distance score between the predicted frame and the real frame at the current moment, where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0103] In this embodiment, the comparison between the distance score and the image frame change threshold determines whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future, and an early warning is given, including:

[0104] When the distance score is greater than or equal to the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

[0105] That is to say, the image frame change threshold is set to ,when When , it is determined that loss of absorption or black spot phenomenon is currently occurring.

[0106] In this embodiment, the method for early warning of aspiration loss and dark spots in femtosecond corneal refractive surgery further includes:

[0107] Inputting a real-time acquired image frame sequence into the forward prediction model, the forward prediction model outputting a predicted image sequence of several frames after the current moment, inputting a predicted image sequence of several frames after the current moment into the backward prediction model, the backward prediction model outputting a backward-inferred image sequence corresponding to the real-time acquired image frame sequence, calculating a distance score between the real-time acquired image frame sequence and an image frame at each moment in the backward-inferred image sequence, and obtaining a maximum distance score;

[0108] When the maximum value of the distance score is greater than the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

[0109] That is to say, the image frame change threshold is set to For the loss of aspiration warning, the results of the backward prediction model are combined with the similarity function to evaluate. Get the maximum distance score. When the value is greater than , it is judged that loss of absorption or black spot will occur at the future time [t+1, t+2].

[0110] It is understandable that in Example 2, the deep learning model is set to include a forward prediction model and a backward prediction model, and miscarriage is judged by forward and backward bidirectional prediction. The forward prediction model predicts the future frame from the past frame, which is a prediction of what will happen. The backward prediction model predicts the past frame from the future frame. If miscarriage occurs in the future frame, then the future frame predicted based on the current normal situation will be significantly different from the actual future frame. Similarly, the past frame obtained by inferring the predicted future frame and the actual past frame will also be significantly different. Then, we can infer whether an abnormality will occur in the future by whether there is a significant difference between the actual past frame and the reversely predicted past frame.

[0111] like Figure 5 As shown, the forward prediction model inputs an image frame sequence and outputs a predicted image sequence of the next few frames. For example, when k=2 and the sequence length is set to 3, the input is , the output is It should be noted that the output here can be either the model directly predicting multiple time frames in one inference, or the model predicting the image frame of the next time each time and repeating the iteration, for example, Prediction , and then by Prediction , until the desired future frame sequence is obtained.

[0112] Subsequently, in the backward prediction model, the predicted frame obtained in the forward prediction model is used as input, that is, the above input , get an estimate of the past frame. For example:

[0113] —> ;

[0114] —> ;

[0115] …….

[0116] At the same time, during the training phase, we also use real sequences for backward prediction, for example:

[0117] —> ;

[0118] —> ;

[0119] …….

[0120] During training, we use the absolute value loss function L1 and the square loss function MSE to calculate the loss of forward prediction and backward prediction respectively as supervision information for model training.

[0121] During inference, we still use the similarity formula in Method 1 to evaluate whether aspiration loss or black spot phenomenon has occurred or will occur.

[0122] For aspiration loss detection, we use To make a judgment, when When , it is determined that loss of absorption or black spot phenomenon is currently occurring.

[0123] For the inhalation warning, we evaluate it by combining the results of the backward prediction model with the similarity function, as follows:

[0124]

[0125] When its value is greater than , it is judged that loss of absorption or black spots will occur at [t+1, t+2].

[0126] In the above-mentioned method for warning of aspiration and black spots in femtosecond corneal refractive surgery, a deep learning model composed of an encoder and a decoder is set up, and the trained deep learning model is used to output the predicted frame at the current moment. By comparing the predicted frame at the current moment with the real frame, it is judged that the real frame has deviated from the predicted direction when there is a deviation between the predicted frame and the real frame. Based on the deviation of the real frame from the predicted direction, it is judged that aspiration or black spots have occurred. This method can timely detect aspiration during the full femtosecond corneal refractive surgery and issue a warning before aspiration occurs, thereby reducing the incidence of aspiration and improving safety.

[0127] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data on early warning of aspiration loss and melasma in femtosecond corneal refractive surgery. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for early warning of aspiration loss and melasma in femtosecond corneal refractive surgery is implemented.

[0128] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0129] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0130] The operation under the surgical microscope is recorded through video stream, and a sequence of image frames is formed in chronological order;

[0131] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence;

[0132] Inputting a sequence of image frames multiple frames before the current moment into the trained deep learning model, the deep learning model outputting a predicted frame at the current moment;

[0133] By comparing the predicted frame at the current moment with the real frame, the distance score is obtained;

[0134] The distance score is compared with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future and an early warning is given.

[0135] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0136] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes:

[0137] A discriminator is set to verify the output result of the deep learning model, wherein the discriminator obtains the predicted frame output by the decoder at the current moment and the real frame at the current moment in the acquired image frame sequence, compares the predicted frame at the current moment with the real frame, and records the first value if they are the same, and records the second value if they are different;

[0138] Whether the deep learning model has completed training is determined based on the ratio of the first value to the second value.

[0139] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0140] The distance score obtained by comparing the predicted frame and the real frame at the current moment includes:

[0141] By formula The distance score is calculated, where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0142] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0143] The step of comparing the distance score with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes and if so, determining that aspiration loss or dark spots will occur in the future and giving an early warning includes:

[0144] Setting the image frame change threshold to include a first threshold and a second threshold, wherein the first threshold is greater than the second threshold;

[0145] When the distance score is greater than the first threshold, it is determined that aspiration loss or black spot occurs at the current moment;

[0146] When the distance score is between the first threshold and the second threshold, it is determined that aspiration loss or dark spot is about to occur in the future, and an early warning is given;

[0147] When the distance score is smaller than the second threshold, it is determined that the current moment is normal.

[0148] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0149] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes:

[0150] Setting the deep learning model to include a forward prediction model and a backward prediction model, wherein both the forward prediction model and the backward prediction model are composed of an encoder and a decoder, the forward prediction model is used to predict future frames from past frames, and the backward prediction model is used to predict past frames from future frames;

[0151] When training the forward prediction model, an image frame sequence from the acquired image frame sequence is input into the forward prediction model, the forward prediction model outputs a predicted image sequence that predicts a number of frames at a subsequent moment, a real image sequence corresponding to the predicted image sequence from the acquired image frame sequence is obtained, a loss of the forward prediction model is calculated using an absolute value loss function, and whether the forward prediction model has completed training is determined based on the value of the absolute value loss function;

[0152] When training the backward prediction model, an image frame sequence in the acquired image frame sequence is input into the backward prediction model, the backward prediction model outputs a predicted image sequence that predicts several frames at the previous moment, and a real image sequence corresponding to the predicted image sequence in the acquired image frame sequence is obtained. The loss of the backward prediction model is calculated using a square loss function, and whether the backward prediction model has completed training is determined based on the value of the square loss function.

[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0154] The distance score obtained by comparing the predicted frame and the real frame at the current moment includes:

[0155] Inputting a sequence of image frames before the current moment that is collected in real time into the forward prediction model, the forward prediction model outputs a predicted frame at the current moment, and obtaining a real frame at the current moment;

[0156] By formula Calculate the distance score between the predicted frame and the real frame at the current moment, where Weight parameters, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] The step of comparing the distance score with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes and if so, determining that aspiration loss or dark spots will occur in the future and giving an early warning includes:

[0159] When the distance score is greater than or equal to the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0161] Inputting a real-time acquired image frame sequence into the forward prediction model, the forward prediction model outputting a predicted image sequence of several frames after the current moment, inputting a predicted image sequence of several frames after the current moment into the backward prediction model, the backward prediction model outputting a backward-inferred image sequence corresponding to the real-time acquired image frame sequence, calculating a distance score between the real-time acquired image frame sequence and an image frame at each moment in the backward-inferred image sequence, and obtaining a maximum distance score;

[0162] When the maximum value of the distance score is greater than the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

[0163] Regarding the specific limitations on the steps implemented when the processor executes the computer program, please refer to the above limitations on the method for warning of aspiration loss and black spots in femtosecond corneal refractive surgery, which will not be repeated here.

[0164] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0165] The operation under the surgical microscope is recorded through video stream, and a sequence of image frames is formed in chronological order;

[0166] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence;

[0167] Inputting a sequence of image frames multiple frames before the current moment into the trained deep learning model, the deep learning model outputting a predicted frame at the current moment;

[0168] By comparing the predicted frame at the current moment with the real frame, the distance score is obtained;

[0169] The distance score is compared with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes. If so, it is determined that aspiration loss or dark spots will occur in the future and an early warning is given.

[0170] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0171] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes:

[0172] A discriminator is set to verify the output result of the deep learning model, wherein the discriminator obtains the predicted frame output by the decoder at the current moment and the real frame at the current moment in the acquired image frame sequence, compares the predicted frame at the current moment with the real frame, and records the first value if they are the same, and records the second value if they are different;

[0173] Whether the deep learning model has completed training is determined based on the ratio of the first value to the second value.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0175] The distance score obtained by comparing the predicted frame and the real frame at the current moment includes:

[0176] By formula The distance score is calculated, where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0178] The step of comparing the distance score with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes and if so, determining that aspiration loss or dark spots will occur in the future and giving an early warning includes:

[0179] Setting the image frame change threshold to include a first threshold and a second threshold, wherein the first threshold is greater than the second threshold;

[0180] When the distance score is greater than the first threshold, it is determined that aspiration loss or black spot occurs at the current moment;

[0181] When the distance score is between the first threshold and the second threshold, it is determined that aspiration loss or dark spot is about to occur in the future, and an early warning is given;

[0182] When the distance score is smaller than the second threshold, it is determined that the current moment is normal.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0184] Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes:

[0185] Setting the deep learning model to include a forward prediction model and a backward prediction model, wherein both the forward prediction model and the backward prediction model are composed of an encoder and a decoder, the forward prediction model is used to predict future frames from past frames, and the backward prediction model is used to predict past frames from future frames;

[0186] When training the forward prediction model, an image frame sequence from the acquired image frame sequence is input into the forward prediction model, the forward prediction model outputs a predicted image sequence that predicts a number of frames at a subsequent moment, a real image sequence corresponding to the predicted image sequence from the acquired image frame sequence is obtained, a loss of the forward prediction model is calculated using an absolute value loss function, and whether the forward prediction model has completed training is determined based on the value of the absolute value loss function;

[0187] When training the backward prediction model, an image frame sequence in the acquired image frame sequence is input into the backward prediction model, the backward prediction model outputs a predicted image sequence that predicts several frames at the previous moment, and a real image sequence corresponding to the predicted image sequence in the acquired image frame sequence is obtained. The loss of the backward prediction model is calculated using a square loss function, and whether the backward prediction model has completed training is determined based on the value of the square loss function.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0189] The distance score obtained by comparing the predicted frame and the real frame at the current moment includes:

[0190] Inputting a sequence of image frames before the current moment that is collected in real time into the forward prediction model, the forward prediction model outputs a predicted frame at the current moment, and obtaining a real frame at the current moment;

[0191] By formula Calculate the distance score between the predicted frame and the real frame at the current moment, where is the weight parameter, is the real frame at the current moment, The predicted frame at the current moment, t is the current moment.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0193] The step of comparing the distance score with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes and if so, determining that aspiration loss or dark spots will occur in the future and giving an early warning includes:

[0194] When the distance score is greater than or equal to the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0196] Inputting a real-time acquired image frame sequence into the forward prediction model, the forward prediction model outputting a predicted image sequence of several frames after the current moment, inputting a predicted image sequence of several frames after the current moment into the backward prediction model, the backward prediction model outputting a backward-inferred image sequence corresponding to the real-time acquired image frame sequence, calculating a distance score between the real-time acquired image frame sequence and an image frame at each moment in the backward-inferred image sequence, and obtaining a maximum distance score;

[0197] When the maximum value of the distance score is greater than the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

[0198] Regarding the specific limitations on the steps implemented when the computer program is executed by the processor, please refer to the above limitations on the method for warning of aspiration loss and melasma in femtosecond corneal refractive surgery, which will not be repeated here.

[0199] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0200] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for early warning of aspiration loss and dark spots in femtosecond corneal refractive surgery, characterized in that: include: The operation under the surgical microscope is recorded through video stream, and a sequence of image frames is formed in chronological order; Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence; Inputting a sequence of image frames multiple frames before the current moment into the trained deep learning model, the deep learning model outputting a predicted frame at the current moment; By comparing the predicted frame at the current moment with the real frame, the distance score is obtained, including: The distance score is calculated, where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment; Determining whether an abnormal change occurs in the real frame at the current moment by comparing the distance score with the image frame change threshold, and if so, determining that aspiration loss or dark spots are about to occur at a future moment and giving a warning, including: setting the image frame change threshold to include a first threshold and a second threshold, the first threshold being greater than the second threshold; determining that aspiration loss or dark spots are about to occur at the current moment when the distance score is greater than the first threshold; determining that aspiration loss or dark spots are about to occur at a future moment and giving a warning when the distance score is between the first threshold and the second threshold; determining that aspiration loss or dark spots are about to occur at a future moment and giving a warning; and determining that the current moment is normal when the distance score is less than the second threshold; The deep learning model is set to include a forward prediction model and a backward prediction model. A real-time acquired image frame sequence is input into the forward prediction model, and the forward prediction model outputs a predicted image sequence of several frames after the current moment. A predicted image sequence of several frames after the current moment is input into the backward prediction model, and the backward prediction model outputs a backward image sequence corresponding to the real-time acquired image frame sequence. The distance score between the real-time acquired image frame sequence and the image frame at each moment in the backward image sequence is calculated to obtain the maximum distance score. When the maximum distance score is greater than the image frame change threshold, it is determined that loss of aspiration or black spots have occurred at the current moment, and an early warning is given.

2. The method for early warning of aspiration loss and melasma in femtosecond corneal refractive surgery according to claim 1, characterized in that: Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes: A discriminator is set to verify the output result of the deep learning model, wherein the discriminator obtains the predicted frame output by the decoder at the current moment and the real frame at the current moment in the acquired image frame sequence, compares the predicted frame at the current moment with the real frame, and records the first value if they are the same, and records the second value if they are different; Whether the deep learning model has completed training is determined based on the ratio of the first value to the second value.

3. The method for early warning of aspiration loss and melasma in femtosecond corneal refractive surgery according to claim 1, characterized in that: Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model using the acquired image frame sequence includes: The forward prediction model and the backward prediction model are both composed of an encoder and a decoder, the forward prediction model is used to predict future frames through past frames, and the backward prediction model is used to predict past frames through future frames; When training the forward prediction model, an image frame sequence from the acquired image frame sequence is input into the forward prediction model, the forward prediction model outputs a predicted image sequence that predicts a number of frames at a subsequent moment, a real image sequence corresponding to the predicted image sequence from the acquired image frame sequence is obtained, a loss of the forward prediction model is calculated using an absolute value loss function, and whether the forward prediction model has completed training is determined based on the value of the absolute value loss function; When training the backward prediction model, an image frame sequence in the acquired image frame sequence is input into the backward prediction model, the backward prediction model outputs a predicted image sequence that predicts several frames at the previous moment, and a real image sequence corresponding to the predicted image sequence in the acquired image frame sequence is obtained. The loss of the backward prediction model is calculated using a square loss function, and whether the backward prediction model has completed training is determined based on the value of the square loss function.

4. The method for early warning of aspiration loss and melasma in femtosecond corneal refractive surgery according to claim 3, characterized in that: The distance score obtained by comparing the predicted frame and the real frame at the current moment includes: The image frame sequence before the current moment that is collected in real time is input into the forward prediction model, and the forward prediction model outputs the predicted frame at the current moment to obtain the real frame at the current moment.

5. The method for early warning of aspiration loss and melasma in femtosecond corneal refractive surgery according to claim 4, characterized in that: The step of comparing the distance score with the image frame change threshold to determine whether the real frame at the current moment has abnormal changes and if so, determining that aspiration loss or dark spots will occur in the future and giving an early warning includes: When the distance score is greater than or equal to the image frame change threshold, it is determined that loss of absorption or black spots occurs at the current moment, and an early warning is given.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images