Femtosecond corneal refractive surgery aspiration and black spot early warning method and device and storage medium

Abstract

The invention relates to a femtosecond corneal refractive surgery aspiration and black spot early warning method and device and a storage medium. According to the invention, the deep learning model composed of the encoder and the decoder is set, the trained deep learning model is used to output the prediction frame at the current moment, the prediction frame at the current moment is compared with the real frame, and when the prediction frame and the real frame are deviated, it is judged that the real frame deviates from the prediction direction. Therefore, the aspiration loss or the black spots can be judged when the real frame deviates from the prediction direction, the aspiration loss can be detected in time in the whole femtosecond operation process, early warning can be carried out before the aspiration loss occurs, the aspiration loss occurrence rate is reduced, and the safety is improved.

Description

Technical Field

[0001] The present application relates to the technical field of detection and early warning, and particularly to a method, device, and storage medium for detecting and warning of aspiration failure and black spots in femtosecond laser refractive surgery. Background Art

[0002] Full femtosecond surgery (SMILE) is an advanced laser myopia correction surgery that is completed entirely using femtosecond laser. It does not require the creation of a traditional corneal flap and is known for its minimally invasive, precise, and safe characteristics. The femtosecond laser directly scans inside the cornea to form a thin lens-shaped tissue, which is then removed through a 2-4 mm micro-incision to reshape the corneal curvature and correct myopia and astigmatism.

[0003] "Aspiration failure" in full femtosecond surgery refers to the phenomenon that during the surgery, the negative pressure suction ring fails to stably fix the eyeball, resulting in the interruption of the laser operation. In mild cases, aspiration failure leads to the interruption of the surgery and the need for repositioning, prolonging the surgery time and potentially increasing the patient's anxiety. However, the surgery can continue with timely treatment. In severe cases, it causes cutting errors, resulting in under-correction, over-correction, or astigmatism, and may also cause corneal damage and postoperative complications.

[0004] Therefore, timely detection of aspiration failure and early warning before the occurrence of aspiration failure during full femtosecond surgery can effectively improve the success rate of the surgery and reduce the surgical risk for patients. Summary of the Invention

[0005] Based on this, a method, device, and storage medium for detecting and warning of aspiration failure and black spots in femtosecond laser refractive surgery are provided to solve the technical problem of the high incidence of aspiration failure caused by the inability to timely detect aspiration failure and give an early warning before the occurrence of aspiration failure during the current full femtosecond surgery process.

[0006] On the one hand, a method for detecting and warning of aspiration failure and black spots in femtosecond laser refractive surgery is provided. The method includes: Recording the operation under the surgical microscope through a video stream to form an image frame sequence in chronological order; Setting up a deep learning model composed of an encoder and a decoder, and training the deep learning model using the collected image frame sequence; Inputting the image frame sequence of multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs a predicted frame at the current moment; Obtaining a distance score by comparing the predicted frame at the current moment with the real frame; Judging whether the real frame at the current moment has an abnormal change according to the comparison between the distance score and the image frame change threshold. If so, it is determined that aspiration failure or black spots will occur in the future moment, and a warning is given.

[0007] In one embodiment, training the deep learning model composed of an encoder and a decoder using the collected image frame sequence includes: Set a discriminator to verify the output result of the deep learning model. The discriminator obtains the predicted frame at the current moment output by the decoder and the real frame at the current moment in the collected image frame sequence, compares the predicted frame at the current moment with the real frame. If they are the same, it is recorded as the first value; if they are different, it is recorded as the second value. Judge whether the deep learning model is trained according to the ratio of the first value and the second value.

[0008] In one embodiment, obtaining the distance score by comparing the predicted frame at the current moment with the real frame includes: Obtain the distance score through the formula where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0009] In one embodiment, judging whether there is an abnormal change in the real frame at the current moment by comparing the distance score with the image frame change threshold. If so, it is determined that air suction loss or black spots will occur in the future, and a warning is given, including: Set the image frame change threshold to include a first threshold and a second threshold, and the first threshold is greater than the second threshold. When the distance score is greater than the first threshold, it is determined that air suction loss or black spots occur at the current moment. When the distance score is between the first threshold and the second threshold, it is determined that air suction loss or black spots will occur in the future, and a warning is given. When the distance score is less than the second threshold, it is determined that the current moment is normal.

[0010] In one embodiment, training the deep learning model composed of an encoder and a decoder using the collected image frame sequence includes: Set the deep learning model to include a forward prediction model and a backward prediction model. Both the forward prediction model and the backward prediction model are composed of an encoder and a decoder. The forward prediction model is used to predict future frames through past frames, and the backward prediction model is used to predict past frames through future frames. When training the forward prediction model, input an image frame sequence in the collected image frame sequences into the forward prediction model. The forward prediction model outputs a predicted image sequence for several frames at subsequent moments. Obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequences, calculate the loss of the forward prediction model through an absolute value loss function, and determine whether the forward prediction model is trained based on the value of the absolute value loss function; When training the backward prediction model, input an image frame sequence in the collected image frame sequences into the backward prediction model. The backward prediction model outputs a predicted image sequence for several frames at previous moments. Obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequences, calculate the loss of the backward prediction model through a squared loss function, and determine whether the backward prediction model is trained based on the value of the squared loss function.

[0011] In one embodiment, the obtaining the distance score by comparing the predicted frame and the real frame at the current moment includes: Input an image frame sequence before the current moment collected in real time into the forward prediction model. The forward prediction model outputs a predicted frame at the current moment, and obtain the real frame at the current moment; Through the formula Calculate the distance score between the predicted frame and the real frame at the current moment, where is a weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0012] In one embodiment, the determining whether there is an abnormal change in the real frame at the current moment based on the comparison between the distance score and the image frame change threshold, and if so, determining that air aspiration or black spot will occur at a future moment and giving an early warning includes: When the distance score is greater than or equal to the image frame change threshold, it is determined that air aspiration or black spot occurs at the current moment, and an early warning is given.

[0013] In one embodiment, the method for early warning of air aspiration and black spot in femtosecond laser in situ keratomileusis further includes: Input an image frame sequence collected in real time into the forward prediction model. The forward prediction model outputs a predicted image sequence for several frames after the current moment. Input the predicted image sequence for several frames after the current moment into the backward prediction model. The backward prediction model outputs a backward-inferred image sequence corresponding to the image frame sequence collected in real time. Calculate the distance score between each moment's image frame in the image frame sequence collected in real time and the backward-inferred image sequence, and obtain the maximum distance score; When the maximum value of the distance score is greater than the image frame change threshold, it is determined that aspiration or black spot occurs at the current moment, and a warning is given.

[0014] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: Record the operation under the surgical microscope through a video stream, and form an image frame sequence in chronological order; Set up a deep learning model composed of an encoder and a decoder, and use the collected image frame sequence to train the deep learning model; Input the image frame sequence of multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs the predicted frame of the current moment; Obtain a distance score by comparing the predicted frame of the current moment with the real frame; Compare the distance score with the image frame change threshold to determine whether the real frame at the current moment has an abnormal change. If so, it is determined that aspiration or black spot will occur at a future moment, and a warning is given.

[0015] On another aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Record the operation under the surgical microscope through a video stream, and form an image frame sequence in chronological order; Set up a deep learning model composed of an encoder and a decoder, and use the collected image frame sequence to train the deep learning model; Input the image frame sequence of multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs the predicted frame of the current moment; Obtain a distance score by comparing the predicted frame of the current moment with the real frame; Compare the distance score with the image frame change threshold to determine whether the real frame at the current moment has an abnormal change. If so, it is determined that aspiration or black spot will occur at a future moment, and a warning is given.

[0016] For the above-mentioned femtosecond laser in situ keratomileusis aspiration and black spot warning method, device and storage medium, by setting up a deep learning model composed of an encoder and a decoder, using the trained deep learning model to output the predicted frame of the current moment, and comparing the predicted frame of the current moment with the real frame, when there is a deviation between the predicted frame and the real frame, it is judged that the real frame deviates from the predicted direction. Thus, based on the fact that the real frame deviates from the predicted direction, it is judged that aspiration or black spot occurs, which can timely detect aspiration during the whole femtosecond surgery process and give a warning before aspiration occurs, reduce the incidence of aspiration, and improve safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0018] Figure 1 It is a schematic diagram of a scenario in an embodiment of the present application where a camera records the operation under a surgical microscope through a video stream to form an image frame sequence; Figure 2 It is a schematic diagram of the principle of the deep learning model as an autoencoder solution in an embodiment of the present application; Figure 3 It is a schematic diagram of the principle of the deep learning model as a solution combining a generative adversarial network in an embodiment of the present application; Figure 4 It is a schematic flowchart of a femtosecond laser in-situ keratomileusis aspiration loss and black spot warning method in an embodiment of the present application; Figure 5 It is a schematic diagram of the principle of judging aspiration loss by using forward and backward bidirectional prediction in an embodiment of the present application; Figure 6 It is an internal structure diagram of a computer device in an embodiment of the present application. Detailed implementation manners

[0019] In order to make the purpose, technical solutions and advantages of the present application more clear, the following further details the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] To solve the above problems, in the embodiments of the present invention, a method for warning of aspiration and black spots in femtosecond laser in situ keratomileusis (FS-LASIK) surgery is creatively proposed. The detection of aspiration refers to detecting the occurrence of aspiration or black spots in the current surgery when the aspiration phenomenon occurs. When the aspiration phenomenon occurs, it will cause the separation and breakage of the lens. The separation and breakage of the lens means that when the breakage occurs during separation, a lens rupture line appears at the head of the separator, the separator breaks into the upper part of the lens from below the lens, and the arc of the original circular lens edge changes significantly. The warning of aspiration refers to predicting in advance that aspiration is about to occur and giving a warning before the aspiration phenomenon occurs. When the aspiration phenomenon has not occurred, the problem of black spots / black areas will appear. The problem of black spots / black areas is as follows: when there are opaque foreign objects or light-deflecting media such as grease, air bubbles, water droplets, etc. on the corneal surface, the photo-induced rupture effect cannot occur, the tissue cannot be cut, and a "black" "uncut" area is left on the cornea, which is the black spot / black area. Therefore, this solution is applicable to the identification of black spots / black areas, lens separation and breakage, and other abnormalities in the femtosecond laser in situ keratomileusis (SMILE) surgery.

[0021] The detection of aspiration refers to detecting the occurrence of aspiration or black spots in the current surgery when the aspiration phenomenon occurs.

[0022] The warning of aspiration refers to predicting in advance that aspiration is about to occur and giving a warning before the aspiration phenomenon occurs.

[0023] As Figure 1 shown, in the scenario of this application, the camera records the operation under the surgical microscope through a video stream to form an image frame sequence. Denote the current moment as t, then the image frame sequence is expressed as .

[0024] Then at the t moment, the detection of aspiration is to judge whether the aspiration or black spot phenomenon occurs at the moment, and mark the occurrence of aspiration or black spot as 1 and normal as 0; the warning of aspiration is to judge whether the aspiration or black spot phenomenon is about to occur within a period of time, for example, within k frames, that is, whether the aspiration or black spot occurs in the frames from t + 1 to t + k. If the aspiration or black spot occurs in the time period [t + 1, t + k], mark it as 1 and normal as 0.

[0025] When the sampling frame rate remains stable, the number of frames can be converted according to the early warning time. Therefore, the larger the warning interval k, the longer the early warning time.

[0026] Embodiment 1

[0027] The method for warning of aspiration and black spots in femtosecond laser in situ keratomileusis (FS-LASIK) surgery provided in Embodiment 1 of this application can be applied to such as Figure 2 , Figure 3In the application environment shown. Among them, the input is an n-frame image sequence before the current moment, and the output is the predicted current frame. By comparing the predicted current frame with the actual current frame, it is determined whether there is an abnormality in the current frame. During the training process, we only use negative samples for training, thus having the advantages of easy acquisition of the data set and no need for a large amount of annotation.

[0028] As Figure 2 , Figure 3 shown, in the prediction of the current frame, the deep learning model can be divided into two schemes, the autoencoder scheme and the scheme combined with the generative adversarial network.

[0029] As Figure 2 shown, for a scheme of an autoencoder, the input is , and after passing through a deep learning model composed of an encoder and a decoder, the output is . Figure 1 An optional deep learning model is UNet.

[0030] As Figure 3 shown, another scheme combined with the generative adversarial network is to add a discriminator on the basis of Figure 2 to improve the quality of the generated images. The difficult point of the discriminator's judgment is not whether there is aspiration failure or black spots, but whether the generated picture is real. By comparing the predicted picture with the real picture, it is judged whether the real picture deviates from the prediction direction, so as to judge that aspiration failure or black spots occur when it deviates from the prediction direction.

[0031] In this embodiment, as Figure 4 shown, a method for warning of aspiration failure and black spots in femtosecond laser in situ keratomileusis is provided, including the following steps: Step S1, recording the operation under the surgical microscope through a video stream and forming an image frame sequence in chronological order; Step S2, setting up a deep learning model composed of an encoder and a decoder, and training the deep learning model by using the collected image frame sequence; Step S3, inputting an image frame sequence of multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs a predicted frame at the current moment; Step S4, obtaining a distance score by comparing the predicted frame at the current moment with the real frame; Step S5, comparing the distance score with the image frame change threshold to judge whether there is an abnormal change in the real frame at the current moment. If so, it is determined that aspiration failure or black spots will occur in the future moment, and a warning is given.

[0032] As Figure 3As shown, in this embodiment, setting up a deep learning model composed of an encoder and a decoder, and training the deep learning model using the collected image frame sequence includes: Setting up a discriminator to verify the output result of the deep learning model. The discriminator obtains the predicted frame at the current moment output by the decoder and the real frame at the current moment in the collected image frame sequence, compares the predicted frame at the current moment with the real frame. If they are the same, it is recorded as the first value; if they are different, it is recorded as the second value; Judging whether the deep learning model has completed training according to the ratio of the first value and the second value.

[0033] It can be understood that the first value is the degree of realism of generating the predicted frame, judging whether it is real. The discriminator is to see whether the generated image (predicted frame) is different from the original image (real frame), and mainly plays an auxiliary role in training.

[0034] In this embodiment, obtaining the distance score by comparing the predicted frame at the current moment with the real frame includes: Obtaining the distance score through the formula where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0035] Among them, the larger the value of the distance score, the less similar the predicted frame at the current moment is to the real frame, and the less similar it is, the more an early warning is given.

[0036] It should be noted that the formula for calculating the distance score is not unique, and any measurement method that can reflect the difference between the predicted frame and the real frame can be used as an alternative for distance calculation, such as peak signal-to-noise ratio, etc.

[0037] In this embodiment, judging whether the real frame at the current moment has an abnormal change by comparing the distance score with the image frame change threshold. If so, it is determined that suction loss or black spots will occur in the future moment, and an early warning is given, including: Setting the image frame change threshold to include a first threshold and a second threshold, and the first threshold is greater than the second threshold; When the distance score is greater than the first threshold, it is determined that suction loss or black spots occur at the current moment; When the distance score is between the first threshold and the second threshold, it is determined that suction loss or black spots will occur in the future moment, and an early warning is given; When the distance score is less than the second threshold, it is determined that the current moment is normal.

[0038] That is to say, setting the first threshold to and the second threshold to , where ; when , it is determined that aspiration loss or black spot occurs at the current moment; when , it is determined that aspiration loss or black spot is about to occur at a future moment, and a warning is given; when , it is determined that the current moment is normal.

[0039] where The selection can be determined by statistically analyzing the distribution in past cases.

[0040] Embodiment 2

[0041] In this embodiment, as Figure 4 shown, a method for warning of aspiration loss and black spot in femtosecond laser in situ keratomileusis is provided, including the following steps: Step S1, recording the operation under the surgical microscope through a video stream, and forming an image frame sequence in chronological order; Step S2, setting a deep learning model composed of an encoder and a decoder, and training the deep learning model by using the collected image frame sequence; Step S3, inputting the image frame sequence of multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs a predicted frame at the current moment; Step S4, obtaining a distance score by comparing the predicted frame at the current moment with the real frame; Step S5, judging whether the real frame at the current moment has abnormal changes according to the comparison between the distance score and the image frame change threshold. If so, it is determined that aspiration loss or black spot is about to occur at a future moment, and a warning is given.

[0042] In this embodiment, the setting of the deep learning model composed of an encoder and a decoder and training the deep learning model by using the collected image frame sequence includes: Setting the deep learning model includes a forward prediction model and a backward prediction model. Both the forward prediction model and the backward prediction model are composed of an encoder and a decoder. The forward prediction model is used to predict future frames through past frames, and the backward prediction model is used to predict past frames through future frames; When training the forward prediction model, input an image frame sequence in the collected image frame sequence into the forward prediction model. The forward prediction model outputs a predicted image sequence of several frames at subsequent moments, obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequence, calculate the loss of the forward prediction model through the absolute value loss function, and judge whether the forward prediction model is completed according to the value of the absolute value loss function; When training the backward prediction model, input an image frame sequence in the collected image frame sequences into the backward prediction model. The backward prediction model outputs a predicted image sequence of several frames at previous times. Obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequences. Calculate the loss of the backward prediction model through a square loss function, and determine whether the backward prediction model has completed training according to the value of the square loss function.

[0043] Where the absolute value loss function is , y' is the image in the predicted image sequence, y is the image in the real image sequence, i is the subscript of the pixel point in the image, and n is the number of pixel points in the image.

[0044] Where the square loss function is , y' is the image in the predicted image sequence, y is the image in the real image sequence, i is the subscript of the pixel point in the image, and n is the number of pixel points in the image.

[0045] In this embodiment, the obtaining the distance score by comparing the predicted frame and the real frame at the current time includes: Input the image frame sequence before the current time collected in real time into the forward prediction model. The forward prediction model outputs the predicted frame at the current time, and obtain the real frame at the current time; Calculate the distance score between the predicted frame and the real frame at the current time through the formula , where is the weight parameter, is the real frame at the current time, is the predicted frame at the current time, and t is the current time.

[0046] In this embodiment, the determining whether there is an abnormal change in the real frame at the current time according to the comparison between the distance score and the image frame change threshold, and if so, determining that air aspiration or black spot will occur at a future time and giving an early warning includes: When the distance score is greater than or equal to the image frame change threshold, it is determined that air aspiration or black spot occurs at the current time, and an early warning is given.

[0047] That is to say, set the image frame change threshold to be , when , it is determined that air aspiration or black spot phenomenon occurs at the current time.

[0048] In this embodiment, the method for early warning of air aspiration and black spot in femtosecond laser in situ keratomileusis further includes: Input the sequence of image frames collected in real time into the forward prediction model. The forward prediction model outputs a sequence of several predicted image frames after the current moment. Input the sequence of several predicted image frames after the current moment into the backward prediction model. The backward prediction model outputs a sequence of backward-inferred images corresponding to the sequence of image frames collected in real time. Calculate the distance score between each image frame in the sequence of image frames collected in real time and the sequence of backward-inferred images, and obtain the maximum value of the distance score. When the maximum value of the distance score is greater than the image frame change threshold, it is determined that air suction loss or black spot occurs at the current moment, and a warning is given.

[0049] That is to say, set the image frame change threshold to , for air suction loss warning, it is evaluated by combining the result of the backward prediction model with the similarity function. By obtain the maximum value of the distance score. When its value is greater than , it is determined that air suction loss or black spot will occur at future moments [t + 1, t + 2].

[0050] It can be understood that in Embodiment 2, the deep learning model is set to include a forward prediction model and a backward prediction model, and the air suction loss is judged by using the forward and backward two-way prediction method. The forward prediction model predicts future frames through past frames, which is a prediction of what will happen. The backward prediction model predicts past frames through future frames. If air suction loss occurs in the future frames, then there will be a large difference between the future frames predicted based on the current normal situation and the real future frames. Similarly, there will also be a large difference between the past frames inferred back from this predicted future frame and the real past frames. Then we can infer whether an abnormality will occur in the future by whether there is a large difference between the real past frames and the backward-predicted past frames.

[0051] As Figure 5 shown, the forward prediction model inputs a sequence of image frames and outputs a sequence of predicted images of the next several frames. Taking k = 2 and the sequence length set to 3 as an example, the input is , and the output is . It should be noted that the output here can either be that the model directly predicts the image frames of multiple moments in one inference, or predicts the image frame of the next moment each time and repeats the iteration. For example, through predict to get , and then from predict to get , until the required future frame sequence is obtained.

[0052] Subsequently, in the backward prediction model, the predicted frames obtained in the forward prediction model will be used as the input, that is, the above input , to obtain an estimate of the past frames. For example: —> ; —> ; ……。

[0053] Meanwhile, during the training phase, we also use real sequences for backward prediction. For example: —> ; —> ; ……。

[0054] During training, we use two loss functions, the absolute value loss function L1 and the squared loss function MSE, to calculate the losses of forward prediction and backward prediction respectively, as the supervision information for model training.

[0055] During inference, we still use the similarity formula in Method 1 to evaluate whether aspiration or black spot phenomenon has occurred or will occur.

[0056] For aspiration detection, we use to make a judgment. When it is determined that the current aspiration or black spot phenomenon has occurred.

[0057] For aspiration warning, we evaluate it by combining the result of the backward prediction model with the similarity function as follows:

[0058] When its value is greater than it is determined that aspiration or black spot will occur at [t + 1, t + 2].

[0059] In the above method for warning aspiration and black spot in femtosecond laser refractive surgery, by setting up a deep learning model composed of an encoder and a decoder, using the trained deep learning model to output the prediction frame at the current moment, and by comparing the prediction frame at the current moment with the real frame, when there is a deviation between the prediction frame and the real frame, it is judged that the real frame deviates from the prediction direction, so as to judge that aspiration or black spot has occurred based on the deviation of the real frame from the prediction direction. It can timely detect aspiration during the SMILE surgery process and give a warning before aspiration occurs, reduce the incidence of aspiration, and improve safety.

[0060] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 6As shown. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store femtosecond laser in situ keratomileusis (FS-LASIK) aspiration failure and black spot warning data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a method for FS-LASIK aspiration failure and black spot warning.

[0061] Those skilled in the art can understand that Figure 6 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0062] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: Record the operations under the surgical microscope through a video stream to form a sequence of image frames in chronological order; Set up a deep learning model composed of an encoder and a decoder, and use the collected sequence of image frames to train the deep learning model; Input the sequence of image frames of multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs a predicted frame at the current moment; By comparing the predicted frame at the current moment with the real frame, obtain a distance score; According to the comparison between the distance score and the image frame change threshold, determine whether there is an abnormal change in the real frame at the current moment. If so, it is determined that aspiration failure or black spots will occur in the future moment, and a warning is given.

[0063] In one embodiment, when the processor executes the computer program, the following steps are also implemented: The setting up of the deep learning model composed of an encoder and a decoder and using the collected sequence of image frames to train the deep learning model includes: Set up a discriminator to verify the output result of the deep learning model. The discriminator obtains the predicted frame at the current moment output by the decoder and the real frame at the current moment in the collected sequence of image frames, and compares the predicted frame at the current moment with the real frame. If they are the same, it is recorded as the first value, and if they are different, it is recorded as the second value; Judge whether the deep learning model has completed training according to the ratio of the first value and the second value.

[0064] In one embodiment, when the processor executes the computer program, the following steps are further implemented: The obtaining of the distance score by comparing the predicted frame and the real frame at the current moment includes: Through the formula Calculate the distance score, where is a weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0065] In one embodiment, when the processor executes the computer program, the following steps are further implemented: The judging whether there is an abnormal change in the real frame at the current moment according to the comparison between the distance score and the image frame change threshold, and if so, determining that air suction loss or black spots will occur at a future moment and giving an early warning includes: Set the image frame change threshold to include a first threshold and a second threshold, and the first threshold is greater than the second threshold; When the distance score is greater than the first threshold, it is determined that air suction loss or black spots occur at the current moment; When the distance score is between the first threshold and the second threshold, it is determined that air suction loss or black spots will occur at a future moment and an early warning is given; When the distance score is less than the second threshold, it is determined that the current moment is normal.

[0066] In one embodiment, when the processor executes the computer program, the following steps are further implemented: The setting of the deep learning model composed of an encoder and a decoder, and the training of the deep learning model using the collected image frame sequence includes: Set the deep learning model to include a forward prediction model and a backward prediction model. Both the forward prediction model and the backward prediction model are composed of an encoder and a decoder. The forward prediction model is used to predict future frames through past frames, and the backward prediction model is used to predict past frames through future frames; When training the forward prediction model, input an image frame sequence in the collected image frame sequence into the forward prediction model. The forward prediction model outputs a predicted image sequence of several frames at subsequent moments. Obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequence. Calculate the loss of the forward prediction model through the absolute value loss function, and judge whether the forward prediction model has completed training according to the value of the absolute value loss function; When training the backward prediction model, input an image frame sequence in the collected image frame sequences into the backward prediction model. The backward prediction model outputs a predicted image sequence of several frames at previous times. Obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequences. Calculate the loss of the backward prediction model through a squared loss function, and determine whether the backward prediction model has completed training according to the value of the squared loss function.

[0067] In one embodiment, when the processor executes the computer program, the following steps are further implemented: The obtaining the distance score by comparing the predicted frame and the real frame at the current time includes: Input an image frame sequence before the current time collected in real time into the forward prediction model. The forward prediction model outputs a predicted frame at the current time. Obtain the real frame at the current time; Through the formula Calculate the distance score between the predicted frame and the real frame at the current time, where Weight parameter, Is the real frame at the current time, Is the predicted frame at the current time, and t is the current time.

[0068] In one embodiment, when the processor executes the computer program, the following steps are further implemented: The determining whether there is an abnormal change in the real frame at the current time according to the comparison between the distance score and the image frame change threshold. If so, it is determined that air suction loss or black spot will occur in the future, and the warning includes: When the distance score is greater than or equal to the image frame change threshold, it is determined that air suction loss or black spot occurs at the current time, and a warning is given.

[0069] In one embodiment, when the processor executes the computer program, the following steps are further implemented: Input an image frame sequence collected in real time into the forward prediction model. The forward prediction model outputs a predicted image sequence of several frames after the current time. Input the predicted image sequence of several frames after the current time into the backward prediction model. The backward prediction model outputs a backward inference image sequence corresponding to the image frame sequence collected in real time. Calculate the distance score between each image frame in the image frame sequence collected in real time and the backward inference image sequence, and obtain the maximum value of the distance score; When the maximum value of the distance score is greater than the image frame change threshold, it is determined that air suction loss or black spot occurs at the current time, and a warning is given.

[0070] For the specific limitations on the implementation steps when the processor executes a computer program, reference can be made to the limitations on the method for suction loss and black spot warning in femtosecond laser in situ keratomileusis described above, which will not be elaborated here.

[0071] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Record the operations under the surgical microscope through a video stream, and form a sequence of image frames in chronological order; Set up a deep learning model composed of an encoder and a decoder, and use the collected sequence of image frames to train the deep learning model; Input the sequence of image frames of multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs a predicted frame at the current moment; By comparing the predicted frame at the current moment with the real frame, obtain a distance score; According to the comparison between the distance score and the image frame change threshold, determine whether the real frame at the current moment has an abnormal change. If so, it is determined that suction loss or black spots will occur in the future, and a warning is given.

[0072] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: The setting up of the deep learning model composed of an encoder and a decoder and using the collected sequence of image frames to train the deep learning model includes: Set up a discriminator to verify the output result of the deep learning model. The discriminator obtains the predicted frame at the current moment output by the decoder and the real frame at the current moment in the collected sequence of image frames, and compares the predicted frame at the current moment with the real frame. If they are the same, it is recorded as the first value, and if they are different, it is recorded as the second value; Judge whether the deep learning model has completed training according to the ratio of the first value and the second value.

[0073] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: The obtaining of the distance score by comparing the predicted frame at the current moment with the real frame includes: Obtain the distance score through the formula where is a weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0074] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: It is determined whether there is an abnormal change in the real frame at the current moment by comparing the distance score with the image frame change threshold. If so, it is determined that air suction loss or black spots will occur in the future moment, and early warnings are given, including: Setting the image frame change threshold includes a first threshold and a second threshold, and the first threshold is greater than the second threshold; When the distance score is greater than the first threshold, it is determined that air suction loss or black spots occur at the current moment; When the distance score is between the first threshold and the second threshold, it is determined that air suction loss or black spots will occur in the future moment, and early warnings are given; When the distance score is less than the second threshold, it is determined that the current moment is normal.

[0075] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: Setting a deep learning model composed of an encoder and a decoder, and training the deep learning model by using the collected image frame sequence includes: Setting the deep learning model includes a forward prediction model and a backward prediction model. Both the forward prediction model and the backward prediction model are composed of an encoder and a decoder. The forward prediction model is used to predict future frames through past frames, and the backward prediction model is used to predict past frames through future frames; When training the forward prediction model, input an image frame sequence in the collected image frame sequence into the forward prediction model. The forward prediction model outputs a predicted image sequence of several frames at subsequent moments, obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequence, calculate the loss of the forward prediction model through the absolute value loss function, and determine whether the forward prediction model is completed training according to the value of the absolute value loss function; When training the backward prediction model, input an image frame sequence in the collected image frame sequence into the backward prediction model. The backward prediction model outputs a predicted image sequence of several frames at previous moments, obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequence, calculate the loss of the backward prediction model through the square loss function, and determine whether the backward prediction model is completed training according to the value of the square loss function.

[0076] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: The obtaining the distance score by comparing the predicted frame and the real frame at the current moment includes: Input the image frame sequence before the current moment collected in real time into the forward prediction model. The forward prediction model outputs the predicted frame at the current moment, and obtain the real frame at the current moment; The distance score between the predicted frame and the real frame at the current moment is calculated through the formula , where is the weight parameter, is the real frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

[0077] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: Determine whether there is an abnormal change in the real frame at the current moment according to the comparison between the distance score and the image frame change threshold. If so, it is determined that aspiration failure or black spot will occur in the future moment, and a warning is given, including: When the distance score is greater than or equal to the image frame change threshold, it is determined that aspiration failure or black spot occurs at the current moment, and a warning is given.

[0078] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: Input the sequence of image frames collected in real time into the forward prediction model. The forward prediction model outputs a sequence of several predicted image frames after the current moment. Input the sequence of several predicted image frames after the current moment into the backward prediction model. The backward prediction model outputs a sequence of backward-inferred images corresponding to the sequence of image frames collected in real time. Calculate the distance score between each moment's image frame in the sequence of image frames collected in real time and the sequence of backward-inferred images, and obtain the maximum value of the distance score; When the maximum value of the distance score is greater than the image frame change threshold, it is determined that aspiration failure or black spot occurs at the current moment, and a warning is given.

[0079] For the specific limitations on the steps implemented when the computer program is executed by the processor, reference can be made to the limitations on the method for warning of aspiration failure and black spot in femtosecond laser in situ keratomileusis described above, which will not be elaborated here.

[0080] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0081] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0082] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of the patent of this application should be subject to the appended claims.

Claims

1. A method for warning of aspiration failure and black spots in femtosecond laser corneal refractive surgery, characterized in that, Including: Recording the operation under the surgical microscope through a video stream to form a sequence of image frames in chronological order; Setting up a deep learning model composed of an encoder and a decoder, and training the deep learning model using the collected sequence of image frames; Inputting the sequence of image frames of multiple frames before the current moment into the trained deep learning model, and the deep learning model outputs the predicted frame of the current moment; Obtaining a distance score by comparing the predicted frame of the current moment with the real frame; Judging whether there is an abnormal change in the real frame of the current moment according to the comparison between the distance score and the image frame change threshold. If so, it is determined that aspiration or black spot will occur in the future moment, and a warning is given.

2. The femtosecond laser in-situ keratomileusis aspiration failure and black spot warning method according to claim 1, wherein The setting up a deep learning model composed of an encoder and a decoder, and training the deep learning model using the collected sequence of image frames includes: Setting up a discriminator to verify the output result of the deep learning model. The discriminator obtains the predicted frame of the current moment output by the decoder and the real frame of the current moment in the collected sequence of image frames, and compares the predicted frame of the current moment with the real frame. If they are the same, it is recorded as the first value. If they are different, it is recorded as the second value; Judging whether the deep learning model is completed training according to the ratio of the first value and the second value.

3. The femtosecond laser in situ keratomileusis aspiration loss and black spot warning method according to claim 2, characterized in that The obtaining a distance score by comparing the predicted frame of the current moment with the real frame includes: The distance score is calculated through the formula where is the weight parameter, is the true frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

4. The femtosecond laser in situ keratomileusis aspiration failure and black spot warning method according to claim 3, wherein The judging whether there is an abnormal change in the real frame of the current moment according to the comparison between the distance score and the image frame change threshold. If so, it is determined that aspiration or black spot will occur in the future moment, and a warning is given includes: Setting the image frame change threshold to include a first threshold and a second threshold, and the first threshold is greater than the second threshold; When the distance score is greater than the first threshold, it is determined that aspiration or black spot occurs at the current moment; When the distance score is between the first threshold and the second threshold, it is determined that aspiration or black spot will occur in the future moment, and a warning is given; When the distance score is less than the second threshold, it is determined that the current moment is normal.

5. The femtosecond laser in-situ keratomileusis aspiration failure and black spot warning method according to claim 1, wherein The setting up a deep learning model composed of an encoder and a decoder, and training the deep learning model using the collected sequence of image frames includes: Setting the deep learning model to include a forward prediction model and a backward prediction model. Both the forward prediction model and the backward prediction model are composed of an encoder and a decoder. The forward prediction model is used to predict future frames through past frames, and the backward prediction model is used to predict past frames through future frames; When training the forward prediction model, inputting an image frame sequence in the collected image frame sequence into the forward prediction model. The forward prediction model outputs a predicted image sequence of several frames in the subsequent moment, obtaining the real image sequence corresponding to the predicted image sequence in the collected image frame sequence, calculating the loss of the forward prediction model through an absolute value loss function, and judging whether the forward prediction model is completed training according to the value of the absolute value loss function; When training the backward prediction model, input an image frame sequence in the collected image frame sequences into the backward prediction model. The backward prediction model outputs a predicted image sequence of several frames at previous times. Obtain the real image sequence corresponding to the predicted image sequence in the collected image frame sequences. Calculate the loss of the backward prediction model through a squared loss function. Determine whether the backward prediction model has completed training according to the value of the squared loss function.

6. The femtosecond laser in situ keratomileusis aspiration failure and black spot warning method according to claim 5, characterized in that The obtaining of the distance score by comparing the predicted frame and the real frame at the current time includes: Input an image frame sequence before the current time collected in real time into the forward prediction model. The forward prediction model outputs a predicted frame at the current time. Obtain the real frame at the current time. The distance score between the predicted frame and the ground truth frame at the current moment is calculated through the formula , where is the weight parameter, is the ground truth frame at the current moment, is the predicted frame at the current moment, and t is the current moment.

7. The femtosecond laser in situ keratomileusis aspiration failure and black spot warning method according to claim 6, wherein The determining whether there is an abnormal change in the real frame at the current time by comparing the distance score with the image frame change threshold. If so, it is determined that air suction loss or black spot will occur at a future time, and the early warning includes: When the distance score is greater than or equal to the image frame change threshold, it is determined that air suction loss or black spot occurs at the current time, and an early warning is given.

8. The femtosecond laser in situ keratomileusis aspiration failure and black spot warning method according to claim 7, characterized in that, The method further includes: Input an image frame sequence collected in real time into the forward prediction model. The forward prediction model outputs a predicted image sequence of several frames after the current time. Input the predicted image sequence of several frames after the current time into the backward prediction model. The backward prediction model outputs a backward inference image sequence corresponding to the image frame sequence collected in real time. Calculate the distance score between each image frame in the image frame sequence collected in real time and the backward inference image sequence. Obtain the maximum value of the distance score. When the maximum value of the distance score is greater than the image frame change threshold, it is determined that air suction loss or black spot occurs at the current time, and an early warning is given.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 8.

Images