Automatic eye movement tracking method, device, equipment and computer storage medium

Abstract

The present invention discloses an automatic eye movement tracking method, apparatus, device, and computer storage medium, comprising: inputting a set of ophthalmoscopic fundus images to be detected into a preset eye movement judgment neural network to obtain a predicted eye movement state corresponding to each ophthalmoscopic fundus image; when it is determined that the eye movement predicted state corresponding to a certain ophthalmoscopic fundus image is the start of an eye movement, determining the ophthalmoscopic fundus image as a first target image and inputting it into a preset optic disc positioning neural network to obtain optic disc positioning information; when it is determined that the eye movement predicted state corresponding to a certain ophthalmoscopic fundus image is the end of an eye movement, determining the ophthalmoscopic fundus image as a second target image, and obtaining offset information corresponding to the optic disc based on the first target image and the second target image. It can be seen that the present invention can continuously track the eyeball, accurately determine the start and end states of an eye movement, and thus calculate accurate eye movement offset information, effectively improving the efficiency and accuracy of eye movement tracking.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an automatic eye movement tracking method, device, equipment and computer storage medium. Background Art

[0002] During fundus image acquisition, eye movement is typically tracked based on ophthalmoscopic fundus images, such as LSO (Line Scanning Opthalmoscope) grayscale images, and then OCT scanning is performed simultaneously to obtain accurate OCT images. However, eye movement can affect OCT imaging quality, resulting in inability to capture valid data due to missed scan targets, blurred scan images, and repeated scans.

[0003] Therefore, it is particularly important to provide an automatic eye tracking method to improve the efficiency and accuracy of eye tracking. Summary of the Invention

[0004] The present invention provides an automatic eye movement tracking method, device, equipment and computer storage medium, which can continuously track the eyeball, accurately determine the start and end states of the eye movement, and then accurately track the movement of the eyeball, thereby calculating accurate eye movement offset information, effectively improving the efficiency and accuracy of eye movement tracking, and facilitating the acquisition accuracy and generation quality of subsequent OCT images. At the same time, combining the advantages of neural networks and registration algorithms, it further improves tracking speed and efficiency.

[0005] In order to solve the above technical problems, the first aspect of the present invention discloses an automatic eye movement tracking method, which includes:

[0006] Inputting the ophthalmoscopic fundus image set to be detected into a preset eye movement judgment neural network, and sequentially obtaining the eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected; wherein the ophthalmoscopic fundus image set to be processed includes a plurality of frames of ophthalmoscopic fundus images to be detected acquired in a time sequence;

[0007] Determining whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement start state;

[0008] When the judgment result is yes, determining the ophthalmoscopic fundus image to be detected as a first target image, and inputting the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image;

[0009] Determine whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement end state;

[0010] When the judgment result is yes, the ophthalmoscopic fundus image to be detected is determined as the second target image, and an offset prediction operation is performed according to the first target image and the second target image to obtain offset information corresponding to the optic disc positioning information.

[0011] As an optional embodiment, in the first aspect of the present invention, inputting the set of ophthalmoscopic fundus images to be detected into a preset eye movement judgment neural network and sequentially obtaining the eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected includes:

[0012] Preprocessing the ophthalmoscopic fundus image set to be detected to obtain a target tensor corresponding to the ophthalmoscopic fundus image set to be detected;

[0013] Inputting the target tensor into a preset eye movement judgment neural network to obtain a predicted probability value corresponding to each ophthalmoscopic fundus image to be detected;

[0014] determining an eye movement prediction state of the ophthalmoscopic fundus image to be detected according to the prediction probability value;

[0015] The preprocessing of the ophthalmoscopic fundus image set to be detected to obtain a target tensor corresponding to the ophthalmoscopic fundus image set to be detected includes:

[0016] splicing all the ophthalmoscopic fundus images to be detected in the ophthalmoscopic fundus image set to be detected into an N*H*W target tensor on the channel, wherein N is the number of all the ophthalmoscopic fundus images to be detected, H is the height of any ophthalmoscopic fundus image to be detected, and W is the width of any ophthalmoscopic fundus image to be detected; or

[0017] Subtracting any two adjacent ophthalmoscopic fundus images to be detected in the set of ophthalmoscopic fundus images to be detected to obtain N-1 intermediate images; and concatenating all the intermediate images into a (N-1)*H*W target tensor on the channel, where N is the number of all the ophthalmoscopic fundus images to be detected, H is the height of any ophthalmoscopic fundus image to be detected, and W is the width of any ophthalmoscopic fundus image to be detected.

[0018] As an optional implementation, in the first aspect of the present invention, the determining whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement start state includes:

[0019] For any frame of the ophthalmoscopic fundus image to be detected, the preset eye movement judgment neural network judges whether the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to a preset eye movement onset threshold;

[0020] When it is determined that the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to the preset eye movement onset threshold, the preset eye movement judgment neural network determines that the eye movement predicted state corresponding to the ophthalmoscopic fundus image to be detected is the eye movement onset state; wherein, the network structure of the preset eye movement judgment neural network includes several convolutional layers, normalization layers, activation layers, attention layers and fully connected layers.

[0021] As an optional implementation, in the first aspect of the present invention, determining whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement end state includes:

[0022] The preset eye movement judgment neural network tracks M prediction probability values ​​corresponding to M consecutive frames of the ophthalmoscopic fundus image to be detected, and determines whether the M prediction probability values ​​are all less than a preset eye movement end threshold;

[0023] When it is determined that the M predicted probability values ​​are all less than the preset eye movement end threshold, the eye movement predicted state corresponding to the last frame of the ophthalmoscopic fundus image to be detected in the M frames of the ophthalmoscopic fundus images to be detected is determined as the eye movement end state.

[0024] As an optional embodiment, in the first aspect of the present invention, inputting the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image includes:

[0025] The preset optic disc positioning neural network performs an optic disc segmentation operation on the first target image to obtain an optic disc segmentation mask corresponding to the first target image;

[0026] obtaining optic disc location information corresponding to the first target image according to the optic disc segmentation mask;

[0027] After obtaining the optic disc location information corresponding to the first target image according to the optic disc segmentation mask, the method further includes:

[0028] performing an information compensation operation on the optic disc segmentation mask to obtain optimized optic disc positioning information corresponding to the optic disc positioning information;

[0029] The step of performing an information compensation operation on the optic disc segmentation mask to obtain optimized optic disc positioning information corresponding to the optic disc positioning information includes:

[0030] Obtaining T scale information corresponding to the first target image; wherein any of the scale information is used to represent features of different dimensions extracted by the preset optic disc positioning neural network and matching the preset optic disc information in the first target image;

[0031] The high-level features and low-level features in the T scale information are fused, and the information lost during the multi-dimensional feature extraction operation is compensated through skip connections to obtain optimized optic disc positioning information corresponding to the optic disc positioning information.

[0032] As an optional implementation manner, in the first aspect of the present invention, performing an offset prediction operation based on the first target image and the second target image to obtain offset information corresponding to the video disc positioning information includes:

[0033] Determining the first target image as a reference image and the second target image as a matching image, performing an offset prediction operation using a preset registration method to complete a matching operation between all regional image blocks in the matching image and a matching template in the reference image, and determining offset information corresponding to the optic disc positioning information; and / or,

[0034] Obtaining offset information corresponding to the optic disc positioning information according to the first target image, the second target image, and a preset offset prediction neural network;

[0035] The performing of an offset prediction operation based on the first target image, the second target image, and a preset offset prediction neural network to obtain offset information corresponding to the video disc positioning information includes:

[0036] Inputting the first target image and the second target image into a preset offset prediction neural network to obtain offset information corresponding to the optic disc positioning information, wherein the offset information includes offsets in the x-direction and the y-direction of the two-dimensional plane, and the preset offset prediction neural network is a convolutional neural network model or a Transformer model; or

[0037] performing an expansion operation on the optic disc segmentation mask to obtain an expanded mask corresponding to the optic disc segmentation mask, wherein the expansion operation is used to perform background denoising and / or pixel enhancement on the optic disc segmentation mask;

[0038] The first target image, the second target image and the expanded mask are input into the preset offset prediction neural network to obtain the offset information corresponding to the optic disc positioning information, wherein the offset information includes the offset in the x-direction and the y-direction of the two-dimensional plane, and the preset offset prediction neural network is a convolutional neural network model or a Transformer model.

[0039] As an optional embodiment, in the first aspect of the present invention, the method further comprises:

[0040] When the offset prediction operation includes the preset registration method and the preset offset prediction neural network method, the offset information corresponding to the optic disc positioning information is corrected according to the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method to obtain corrected offset information corresponding to the optic disc positioning information;

[0041] The step of correcting the offset information corresponding to the video disc positioning information to obtain the corrected offset information corresponding to the video disc positioning information includes:

[0042] When a result error between a prediction result corresponding to the preset registration method and a prediction result corresponding to the preset offset prediction neural network method is less than or equal to a first error threshold, an average value of the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information;

[0043] When the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is greater than the first error threshold and less than the second error threshold, the prediction result corresponding to the preset registration method is determined as the corrected offset information;

[0044] When the result error between the prediction result corresponding to the preset alignment method and the prediction result corresponding to the preset offset prediction neural network method is greater than or equal to the second error threshold, the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information.

[0045] A second aspect of the present invention discloses an automatic eye movement tracking device, comprising:

[0046] An eye movement prediction module is configured to input a set of ophthalmoscopic fundus images to be detected into a preset eye movement judgment neural network, and sequentially obtain an eye movement prediction state corresponding to each of the ophthalmoscopic fundus images to be detected; wherein the set of ophthalmoscopic fundus images to be processed includes a plurality of frames of ophthalmoscopic fundus images to be detected acquired in a time sequence;

[0047] A first judgment module is used to judge whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement start state;

[0048] an optic disc positioning module, configured to, when the determination result of the first determination module is yes, determine the ophthalmoscopic fundus image to be detected as a first target image, and input the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image;

[0049] The second judgment module is used to judge whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement end state;

[0050] an offset prediction module for determining the ophthalmoscopic fundus image to be detected as a second target image when the judgment result of the second judgment module is yes, and performing an offset prediction operation based on the first target image and the second target image to obtain offset information corresponding to the optic disc positioning information.

[0051] As an optional implementation, in the second aspect of the present invention, the eye movement prediction module includes:

[0052] A preprocessing submodule, configured to preprocess the ophthalmoscopic fundus image set to be detected to obtain a target tensor corresponding to the ophthalmoscopic fundus image set to be detected;

[0053] A calculation submodule, configured to input the target tensor into a preset eye movement judgment neural network to obtain a predicted probability value corresponding to each ophthalmoscopic fundus image to be detected;

[0054] a determination submodule, configured to determine an eye movement prediction state of the ophthalmoscopic fundus image to be detected according to the prediction probability value;

[0055] The specific method of the preprocessing submodule for preprocessing the ophthalmoscopic fundus image set to be detected and obtaining the target tensor corresponding to the ophthalmoscopic fundus image set to be detected is as follows:

[0056] splicing all the ophthalmoscopic fundus images to be detected in the ophthalmoscopic fundus image set to be detected into an N*H*W target tensor on the channel, wherein N is the number of all the ophthalmoscopic fundus images to be detected, H is the height of any ophthalmoscopic fundus image to be detected, and W is the width of any ophthalmoscopic fundus image to be detected; or

[0057] Subtracting any two adjacent ophthalmoscopic fundus images to be detected in the set of ophthalmoscopic fundus images to be detected to obtain N-1 intermediate images; and concatenating all the intermediate images into a (N-1)*H*W target tensor on the channel, where N is the number of all the ophthalmoscopic fundus images to be detected, H is the height of any ophthalmoscopic fundus image to be detected, and W is the width of any ophthalmoscopic fundus image to be detected.

[0058] As an optional implementation, in the second aspect of the present invention, the first judgment module determines whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement start state in a specific manner as follows:

[0059] For any frame of the ophthalmoscopic fundus image to be detected, the preset eye movement judgment neural network judges whether the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to a preset eye movement onset threshold;

[0060] When it is determined that the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to the preset eye movement onset threshold, the preset eye movement judgment neural network determines that the eye movement predicted state corresponding to the ophthalmoscopic fundus image to be detected is the eye movement onset state; wherein, the network structure of the preset eye movement judgment neural network includes several convolutional layers, normalization layers, activation layers, attention layers and fully connected layers.

[0061] As an optional implementation, in the second aspect of the present invention, the second judgment module determines whether there is an eye movement prediction state corresponding to the ophthalmoscopic fundus image to be detected that is an eye movement end state in a specific manner:

[0062] The preset eye movement judgment neural network tracks M prediction probability values ​​corresponding to M consecutive frames of the ophthalmoscopic fundus image to be detected, and determines whether the M prediction probability values ​​are all less than a preset eye movement end threshold;

[0063] When it is determined that the M predicted probability values ​​are all less than the preset eye movement end threshold, the eye movement predicted state corresponding to the last frame of the ophthalmoscopic fundus image to be detected in the M frames of the ophthalmoscopic fundus images to be detected is determined as the eye movement end state.

[0064] As an optional embodiment, in the second aspect of the present invention, the optic disc positioning module includes:

[0065] a segmentation submodule, configured to use the preset optic disc positioning neural network to perform an optic disc segmentation operation on the first target image to obtain an optic disc segmentation mask corresponding to the first target image;

[0066] a compensation submodule, configured to perform an information compensation operation on the optic disc segmentation mask to obtain optic disc positioning information corresponding to the first target image;

[0067] The compensation submodule performs information compensation on the optic disc segmentation mask to obtain the optic disc positioning information corresponding to the first target image in the following specific manner:

[0068] Obtaining T scale information corresponding to the first target image; wherein any of the scale information is used to represent features of different dimensions extracted by the preset optic disc positioning neural network and matching the preset optic disc information in the first target image;

[0069] The high-level features and low-level features in the T scale information are fused, and information lost during the multi-dimensional feature extraction operation is compensated through skip connections to obtain optic disc positioning information corresponding to the first target image.

[0070] As an optional implementation, in the second aspect of the present invention, the offset prediction module performs an offset prediction operation based on the first target image and the second target image to obtain the offset information corresponding to the video disc positioning information in the following specific manner:

[0071] Determining the first target image as a reference image and the second target image as a matching image, performing an offset prediction operation using a preset registration method to complete a matching operation between all regional image blocks in the matching image and a matching template in the reference image, and determining offset information corresponding to the optic disc positioning information; and / or,

[0072] Obtaining offset information corresponding to the optic disc positioning information according to the first target image, the second target image, and a preset offset prediction neural network;

[0073] The performing of an offset prediction operation based on the first target image, the second target image, and a preset offset prediction neural network to obtain offset information corresponding to the video disc positioning information includes:

[0074] Inputting the first target image and the second target image into a preset offset prediction neural network to obtain offset information corresponding to the optic disc positioning information, wherein the offset information includes offsets in the x-direction and the y-direction of the two-dimensional plane, and the preset offset prediction neural network is a convolutional neural network model or a Transformer model; or

[0075] performing an expansion operation on the optic disc segmentation mask to obtain an expanded mask corresponding to the optic disc segmentation mask, wherein the expansion operation is used to perform background denoising and / or pixel enhancement on the optic disc segmentation mask;

[0076] The first target image, the second target image and the expanded mask are input into the preset offset prediction neural network to obtain the offset information corresponding to the optic disc positioning information, wherein the offset information includes the offset in the x-direction and the y-direction of the two-dimensional plane, and the preset offset prediction neural network is a convolutional neural network model or a Transformer model.

[0077] As an optional embodiment, in the second aspect of the present invention, the device further includes:

[0078] a correction module configured to, when the offset prediction operation includes the preset registration method and the preset offset prediction neural network method, correct the offset information corresponding to the optic disc positioning information based on the prediction results corresponding to the preset registration method and the prediction results corresponding to the preset offset prediction neural network method, to obtain corrected offset information corresponding to the optic disc positioning information;

[0079] The correction module corrects the offset information corresponding to the video disc positioning information to obtain the corrected offset information corresponding to the video disc positioning information in the following specific manner:

[0080] When a result error between a prediction result corresponding to the preset registration method and a prediction result corresponding to the preset offset prediction neural network method is less than or equal to a first error threshold, an average value of the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information;

[0081] When the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is greater than the first error threshold and less than the second error threshold, the prediction result corresponding to the preset registration method is determined as the corrected offset information;

[0082] When the result error between the prediction result corresponding to the preset alignment method and the prediction result corresponding to the preset offset prediction neural network method is greater than or equal to the second error threshold, the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information.

[0083] A third aspect of the present invention discloses another automatic eye tracking device, comprising:

[0084] a memory storing executable program code;

[0085] a processor coupled to the memory;

[0086] The processor calls the executable program code stored in the memory to execute part or all of the steps in any one of the automatic eye tracking methods disclosed in the first aspect of the present invention.

[0087] A fourth aspect of the present invention discloses a computer storage medium storing computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in any one of the automatic eye tracking methods disclosed in the first aspect of the present invention.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] The present invention discloses an automatic eye movement tracking method and device. The method comprises: inputting a set of ophthalmoscopic fundus images to be detected into a preset eye movement judgment neural network, sequentially obtaining an eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected; when it is determined that the eye movement prediction state corresponding to a certain ophthalmoscopic fundus image is an eye movement start state, determining the ophthalmoscopic fundus image to be detected as a first target image, and inputting the image into a preset optic disc positioning neural network to obtain corresponding optic disc positioning information; when it is determined that the eye movement prediction state corresponding to a certain ophthalmoscopic fundus image is an eye movement end state, determining the ophthalmoscopic fundus image to be detected as a second target image, and performing an offset prediction operation based on the first target image and the second target image to obtain offset information corresponding to the optic disc positioning information. It can be seen that the present invention can continuously track the eyeball, accurately determine the start state and end state of the eye movement, and then accurately track the eyeball's movement, thereby calculating accurate eye movement offset information, effectively improving the efficiency and accuracy of eye movement tracking, and facilitating the acquisition accuracy and generation quality of subsequent OCT images. At the same time, combining the advantages of neural networks and registration algorithms, it further improves tracking speed and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0091] Figure 1 This is a flow chart of an automatic eye movement tracking method disclosed in an embodiment of the present invention;

[0092] Figure 2 is a flow chart of another automatic eye movement tracking method disclosed in an embodiment of the present invention;

[0093] Figure 3 1 is a schematic structural diagram of an automatic eye tracking device disclosed in an embodiment of the present invention;

[0094] Figure 4 1 is a schematic structural diagram of another automatic eye tracking device disclosed in an embodiment of the present invention;

[0095] Figure 5 It is a structural diagram of an automatic eye movement tracking device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0096] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0097] The terms "first," "second," and so on, in the description and claims of the present invention and the accompanying drawings are used to distinguish between different items, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or end comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed therein, or may optionally include other steps or elements inherent to such process, method, product, or end.

[0098] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0099] The present invention discloses an automatic eye movement tracking method, device, equipment, and computer storage medium. The method and device can continuously track the eye, accurately determine the start and end states of eye movements, and thus accurately track the eye movements, thereby calculating precise eye movement offset information. This effectively improves the efficiency and accuracy of eye movement tracking, facilitates the accuracy and quality of subsequent OCT image acquisition, and combines the advantages of neural networks and registration algorithms to further improve tracking speed and efficiency. A detailed description of each method and device follows below.

[0100] Example 1

[0101] See also Figure 1 , Figure 1 This is a flow chart of an automatic eye tracking method disclosed in an embodiment of the present invention. Figure 1 The described method can be applied to an automatic eye tracking device, which can be an independent device or integrated into an automatic eye tracking processing device, and the embodiment of the present invention does not limit this. Figure 1 As shown, the automatic eye movement tracking method may include the following operations:

[0102] 101. Input the ophthalmoscopic fundus image set to be detected into a preset eye movement judgment neural network, and sequentially obtain the eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected.

[0103] In an embodiment of the present invention, the present invention is mainly composed of three neural network models, wherein the first neural network model is a preset eye movement judgment neural network, which can automatically learn the difference information between two images and output the predicted state of the relative eye movement of the two images. Among them, the ophthalmoscope fundus image is a planar two-dimensional imaging of the fundus, which can be a laser scanning ophthalmoscope, an LSO (Line Scanning Opthalmoscope) image, or a confocal cLSO image (confocal scanning ophthalmoscope), etc., which is not limited in the embodiment of the present invention. It should be noted that the present invention does not limit the number of images or the dimension of the tensor input into the preset eye movement judgment neural network. In an embodiment of the present invention, the ophthalmoscope fundus image set to be processed can include several frames of ophthalmoscope fundus images to be detected arranged in time sequence. After the image set is input into the preset eye movement judgment neural network, the eye movement prediction state corresponding to each ophthalmoscope fundus image to be detected can be obtained.

[0104] 102. Determine whether an eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected is an eye movement start state.

[0105] 103. When the judgment result is yes, determine the ophthalmoscopic fundus image to be detected as a first target image, and input the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image.

[0106] In an embodiment of the present invention, a preset eye movement judgment neural network performs eye movement judgment on each frame of the ophthalmoscope fundus image. When eye movement is determined to have occurred, the ophthalmoscope fundus image of the current frame is recorded as the first target image, and the first target image is input into the preset optic disc positioning neural network. The optic disc in the first target image is masked and segmented to obtain the segmentation mask of the optic disc and the coordinate position information of the optic disc.

[0107] 104. Determine whether there is an eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected that is an eye movement end state.

[0108] 105. When the judgment result is yes, determine the ophthalmoscopic fundus image to be detected as the second target image, and perform an offset prediction operation based on the first target image and the second target image to obtain offset information corresponding to the optic disc positioning information.

[0109] In this embodiment of the present invention, after determining that the eye movement has ended, an offset prediction operation is performed on the first target image and the second target image to determine the offset of the position of the optic disc on the second target image relative to the first target image. For example, the offset of the pixel coordinate positions in the x and y directions is determined.

[0110] It can be seen that the method described in the embodiment of the present invention can continuously track the eyeball, accurately determine the starting state and the ending state of the eye movement, and then accurately track the movement of the eyeball, thereby calculating accurate eye movement offset information, effectively improving the efficiency and accuracy of eye movement tracking, and is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0111] In an optional embodiment, inputting the set of ophthalmoscopic fundus images to be detected into a preset eye movement judgment neural network and sequentially obtaining the eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected may include the following operations:

[0112] Preprocessing the ophthalmoscopic fundus image set to be detected to obtain a target tensor corresponding to the ophthalmoscopic fundus image set to be detected;

[0113] Input the target tensor into the preset eye movement judgment neural network to obtain the predicted probability value corresponding to each ophthalmoscopic fundus image to be detected;

[0114] determining an eye movement prediction state of the ophthalmoscopic fundus image to be detected according to the prediction probability value;

[0115] The ophthalmoscope fundus image set to be detected is preprocessed to obtain a target tensor corresponding to the ophthalmoscope fundus image set to be detected, including:

[0116] All the ophthalmoscopic fundus images to be tested in the ophthalmoscopic fundus image set to be tested are stitched together into a target tensor of N*H*W on the channels, where N is the number of all ophthalmoscopic fundus images to be tested, H is the height of any ophthalmoscopic fundus image to be tested, and W is the width of any ophthalmoscopic fundus image to be tested; or

[0117] Subtract any two adjacent ophthalmoscopic fundus images in the ophthalmoscopic fundus image set to obtain N-1 intermediate images; concatenate all the intermediate images into a (N-1)*H*W target tensor on the channel, where N is the number of all ophthalmoscopic fundus images to be tested, H is the height of any ophthalmoscopic fundus image to be tested, and W is the width of any ophthalmoscopic fundus image to be tested.

[0118] In an embodiment of the present invention, the ophthalmoscope fundus image set to be detected is preprocessed to obtain channel stitching of N images arranged in time sequence or N-1 processed images obtained by subtracting adjacent two images, both of which can be used as input tensors for the preset eye movement judgment neural network. For example, for the channel stitching of N images, because the vector information of a single ophthalmoscope fundus image is 1*H*W (ophthalmoscope fundus images are usually grayscale images, so they are single-channel images), for example, 4 images at adjacent moments t, t-1, t-2, and t-3 are stitched on the channel into a tensor of 4*H*W (H, W are the pixel height and pixel width of the image), and the input network is this tensor. For N-1 processed images obtained by subtracting adjacent two images, for example, 4 images I at moments t, t-1, t-2, and t-3 are taken. t , I t-1 , I t-2 , I t-3 , and we get I by subtracting them two by two a =I t -I t-1 , I b =I t-1 -I t-2 , I c =I t-2 -I t-3 Three images (I a , I b , I c ), the three images are spliced ​​into a 3*H*W tensor input network.

[0119] It can be seen that the method described in the embodiment of the present invention can provide various forms of images or tensors to input into the eye movement judgment neural network to determine the eye movement state, which can improve the versatility, applicability and flexibility of the present solution, and is conducive to improving the accuracy of tracking eye movements, effectively improving the efficiency and accuracy of eye tracking, thereby calculating accurate eye movement offset information, which is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0120] In another optional embodiment, determining whether the eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected is the eye movement start state may include the following operations:

[0121] For any frame of ophthalmoscopic fundus image to be detected, the preset eye movement judgment neural network determines whether the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to a preset eye movement start threshold;

[0122] When it is determined that the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to the preset eye movement onset threshold, the preset eye movement judgment neural network determines that the eye movement predicted state corresponding to the ophthalmoscopic fundus image to be detected is the eye movement onset state.

[0123] In an embodiment of the present invention, the preset eye movement judgment neural network is a network based on a convolutional neural network or a Transformer model, and its network structure includes several convolutional layers, normalization layers, activation layers, attention layers, and fully connected layers. The paired images in the training process simulate the eye movements that occur in adjacent images in the actual acquisition as much as possible. It should be noted that its output instance relative to the eye movement prediction can be a predicted probability value. The smaller the predicted probability value, the less eye movement occurs, and the larger the predicted probability value, the more eye movement occurs. In addition, an argmax operation is performed on the predicted probability value to map the predicted probability value to the {0,1} interval, that is, 0 represents no eye movement and 1 represents eye movement. When the probability value output by the network is greater than a certain threshold, it can be determined that the current state is the eye movement start state.

[0124] It can be seen that the method described in the embodiment of the present invention can infer the accurate eye movement onset state through the quantified prediction probability value, and accurately determine which frame of the image is at the specific eye movement onset moment, which is conducive to improving the accuracy of tracking eye movements and effectively improving the efficiency and accuracy of eye tracking, thereby calculating accurate eye movement offset information, which is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0125] In another optional embodiment, determining whether the eye movement predicted state corresponding to a fundus ophthalmoscopy image to be detected is the eye movement end state may include the following operations:

[0126] The preset eye movement judgment neural network tracks M prediction probability values ​​corresponding to M consecutive frames of ophthalmoscopic fundus images to be detected, and judges whether the M prediction probability values ​​are all less than a preset eye movement end threshold;

[0127] When it is determined that the M predicted probability values ​​are all less than the preset eye movement end threshold, the eye movement predicted state corresponding to the last frame of the ophthalmoscopic fundus image to be detected in the M frames of ophthalmoscopic fundus images to be detected is determined as the eye movement end state.

[0128] In this embodiment of the present invention, a preset eye movement determination neural network continuously tracks the predicted probability values ​​output by the network for a plurality of ophthalmoscopic fundus images. When it is determined that the predicted probability values ​​have returned to a preset eye movement end threshold, the current state is determined to be an eye movement end state. Simultaneously, the last frame of the plurality of ophthalmoscopic fundus images is marked as the image corresponding to the eye movement end state.

[0129] It can be seen that the method described in the embodiment of the present invention can infer the accurate eye movement end state through the quantified prediction probability value, and accurately determine which frame of the image at the specific eye movement end moment is, which is conducive to improving the accuracy of tracking eye movements, effectively improving the efficiency and accuracy of eye tracking, and thus calculating accurate eye movement offset information, which is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0130] In yet another optional embodiment, inputting the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image may include:

[0131] The preset optic disc positioning neural network performs an optic disc segmentation operation on the first target image to obtain an optic disc segmentation mask corresponding to the first target image;

[0132] Obtaining optic disc location information corresponding to the first target image according to the optic disc segmentation mask;

[0133] After obtaining the optic disc location information corresponding to the first target image according to the optic disc segmentation mask, the method may further include the following operations:

[0134] performing an information compensation operation on the optic disc segmentation mask to obtain optimized optic disc positioning information corresponding to the optic disc positioning information;

[0135] The information compensation operation is performed on the optic disc segmentation mask to obtain optimized optic disc positioning information corresponding to the optic disc positioning information, including:

[0136] Obtaining T scale information corresponding to the first target image; wherein any scale information is used to represent features of different dimensions extracted by the preset optic disc positioning neural network and matching the preset optic disc information in the first target image;

[0137] The high-level features and low-level features in the T scale information are fused, and the information lost during the multi-dimensional feature extraction operation is compensated through skip connections to obtain the optimized optic disc positioning information corresponding to the optic disc positioning information.

[0138] In an embodiment of the present invention, after the ophthalmoscopic fundus image determined to be in an eye movement onset state is determined as a first target image, an optic disc segmentation operation is performed on the target image using an optic disc annotation frame to extract optic disc features. The first target image is segmented to obtain an optic disc segmentation mask corresponding to the first target image. In other words, optic disc positioning information corresponding to the first image, i.e., pixel coordinate position information of the optic disc in the first target image, can be obtained.

[0139] In this optional embodiment, the boundary of the optic disc segmentation mask is then optimized, that is, the above-mentioned information compensation operation is performed to obtain optimized optic disc positioning information corresponding to the optic disc positioning information, that is, more accurate pixel coordinate position information of the optic disc in the first target image.

[0140] In this optional embodiment, the preset optic disc localization neural network can be a Resnet18Unet model that combines Resnet18 and Unet. The more powerful ResNet is used as the backbone for the UNet encoder, replacing Unet's traditional (Conv+BN+ReLU) stacked module structure. This can achieve faster and more effective optic disc localization and segmentation. The Resnet18-Unet network feature extraction component pools the original image five times in sequence to obtain information at five scales. The upsampling component fuses shallow high-resolution and deep low-resolution features at different scales and utilizes skip connections to supplement information lost by downsampling, resulting in better optic disc segmentation.

[0141] It can be seen that the method described in the embodiment of the present invention can perform optic disc segmentation and positioning on the first target image through the preset optic disc positioning neural network, and obtain accurate positioning information of the optic disc segmentation mask. At the same time, combined with the Resnet18Unet network model, the learning ability of the model is improved, the speed and effect of positioning and segmentation are improved, and the efficiency and accuracy of eye tracking are further effectively improved, which is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0142] In yet another optional embodiment, performing an offset prediction operation based on the first target image and the second target image to obtain offset information corresponding to the video disc positioning information may include the following operations:

[0143] Determining the first target image as the reference image and the second target image as the matching image, performing an offset prediction operation using a preset registration method to complete a matching operation between all regional image blocks in the matching image and a matching template in the reference image, and determining offset information corresponding to the optic disc positioning information; and / or,

[0144] Obtaining offset information corresponding to the optic disc positioning information according to the first target image, the second target image, and a preset offset prediction neural network;

[0145] The offset prediction operation is performed according to the first target image, the second target image, and a preset offset prediction neural network to obtain offset information corresponding to the video disc positioning information, including:

[0146] Inputting the first target image and the second target image into a preset offset prediction neural network to obtain offset information corresponding to the optic disc positioning information, the offset information including offsets in the x-direction and the y-direction of the two-dimensional plane, wherein the preset offset prediction neural network is a convolutional neural network model or a Transformer model; or

[0147] Performing an expansion operation on the optic disc segmentation mask to obtain an expanded mask corresponding to the optic disc segmentation mask, wherein the expansion operation is used to perform background denoising and / or pixel enhancement on the optic disc segmentation mask;

[0148] The first target image, the second target image and the expanded mask are input into a preset offset prediction neural network to obtain offset information corresponding to the optic disc positioning information. The offset information includes the offset in the x-direction and the y-direction of the two-dimensional plane. The preset offset prediction neural network is a convolutional neural network model or a Transformer model.

[0149] In the embodiment of the present invention, the preset registration method is an image registration method, including template matching, rigid registration, affine registration, and bspline registration, etc., which are not limited in the embodiment of the present invention. In the embodiment of the present invention, the preset registration method and the preset offset prediction neural network method can be combined to obtain offset information corresponding to the optic disc positioning information.

[0150] In the embodiment of the present invention, for the preset registration method, the optic disc is used as the aiming point, and i template matching areas are selected. The i areas are selected based on the principle of covering the optic disc and retinal blood vessels as much as possible. Based on the characteristics of the fundus image and the ophthalmoscope fundus image, the selection of the optic disc matching frame considers three situations: (1) Selection of the optic disc center reference frame: If it is the right eye, the center position of the matching frame is the optic disc center and needs to be shifted to the left by several pixels. If it is the left eye, it needs to be shifted to the right by several pixels. The size is about several times the area of ​​the optic disc; (2) Selection of the matching frame containing both blood vessels and optic disc: The center of the optic disc is shifted up and down by several pixels respectively, and the upper and lower matching frames of the same size as the optic disc reference frame are selected. The size of the matching frame is about several times the area of ​​the optic disc; (3) Selection of the vascular area matching frame: For the right eye, the upper left and lower left sides of the optic disc are selected. For the left eye, the upper right and lower right sides of the optic disc are selected. The size of the matching frame is about several times the area of ​​the optic disc that is larger than the optic disc center reference frame. The criterion for selecting the value of the optic disc template matching is as follows: the median of the i template matching prediction values ​​is used as the baseline, and the results of the remaining matching boxes whose error with the median is within 3 pixels are taken. The final template matching result is the average of these results, and the x and y direction offsets are calculated independently.

[0151] In embodiments of the present invention, for a preset offset prediction neural network approach, the first and second target images can optionally be used as inputs to the preset offset prediction neural network to output offset information. Furthermore, an expansion operation can be performed on the optic disc segmentation mask to obtain a corresponding expanded mask. The first and second target images, along with the expanded mask, can then be used as inputs to the preset offset prediction neural network to output offset information. It should be noted that the expansion operation is used to perform background denoising and / or pixel enhancement on the optic disc segmentation mask, such as the erosion and dilation operations in morphological filtering, and is not limited in this embodiment of the present invention.

[0152] It can be seen that the method described in the embodiment of the present invention can construct a preset offset prediction neural network with diverse input forms to achieve accurate prediction of offset information. At the same time, it also provides a registration algorithm to calculate the offset information, which is conducive to the subsequent combination and correction of the calculation results of the two, further effectively improving the efficiency and accuracy of eye tracking, and is beneficial to the subsequent OCT image acquisition accuracy and generation quality.

[0153] Example 2

[0154] See also Figure 2 , Figure 2 This is another flow chart of automatic eye tracking disclosed in an embodiment of the present invention. Figure 2 The described method can be applied to an automatic eye tracking device, which can be an independent device or integrated into an automatic eye tracking processing device, and the embodiment of the present invention does not limit this. Figure 2 As shown, the automatic eye movement tracking method may include the following operations:

[0155] 201. Input the ophthalmoscopic fundus image set to be detected into a preset eye movement judgment neural network, and sequentially obtain the eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected.

[0156] 202. Determine whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement start state.

[0157] 203. When the judgment result is yes, determine the ophthalmoscopic fundus image to be detected as the first target image, and input the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image.

[0158] 204. Determine whether there is an eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected that is an eye movement end state.

[0159] 205. When the judgment result is yes, determine the ophthalmoscopic fundus image to be detected as the second target image, and perform an offset prediction operation based on the first target image and the second target image to obtain offset information corresponding to the optic disc positioning information.

[0160] In the embodiment of the present invention, for other descriptions of step 201 to step 205, please refer to the detailed description of step 101 to step 105 in embodiment 1 respectively, and the embodiment of the present invention will not be repeated.

[0161] 206. Correct the offset information corresponding to the optic disc positioning information according to the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method to obtain corrected offset information corresponding to the optic disc positioning information.

[0162] In an embodiment of the present invention, specifically, when the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is less than or equal to a first error threshold, the average value of the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information; when the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is greater than the first error threshold and less than the second error threshold, the prediction result corresponding to the preset registration method is determined as the corrected offset information; when the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is greater than or equal to the second error threshold, the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information.

[0163] For example, when the error between the offset prediction network and the template matching result is within 3 pixels, the offset is the average of the two; if the two prediction results are within 20 pixels but the error is greater than 3 pixels, the result is the template matching prediction result; if the two prediction results exceed 20 pixels and the error is greater than 3 pixels, the result is the neural network prediction result.

[0164] It can be seen that the method described in the embodiment of the present invention can combine the advantages of deep neural networks and template matching to automatically perform eye tracking judgment and offset information correction on the time series images of LSO, improve the tracking effect and tracking accuracy, and further improve the accuracy of the obtained eye movement offset information.

[0165] In another optional embodiment, the method further includes a training method for three neural network models, wherein the three neural network models are all trained and learned independently.

[0166] For the preset eye movement judgment neural network, its input is the channel splicing of N-channel images, using the Imagenet pre-training parameters, and replacing the 3 channels of the first layer of convolution with random initialization parameters of N-channel input. The loss function is binary cross loss.

[0167] For the preset optic disc localization neural network, its input is a 1-channel grayscale image. The pre-trained parameters of Imagenet are also used, and the 3 channels of the first layer of convolution are replaced by random initialization parameters of a single channel input. The loss function is a combination of cross entropy loss and Dice loss.

[0168] For the preset offset prediction neural network, its input is the channel concatenation of 2- or 3-channel images, the pre-trained parameters of Imagenet are used to replace the first layer of convolution, and the loss function is a combination of the mean square error loss function of the x-direction and y-direction offsets.

[0169] It can be seen that the method described in the embodiment of the present invention can improve the convergence speed of the training process through the pre-training parameters of Imagenet, and fine-tune the weight parameters of all layers during the training process, which is conducive to reducing the loss function through gradient backpropagation and improving the parameter learning and updating capabilities. At the same time, the dynamic learning rate adjustment technology is adopted, and the learning rate is dynamically adjusted downward using the loss indicator to improve the speed and efficiency of training, further effectively improve the efficiency and accuracy of eye tracking, and are beneficial to the acquisition accuracy and generation quality of OCT images.

[0170] Example 3

[0171] See also Figure 3 , Figure 3 : is a schematic diagram of the structure of an automatic eye tracking device disclosed in an embodiment of the present invention. Figure 3 The described device can be an independent device or integrated into an automatic eye tracking processing device, which is not limited in the embodiment of the present invention. It should be noted that the automatic eye tracking device refers to the steps of an automatic eye tracking method described in the first and second embodiments, and the detailed description is not repeated in this embodiment. Figure 3 As shown, the automatic eye tracking device may include:

[0172] The eye movement prediction module 301 is configured to input the set of ophthalmoscopic fundus images to be detected into a preset eye movement judgment neural network, and sequentially obtain an eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected; wherein the set of ophthalmoscopic fundus images to be processed includes a plurality of frames of ophthalmoscopic fundus images to be detected acquired in a time sequence;

[0173] The first judgment module 302 is used to judge whether there is an eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected that is an eye movement start state;

[0174] an optic disc positioning module 303 for determining the ophthalmoscopic fundus image to be detected as a first target image when the determination result of the first determination module 302 is yes, and inputting the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image;

[0175] The second judgment module 304 is used to judge whether there is an eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected that is an eye movement end state;

[0176] The offset prediction module 305 is configured to determine the ophthalmoscopic fundus image to be detected as the second target image when the judgment result of the second judgment module 304 is yes, and perform an offset prediction operation based on the first target image and the second target image to obtain offset information corresponding to the optic disc positioning information.

[0177] It can be seen that the device described in the embodiment of the present invention is able to continuously track the eyeball, accurately determine the starting state and the ending state of the eye movement, and then accurately track the movement of the eyeball, thereby calculating accurate eye movement offset information, effectively improving the efficiency and accuracy of eye movement tracking, and is beneficial to the subsequent OCT image acquisition accuracy and generation quality.

[0178] In an optional embodiment, if Figure 4 As shown, the eye movement prediction module 301 may include:

[0179] A preprocessing submodule 3011 is used to preprocess the ophthalmoscopic fundus image set to be detected to obtain a target tensor corresponding to the ophthalmoscopic fundus image set to be detected;

[0180] The calculation submodule 3012 is used to input the target tensor into the preset eye movement judgment neural network to obtain the predicted probability value corresponding to each ophthalmoscopic fundus image to be detected;

[0181] a determination submodule 3013 for determining an eye movement prediction state of the ophthalmoscopic fundus image to be detected according to the prediction probability value;

[0182] The preprocessing submodule 3011 preprocesses the ophthalmoscopic fundus image set to be detected, and obtains the target tensor corresponding to the ophthalmoscopic fundus image set to be detected in the following manner:

[0183] All the ophthalmoscopic fundus images to be tested in the ophthalmoscopic fundus image set to be tested are stitched together into a target tensor of N*H*W on the channels, where N is the number of all ophthalmoscopic fundus images to be tested, H is the height of any ophthalmoscopic fundus image to be tested, and W is the width of any ophthalmoscopic fundus image to be tested; or

[0184] Subtract any two adjacent ophthalmoscopic fundus images in the ophthalmoscopic fundus image set to obtain N-1 intermediate images; concatenate all the intermediate images into a (N-1)*H*W target tensor on the channel, where N is the number of all ophthalmoscopic fundus images to be tested, H is the height of any ophthalmoscopic fundus image to be tested, and W is the width of any ophthalmoscopic fundus image to be tested.

[0185] It can be seen that the device described in the embodiment of the present invention can provide various forms of images or tensors to input the eye movement judgment neural network to determine the eye movement state, which can improve the versatility, applicability and flexibility of the present solution, and is conducive to improving the accuracy of tracking eye movements, effectively improving the efficiency and accuracy of eye tracking, thereby calculating accurate eye movement offset information, which is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0186] In another optional embodiment, Figure 4 As shown, the specific manner in which the first judgment module 302 judges whether there is an eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected that is an eye movement start state is:

[0187] For any frame of ophthalmoscopic fundus image to be detected, the preset eye movement judgment neural network determines whether the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to a preset eye movement start threshold;

[0188] When it is determined that the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to a preset eye movement onset threshold, the preset eye movement judgment neural network determines that the eye movement predicted state corresponding to the ophthalmoscopic fundus image to be detected is an eye movement onset state; wherein, the network structure of the preset eye movement judgment neural network includes several convolutional layers, normalization layers, activation layers, attention layers and fully connected layers.

[0189] It can be seen that the device described in the embodiment of the present invention can infer the accurate eye movement onset state through the quantified predicted probability value, and accurately determine which frame of the image is at the specific eye movement onset moment, which is conducive to improving the accuracy of tracking eye movements, effectively improving the efficiency and accuracy of eye tracking, and thus calculating accurate eye movement offset information, which is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0190] In another optional embodiment, Figure 4 As shown, the specific manner in which the second judgment module 304 judges whether there is an eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected that is an eye movement end state is:

[0191] The preset eye movement judgment neural network tracks M prediction probability values ​​corresponding to M consecutive frames of ophthalmoscopic fundus images to be detected, and judges whether the M prediction probability values ​​are all less than a preset eye movement end threshold;

[0192] When it is determined that the M predicted probability values ​​are all less than the preset eye movement end threshold, the eye movement predicted state corresponding to the last frame of the ophthalmoscopic fundus image to be detected in the M frames of ophthalmoscopic fundus images to be detected is determined as the eye movement end state.

[0193] It can be seen that the device described in the embodiment of the present invention can calculate the accurate eye movement end state through the quantified predicted probability value, and accurately determine which frame of the image at the specific eye movement end moment is, which is conducive to improving the accuracy of tracking eye movements, effectively improving the efficiency and accuracy of eye tracking, and thus calculating accurate eye movement offset information, which is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0194] In another optional embodiment, Figure 4 As shown, the video disc positioning module 303 may include:

[0195] The segmentation submodule 3031 is configured to perform an optic disc segmentation operation on the first target image using a preset optic disc positioning neural network to obtain an optic disc segmentation mask corresponding to the first target image;

[0196] The compensation submodule 3032 is configured to perform an information compensation operation on the optic disc segmentation mask to obtain optic disc positioning information corresponding to the first target image;

[0197] The compensation submodule 3032 performs information compensation on the optic disc segmentation mask to obtain the optic disc location information corresponding to the first target image in the following manner:

[0198] Obtaining T scale information corresponding to the first target image; wherein any scale information is used to represent features of different dimensions extracted by the preset optic disc positioning neural network and matching the preset optic disc information in the first target image;

[0199] The high-level features and low-level features in the T scale information are fused, and the information lost during the multi-dimensional feature extraction operation is compensated through skip connections to obtain the optic disc positioning information corresponding to the first target image.

[0200] It can be seen that the device described in the embodiment of the present invention can perform optic disc segmentation and positioning on the first target image through the preset optic disc positioning neural network, and obtain accurate positioning information of the optic disc segmentation mask. At the same time, combined with the Resnet18Unet network model, the learning ability of the model is improved, the speed and effect of positioning and segmentation are improved, and the efficiency and accuracy of eye tracking are further effectively improved, which is beneficial to the acquisition accuracy and generation quality of subsequent OCT images.

[0201] In another optional embodiment, Figure 4As shown, the offset prediction module 305 performs an offset prediction operation based on the first target image and the second target image to obtain the offset information corresponding to the video disc positioning information in the following manner:

[0202] Determining the first target image as the reference image and the second target image as the matching image, performing an offset prediction operation using a preset registration method to complete a matching operation between all regional image blocks in the matching image and a matching template in the reference image, and determining offset information corresponding to the optic disc positioning information; and / or,

[0203] Obtaining offset information corresponding to the optic disc positioning information according to the first target image, the second target image, and a preset offset prediction neural network;

[0204] The offset prediction operation is performed according to the first target image, the second target image, and a preset offset prediction neural network to obtain offset information corresponding to the video disc positioning information, including:

[0205] Inputting the first target image and the second target image into a preset offset prediction neural network to obtain offset information corresponding to the optic disc positioning information, the offset information including offsets in the x-direction and the y-direction of the two-dimensional plane, wherein the preset offset prediction neural network is a convolutional neural network model or a Transformer model; or

[0206] Performing an expansion operation on the optic disc segmentation mask to obtain an expanded mask corresponding to the optic disc segmentation mask, wherein the expansion operation is used to perform background denoising and / or pixel enhancement on the optic disc segmentation mask;

[0207] The first target image, the second target image and the expanded mask are input into a preset offset prediction neural network to obtain offset information corresponding to the optic disc positioning information. The offset information includes the offset in the x-direction and the y-direction of the two-dimensional plane. The preset offset prediction neural network is a convolutional neural network model or a Transformer model.

[0208] It can be seen that the device described in the embodiment of the present invention is able to construct a preset offset prediction neural network with diverse input forms to achieve accurate prediction of offset information. It also provides a template matching method to calculate the offset information, which is conducive to the subsequent combination and correction of the calculation results of the two, further effectively improving the efficiency and accuracy of eye tracking, and is beneficial to the subsequent OCT image acquisition accuracy and generation quality.

[0209] In another optional embodiment, Figure 4 As shown, the device may also include:

[0210] a correction module 306 configured to correct the offset information corresponding to the optic disc positioning information based on the prediction results corresponding to the preset registration method and the preset offset prediction neural network method when the offset prediction operation includes the preset registration method and the preset offset prediction neural network method, thereby obtaining corrected offset information corresponding to the optic disc positioning information;

[0211] The correction module 306 corrects the offset information corresponding to the video disc positioning information to obtain the corrected offset information corresponding to the video disc positioning information in the following manner:

[0212] When the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is less than or equal to the first error threshold, the average value of the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information;

[0213] When the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is greater than the first error threshold and less than the second error threshold, the prediction result corresponding to the preset registration method is determined as the corrected offset information;

[0214] When the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is greater than or equal to the second error threshold, the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information.

[0215] It can be seen that the device described in the embodiment of the present invention can combine the advantages of deep neural networks and registration algorithms to automatically perform eye movement tracking judgment and offset information correction on time series images of ophthalmoscope fundus images, improve the tracking effect and tracking accuracy, and further improve the accuracy of the obtained eye movement offset information.

[0216] Example 4

[0217] See also Figure 5 , Figure 5 : is a schematic diagram of the structure of an automatic eye tracking device disclosed in an embodiment of the present invention. Figure 5 The device described can be an independent device or integrated into an automatic eye tracking processing system, which is not limited in the embodiment of the present invention. Figure 5 As shown, the automatic eye tracking device may include:

[0218] A memory 401 storing executable program code;

[0219] a processor 402 coupled to the memory 401;

[0220] The processor 402 calls the executable program code stored in the memory 401 to execute part or all of the steps in the automatic eye tracking method disclosed in the first or second embodiment of the present invention.

[0221] Example 5

[0222] An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps of the automatic eye tracking method disclosed in the first or second embodiment of the present invention.

[0223] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Those skilled in the art can understand and implement the present invention without inventive effort.

[0224] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the above technical solution, in essence, or the portion that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0225] It should be noted that the computer program code required for the operation of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C language, Visual Basic, Fortran2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run entirely on a computer (PC, embedded intelligent device, etc.), or as an independent software package on a user's computer, or partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0226] Finally, it should be noted that the automatic eye tracking method, device, equipment and computer storage medium disclosed in the embodiments of the present invention are only preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An automatic eye movement tracking method, characterized in that: The method comprises: Inputting the ophthalmoscopic fundus image set to be detected into a preset eye movement judgment neural network, and sequentially obtaining the eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected; wherein the ophthalmoscopic fundus image set to be detected includes a plurality of frames of ophthalmoscopic fundus images to be detected acquired in a time sequence; Determining whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement start state; When the judgment result is yes, determining the ophthalmoscopic fundus image to be detected as a first target image, and inputting the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image; Determine whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement end state; When the judgment result is yes, the ophthalmoscopic fundus image to be detected is determined as the second target image, and an offset prediction operation is performed according to the first target image and the second target image to obtain offset information corresponding to the optic disc positioning information.

2. The automatic eye tracking method according to claim 1, wherein: The step of inputting the ophthalmoscopic fundus image set to be detected into a preset eye movement judgment neural network and sequentially obtaining the eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected comprises: Preprocessing the ophthalmoscopic fundus image set to be detected to obtain a target tensor corresponding to the ophthalmoscopic fundus image set to be detected; Inputting the target tensor into a preset eye movement judgment neural network to obtain a predicted probability value corresponding to each ophthalmoscopic fundus image to be detected; determining an eye movement prediction state of the ophthalmoscopic fundus image to be detected according to the prediction probability value; The preprocessing of the ophthalmoscopic fundus image set to be detected to obtain a target tensor corresponding to the ophthalmoscopic fundus image set to be detected includes: splicing all the ophthalmoscopic fundus images to be detected in the ophthalmoscopic fundus image set to be detected into an N*H*W target tensor on the channel, wherein N is the number of all the ophthalmoscopic fundus images to be detected, H is the height of any ophthalmoscopic fundus image to be detected, and W is the width of any ophthalmoscopic fundus image to be detected; or Subtracting any two adjacent ophthalmoscopic fundus images to be detected in the set of ophthalmoscopic fundus images to be detected to obtain N-1 intermediate images; and concatenating all the intermediate images into a (N-1)*H*W target tensor on the channel, where N is the number of all the ophthalmoscopic fundus images to be detected, H is the height of any ophthalmoscopic fundus image to be detected, and W is the width of any ophthalmoscopic fundus image to be detected.

3. The automatic eye tracking method according to claim 2, characterized in that: The determining whether the eye movement prediction state corresponding to a fundus ophthalmoscopy image to be detected is an eye movement start state includes: For any frame of the ophthalmoscopic fundus image to be detected, the preset eye movement judgment neural network judges whether the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to a preset eye movement onset threshold; When it is determined that the predicted probability value corresponding to the ophthalmoscopic fundus image to be detected is greater than or equal to the preset eye movement onset threshold, the preset eye movement judgment neural network determines that the eye movement predicted state corresponding to the ophthalmoscopic fundus image to be detected is the eye movement onset state; wherein, the network structure of the preset eye movement judgment neural network includes several convolutional layers, normalization layers, activation layers, attention layers and fully connected layers.

4. The automatic eye tracking method according to claim 3, wherein: The determining whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement end state includes: The preset eye movement judgment neural network tracks M prediction probability values ​​corresponding to M consecutive frames of the ophthalmoscopic fundus image to be detected, and determines whether the M prediction probability values ​​are all less than a preset eye movement end threshold; When it is determined that the M predicted probability values ​​are all less than the preset eye movement end threshold, the eye movement predicted state corresponding to the last frame of the ophthalmoscopic fundus image to be detected in the M frames of the ophthalmoscopic fundus images to be detected is determined as the eye movement end state.

5. The automatic eye tracking method according to any one of claims 1 to 4, characterized in that: Inputting the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image includes: The preset optic disc positioning neural network performs an optic disc segmentation operation on the first target image to obtain an optic disc segmentation mask corresponding to the first target image; obtaining optic disc location information corresponding to the first target image according to the optic disc segmentation mask; After obtaining the optic disc location information corresponding to the first target image according to the optic disc segmentation mask, the method further includes: performing an information compensation operation on the optic disc segmentation mask to obtain optimized optic disc positioning information corresponding to the optic disc positioning information; The step of performing an information compensation operation on the optic disc segmentation mask to obtain optimized optic disc positioning information corresponding to the optic disc positioning information includes: Obtaining T scale information corresponding to the first target image; wherein any of the scale information is used to represent features of different dimensions extracted by the preset optic disc positioning neural network and matching the preset optic disc information in the first target image; High-level features and low-level features in the T scale information are fused, and information lost in an operation of extracting features of different dimensions that match preset optic disc information in the first target image is compensated through skip connections to obtain optimized optic disc positioning information corresponding to the optic disc positioning information.

6. The automatic eye tracking method according to claim 5, characterized in that: The performing of the offset prediction operation according to the first target image and the second target image to obtain the offset information corresponding to the video disc positioning information includes: Preset registration method: determining the first target image as a reference image and the second target image as a matching image, performing an offset prediction operation using the preset registration method to complete a matching operation between all regional image blocks in the matching image and a matching template in the reference image, and determining offset information corresponding to the optic disc positioning information; and / or, Preset offset prediction neural network method: performing an offset prediction operation based on the first target image, the second target image and a preset offset prediction neural network to obtain offset information corresponding to the video disc positioning information; The performing of an offset prediction operation based on the first target image, the second target image, and a preset offset prediction neural network to obtain offset information corresponding to the video disc positioning information includes: Inputting the first target image and the second target image into a preset offset prediction neural network to obtain offset information corresponding to the optic disc positioning information, wherein the offset information includes offsets in the x-direction and the y-direction of the two-dimensional plane, and the preset offset prediction neural network is a convolutional neural network model or a Transformer model; or performing an expansion operation on the optic disc segmentation mask to obtain an expanded mask corresponding to the optic disc segmentation mask, wherein the expansion operation is used to perform background denoising and / or pixel enhancement on the optic disc segmentation mask; The first target image, the second target image and the expanded mask are input into the preset offset prediction neural network to obtain the offset information corresponding to the optic disc positioning information, wherein the offset information includes the offset in the x-direction and the y-direction of the two-dimensional plane, and the preset offset prediction neural network is a convolutional neural network model or a Transformer model.

7. The automatic eye tracking method according to claim 6, characterized in that: The method further comprises: When the offset prediction operation includes the preset registration method and the preset offset prediction neural network method, the offset information corresponding to the optic disc positioning information is corrected according to the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method to obtain corrected offset information corresponding to the optic disc positioning information; The step of correcting the offset information corresponding to the video disc positioning information to obtain the corrected offset information corresponding to the video disc positioning information includes: When a result error between a prediction result corresponding to the preset registration method and a prediction result corresponding to the preset offset prediction neural network method is less than or equal to a first error threshold, an average value of the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information; When the result error between the prediction result corresponding to the preset registration method and the prediction result corresponding to the preset offset prediction neural network method is greater than the first error threshold and less than the second error threshold, the prediction result corresponding to the preset registration method is determined as the corrected offset information; When the result error between the prediction result corresponding to the preset alignment method and the prediction result corresponding to the preset offset prediction neural network method is greater than or equal to the second error threshold, the prediction result corresponding to the preset offset prediction neural network method is determined as the corrected offset information.

8. An automatic eye tracking device, characterized in that: The device comprises: An eye movement prediction module is configured to input a set of ophthalmoscopic fundus images to be detected into a preset eye movement judgment neural network, and sequentially obtain an eye movement prediction state corresponding to each ophthalmoscopic fundus image to be detected; wherein the set of ophthalmoscopic fundus images to be detected includes a plurality of frames of ophthalmoscopic fundus images to be detected acquired in a time sequence; A first judgment module is used to judge whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement start state; an optic disc positioning module, configured to, when the determination result of the first determination module is yes, determine the ophthalmoscopic fundus image to be detected as a first target image, and input the first target image into a preset optic disc positioning neural network to obtain optic disc positioning information corresponding to the first target image; The second judgment module is used to judge whether there is an eye movement prediction state corresponding to a certain ophthalmoscopic fundus image to be detected that is an eye movement end state; an offset prediction module for determining the ophthalmoscopic fundus image to be detected as a second target image when the judgment result of the second judgment module is yes, and performing an offset prediction operation based on the first target image and the second target image to obtain offset information corresponding to the optic disc positioning information.

9. An automatic eye tracking device, characterized in that: The device comprises: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the automatic eye tracking method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium stores computer instructions, which, when called, are used to execute the automatic eye tracking method according to any one of claims 1 to 7.

Images