System, method and medium for detecting ophthalmic fissure image

Abstract

The invention relates to the technical field of ophthalmic disease diagnosis, and discloses a system for ophthalmic slit image detection, which comprises a video acquisition module, a slit lamp microscope, a high-definition camera and an image processing module, and is characterized in that a slit lamp microscope is integrated with the high-definition camera to acquire examination videos of eyes of a patient; the key frame extraction module is used for carrying out key frame extraction on the examination video of the eyes of the patient; the semantic segmentation module is used for dividing each key frame image into different sub-regions through a semantic segmentation model; the global graph data construction module is used for constructing global graph data according to the sub-regions of the key frame image; the diagnosis result output module is used for inputting the global graph data into the trained diagnosis model, and an output value represents a diagnosis result of the eyes of the patient; according to the method, the inspection video of the eyes of the patient is constructed into global graph data representation through key frame extraction and image semantic segmentation, and aggregation analysis is performed on different sub-regions of each key frame through the diagnosis model, so that semantic understanding of a plurality of eye crack images is improved, and the precision of a diagnosis result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of ophthalmic disease diagnosis, and more specifically, to a system, method and medium for ophthalmic fissure image detection. Background Art

[0002] The slit lamp microscope is an indispensable tool in the diagnosis of eye diseases. It uses a high-intensity light source to form a parallel light beam through a set of lenses, and then forms a slit light band through an adjustable aperture to project it into the patient's eyes. The length, width, direction and brightness of the slit light band can be adjusted. Therefore, the doctor can observe the front segment of the patient's eye, including the cornea, iris and lens, through the slit lamp microscope, and discover the pathological signs of the patient's eye based on the doctor's professional experience. However, this diagnostic method is highly dependent on the doctor's professional experience, and different doctors may have subjective differences in the diagnostic results.

[0003] With the development of artificial intelligence, more and more intelligent models are applied in the field of medical diagnosis. Currently, CNN (convolutional neural network model) is used to extract features from a single eye fissure image to identify and classify eye diseases. In theory, it can provide consistent diagnostic results. However, there are certain limitations to analyzing only a single eye fissure image, namely, the imaging quality requirements for the image are high, a single eye fissure image may not be able to fully display complete signs of eye pathology, and there is a lack of semantic understanding of multiple eye fissure images, resulting in low accuracy of diagnostic results. Summary of the invention

[0004] The present invention provides a system, method and medium for ophthalmic fissure image detection, which solve the technical problems in the above-mentioned background technology.

[0005] The present invention provides a system for ophthalmic fissure image detection, comprising: A video acquisition module, which is used to integrate a high-definition camera into a slit lamp microscope to acquire a patient's eye examination video; A key frame extraction module is used to extract key frames from the patient's eye examination video to obtain an image data set with M key frame images; Where M is a custom parameter and is a positive integer greater than 1; A semantic segmentation module is used to traverse the M key frame images of the image dataset and divide each key frame image into different sub-regions through a semantic segmentation model; Subregions include: eyelid, conjunctiva, sclera, cornea, anterior chamber, iris, pupil, lens, lens capsule, and anterior vitreous; A global graph data construction module, which is used to construct global graph data according to the sub-regions of the M key frame images; Global graph data includes: nodes and edges between nodes; The nodes of the global graph data establish data connections with the sub-regions of each key frame image; The nodes corresponding to adjacent sub-regions of the same key frame image form edges; The nodes corresponding to the same sub-region of adjacent key frame images form edges; A diagnostic model training module, which is used to build and train a diagnostic model; A diagnosis result output module, which is used to input the global image data into the trained diagnosis model, and the output value represents the diagnosis result of the patient's eye; Diagnoses include: refractive error, conjunctival disease, corneal disease, anterior chamber disease, iris disease, and ocular trauma.

[0006] Furthermore, performing key frame extraction on the patient's eye examination video to obtain an image dataset with M key frame images includes the following steps: Step S201, dividing the patient's eye examination video into N frame images at equal time intervals, where N is a custom parameter greater than M; Step S202, calculating the similarity value of two adjacent sub-frame images among the N sub-frame images by using the structural similarity index, and determining that the similarity value is less than or equal to a similarity threshold, then taking the next sub-frame image as a key frame, wherein the similarity threshold is a custom parameter between 0 and 1; Step S203: when the number of key frame images is greater than M, the similarity threshold is increased; when the number of key frame images is less than M, the similarity threshold is decreased until M key frame images are obtained.

[0007] Furthermore, the semantic segmentation model is TransUNet, and sample labels of training samples used to train the semantic segmentation model are obtained by annotation by ophthalmologists.

[0008] Further, the diagnostic model includes: a first hidden layer, a second hidden layer and a first classifier; The first hidden layer is used to extract the image features of the sub-regions connected to each node of the global graph data, and expand the image features into vector representation; The second hidden layer inputs global graph data and outputs comprehensive features; The comprehensive features are input to the first classifier, and the classification space of the first classifier represents the diagnosis result of the patient's eye.

[0009] Furthermore, the calculation formula of the first hidden layer includes: ; ; ; Where 1≤c≤C, C represents the number of channels of the key frame image, which is assigned a value of 3. represents the image features output by the first hidden layer, , and Represents the original image, channel score and global average pooling score of the cth channel of the sub-region connected by each node. and represent the first weight parameter and the first bias parameter respectively, , , and They respectively represent the first weight parameter, the second weight parameter, the first bias parameter and the second bias parameter of the cth channel of the sub-region to which each node is connected. represents the pixel value of the i-th row and j-th column of the original image of the c-th channel of the sub-region to which each node is connected. H and W represent the number of rows and columns of the original image of the sub-region, respectively. Indicates that the original images of C channels are spliced ​​together. ReLU represents the ReLU activation function.

[0010] Furthermore, the calculation formula of the second hidden layer includes: ; ; Where 1≤m≤M, 1≤u≤U, U represents the number of nodes in each key frame image, Feature represents the comprehensive features of the output of the second hidden layer, represents the set of nodes that have edges connected to the u-th node in the m-th key frame image, and Respectively represent the vector representation of the image features of the u-th and v-th nodes of the m-th key frame image, represents the updated features of the u-th node of the m-th key frame image, , and represent the second weight parameter, the third weight parameter and the fourth weight parameter respectively, It means stacking the updated features of U nodes of M key frame images.

[0011] Furthermore, the first hidden layer is pre-trained before the diagnosis model is trained, and the second classifier is connected during the pre-training of the first hidden layer, and the classification space of the second classifier indicates whether there is anterior chamber bleeding.

[0012] Furthermore, sample labels of training samples used to train the diagnostic model are obtained by annotation by ophthalmologists.

[0013] The present invention provides a method for detecting ophthalmic fissure images, comprising the following steps: Step S301, collecting a patient's eye examination video using a high-definition camera integrated into a slit lamp microscope; Step S302, extracting key frames from the patient's eye examination video to obtain an image data set with M key frame images; Step S303, traversing the M key frame images of the image data set, and dividing each key frame image into different sub-regions through a semantic segmentation model; Step S304, constructing global graph data according to the sub-regions of the M key frame images; Step S305, constructing and training a diagnostic model; Step S306, inputting the global image data into the trained diagnostic model, and the output value represents the diagnosis result of the patient's eye.

[0014] The present invention provides a storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the steps in the above-mentioned method for ophthalmic fissure image detection are executed.

[0015] The beneficial effects of the present invention are as follows: the present invention constructs the patient's eye examination video into a global graph data representation by extracting key frames and performing image semantic segmentation, and performs aggregation analysis on different sub-regions of each key frame through a diagnostic model, thereby improving the semantic understanding of multiple eye fissure images and improving the accuracy of diagnostic results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of a system for ophthalmic slit image detection according to the present invention; Figure 2 It is a flow chart of extracting key frames from a patient's eye examination video according to the present invention; Figure 3 It is a flow chart of a method for ophthalmic fissure image detection of the present invention.

[0017] In the figure: video acquisition module 101, key frame extraction module 102, semantic segmentation module 103, global graph data construction module 104, diagnosis model training module 105, diagnosis result output module 106. DETAILED DESCRIPTION

[0018] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0019] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in one or more embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] like Figure 1 to Figure 3 As shown, a system for ophthalmic fissure image detection comprises: A video acquisition module 101, which is used to integrate a high-definition camera into a slit lamp microscope to acquire a patient's eye examination video; A key frame extraction module 102, which is used to extract key frames from the patient's eye examination video to obtain an image data set with M key frame images; Where M is a custom parameter and is a positive integer greater than 1; A semantic segmentation module 103, which is used to traverse the M key frame images of the image data set and divide each key frame image into different sub-regions through a semantic segmentation model; Subregions include: eyelid, conjunctiva, sclera, cornea, anterior chamber, iris, pupil, lens, lens capsule, and anterior vitreous; A global graph data construction module 104, which is used to construct global graph data according to the sub-regions of the M key frame images; Global graph data includes: nodes and edges between nodes; The nodes of the global graph data establish data connections with the sub-regions of each key frame image; The nodes corresponding to adjacent sub-regions of the same key frame image form edges; The nodes corresponding to the same sub-region of adjacent key frame images form edges; A diagnostic model training module 105, which is used to build and train a diagnostic model; A diagnosis result output module 106, which is used to input the global image data into the trained diagnosis model, and the output value represents the diagnosis result of the patient's eye; Diagnoses include: refractive error, conjunctival disease, corneal disease, anterior chamber disease, iris disease, and ocular trauma.

[0021] It should be noted that the eyelids include the skin, eyelashes, meibomian glands, etc. of the upper and lower eyelids; the conjunctiva is the transparent mucous membrane covering the inside of the eyelids and the surface of the eyeball; the sclera is the white outer tough layer of the eyeball; the cornea is the transparent refractive structure in the front of the eyeball, which is responsible for most of the eye's focusing ability; the anterior chamber is the space filled with aqueous humor between the cornea and the iris; the iris contains the colored circular muscle tissue of the pupil, which is used to control the size of the pupil; the pupil is the opening in the center of the iris, through which light enters the eye; the lens is located behind the iris, responsible for further focusing light to the retina; the lens capsule is the transparent bag that wraps the lens; the anterior vitreous body is the front 1 / 3 of the transparent jelly-like substance in the eyeball.

[0022] It should be noted that refractive errors can be identified by observing the corneal curvature and lens shape. Refractive errors include myopia, hyperopia, astigmatism and presbyopia. Conjunctival diseases may be conjunctivitis, which manifests as redness and swelling of the conjunctiva, increased secretions and other symptoms. Corneal diseases can be diagnosed by observing the transparency, thickness and surface condition of the cornea. Corneal diseases include keratitis, corneal ulcers, corneal scars and corneal degeneration. Anterior chamber diseases can be identified by observing the clarity and cellular composition of the anterior chamber. Anterior chamber diseases include hypopyon and hyphema. Iris diseases can be diagnosed by observing changes in the color, texture and morphology of the iris. Iris diseases include iritis and iris atrophy. The slit lamp microscope can clearly show the location and degree of eye trauma, such as corneal abrasions and foreign body entry.

[0023] In one embodiment of the present invention, the patient's eye diagnosis results may also include: lens disease, eyelid disease, dry eye, glaucoma, etc.

[0024] It should be noted that lens diseases can be cataracts; eyelid diseases can include: meibomian gland dysfunction, blepharitis, inverted eyelashes, ptosis, etc.; xerophthalmia can be diagnosed by observing the moisture level of the cornea; although slit lamp microscopes are mainly used to observe the anterior structure of the eye, during the examination, you may also notice signs that indirectly indicate glaucoma, such as corneal edema and iris abnormalities.

[0025] In one embodiment of the present invention, a high-definition camera is installed at the eyepiece of a slit lamp microscope or on an optical path adapter to collect a patient's eye examination video, and the examination video is transmitted to a key frame extraction module for processing via wireless or wired transmission. It can also be transmitted to a display for doctors to observe. In addition, video files can be stored for doctors to review or re-examine, eliminating the need for patients to repeat multiple examinations, thereby improving patient satisfaction with the medical visit.

[0026] In one embodiment of the present invention, Figure 2 As shown, performing key frame extraction on a patient's eye examination video to obtain an image dataset with M key frame images includes the following steps: Step S201, dividing the patient's eye examination video into N frame images at equal time intervals, where N is a custom parameter greater than M; Step S202, calculating the similarity value of two adjacent sub-frame images among the N sub-frame images by SSIM (Structural Similarity Index), and determining that the similarity value is less than or equal to a similarity threshold, then taking the latter sub-frame image as a key frame, wherein the similarity threshold is a custom parameter between 0 and 1, and preferably, the similarity threshold is set to 0.5; Step S203: when the number of key frame images is greater than M, the similarity threshold is increased; when the number of key frame images is less than M, the similarity threshold is decreased until M key frame images are obtained.

[0027] It should be noted that N is set according to the duration of the patient's eye examination video. Generally, the duration of eye disease diagnosis through a slit lamp microscope is about 3 to 5 minutes. If the duration is 4 minutes, then 120 frame images can be obtained by performing frame processing at intervals of 2 seconds. Preferably, M is set to 20 and N is set to 100.

[0028] In one implementation of the present invention, the sum of the absolute values ​​of the differences between each pixel of two adjacent frame images may also be calculated as the similarity value.

[0029] In one embodiment of the present invention, the semantic segmentation model is TransUNet, or Mask R-CNN or PSPNet, etc., and the sample labels of the training samples used to train the semantic segmentation model are obtained by annotation by ophthalmologists.

[0030] In one implementation of the present invention, the diagnostic model includes: a first hidden layer, a second hidden layer, and a first classifier; The first hidden layer is used to extract the image features of the sub-regions connected to each node of the global graph data, and expand the image features into vector representation; The second hidden layer inputs global graph data and outputs comprehensive features; The comprehensive features are input to the first classifier, and the classification space of the first classifier represents the diagnosis result of the patient's eye.

[0031] In one implementation of the present invention, the calculation formula of the first hidden layer includes: ; ; ; Where 1≤c≤C, C represents the number of channels of the key frame image, which is assigned a value of 3. represents the image features output by the first hidden layer, , and Represents the original image, channel score and global average pooling score of the cth channel of the sub-region connected by each node. and represent the first weight parameter and the first bias parameter respectively, , , and They respectively represent the first weight parameter, the second weight parameter, the first bias parameter and the second bias parameter of the cth channel of the sub-region to which each node is connected. represents the pixel value of the i-th row and j-th column of the original image of the c-th channel of the sub-region to which each node is connected. H and W represent the number of rows and columns of the original image of the sub-region, respectively. Indicates that the original images of C channels are spliced ​​together. ReLU represents the ReLU activation function.

[0032] It should be noted that the number of channels of the key frame image is 3, corresponding to the three channels of RGB. If the key frame image is processed in grayscale, the number of channels of the key frame image is 1.

[0033] In one embodiment of the present invention, the first hidden layer can be constructed based on a convolutional neural network model.

[0034] In one implementation of the present invention, the calculation formula of the second hidden layer includes: ; ; Where 1≤m≤M, 1≤u≤U, U represents the number of nodes in each key frame image, Feature represents the comprehensive features of the output of the second hidden layer, represents the set of nodes that have edges connected to the u-th node in the m-th key frame image, and Respectively represent the vector representation of the image features of the u-th and v-th nodes of the m-th key frame image, represents the updated features of the u-th node of the m-th key frame image, , and represent the second weight parameter, the third weight parameter and the fourth weight parameter respectively, It means stacking the updated features of U nodes of M key frame images.

[0035] In one embodiment of the present invention, the first hidden layer is pre-trained before the diagnostic model is trained, and the second classifier is connected during the pre-training of the first hidden layer, and the classification space of the second classifier indicates whether there is anterior chamber bleeding.

[0036] In one embodiment of the present invention, the first hidden layer is pre-trained before the diagnostic model is trained, and the third classifier is connected during the pre-training of the first hidden layer, and the classification space of the third classifier represents whether the lens is opaque.

[0037] It should be noted that pre-training the first hidden layer can capture important features of the key frame image, which helps to speed up the training of the diagnosis model.

[0038] In one embodiment of the present invention, sample labels of training samples used to train the diagnostic model are obtained by annotation by ophthalmologists.

[0039] It should be noted that the maximum number of iterations of the diagnostic model during training is set, the difference between the value output by the diagnostic model in each iteration and the sample label of the training sample is specified as the loss function, and the weight parameters and bias parameters of the diagnostic model are updated through back propagation and gradient descent algorithms until the maximum number of iterations is reached or the loss value of the diagnostic model is within the preset range within multiple consecutive iterations, which means that the diagnostic model has reached a convergence state, and the training is terminated.

[0040] In one embodiment of the present invention, Figure 3 As shown, the present invention provides a method for ophthalmic fissure image detection, comprising the following steps: Step S301, collecting a patient's eye examination video using a high-definition camera integrated into a slit lamp microscope; Step S302, extracting key frames from the patient's eye examination video to obtain an image data set with M key frame images; Step S303, traversing the M key frame images of the image data set, and dividing each key frame image into different sub-regions through a semantic segmentation model; Step S304, constructing global graph data according to the sub-regions of the M key frame images; Step S305, constructing and training a diagnostic model; Step S306, inputting the global image data into the trained diagnostic model, and the output value represents the diagnosis result of the patient's eye.

[0041] In one embodiment of the present invention, the present invention provides a storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, a method for ophthalmic slit image detection as described above can be performed.

[0042] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are within the protection of the present embodiment.

Claims

1. A system for ophthalmic fissure image detection, characterized in that: include: A video acquisition module, which is used to integrate a high-definition camera into a slit lamp microscope to acquire a patient's eye examination video; A key frame extraction module is used to extract key frames from the patient's eye examination video to obtain an image data set with M key frame images; Where M is a custom parameter and is a positive integer greater than 1; A semantic segmentation module is used to traverse the M key frame images of the image dataset and divide each key frame image into different sub-regions through a semantic segmentation model; Subregions include: eyelid, conjunctiva, sclera, cornea, anterior chamber, iris, pupil, lens, lens capsule, and anterior vitreous; A global graph data construction module, which is used to construct global graph data according to the sub-regions of the M key frame images; Global graph data includes: nodes and edges between nodes; The nodes of the global graph data establish data connections with the sub-regions of each key frame image; The nodes corresponding to adjacent sub-regions of the same key frame image form edges; The nodes corresponding to the same sub-region of adjacent key frame images form edges; A diagnostic model training module, which is used to build and train a diagnostic model; A diagnosis result output module, which is used to input the global image data into the trained diagnosis model, and the output value represents the diagnosis result of the patient's eye; Diagnoses include: refractive error, conjunctival disease, corneal disease, anterior chamber disease, iris disease, and ocular trauma.

2. A system for ophthalmic fissure image detection according to claim 1, characterized in that: Extracting key frames from a patient's eye examination video to obtain an image dataset with M key frame images includes the following steps: Step S201, dividing the patient's eye examination video into N frame images at equal time intervals, where N is a custom parameter greater than M; Step S202, calculating the similarity value of two adjacent sub-frame images among the N sub-frame images by using the structural similarity index, and determining that the similarity value is less than or equal to a similarity threshold, then taking the next sub-frame image as a key frame, wherein the similarity threshold is a custom parameter between 0 and 1; Step S203: when the number of key frame images is greater than M, the similarity threshold is increased; when the number of key frame images is less than M, the similarity threshold is decreased until M key frame images are obtained.

3. A system for ophthalmic fissure image detection according to claim 1, characterized in that: The semantic segmentation model is TransUNet, and the sample labels of the training samples used to train the semantic segmentation model are obtained by annotation by ophthalmologists.

4. A system for ophthalmic fissure image detection according to claim 1, characterized in that: The diagnostic model includes: a first hidden layer, a second hidden layer and a first classifier; The first hidden layer is used to extract the image features of the sub-regions connected to each node of the global graph data, and expand the image features into vector representation; The second hidden layer inputs global graph data and outputs comprehensive features; The comprehensive features are input to the first classifier, and the classification space of the first classifier represents the diagnosis result of the patient's eye.

5. A system for ophthalmic fissure image detection according to claim 4, characterized in that: The calculation formula for the first hidden layer includes: ； ； ； Where 1≤c≤C, C represents the number of channels of the key frame image, which is assigned a value of 3. represents the image features output by the first hidden layer, , and Represents the original image, channel score and global average pooling score of the cth channel of the sub-region connected by each node. and represent the first weight parameter and the first bias parameter respectively, , , and They respectively represent the first weight parameter, the second weight parameter, the first bias parameter and the second bias parameter of the cth channel of the sub-region to which each node is connected. represents the pixel value of the i-th row and j-th column of the original image of the c-th channel of the sub-region to which each node is connected. H and W represent the number of rows and columns of the original image of the sub-region, respectively. Indicates that the original images of C channels are spliced ​​together. ReLU represents the ReLU activation function.

6. A system for ophthalmic fissure image detection according to claim 4, characterized in that: The calculation formula for the second hidden layer includes: ； ； Where 1≤m≤M, 1≤u≤U, U represents the number of nodes in each key frame image, Feature represents the comprehensive features of the output of the second hidden layer, represents the set of nodes that have edges connected to the u-th node in the m-th key frame image, and Respectively represent the vector representation of the image features of the u-th and v-th nodes of the m-th key frame image, represents the updated features of the u-th node of the m-th key frame image, , and represent the second weight parameter, the third weight parameter and the fourth weight parameter respectively, It means stacking the updated features of U nodes of M key frame images.

7. A system for ophthalmic fissure image detection according to claim 4, characterized in that: The first hidden layer is pre-trained before the diagnosis model is trained. During the pre-training of the first hidden layer, the second classifier is connected. The classification space of the second classifier indicates whether there is anterior chamber bleeding.

8. The system for ophthalmic fissure image detection according to claim 1, characterized in that: The sample labels of the training samples used to train the diagnosis model are obtained by annotation by ophthalmologists.

9. A storage medium, characterized in that: The device stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, a method for ophthalmic fissure image detection is performed, comprising the following steps: Step S301, collecting a patient's eye examination video using a high-definition camera integrated into a slit lamp microscope; Step S302, extracting key frames from the patient's eye examination video to obtain an image data set with M key frame images; Step S303, traversing the M key frame images of the image data set, and dividing each key frame image into different sub-regions through a semantic segmentation model; Step S304, constructing global graph data according to the sub-regions of the M key frame images; Step S305, constructing and training a diagnostic model; Step S306, inputting the global image data into the trained diagnostic model, and the output value represents the diagnosis result of the patient's eye.

Images