A multi-eye disease integrated screening and diagnosis method and system based on multi-angle multi-spectrum multi-luminance eccentric photography

Abstract

The present application belongs to the technical field of ophthalmic disease screening and diagnosis, and provides a multi-eye disease integrated screening and diagnosis method and system based on multi-angle multi-spectrum multi-luminance eccentric photography, which comprises: multi-acquisition condition image sequence acquisition, individual information acquisition, multi-eye disease recognition AI model analysis; the present application uses a multi-spectrum eccentric light source to irradiate the patient's eyeball in turn at different angles, wavelengths or luminances, acquires the patient's eye images under the irradiation of each angle, wavelength or luminance light source, and can record the response characteristics of the pupil, ocular surface and related structures under different acquisition conditions; through the joint analysis of the collected image sequence and personal information by the pre-trained AI model, the image information and response difference characteristics under different acquisition conditions are fused, the recognition and prediction of multiple eye diseases are realized, the eye tissue information expression capability is improved, the screening disease range is expanded, and the dependence on professional operation experience, the cost of equipment and the diagnosis time are reduced.

Description

Technical Field

[0001] This invention relates to the field of ophthalmic disease screening and diagnosis technology, and in particular to an integrated screening and diagnosis method and system for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness off-center photography. Background Technology

[0002] Cataracts, refractive errors, glaucoma, and dry eye syndrome are common eye diseases among middle-aged and elderly people, characterized by high incidence, insidious progression, and atypical early symptoms. According to publicly available statistics, the prevalence of cataracts in people over 60 years old exceeds 80%, the blindness rate from glaucoma is approximately 30%, and the incidence of fundus complications related to high refractive errors continues to rise. Currently, traditional screening relies on ophthalmologists using specialized equipment configured for different diseases: cataracts are diagnosed by observing lens opacity under different illumination angles with a slit lamp and combining this with grading standards; refractive errors rely on computer-assisted refraction measurements; and dry eye syndrome requires ocular surface analyzers to assess indicators such as tear meniscus and tear film breakup time. In primary healthcare institutions or large-scale screening scenarios, this model generally suffers from the following problems: 1) There are many types of equipment, which are costly. Each device can only support the detection of a single eye disease, resulting in low system integration and high equipment investment costs; 2) The examination time is long. Traditional methods for the three common examinations of visual acuity, refractive error, cataracts, and dry eye require more than 45 minutes. Patients need to queue between multiple devices, and doctors need to repeatedly adjust the device parameters; 3) The accuracy of the results is highly dependent on the doctor's experience. For example, the diagnosis of cataracts requires doctors to accurately control the incident angle of the slit lamp and subjectively judge the degree of turbidity; 4) There is a shortage of professional ophthalmologists, and their distribution is uneven, so people at the grassroots level cannot receive timely and effective screening for eye diseases.

[0003] Patent CN110327013A proposes an AI-based screening method for multiple eye diseases based on fundus images. This method extracts sub-images through region segmentation and uses a multi-neural network model to detect cataracts, glaucoma, etc. Its core problems are: 1) It relies on the professional operation of ophthalmologists, requiring alignment, focusing, and refractive compensation, which is difficult for ordinary physicians to handle; 2) It requires high patient cooperation, needing to remain stable under strong light, which is not suitable for elderly patients; 3) The examination efficiency is low, with dilated fundus photography taking one hour and non-dilated fundus photography taking several minutes. Patent CN116563932A proposes a combined refractive and cataract screening method based on infrared off-center photography. This method acquires images using a single wavelength, fixed brightness, and limited angle conditions, and predicts refractive error and cataracts through a multi-task learning model. This method still suffers from inherent limitations due to its singular imaging conditions. Existing literature shows that when screening for cataracts under single wavelength and brightness conditions, the sensitivity is 81.58% and the specificity is 79.87%, with a screening rate of only 53.57% for capsular cataracts and a false positive rate of 20.13%. Furthermore, this method can only identify refractive errors and cataracts, making it difficult to extend to other types of eye diseases such as glaucoma, dry eye, and eye strain. The problems mainly stem from the relatively singular design of imaging conditions, specifically: 1) Insufficient acquisition of lesion information. A single infrared band cannot effectively acquire lesion information from some tissue layers, resulting in insufficient ability to identify specific types of lesions, such as the low screening rate for capsular cataracts. 2) Severe feature superposition and confusion. Images acquired under single physical conditions mix scattering and reflection features from different tissue sites, lacking response difference information caused by changes in conditions, making it difficult to distinguish lesions in different locations. For example, it is difficult to distinguish the characteristics of cataract opacity from those of corneal or vitreous opacity, resulting in a high false positive rate. 3) Limited spatial coverage. Infrared off-center imaging equipment has a fixed overall light source position, mainly illuminating the central region of the eyeball. The image only characterizes a small area of ​​structure in the center of the lens. Early lesions in the periphery of the lens or other non-central areas are difficult to cover adequately, leading to a higher false negative rate. 4) Single feature type and lack of dynamic response information: Because the features acquired by traditional infrared off-center imaging equipment characterize the pupil brightness distribution, it can be used to identify diseases that affect pupil brightness distribution, such as brightness gradient changes caused by refractive errors and scotomas caused by cataracts. However, for eye diseases that depend on dynamic changes, such as glaucoma and dry eye, it is difficult to effectively acquire and characterize relevant dynamic features under single imaging conditions. For example, the nerve damage response caused by glaucoma is abnormal pupil dynamic changes in the light reaction test, and the tear film breakage caused by dry eye is abnormal. The common root cause of the above problems is that the existing technology only acquires images under single or limited incident angle, spectral wavelength and brightness intensity conditions, without forming a multi-dimensional physical acquisition parameter combination design, resulting in the lack of cross-condition response difference information in the acquired images. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide an integrated screening and diagnosis method and system for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness eccentric photography, which solves the problems of existing technologies having a variety of equipment, high cost, long time for a single eye health examination, and high dependence of the accuracy of results on the experience of ophthalmologists.

[0005] To achieve the above objectives, the present invention provides the following solution: A multi-eye disease integrated screening and diagnostic method based on multi-angle, multi-spectral, and multi-brightness eccentric photography includes: According to a preset order or rule, the target patient's eyeball is irradiated with incident light of different wavelengths, incident angles, and brightness in sequence, and eye images are acquired to obtain a multi-acquisition condition image sequence; the acquisition conditions include: acquisition parameters corresponding to different incident angles, wavelengths, and brightness. Collect the individual information of the target patient; The multi-acquisition condition image sequence and the individual information are input into a pre-trained multi-eye disease recognition AI model for joint recognition to obtain screening and diagnosis results; the multi-eye disease recognition AI model includes: at least one feature encoder for extracting and fusing multi-dimensional image features and patient individual information features from the multi-acquisition condition image sequence and the individual information, and at least one multi-eye disease joint predictor.

[0006] Preferably, the screening and diagnostic results are in the form of any one or more combinations of binary form, multi-value form, and continuous numerical form; wherein, the binary form is used to represent the presence status of the target eye disease; the multi-value form is used to represent the severity level of the eye disease; the continuous numerical form is used to represent the degree of the eye disease or the refractive parameter prediction result; the refractive parameter prediction result includes: spherical, cylindrical, and axial values ​​of the refractive error.

[0007] Preferably, the feature fusion method of the feature encoder includes one or more of the following: staged fusion, joint fusion, conditional modulation fusion, and attention-weighted fusion; the feature encoder includes: an image sequence structuring construction unit, a condition-aware feature extraction unit, a condition-intra-temporal feature modeling unit, a condition-inter-condition response law feature extraction unit, a patient individual information feature modeling unit, and a multi-source feature fusion unit connected in sequence. The image sequence structuring unit is used to identify and group the multi-acquisition condition image sequences to obtain multiple intra-condition image sequences and cross-condition image sets.

[0008] Preferably, the processing procedure of the multi-eye disease combined predictor includes: The weights of the features output by the feature encoder for each eye disease are calculated using an attention network to obtain eye disease sensitivity features; the expression for the eye disease sensitivity features is as follows: ;in, The aforementioned eye disease sensitivity characteristics; Represents an attention network; The query vector corresponding to each eye disease; The features output by the feature encoder; The predicted value for each eye disease is calculated using the aforementioned eye disease sensitivity features to obtain the screening and diagnostic results.

[0009] Preferably, the process of constructing the training dataset for the multi-eye disease recognition AI model includes: Recruit target volunteers; the target volunteers include: healthy volunteers and volunteers suffering from one or more target eye diseases; the target eye diseases include: cataracts, glaucoma, eye strain, dry eye syndrome, and refractive errors; Obtain the individual information of the target volunteer; The target volunteers underwent ophthalmological examinations to obtain ophthalmic disease diagnostic information; the ophthalmological examinations included: slit-lamp examination, intraocular pressure measurement, fundus examination, OCT examination, ocular surface examination, and refractive examination; the ophthalmic disease diagnostic information included: the presence and severity of ophthalmic diseases; The eyes of the target volunteer were scanned using a multi-angle, multi-spectral, and multi-brightness off-center light source to obtain a multi-angle, multi-spectral, and multi-brightness image sequence; The training dataset is obtained by integrating the individual information of the target volunteers, the eye disease diagnosis information, and the multi-angle, multispectral, and multi-brightness image sequences.

[0010] Preferably, the expression of the condition-aware feature extraction unit is: ;in, Perceive features based on target conditions; This is a target condition-aware feature mapping function used to extract spatial features from an image; For image frames; This refers to the acquisition conditions corresponding to the current frame.

[0011] Preferably, the processing procedure of the conditional temporal feature modeling unit includes: Extracting the temporal features from the output features of the condition-aware feature extraction unit yields initial temporal features; the initial temporal features include any one or more combinations of time stability features and time fluctuation pattern features; the expression for the time stability feature is: The expression for the time fluctuation pattern characteristics is: ;in, , These respectively represent the time stability characteristic and the time fluctuation pattern characteristic; A time stability modeling function is used to model the overall distribution characteristics of a feature sequence over time. This is a time fluctuation function used to model features that change in direction, magnitude, or speed over time. The condition-aware feature extraction unit outputs features; The acquisition conditions corresponding to the current frame; The initial temporal features are combined to obtain the temporal features within the target conditions; the expression for the temporal features within the target conditions is: ;in, The temporal features within the target conditions; This is the characteristic combination function.

[0012] Preferably, the processing procedure of the conditional response pattern feature extraction unit includes: A preset set of acquisition conditions; the set of acquisition conditions includes multiple acquisition condition pairs; the acquisition condition pair includes: two different combinations of acquisition parameters of incident angle, wavelength and brightness; The response characteristics between each pair of acquisition conditions in the set of acquisition conditions are calculated to obtain the response characteristics between the target acquisition conditions; the expression of the response characteristics between the target acquisition conditions is: ;in, The response characteristics between the target acquisition conditions; This is a feature difference capture function used to calculate the directionality, consistency, and conditional sensitivity of response changes; and These represent the i-th and j-th sets of collection conditions in the collection condition pair, respectively. , The corresponding outputs of the time series feature modeling unit within the conditions are respectively and The conditional temporal characteristics.

[0013] Preferably, the processing procedure of the multi-source feature fusion unit includes: Modulation parameters are generated based on the patient individual information features output by the patient individual information feature modeling unit; the expression of the modulation parameters includes: , ;in, , , , These are the first parameter, the second parameter, the third parameter, and the fourth parameter, respectively. It is a multi-layer neural network; This represents the individual information characteristics of the patient; The features output by the conditional temporal feature modeling unit and the conditional response pattern feature extraction unit are modulated using the modulation parameters to obtain multi-source fusion features; the expression of the multi-source fusion features includes: , ;in, , These are the modulated conditional intra-conditional temporal characteristics and conditional response regularity characteristics, respectively. , These are the conditional time series features output by the conditional time series feature modeling unit and the conditional response pattern features output by the conditional response pattern feature extraction unit, respectively.

[0014] Preferably, a multi-eye disease integrated screening and diagnostic system based on multi-angle, multi-spectral, and multi-brightness off-center photography includes: a multi-spectral off-center light source module, a scanning module, an imaging module, an examination data acquisition module, and a multi-eye disease AI recognition module; the multi-spectral off-center light source module includes: multiple sets of multi-spectral light-emitting units with different off-center distances; The multispectral light-emitting unit is used to emit single or combined wavelength light within a preset wavelength range; the multispectral light-emitting unit can adjust the emission brightness; the scanning module is used to adjust the spatial position or emission direction of the multispectral eccentric light source module according to a preset order or rule; the imaging module is used to acquire images of the patient's eyes under the illumination of the multispectral eccentric light source; the examination data acquisition module is used to coordinate and control the multispectral eccentric light source module, the scanning module, and the imaging module to sequentially irradiate the target patient's eyeball with incident light of different wavelengths, incident angles, and brightness according to a preset order or rule and acquire eye images to obtain a multi-acquisition condition image sequence; the acquisition conditions include: acquisition parameters corresponding to different incident angles, wavelengths, and brightness; the multi-eye disease AI recognition module is used to receive the multi-acquisition condition image sequence and individual patient information, and to jointly model the eye response characteristics under fixed acquisition conditions and the response patterns caused by changes in different acquisition conditions based on the received multi-acquisition condition image sequence to obtain prediction results for multiple eye diseases in the patient.

[0015] The present invention discloses the following technical effects: This invention provides an integrated screening and diagnostic method and system for multiple eye diseases based on multi-angle, multispectral, and multi-brightness eccentric photography. By applying controlled optical stimulation to the same eyeball under different incident angles, wavelengths, and brightness, it solves the problem of insufficient information dimension caused by existing eccentric photography methods that only collect information at a single wavelength and angle. It achieves the active construction of response image sequences of eye tissue under different physical acquisition conditions. By using a deep neural network model to jointly model the stable physiological response under the same physical acquisition condition and the response differences caused by changes in different physical acquisition conditions, it solves the problems of insufficient lesion information, confusion due to superposition of different features, and too small examination range in existing technologies, and achieves the extraction of response features related to different eye diseases. By introducing individual patient information as a modulating factor for conditional response, it solves the problem of poor adaptability of existing technologies to different populations and the risk of misjudgment caused by differences in population distribution, and achieves joint modeling and comprehensive prediction of multiple eye diseases. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an integrated screening and diagnosis process for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness eccentric photography, provided in an embodiment of the present invention. Figure 2 A model architecture diagram provided for embodiments of the present invention; Figure 3 This is a structural diagram of a feature encoder provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The purpose of this invention is to provide an integrated screening and diagnostic method and system for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness off-center photography, which solves the problems of existing technologies such as the large number of equipment required, high cost, long examination time, and high dependence of accuracy on ophthalmologists' experience.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Figure 1 This is a schematic diagram of an integrated screening and diagnosis process for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness off-center photography provided in an embodiment of the present invention. Figure 1 As shown, this invention provides an integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multispectral, and multi-brightness off-center photography, comprising: Step 100: Irradiate the target patient's eyeball with incident light of different wavelengths, incident angles, and brightness according to a preset order or rule, and acquire eye images to obtain a multi-acquisition condition image sequence; the acquisition conditions include: acquisition parameters corresponding to different incident angles, wavelengths, and brightness; Step 200: Collect the individual information of the target patient; Step 300: Input the multi-acquisition condition image sequence and the individual information into a pre-trained multi-eye disease recognition AI model for joint recognition to obtain screening and diagnosis results; the multi-eye disease recognition AI model includes: at least one feature encoder for extracting and fusing multi-dimensional image features and patient individual information features from the multi-acquisition condition image sequence and the individual information, and at least one multi-eye disease joint predictor.

[0022] Preferably, the screening and diagnostic results are in the form of any one or more combinations of binary form, multi-value form, and continuous numerical form; wherein, the binary form is used to represent the presence status of the target eye disease; the multi-value form is used to represent the severity level of the eye disease; the continuous numerical form is used to represent the degree of the eye disease or the refractive parameter prediction result; the refractive parameter prediction result includes: spherical, cylindrical, and axial values ​​of the refractive error.

[0023] Optionally, the feature fusion method of the feature encoder includes one or more of the following: staged fusion, joint fusion, conditional modulation fusion, and attention-weighted fusion; the feature encoder includes: an image sequence structuring construction unit, a condition-aware feature extraction unit, an intra-conditional temporal feature modeling unit, an inter-conditional response law feature extraction unit, a patient individual information feature modeling unit, and a multi-source feature fusion unit connected in sequence. The image sequence structuring unit is used to identify and group the multi-acquisition condition image sequences to obtain multiple intra-condition image sequences and cross-condition image sets.

[0024] Preferably, the processing procedure of the multi-eye disease combined predictor includes: The weights of the features output by the feature encoder for each eye disease are calculated using an attention network to obtain eye disease sensitivity features; the expression for the eye disease sensitivity features is as follows: ;in, The aforementioned eye disease sensitivity characteristics; Represents an attention network; The query vector corresponding to each eye disease; The features output by the feature encoder; The predicted value for each eye disease is calculated using the aforementioned eye disease sensitivity features to obtain the screening and diagnostic results.

[0025] Furthermore, the process of constructing the training dataset for the multi-eye disease recognition AI model includes: Recruit target volunteers; the target volunteers include: healthy volunteers and volunteers suffering from one or more target eye diseases; the target eye diseases include: cataracts, glaucoma, eye strain, dry eye syndrome, and refractive errors; Obtain the individual information of the target volunteer; The target volunteers underwent ophthalmological examinations to obtain ophthalmic disease diagnostic information; the ophthalmological examinations included: slit-lamp examination, intraocular pressure measurement, fundus examination, OCT examination, ocular surface examination, and refractive examination; the ophthalmic disease diagnostic information included: the presence and severity of ophthalmic diseases; The eyes of the target volunteer were scanned using a multi-angle, multi-spectral, and multi-brightness off-center light source to obtain a multi-angle, multi-spectral, and multi-brightness image sequence; The training dataset is obtained by integrating the individual information of the target volunteers, the eye disease diagnosis information, and the multi-angle, multispectral, and multi-brightness image sequences.

[0026] Specifically, the expression for the condition-aware feature extraction unit is: ;in, Perceive features based on target conditions; This is a target condition-aware feature mapping function used to extract spatial features from an image; For image frames; This refers to the acquisition conditions corresponding to the current frame.

[0027] Furthermore, the processing procedure of the conditional temporal feature modeling unit includes: Extracting the temporal features from the output features of the condition-aware feature extraction unit yields initial temporal features; the initial temporal features include any one or more combinations of time stability features and time fluctuation pattern features; the expression for the time stability feature is: The expression for the time fluctuation pattern characteristics is: ;in, , These respectively represent the time stability characteristic and the time fluctuation pattern characteristic; A time stability modeling function is used to model the overall distribution characteristics of a feature sequence over time. This is a time fluctuation function used to model features that change in direction, magnitude, or speed over time. The condition-aware feature extraction unit outputs features; The acquisition conditions corresponding to the current frame; The initial temporal features are combined to obtain the temporal features within the target conditions; the expression for the temporal features within the target conditions is: ;in, The temporal features within the target conditions; This is the characteristic combination function.

[0028] Specifically, the processing procedure of the conditional response pattern feature extraction unit includes: A preset set of acquisition conditions; the set of acquisition conditions includes multiple acquisition condition pairs; the acquisition condition pair includes: two different combinations of acquisition parameters of incident angle, wavelength and brightness; The response characteristics between each pair of acquisition conditions in the set of acquisition conditions are calculated to obtain the response characteristics between the target acquisition conditions; the expression of the response characteristics between the target acquisition conditions is: ;in, The response characteristics between the target acquisition conditions; This is a feature difference capture function used to calculate the directionality, consistency, and conditional sensitivity of response changes; and These represent the i-th and j-th sets of collection conditions in the collection condition pair, respectively. , The corresponding outputs of the time series feature modeling unit within the conditions are respectively and The conditional temporal characteristics.

[0029] Preferably, the processing procedure of the multi-source feature fusion unit includes: Modulation parameters are generated based on the patient individual information features output by the patient individual information feature modeling unit; the expression of the modulation parameters includes: , ;in, , , , These are the first parameter, the second parameter, the third parameter, and the fourth parameter, respectively. It is a multi-layer neural network; This represents the individual information characteristics of the patient; The features output by the conditional temporal feature modeling unit and the conditional response pattern feature extraction unit are modulated using the modulation parameters to obtain multi-source fusion features; the expression of the multi-source fusion features includes: , ;in, , These are the modulated conditional intra-conditional temporal characteristics and conditional response regularity characteristics, respectively. , These are the conditional time series features output by the conditional time series feature modeling unit and the conditional response pattern features output by the conditional response pattern feature extraction unit, respectively.

[0030] Preferably, from a physical mechanism perspective, ocular tissues exhibit a significant conditional dependence on optical stimuli. The propagation depth and scattering characteristics of light of different wavelengths vary in different tissues; different incident angles alter the propagation path and reflection area of ​​light within the eye; and stimuli of different brightness may elicit different physiological responses, such as the amplitude and dynamic changes in pupil constriction. If image acquisition is performed only under a single imaging condition, it is difficult to fully obtain information on the differences in ocular tissue responses under different physical conditions.

[0031] Specifically, when certain parts of the eyeball develop lesions, their optical properties, such as transmittance and refractive index, change, leading to alterations in the features of the eye image under illumination. The sequence features of eye image data under illumination of different wavelengths and brightness levels can characterize the type and severity of various eye diseases. Illumination from different angles can penetrate different tissues of the eyeball, acquiring feature information from different parts of the patient's eye. Based on the collected eye image sequence data and the ophthalmologist's diagnostic results, a deep neural network model automatically learns and extracts the eye disease-related features represented by the image sequence, as well as the mapping relationship between these features and the ophthalmologist's diagnostic results, thus achieving integrated intelligent screening and diagnosis of multiple eye diseases. This embodiment designs an integrated intelligent screening and diagnosis method for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness eccentric photography imaging, including the following steps: Step S1: Acquisition of multi-angle, multi-spectral, and multi-brightness image sequences; Step S2: Collection of individual patient information; Step S3: Joint AI-based identification of multiple eye diseases.

[0032] Specifically, step S1: Acquisition of multi-angle, multi-spectral, and multi-brightness image sequences: During the pre-set examination process, the patient's eyeball is illuminated sequentially with light of different wavelengths, incident angles, and brightness according to a pre-set order or rules. Eye images are then acquired under the corresponding acquisition conditions, forming a multi-acquisition-condition image sequence. Specifically: different incident angles are used to alter the propagation path and spatial area of ​​light within the eyeball to obtain the reflection characteristics of different tissues; different wavelengths are used to obtain the differences in the response of ocular tissues to light of different wavelengths; and different brightness levels are used to obtain the response characteristics of ocular tissues to light of different brightness levels and the trend of pupillary state changes.

[0033] Step S2: Patient Individual Information Collection: Obtain individual patient information, including but not limited to age, gender, occupation, eye habits, and past medical history. Use this information to assist the model in making targeted predictions, improving the model's adaptability to different populations.

[0034] Step S3: Joint AI-based identification of multiple eye diseases: The collected examination data, including multi-angle, multispectral, and multi-brightness image sequences and individual patient information, is input into a pre-trained multi-eye disease recognition AI model. The model outputs predicted eye diseases, including but not limited to refractive errors, cataracts, glaucoma, eye strain, and dry eye syndrome. The information includes the probability and severity of a particular eye disease. This multi-eye disease recognition AI model is trained using massive amounts of examination data and corresponding ophthalmologist diagnoses. It can automatically learn the static and dynamic changes in the eye under different angles, brightness levels, and wavelengths represented by image sequences, as well as the complex mapping relationship between these features and ophthalmologist diagnoses.

[0035] refer to Figure 2 AI models for recognizing multiple eye diseases. For example... Figure 2 As shown, the model comprises three parts: an input layer, a feature encoding unit (i.e., the aforementioned feature encoder), and a multi-eye disease joint prediction unit (i.e., the aforementioned multi-eye disease joint predictor). The feature encoding unit extracts and fuses multi-dimensional image features and patient-specific information features from multi-angle, multispectral, and multi-brightness image sequences and patient-specific information. These multi-dimensional image features include image spatial features, temporal features of the image sequence, and response patterns under changing acquisition conditions. The multi-eye disease joint prediction unit uses the fused features extracted by the feature encoding unit to calculate the probability and severity of each eye disease. The input to the multi-eye disease recognition AI model is a multi-angle, multispectral, and multi-brightness image sequence. },in This represents the set of data collection conditions. Represents time frames under the same acquisition conditions; the acquisition condition set includes combinations of eccentric light with different angles, wavelengths, and brightness; individual patient information. This includes patient age, gender, occupation, eye habits, and past medical history. The model outputs various ophthalmic disease labels, including category labels. ,in This represents the predictive information for the i-th type of eye disease. It can be in binary form, representing whether the i-th type of eye disease exists; It can also be in a multi-value form, representing the classification of eye diseases in terms of multiple severity levels; It can also be in continuous numerical form, representing the degree of eye disease, or the spherical, cylindrical, or axial values ​​of refractive errors.

[0036] Specifically, the feature encoding unit is used to extract and fuse multi-dimensional image features and patient-specific information features from multi-angle, multi-spectral, and multi-brightness image sequences and patient-specific information. These multi-dimensional image features include image spatial features, temporal features of the image sequence, and response patterns when acquisition conditions change. Multi-angle, multi-spectral, and multi-brightness image sequences contain eye images stimulated by light sources with different incident directions, wavelengths, and brightness levels. Therefore, depending on the combination and order of feature extraction and fusion, there are various implementation methods. This embodiment does not limit the fusion method to a single approach. In different embodiments, the feature fusion method may include, but is not limited to, one or more of the following: staged fusion: sequentially performing fusion across time / direction / wavelength / brightness / patient-specific information (the order can be interchanged); joint fusion: stitching features from different acquisition conditions together and then performing overall modeling through a unified network; conditional modulation fusion: using brightness, wavelength, or direction information as conditions to modulate features before stitching and fusion; attention-weighted fusion: dynamically adjusting the weights of different conditional features through an attention mechanism.

[0037] refer to Figure 3 One implementation of the feature coding unit is as follows: Figure 3 As shown, the spatial features of a single frame image are first extracted; then, image features under the same acquisition conditions are fused from the temporal dimension to extract temporal features; next, image features under different acquisition conditions are modeled to extract response pattern features when acquisition conditions change; finally, patient individual information features are fused to obtain the final feature representation. The feature encoding unit includes: an image sequence structuring unit, a condition-aware feature extraction unit, a condition-intra-condition temporal feature modeling unit, a condition-inter-condition response pattern feature extraction unit, a patient individual information feature modeling unit, and a multi-source feature fusion unit.

[0038] Furthermore, the image sequence structuring unit identifies and groups the acquired eye images according to the light source angle, wavelength, and brightness parameters, and constructs multiple intra-condition image sequences and cross-condition image sets.

[0039] Specifically, the Conditional Aware Feature Extraction (CAF) unit is used to perform feature modeling on a single frame of an eye image, with the input being a sequence of patient eye images. This method extracts features from individual image frames and explicitly incorporates acquisition condition information, allowing the features to reflect the impact of different acquisition conditions on imaging characteristics during the extraction stage. That is, for each image frame... Feature extraction : .

[0040] in This is the conditional feature mapping function.

[0041] This mapping function depends on both the image content and the acquisition condition parameters.

[0042] Furthermore, the Conditional Aware Feature Extraction Unit (ICTM) can extract spatial features from images to characterize local and global structural features related to eye diseases, such as the cornea, lens, pupillary region, and retinal reflectance distribution. The In-Condition Temporal Feature Modeling Unit (ICTM) takes a sequence of image features from multiple frames acquired under the same conditions and performs temporal modeling to characterize the physiological response characteristics of ocular tissues over time under fixed physical stimuli. This unit takes the acquisition conditions as input. The feature sequence output by the conditionally aware feature extraction unit after the image is acquired is... ,in This represents the number of time frames under the acquisition conditions. ICTM models the feature sequence to obtain the in-condition temporal features: .

[0043] ICTM models the feature sequences of multiple image frames to extract one or a combination of the following two types of information: 1) Temporal stability characteristics: These characterize the overall state of image features stabilizing over time under continuous stimulation under fixed acquisition conditions. They can be expressed as: .

[0044] in This is a time-stability modeling function used to model the overall distribution characteristics of the feature sequence over time. This feature reflects the steady-state physiological response of ocular tissues under fixed stimulus conditions.

[0045] 2) Temporal fluctuation pattern features: These are used to characterize the changing trends of image features within adjacent time frames or short time windows during continuous stimulation. They can be represented as: .

[0046] in This feature is used to model the direction, magnitude, or rate of change of a feature over time. It reflects the dynamic physiological characteristics of ocular tissue during stimulation.

[0047] The final conditional temporal feature representation can be expressed as: .

[0048] in The feature combination function can be implemented by concatenation, weighted summation or other mapping methods.

[0049] Through the above modeling method, ICTM can simultaneously characterize the steady-state response characteristics and dynamic change characteristics of ocular tissues under fixed acquisition conditions, providing a more robust input representation for subsequent inter-condition response law modeling.

[0050] Specifically, the Conditional Response Pattern Extraction Unit (CRP) is used to extract the response patterns of feature changes under different physical stimuli such as different wavelengths, angles, and brightness. That is, if the acquisition conditions change from... → How do the features change? The set is compared and collected according to pre-defined conditions. Calculate the response characteristics between each pair of acquisition conditions: .

[0051] CRP captures the differences between features, including the directionality, consistency, and conditional sensitivity of response changes.

[0052] This response pattern reflects the changes in the functional state of ocular tissues under different physical stimuli, and is an important basis for distinguishing different eye diseases.

[0053] Optionally, a patient individual information feature modeling unit. The model receives patient individual information data, and transforms the discrete features of the patient individual information into continuous vectors through encoding, normalization, and mapping, thereby obtaining the patient individual information feature representation. This information is used to assist in the identification of eye diseases. Patient-specific information includes at least age, gender, occupation, eye habits, and past medical history. This patient-specific information is not directly used to generate image features, but rather to modulate how the model interprets conditional response features, enabling the model to distinguish between normal response differences and pathological abnormal responses caused by age, physiological state, or long-term eye habits.

[0054] Furthermore, the multi-source feature fusion unit integrates conditional temporal features Response characteristics between data collection conditions Patient individual information characteristics By fusing the data, a global representation can be obtained. : .

[0055] in This is the feature fusion mapping function.

[0056] Furthermore, the multi-eye disease joint prediction unit performs joint prediction of multiple eye diseases based on global feature Z, and outputs the predicted value for each eye disease: .

[0057] in This represents the predictive information for the i-th type of eye disease.

[0058] For each eye disease, one or more prediction branches can be set for prediction. Each prediction branch can output classification or regression results, representing the risk prediction, severity level, degree of severity, or refractive parameter prediction results of the eye disease. The implementation of the multi-eye disease joint prediction unit includes: each branch can use a multilayer perceptron (MLP) for prediction, or use an attention network to calculate the weights of different features for each eye disease, that is, generate an eye disease sensitive feature for each prediction branch. And then based on Prediction is performed using a multilayer perceptron (MLP).

[0059] Specifically, dataset construction. To enable the model to predict various eye diseases, a certain amount of examination data and ophthalmologist diagnoses need to be collected for model training and evaluation. The data collection process includes: Volunteer Recruitment: Volunteers include those with one or more eye diseases and healthy volunteers. Eye diseases include, but are not limited to, cataracts, glaucoma, eye strain, dry eye syndrome, and refractive errors.

[0060] Obtain information such as the age, gender, occupation, and medical history of volunteers.

[0061] Volunteers underwent ophthalmic examinations, including slit-lamp examination, intraocular pressure measurement, fundus examination, OCT examination, ocular surface examination, and refractive examination. The results were compiled to obtain ophthalmic disease diagnostic information, including but not limited to refractive errors, cataracts, glaucoma, and dry eye syndrome. The information included the presence and severity of any ophthalmic disease.

[0062] By scanning volunteers' eyes with a multi-angle, multi-spectral, and multi-brightness off-center light source, a multi-angle, multi-spectral, and multi-brightness image sequence of the volunteers was obtained.

[0063] Building a training dataset ,in For collecting multi-angle, multispectral, and multi-brightness image sequences from volunteers. Basic information about volunteers, including age, gender, occupation, medical history, etc. and This involves professional ophthalmologists assigning predictive labels to volunteers to indicate the type and severity of their eye diseases. Numerical labels for predicting eye diseases in volunteers, including the severity of the eye disease and refractive error parameters (spherical, cylindrical, and axial, etc.).

[0064] Furthermore, the model predicts loss through multiple eye diseases. Supervised training can be conducted, and training objectives can include minimizing classification loss for predicting the presence or severity (grading) of eye diseases, and minimizing regression loss for predicting continuous parameters, depending on the type of eye disease, sample distribution, or clinical importance.

[0065] Specifically, in this embodiment, the image sequence structuring unit first constructs a conditional structure for the acquired eye images. This includes: generating a corresponding conditional identifier for each frame of the image based on the light source angle, wavelength, and brightness parameters; dividing the image into multiple conditional subsequences according to the conditional identifiers; and each conditional subsequence containing multiple consecutively acquired frames of images under the same light source configuration. In this way, the model can clearly distinguish the data sources under different physical imaging conditions during subsequent processing, avoiding interference introduced by mixing images with different conditions for modeling.

[0066] Furthermore, one implementation of the Conditional Aware Feature Extraction Unit (CAF) is to use conditional modulation convolution for image frames. The processing is as follows: Image features are extracted using a multi-layer convolutional neural network that shares weights under different acquisition conditions. ; Condition-aware modulation is applied to the image features acquired under each acquisition condition c: .

[0067] in Indicates a data collection condition Parameterized modulation functions, such as using a multilayer perceptron (MLP) or a feature dictionary lookup table, for each acquisition condition. Generate a set of affine parameters : .

[0068] Conditional sensing modulation function This allows the model to learn to adjust its feature extraction strategy based on the acquisition conditions early in the feature extraction process. Convolutional feature extraction uses a convolutional neural network with shared weights, avoiding model training problems caused by too many model parameters and ensuring consistency among extracted features.

[0069] Specifically, in one embodiment, the conditional temporal feature modeling unit includes a temporal stability modeling subunit and a temporal fluctuation pattern modeling subunit, which respectively perform temporal modeling with different emphases on the image feature sequence under the same acquisition conditions.

[0070] 1) Time stability modeling sub-unit: Input acquisition conditions for time stability modeling subunit The feature sequence output by the conditionally aware feature extraction unit after the image is acquired is... , This represents the number of time frames under the acquisition conditions, used to model the overall distribution characteristics of the feature sequence in the time dimension.

[0071] In one specific implementation, the feature sequence can first be time-aggregated to obtain a stability representation: .

[0072] Furthermore, time stability characteristics are obtained through nonlinear mapping: .

[0073] in It can be implemented using multilayer perceptrons, attention-weighted mapping, or a combination thereof.

[0074] This modeling approach is used to characterize the overall response state of ocular tissues after they have stabilized under fixed stimulus conditions.

[0075] In another specific implementation, the time stability modeling subunit can be implemented using sequence modeling network structures such as Long Short-Term Memory (LSTM) and Transformer networks. 2) Time fluctuation modeling sub-unit: Input sequence of time stability modeling subunit It is used to characterize the changing trend of features over time.

[0076] In one specific implementation, the feature sequence can first be subjected to multi-scale differencing. For example, a set of differencing scales {1, 2, 4, 8} can be defined, and the differencing sequence can be calculated for each scale s. .

[0077] Then, for each scale difference sequence, a set of LSTM / Transformer networks is used to extract the fluctuation features at scale s. ; The fluctuation characteristics of each scale s The final time fluctuation pattern features are obtained by splicing. .

[0078] This feature is used to characterize the directionality, speed, and amplitude of changes in the response of ocular tissues during continuous stimulation.

[0079] 3) Conditional temporal feature fusion, in one embodiment, The conditional temporal features can be obtained by splicing, summing after nonlinear mapping of multilayer perceptrons. Through the above modeling method, ICTM can simultaneously characterize the steady-state response characteristics and dynamic change characteristics of ocular tissues under fixed acquisition conditions, providing a more robust input representation for subsequent inter-conditional response modeling.

[0080] Furthermore, in one embodiment, the conditional response pattern feature extraction unit includes: 1) Based on pre-set comparison and collection conditions, the dataset is... Calculate the similarity of features between each pair of acquisition conditions.

[0081] 2) Calculate response difference features by combining comprehensive features and the similarity between features: .

[0082] in This represents various similarity measurement algorithms, including cosine similarity. Structural similarity wait; To respond to the pattern difference fusion function, a multilayer perceptron (MLP) can be used, where: .

[0083] Preferably, one implementation of the patient individual information feature modeling unit is to embed the patient individual information features I (age, gender, occupation, eye habits, and past medical history, etc.) using a multilayer perceptron (MLP) to obtain continuous features. .

[0084] Furthermore, one implementation of the multi-source feature fusion unit involves modulating the conditional temporal features and conditional response pattern features using individual patient information features, and then concatenating them. In a specific implementation, the individual patient information features can be... Input a multilayer neural network (MLP) to generate modulation parameters, i.e.: .

[0085] .

[0086] .

[0087] .

[0088] in, , These represent the intra-conditional temporal characteristics and conditional response regularity characteristics after modulation of individual patient information, respectively. , This represents the modulated parameters for calculating individual patient information.

[0089] Specifically, one implementation of the multi-eye disease joint prediction unit is as follows: First, an attention network is used to calculate the weights of different features for each eye disease, that is, to generate an eye disease-sensitive feature for each prediction branch. : .

[0090] in The query vector corresponding to each eye disease.

[0091] Use of eye disease sensitivity characteristics Calculate the predicted value for each eye disease.

[0092] Alternatively, one approach to model training is to minimize the model's performance on the training dataset. Losses : .

[0093] in This is used to measure the difference between the model's output class distribution and the diagnostic result class labels. Multi-label prediction losses can be used, such as binary cross-entropy loss or multi-label softmax loss. This is used to measure the difference between continuous values ​​output by the model and labels of diagnostic results. Calculations such as mean squared error and mean absolute error can be used. The weight of the loss.

[0094] Preferably, this embodiment also provides an integrated intelligent screening and diagnostic system for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness eccentric photography. This system is used to actively construct multi-acquisition condition image sequences of ocular tissues under controlled physical stimulation conditions, and to achieve joint intelligent identification of multiple eye diseases based on the response sequences. The core of the system includes: Multispectral off-center light source module: Contains multiple sets of multispectral light-emitting units with different off-center distances. Each light-emitting unit can emit single or combined wavelengths of light within a preset wavelength range and supports precise adjustment of the emitted brightness. Different off-center distances are used to change the incident position of light relative to the center of the pupil, thereby changing the propagation path of light within the eyeball; different wavelengths and brightness are used to construct the response differences of ocular tissues under different physical stimuli.

[0095] The scanning module is used to adjust the spatial position or emission direction of the multispectral eccentric light source module according to a preset sequence or rule during the examination, so that different eccentric light rays sequentially illuminate the patient's eyeball at different incident angles. By changing the incident angle of the light, the scanning module is used to selectively excite different spatial regions and different tissue structures of the eyeball, thereby acquiring image information reflecting the response characteristics of different tissues.

[0096] Imaging module: The imaging module is used to acquire images of the patient's eyes under multispectral eccentric light source illumination, and to synchronize or conditionally label them with the current light source wavelength, brightness and incident angle parameters.

[0097] The examination data acquisition module coordinates and controls the multispectral off-center light source module, scanning module, and imaging module to sequentially irradiate and image the patient's eyeball according to a preset multi-angle, multispectral, and multi-brightness acquisition strategy, constructing a multi-acquisition condition image sequence with clearly defined acquisition condition identifiers. Simultaneously, the examination data acquisition module collects individual patient information, including age, gender, occupation, eye habits, and past medical history, for individualized modeling in subsequent eye disease identification processes.

[0098] Multi-eye disease AI recognition module: This module receives multi-angle, multi-spectral, and multi-brightness image sequences and individual patient information. It jointly models the ocular response characteristics under fixed acquisition conditions and the response patterns caused by changes in different acquisition conditions, and outputs prediction results for various eye diseases in the patient. The prediction results include, but are not limited to, the probability and severity of the presence of eye diseases such as refractive errors, cataracts, glaucoma, and dry eye syndrome.

[0099] Specifically, the multispectral eccentric light source module includes multiple sets of eccentric multispectral LED light sources. These multispectral LEDs can have their emitted light wavelength and brightness adjusted via programming under the control of a circuit. In one implementation, multiple sets of multispectral LEDs with different eccentric distances are arranged around the imaging module. Each multispectral LED contains multiple monochromatic light emission channels; different wavelengths of light are emitted by selecting different channels, and the brightness of the light is varied by adjusting the driving current.

[0100] Furthermore, the scanning module is used to adjust the angle at which the light emitted from the multispectral eccentric light source module is incident on the eyeball, enabling the light to illuminate different areas of the eyeball, thereby achieving scanning imaging of different parts of the patient's eyeball. In one implementation, the scanning module includes a mechanical adjustment mechanism for controlling the position and orientation of the multispectral eccentric light source module relative to the eyeball, thereby changing the angle at which the light is incident on the eyeball. Through the above methods, eye illumination under multi-angle conditions is achieved.

[0101] Specifically, the imaging module can use traditional CMOS or CCD imaging technology to acquire eye image sequences under multispectral eccentric light source illumination.

[0102] Furthermore, the examination data acquisition module collects multi-angle, multi-spectral, and multi-brightness image sequences and individual patient information. The data acquisition module controls the scanning module and the multi-spectral off-center light source module to sequentially illuminate the patient's eyeball at different angles, wavelengths, and brightness levels. The imaging module simultaneously acquires eye images under each light source configuration, resulting in a multi-angle, multi-spectral, and multi-brightness eye image sequence. The collected individual patient information includes age, gender, occupation, and medical history. One implementation of the module involves pre-setting multiple incident angles, wavelengths, and brightness configurations. Based on different configuration combinations, the scanning module and the multi-spectral off-center light source module are controlled to illuminate the eyeball at specified angles, wavelengths, and brightness levels, acquiring eye images under each configuration. For example, incident angles include 0°, 15°, and 30°; wavelengths include 450nm blue light, 550nm green light, 620nm red light, and 850nm infrared light; and brightness levels include 10 lux, 100 lux, and 200 lux. The module collects individual patient information including age, gender, occupation, and medical history. It supports retrieving individual patient information from the network based on patient number, QR code, etc., and also supports manual input by device users.

[0103] Preferably, compared to eccentric photography methods based solely on a single wavelength or single acquisition condition, this embodiment improves the accuracy of multi-eye disease screening by acquiring and jointly modeling images under multiple angles, spectra, and brightness conditions. This is due to reasons including, but not limited to: different tissues of the eye exhibit different transmission and reflection characteristics to different wavelengths of light; a single wavelength can only represent limited lesion information, while comprehensive multi-wavelength images can acquire richer eye disease-related information; under different wavelengths and angles, the response differences of different parts of the eyeball vary, helping to distinguish lesion features from different tissue structures and reducing the risk of misidentification caused by the aliasing of different lesion features; by changing the incident angle of the light source, the coverage of different areas of the eyeball can be expanded, which is beneficial for acquiring lesion information in areas such as the lens periphery and ocular surface, improving the ability to identify early or localized lesions. For example, regarding cataract screening and diagnosis, to verify the performance of the algorithm described in this embodiment, the inventors used the method of this invention to acquire multi-acquisition condition image sequences of the eyes of patients with different types and degrees of cataracts and the eyes of healthy volunteers at multiple ophthalmology hospitals and optometry centers, and obtained the corresponding cataract diagnosis results, wherein the cataract diagnosis results were given by ophthalmologists through slit-lamp examination. During data partitioning, the "eye" was used as the smallest independent unit to ensure that all image sequences of the same eye appeared in only one of the training or test sets, avoiding bias caused by data overlap. A test set was randomly selected from 1556 eyes with multiple acquisition conditions and their labels, including 420 eyes with cataracts and 1136 eyes from healthy volunteers. The remaining eye data was used for algorithm training. The experimental results of the algorithm are shown in Table 1.

[0104] Table 1

[0105] Table 1 shows the test results of this embodiment in the cataract screening task. It can be seen that this embodiment achieved high sensitivity, specificity, and AUC in nuclear cataract, cortical cataract, posterior capsule cataract, and cataract presence / absence identification tasks, indicating that the method based on multi-angle, multi-spectral, and multi-luminance acquisition and joint modeling has good screening performance. Furthermore, compared with the cataract screening results reported in related literature on the application of binocular vision screening instruments in screening for congenital cataracts, the sensitivity for cataract screening (presence / absence of cataract) was 81.58%, the specificity was 79.87%, and the sensitivity for capsule cataract was 53.57%. This embodiment showed higher detection indicators in cataract presence / absence identification and posterior capsule cataract identification. Specifically, the sensitivity for posterior capsule cataract increased from 53.57% to 92.80%, an increase of 39.23 percentage points. It should be noted that, given the potential differences in subject population, imaging equipment, acquisition conditions, and evaluation procedures among different studies, the above comparative results are mainly used to illustrate the superiority and reference value of this embodiment.

[0106] Furthermore, Table 2 shows that this embodiment expands the number of identifiable eye diseases from the existing two (refractive errors and cataracts) to at least five (refractive errors, cataracts, glaucoma, dry eye syndrome, and eye strain), and the method can also be extended to identify other eye diseases. This embodiment effectively solves the problem of limited identifiable eye diseases in existing technologies by using multi-angle, multi-spectral, and multi-brightness acquisition methods, providing a technical foundation for the early screening of various eye diseases.

[0107] Table 2

[0108] The beneficial effects of this invention are as follows: (1) This invention automatically completes the acquisition of eye images through a multi-angle, multi-spectral, and multi-brightness eccentric photography module, and analyzes and predicts them using a pre-trained multi-eye disease AI recognition model. This achieves standardization of the examination process and simplification of the operation steps, significantly reducing the reliance on professional ophthalmologists. Non-professionals can independently complete the initial screening of eye diseases, effectively solving the problem of the shortage of ophthalmologists in primary medical institutions.

[0109] (2) The eye image sequences acquired under different incident angles, wavelengths and brightness conditions can characterize the characteristics of various ophthalmic diseases, including different wavelengths to enhance the response characteristics of tissues such as the lens, cornea and ocular surface, different angles to expand the coverage of the eyeball, and different brightness to characterize the dynamic state characteristics of the pupil and ocular surface. By using a multi-eye disease AI recognition model to jointly model the above-mentioned multi-acquisition condition image sequences and individual patient information, the simultaneous screening of multiple eye diseases such as refractive errors, cataracts, glaucoma, dry eye syndrome and eye fatigue is realized.

[0110] (3) The screening system of the present invention can obtain the diagnostic results of multiple eye diseases with only one shot, and the entire examination process only takes a few seconds, which greatly improves efficiency compared with traditional methods.

[0111] (4) This invention significantly improves the accuracy and stability of screening by acquiring and jointly modeling images under multiple angles, multiple spectra, and multiple brightness conditions. The sensitivity of cataract screening is increased from 81.58% to 97.5% in the prior art, the specificity is increased from 79.87% to 99.3%, and the recall rate of capsular cataract screening is increased from 53.57% to 92.8%. The advantages are: (i) multispectral coverage of the differences in transmittance of different tissues, obtaining richer lesion information; (ii) multi-angle differentiation of the response characteristics of different tissue structures, reducing feature aliasing; (iii) multi-brightness capture of dynamic physiological response, improving the ability to identify early lesions.

[0112] (5) This invention expands the scope of eye disease identification. By acquiring dynamic response features through multi-condition acquisition, it can effectively characterize optic nerve dysfunction, such as abnormal pupillary response in glaucoma, abnormal ocular surface condition, and abnormal tear film breakage in dry eye. This expands the types of eye diseases that can be identified from refractive errors and cataracts in the prior art to refractive errors, cataracts, glaucoma, dry eye, and eye strain, and supports future expansion to the screening of other eye diseases.

[0113] (6) The multi-eye disease AI recognition model of the present invention adopts a specially designed feature encoder. It extracts static features under a single acquisition condition through a condition-aware feature extraction unit and a condition-in-condition temporal feature modeling unit, such as pupil spot information of cataracts and pupil brightness changes of refractive errors. It also extracts dynamic features under multiple acquisition condition changes through a condition-inter-condition response law feature extraction unit, such as abnormal pupil reaction of glaucoma and abnormal tear film breakage of dry eye. It automatically learns the contribution relationship of various features to the prediction of different eye diseases, and realizes accurate joint prediction of multiple eye diseases.

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0115] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-eye disease integrated screening and diagnostic method based on multi-angle, multi-spectral, and multi-brightness eccentric photography, characterized in that, include: According to a preset order or rule, the target patient's eyeball is irradiated with incident light of different wavelengths, incident angles, and brightness in sequence, and eye images are collected to obtain a multi-acquisition condition image sequence. The acquisition conditions include: acquisition parameters corresponding to different incident angles, wavelengths, and brightness; Collect the individual information of the target patient; The multi-acquisition condition image sequence and the individual information are input into a pre-trained multi-eye disease recognition AI model for joint recognition to obtain screening and diagnosis results; the multi-eye disease recognition AI model includes: at least one feature encoder for extracting and fusing multi-dimensional image features and patient individual information features from the multi-acquisition condition image sequence and the individual information, and at least one multi-eye disease joint predictor.

2. The integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness eccentric photography according to claim 1, characterized in that, The screening and diagnostic results can take the form of any one or more combinations of binary, multi-valued, and continuous numerical forms; wherein, the binary form is used to represent the presence of the target eye disease; the multi-valued form is used to represent the severity level of the eye disease; the continuous numerical form is used to represent the degree of the eye disease or the refractive parameter prediction results; the refractive parameter prediction results include: spherical, cylindrical, and axial values ​​of the refractive error.

3. The integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multispectral, and multi-brightness eccentric photography according to claim 1, characterized in that, The feature fusion method of the feature encoder includes one or more of the following: staged fusion, joint fusion, conditional modulation fusion, and attention-weighted fusion; the feature encoder includes: an image sequence structuring construction unit, a condition-aware feature extraction unit, an intra-conditional temporal feature modeling unit, an inter-conditional response law feature extraction unit, a patient individual information feature modeling unit, and a multi-source feature fusion unit connected in sequence. The image sequence structuring unit is used to identify and group the multi-acquisition condition image sequences to obtain multiple intra-condition image sequences and cross-condition image sets.

4. The integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness eccentric photography according to claim 1, characterized in that, The processing procedure of the multi-eye disease combined predictor includes: The weights of the features output by the feature encoder for each eye disease are calculated using an attention network to obtain eye disease sensitivity features; the expression for the eye disease sensitivity features is as follows: ;in, The aforementioned eye disease is characterized by sensitivity. Represents an attention network; The query vector corresponding to each eye disease; The features output by the feature encoder; The predicted value for each eye disease is calculated using the aforementioned eye disease sensitivity features to obtain the screening and diagnostic results.

5. The integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multi-spectral, and multi-brightness eccentric photography according to claim 1, characterized in that, The process of constructing the training dataset for the multi-eye disease recognition AI model includes: Recruit target volunteers; the target volunteers include: healthy volunteers and volunteers suffering from one or more target eye diseases; the target eye diseases include: cataracts, glaucoma, eye strain, dry eye syndrome, and refractive errors; Obtain the individual information of the target volunteer; The target volunteers underwent ophthalmological examinations to obtain ophthalmic disease diagnostic information; the ophthalmological examinations included: slit-lamp examination, intraocular pressure measurement, fundus examination, OCT examination, ocular surface examination, and refractive examination; the ophthalmic disease diagnostic information included: the presence and severity of ophthalmic diseases; The eyes of the target volunteer were scanned using a multi-angle, multi-spectral, and multi-brightness off-center light source to obtain a multi-angle, multi-spectral, and multi-brightness image sequence; The training dataset is obtained by integrating the individual information of the target volunteers, the eye disease diagnosis information, and the multi-angle, multispectral, and multi-brightness image sequences.

6. The integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multispectral, and multi-brightness eccentric photography according to claim 3, characterized in that, The expression for the condition-aware feature extraction unit is: ;in, Perceive features based on target conditions; This is a target condition-aware feature mapping function used to extract spatial features from an image; For image frames; This refers to the acquisition conditions corresponding to the current frame.

7. The integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multispectral, and multi-brightness eccentric photography according to claim 3, characterized in that, The processing procedure of the conditional temporal feature modeling unit includes: Extracting the temporal features from the output features of the condition-aware feature extraction unit yields initial temporal features; the initial temporal features include any one or more combinations of time stability features and time fluctuation pattern features; the expression for the time stability feature is: The expression for the time fluctuation pattern characteristics is: ;in, , These respectively represent the time stability characteristic and the time fluctuation pattern characteristic; A time stability modeling function is used to model the overall distribution characteristics of a feature sequence over time. This is a time fluctuation function used to model features that change in direction, magnitude, or speed over time. The condition-aware feature extraction unit outputs features; The acquisition conditions corresponding to the current frame; The initial temporal features are combined to obtain the temporal features within the target conditions; the expression for the temporal features within the target conditions is: ;in, The temporal features within the target conditions; This is the characteristic combination function.

8. The integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multispectral, and multi-brightness eccentric photography according to claim 3, characterized in that, The processing procedure of the conditional response pattern feature extraction unit includes: A preset set of acquisition conditions; the set of acquisition conditions includes multiple acquisition condition pairs; the acquisition condition pair includes: two different combinations of acquisition parameters of incident angle, wavelength and brightness; The response characteristics between each pair of acquisition conditions in the set of acquisition conditions are calculated to obtain the response characteristics between the target acquisition conditions; the expression of the response characteristics between the target acquisition conditions is: ;in, The response characteristics between the target acquisition conditions; This is a feature difference capture function used to calculate the directionality, consistency, and conditional sensitivity of response changes; and These represent the i-th and j-th sets of collection conditions in the collection condition pair, respectively. , The corresponding outputs of the time series feature modeling unit within the conditions are respectively and The conditional temporal characteristics.

9. The integrated screening and diagnostic method for multiple eye diseases based on multi-angle, multispectral, and multi-brightness eccentric photography according to claim 3, characterized in that, The processing procedure of the multi-source feature fusion unit includes: Modulation parameters are generated based on the patient individual information features output by the patient individual information feature modeling unit; the expression of the modulation parameters includes: , ;in, , , , These are the first parameter, the second parameter, the third parameter, and the fourth parameter, respectively. It is a multi-layer neural network; This represents the individual information characteristics of the patient; The features output by the conditional temporal feature modeling unit and the conditional response pattern feature extraction unit are modulated using the modulation parameters to obtain multi-source fusion features; the expression of the multi-source fusion features includes: , ;in, , These are the modulated conditional intra-conditional temporal characteristics and conditional response regularity characteristics, respectively. , These are the conditional time series features output by the conditional time series feature modeling unit and the conditional response pattern features output by the conditional response pattern feature extraction unit, respectively.

10. A multi-eye disease integrated screening and diagnostic system based on multi-angle, multi-spectral, and multi-brightness eccentric photography, characterized in that, To implement the integrated screening and diagnosis method for multiple eye diseases based on multi-angle, multispectral, and multi-brightness off-center photography as described in claim 1, the system includes: a multispectral off-center light source module, a scanning module, an imaging module, an examination data acquisition module, and a multi-eye disease AI recognition module; the multispectral off-center light source module includes: multiple sets of multispectral emitting units with different off-center distances; The multispectral light-emitting unit is used to emit single or combined wavelength light within a preset wavelength range; the multispectral light-emitting unit can adjust the emission brightness; the scanning module is used to adjust the spatial position or emission direction of the multispectral eccentric light source module according to a preset order or rule; the imaging module is used to acquire images of the patient's eyes under the illumination of the multispectral eccentric light source; the examination data acquisition module is used to coordinate and control the multispectral eccentric light source module, the scanning module, and the imaging module to sequentially irradiate the target patient's eyeball with incident light of different wavelengths, incident angles, and brightness according to a preset order or rule and acquire eye images to obtain a multi-acquisition condition image sequence; the acquisition conditions include: acquisition parameters corresponding to different incident angles, wavelengths, and brightness; the multi-eye disease AI recognition module is used to receive the multi-acquisition condition image sequence and individual patient information, and to jointly model the eye response characteristics under fixed acquisition conditions and the response patterns caused by changes in different acquisition conditions based on the received multi-acquisition condition image sequence to obtain prediction results for multiple eye diseases in the patient.

Images