Computing devices for ophthalmic auxiliary diagnosis and treatment

Abstract

This invention generally relates to executing models on a computing system and discloses a computing device for ophthalmological auxiliary diagnosis and treatment. A computing device for ophthalmological auxiliary diagnosis and treatment includes computing resources, a first intelligent agent, a second intelligent agent, and a third intelligent agent. The first intelligent agent is configured to acquire a user's facial image and generate a first eye image including the left and right eyes based on the facial image. The second intelligent agent is configured to determine ophthalmological screening results based on the first eye image. The third intelligent agent is configured to generate ophthalmological diagnosis and treatment recommendations based on the ophthalmological screening results. According to some embodiments, the computing device automates the entire process of ophthalmological screening and ophthalmological diagnosis and treatment opinion output based on user input through multi-agent fusion, thus promoting ophthalmological auxiliary diagnosis and treatment.

Description

Technical Field

[0001] This invention generally relates to executing models on computing systems, and more specifically to agent-based ophthalmic auxiliary diagnosis and treatment technology. Background Technology

[0002] Eye diseases are common problems affecting human health and quality of life. Traditional eye care relies heavily on the knowledge and experience of specialized ophthalmologists. Patients typically need to go to the hospital, go through a series of procedures including registration, waiting, and examinations, before receiving diagnosis and treatment advice. This process is not only time-consuming and labor-intensive, but the uneven distribution of quality medical resources also makes it difficult for patients in many areas to obtain timely and professional eye care advice. Significant efficiency bottlenecks exist, particularly in initial screening and routine health management.

[0003] In recent years, with the development of deep learning technology, using artificial intelligence for medical image analysis has become a research hotspot. Some studies have proposed using models such as convolutional neural networks (CNNs) to detect specific diseases (e.g., diabetic retinopathy, glaucoma) in fundus photographs and OCT images. However, these methods are usually based on independent models for single tasks, making it difficult to form a complete and automated diagnostic recommendation process. They suffer from the following problems: 1) Fragmented process: Existing artificial intelligence models usually only solve a single problem in the diagnosis and treatment process, such as only detecting the eye area or only diagnosing a specific disease. They lack a comprehensive solution that effectively connects multiple links such as image preprocessing, screening for multiple diseases, fusion of medical record information, and generation of personalized suggestions.

[0004] 2) Poor interactivity: Most models work in a "black box" manner, directly obtaining results after the user inputs an image. They cannot be dynamically adjusted according to the user's specific requirements or supplementary information, and it is also difficult to effectively combine the screening results of artificial intelligence with the patient's personal medical history, lifestyle, and other information.

[0005] 3) Limited recommendations: Most existing technologies focus on "diagnosis," that is, determining whether a certain disease exists, but they do not adequately cover the "service" aspects such as subsequent health management, preventive measures, and treatment plan recommendations, and cannot generate systematic and personalized management opinions.

[0006] Therefore, there is an urgent need for an ophthalmology-assisted diagnosis and treatment technology that can simulate the diagnostic thinking of doctors, integrate the capabilities of multiple specialized artificial intelligence models, and achieve full-process automation from image input to personalized health management advice output. Summary of the Invention

[0007] This invention is provided to offer a fully automated ophthalmic auxiliary diagnosis and treatment technology.

[0008] This invention provides a computing device for ophthalmic auxiliary diagnosis and treatment, comprising: computing resources; a first agent configured to invoke the computing resources to: obtain a user's facial image; generate a first eye image including a left eye and a right eye based on the facial image; and a second agent configured to invoke the computing resources to: determine an ophthalmic screening result based on the first eye image, wherein the second agent is trained based on the following steps: providing a third eye image and a fourth eye image associated with different viewpoints of the second eye image, the second eye image being obtained by cropping an image including the eyes from a face image in an open-source face dataset; cropping a portion of the third eye image and a portion of the fourth eye image to generate a set of local images; generating a first patch primitive set based on the third eye image and the fourth eye image; randomly masking the first patch primitive set to form a second patch primitive set; causing a first network to generate a first prediction probability set based on the third eye image and the fourth eye image; and causing a second network to... A second prediction probability set is generated from the local image set; the first network generates a third prediction probability set based on the first patch primitive set; the prediction probabilities at positions corresponding to the masked patch primitive positions in the third prediction probability set are retained to obtain a subset of the third prediction probability set; the second network generates a fourth prediction probability set based on the second patch primitive set; the prediction probabilities at positions corresponding to the masked patch primitive positions in the fourth prediction probability set are retained to obtain a subset of the fourth prediction probability set; a loss function associated with the second network is determined based on the first prediction probability set, the second prediction probability set, the subset of the third prediction probability set, and the subset of the fourth prediction probability set; and the parameters associated with the second network are iteratively updated based on the loss function to train the second network, wherein the trained second network is used as the second agent; and a third agent is configured to invoke the computing resources to: generate ophthalmological treatment recommendations based on the ophthalmological screening results.

[0009] In the computing device described above, the first agent is configured to: detect multiple bounding boxes including eyes from the face image and confidence scores indicating that each bounding box includes a left eye and a right eye; divide the multiple bounding boxes into a set of left eye bounding boxes and a set of right eye bounding boxes based on the center position of the coordinates of each bounding box and the position of the central axis of the face image; select candidate left eye bounding boxes from the set of left eye bounding boxes and candidate right eye bounding boxes from the set of right eye bounding boxes based on the confidence scores and the intersection-union ratio of the left eye bounding boxes and the right eye bounding boxes; and generate the first eye image based on the candidate left eye bounding boxes and the candidate right eye bounding boxes.

[0010] In the computing device described in any of the above, the first agent is configured to iteratively select left-eye bounding boxes from the set of left-eye bounding boxes and right-eye bounding boxes from the set of right-eye bounding boxes in descending order of confidence scores, until the intersection-union ratio of the selected left-eye bounding boxes and right-eye bounding boxes is zero.

[0011] The computing device described in any of the above embodiments, wherein the third eye image and the fourth eye image are obtained by performing an enhancement algorithm on the second eye image, the enhancement algorithm including at least one of the following algorithms: random cropping, horizontal flipping, color perturbation, grayscale conversion, and Gaussian perturbation.

[0012] In the computing device described in any of the preceding claims, the loss function is determined based on a first loss function and a second loss function, the first loss function being determined based on a first set of predicted probabilities and a second set of predicted probabilities, and the second loss function being determined based on a subset of the third set of predicted probabilities and a subset of the fourth set of predicted probabilities.

[0013] The computing device described in any of the above embodiments, wherein the loss function is further determined based on a third loss function, which is determined based on Koleo regularization.

[0014] The computing device described in any of the above embodiments, wherein the second agent is trained using a first training dataset, the first training dataset comprising a fifth eye image, the fifth eye image comprising at least one of the following labels: a label indicating whether or not myopia exists, a label indicating the degree of myopia, a label indicating the refractive power, and a label indicating an eye condition.

[0015] The computing device described in any of the above embodiments, wherein the third agent is trained using a second training dataset, the second training dataset including task instructions, input information, and diagnostic suggestion text.

[0016] The input information of the computing device described in any of the above includes ophthalmological screening results, personal information, auxiliary examination results, and biometric data.

[0017] The computing device described in any of the above embodiments, wherein the input information is obtained from the patient's medical record, wherein missing biometric data in the medical record is determined based on the personal information and the results of the auxiliary examination.

[0018] The computing device as described in any of the above, wherein the biometric data includes the axial length of both eyes.

[0019] The missing binocular axial length in the medical record is determined by the following steps using the computing device described in any of the above: obtaining the baseline monocular axial length associated with the left or right eye for a non-myopic person of a predetermined age group and specified gender; for one eye in the left or right eye, determining a first monocular axial length associated with the said eye based on the baseline axial length associated with the said eye, according to a predetermined relationship between monocular axial length and refractive power; for the other eye in the left or right eye, determining an axial length difference based on the refractive power difference between the refractive power of the other eye and the refractive power of the said eye; for the other eye in the left or right eye, determining a second monocular axial length associated with the other eye based on a predetermined relationship between monocular axial length and refractive power and based on the axial length difference; and determining the binocular axial length based on the first monocular axial length and the second monocular axial length.

[0020] The computing device as described in any of the above, wherein the biometric data includes corneal curvature of both eyes, and the corneal curvature of each eye includes flat curvature and steep curvature.

[0021] The corneal curvature of the missing eyes in the medical record is determined by the following steps using the computing device described in any of the preceding claims: obtaining a baseline flat curvature associated with the left or right eye for a person of a predetermined age and specified sex; for one eye (left or right), determining a steep curvature associated with the said eye based on the baseline flat curvature associated with the said eye, according to a constraint between a flat curvature associated with the said eye and a steep curvature associated with the said eye; for the other eye (left or right), determining a flat curvature associated with the said eye based on the baseline flat curvature associated with the said eye, according to a constraint between a flat curvature associated with the said eye and a flat curvature associated with the said eye; for the other eye (left or right), determining a steep curvature associated with the said eye based on a constraint between a steep curvature associated with the said eye and a steep curvature associated with the said eye, according to a constraint between a steep curvature associated with the said eye and a steep curvature associated with the said eye.

[0022] The computing device as described in any of the above, the third intelligent agent is further configured to: receive user input, and wherein the ophthalmological treatment suggestion is further generated based on the user input.

[0023] The computing device as described in any of the above, wherein the first agent is further configured to: delete the face image and the first eye image after providing the first eye image to the second agent, and the second agent is configured to: delete the first eye image after determining the ophthalmological screening result.

[0024] The computing device as described in any of the above embodiments, wherein the first intelligent agent is further configured to prompt the user before collecting the facial image from the user, and the first intelligent agent is only allowed to obtain the facial image with the user's authorization.

[0025] According to some embodiments, the computing device achieves full automation of the ophthalmic screening and ophthalmic diagnosis and treatment opinion output process based on user input through multi-agent fusion, which promotes ophthalmic auxiliary diagnosis and treatment. Attached Figure Description

[0026] Various embodiments of the present invention are described in conjunction with the accompanying drawings.

[0027] Figure 1 This is a block diagram of a computing device for ophthalmic auxiliary diagnosis and treatment according to an embodiment of the present invention.

[0028] Figure 2 This is a flowchart of the overall framework for training a second intelligent agent according to an embodiment of the present invention.

[0029] Figure 3 This is a flowchart of the first process of training a second intelligent agent according to an embodiment of the present invention.

[0030] Figure 4 This is a flowchart of the second process for training a second intelligent agent according to an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of the operation process of the first intelligent agent according to an embodiment of the present invention.

[0032] Figure 6 This is a flowchart of the process for determining the missing axial length of both eyes in a case according to an embodiment of the present invention.

[0033] Figure 7 This is a flowchart illustrating the process of determining the corneal curvature of the missing eyes in a case according to an embodiment of the present invention. Detailed Implementation

[0034] In this application, the term "patch token" refers to the basic unit after image tokenization. The term "prediction probability" refers to the result of the prediction network generating a logit based on the patch token input and passing it through a softmax layer.

[0035] According to one aspect of the present invention, a computing device for ophthalmic auxiliary diagnosis and treatment is provided.

[0036] Figure 1 This is a block diagram of a computing device 100 for ophthalmic auxiliary diagnosis and treatment according to an embodiment of the present invention.

[0037] In some embodiments, the computing device 100 may include computing resources 110, a first agent 130, a second agent 150, and a third agent 170.

[0038] In some embodiments, computing resources 110 may include various general-purpose or special-purpose computing resources. As an example, computing resources 110 may include a central processing unit (CPU), a graphics processing unit (GPU), a ray tracing core, a tensor core, a vector processing unit, a matrix processing unit, an integer processing unit, a floating-point processing unit, etc.

[0039] The first intelligent agent 130, the second intelligent agent 150, and the third intelligent agent 170 can each call upon computing resources 110 to perform corresponding operations.

[0040] In some embodiments, a first agent 130 may be configured to obtain a user's facial image 120. The first agent 130 may be configured to generate a first eye image 140 including the left and right eyes based on the facial image 120. The generated first eye image 140 may be provided to a second agent 150. The following will be combined with... Figure 5 The specific implementation method of the first intelligent agent 130 generating the first eye image 140 based on the face image 120 is described.

[0041] In some embodiments, the first agent 130 may be configured to prompt the user before collecting the user's facial image 120. As an example, the prompt may include informing the user that the computing device 100 will collect the user's facial image and the method for subsequent processing of the facial image. In some embodiments, the first agent 130 is only allowed to acquire the user's facial image 120 with the user's authorization. As an example, if no user authorization is received, the computing device 100 may prohibit the first agent 130 from accessing the computing device 100's camera (not shown). Thus, the first agent 130 only acquires the user's facial image for subsequent processing after informing the user in advance and obtaining the user's authorization.

[0042] In some embodiments, the first agent 130 may be configured to delete the face image 120 and the first eye image 140 after providing the first eye image 140 to the second agent 150. Thus, the first agent 130 only analyzes and processes the face image 120, and after generating and outputting the analysis results, it no longer saves the face image 120 and the first eye image 140, avoiding the leakage of user privacy data.

[0043] In some embodiments, the second agent 150 may be configured to generate an ophthalmological screening result 160 based on a first eye image 140.

[0044] Eye disease symptoms typically manifest in eye images. In some embodiments, a first training dataset can be used to train a second agent 150. The trained second agent 150 learns the relationship between visual features of eye images and eye disease symptoms, thereby enabling it to generate ophthalmological screening results based on eye images.

[0045] In some embodiments, the first training dataset may include labeled images of the eye. As an example, the labels may include labels indicating whether the eye is myopic. Alternatively or additionally, the labels may include labels indicating the degree of myopia (e.g., low myopia, moderate myopia, high myopia). Alternatively or additionally, the labels may include labels indicating the refractive error. Alternatively or additionally, the labels may include labels indicating eye conditions (e.g., ptosis, strabismus, etc.).

[0046] In some embodiments, the second agent 150 can be obtained by training the student model within a "teacher-student model" joint architecture. The following will combine... Figures 2-4 Describe the specific process of training the second agent 150.

[0047] In some embodiments, the third agent 170 may be configured to generate ophthalmological treatment recommendations 180 based on ophthalmological screening results 160.

[0048] In some embodiments, the third agent 170 may be trained using a second training dataset. In some embodiments, the second training dataset may include task instructions, input information, and diagnostic suggestion text.

[0049] In some embodiments, the input information may include eye screening results, personal information, auxiliary test results, biometric data, etc.

[0050] In some embodiments, the input information may be obtained from the patient's medical records. To protect patient privacy, embodiments of the present invention obtain the patient's medical records with the patient's authorization or from compliant channels, wherein patient privacy information (e.g., the patient's name, identification number, etc.) in the medical records is removed.

[0051] In some embodiments, missing biometric data in the medical record are identified based on personal information and ancillary testing results. The following will combine... Figure 6 Describe the process for determining the missing axial length of both eyes in the case, and combine it with... Figure 7 Describe the process for determining the corneal curvature of the missing eyes in a case.

[0052] In some embodiments, unstructured text in medical records can be transformed into a uniform format using information extraction and structuring techniques to form a second training dataset. As an example, this uniform format could be triples (…). I , X , Y ),in I It is a task instruction. X It is input information. Y This is the corresponding standardized treatment recommendation text.

[0053] As an example, supervised fine-tuning (SFT) of the third agent 170 can be performed by constructing a second training dataset. As another example, maximum likelihood estimation can be used to optimize the parameters associated with the third agent 170. As an example, the loss function associated with training the third agent 170 can be expressed by the following formula (1): (1) Where T is the length of the output text sequence. y t It is the t-th word element. y <t Indicates all preceding word elements, Indicates the training dataset D train Sample triples ( I , X , Y ) Calculate the expected value. This is done by adjusting the loss function. L SFT Minimize, the third agent 170 can follow task instructions I and input information X Generate treatment recommendations.

[0054] In some embodiments, the third agent 170 may also be configured to receive user input. For example, user input may include descriptions related to symptoms of an eye disease, descriptions of a history of eye diseases, etc. In some embodiments, the third agent 170 may be configured to generate an ophthalmological treatment recommendation 180 based on both the ophthalmological screening result 160 and the user input.

[0055] Some embodiments integrate multiple processes, such as eye image preprocessing and recognition, eye screening result generation, and eye treatment suggestion generation, through the fusion of multiple intelligent agents, thereby promoting fully automated ophthalmic assisted diagnosis and treatment.

[0056] Figure 2This is a flowchart of the overall framework for training a second agent according to an embodiment of the present invention. The overall framework may employ a first network 210 and a second network 230. In some embodiments, the first network 210 may be a teacher network in a "teacher-student model" architecture, and the second network 230 may be a student network in a "teacher-student model" architecture. Some embodiments may use a "teacher-student model" architecture to train the student model (i.e., the second network 230) to use the trained second network 230 as a second agent (e.g., ...). Figure 1 The second intelligent agent 150 in the middle.

[0057] In some embodiments, a second eye image 220 may be provided. This second eye image 220 may be a training input image used to train a second network 230. The second eye image 220 may be obtained by cropping an image including the eyes from face images in an open-source face dataset.

[0058] As an example, open-source face datasets can include the ffhq dataset, a high-quality open-source face dataset released by NVIDIA in 2019, which contains 70,000 high-quality face images. The images are licensed under the Flickr license, and users can obtain the dataset download script from the GitHub repository.

[0059] As an example, open-source face datasets can include the CelebA dataset, an open-source face dataset released by the Multimedia Lab at the Chinese University of Hong Kong, which contains 202,599 celebrity face images. Users can download the dataset from its official homepage.

[0060] In some embodiments, eyes can be identified from face images in an open-source face dataset and the associated region can be cropped out using an object detection algorithm to obtain a second eye image 220.

[0061] In some embodiments, a third eye image 222 and a fourth eye image 224 may be provided that are associated with different perspectives of the second eye image 220.

[0062] In some embodiments, portions of the third eye image 222 and the fourth eye image 224 can be cropped to obtain a first set of partial images 242 and a second set of partial images 244, respectively. The first set of partial images 242 and the second set of partial images 244 can constitute a set of partial images.

[0063] In some embodiments, a first patch primitive set 262 may be generated based on the third eye image 222 and the fourth eye image 224.

[0064] In some embodiments, the first patch primitive set 262 can be randomly masked to form the second patch primitive set 264. Figure 2 The slashes within squares represent masked patch primitives. As an example, alternative patch primitives can be used to replace the original patch primitives at the masked locations.

[0065] In some embodiments, the first network 210 may generate a first set of predicted probabilities 282 based on the third eye image 222 and the fourth eye image 224.

[0066] In some embodiments, the second network 230 may generate a second set of predicted probabilities 284 based on a set of local images (including a first set of local images 242 and a second set of local images 244).

[0067] In some embodiments, the first network 210 may generate a third prediction probability set based on the first patch primitive set 262. In some embodiments, the prediction probabilities at positions corresponding to the masked patch primitive positions in the third prediction probability set may be retained to obtain a subset 286 of the third prediction probability set.

[0068] In some embodiments, the second network 230 may generate a fourth prediction probability set based on the second patch primitive set 264. In some embodiments, the prediction probabilities at positions corresponding to the masked patch primitive positions in the fourth probability set may be retained to obtain a subset 288 of the fourth prediction probability set.

[0069] In some embodiments, the loss function associated with the second network 230 may be determined based on a first set of predicted probabilities 282, a second set of predicted probabilities 284, a subset 286 of a third set of predicted probabilities, and a subset 288 of a fourth set of predicted probabilities. L .

[0070] In some embodiments, it can be based on a loss function L The parameters associated with the second network 230 are iteratively updated to train the second network 230. The trained second network 230 can be used as a second agent (e.g., Figure 1 The second intelligent agent 150 in the middle.

[0071] Some implementations incorporate a set of prediction probabilities reflecting the global state of the image, a set of prediction probabilities reflecting the local state of the image, and prediction probabilities at the patch primitive level into the calculation of the loss function. This enables the model to learn the local-global correlation of the image and complete the semantics during the inference process, thereby improving the model's generalization and robustness.

[0072] In some embodiments, the third eye image 222 and the fourth eye image 224 may be obtained by performing an enhancement algorithm on the second eye image 220. In some embodiments, the enhancement algorithm may include random cropping. Additionally or alternatively, the enhancement algorithm may include horizontal flipping. Additionally or alternatively, the enhancement algorithm may include color perturbation. Additionally or alternatively, the enhancement algorithm may include grayscale conversion. Additionally or alternatively, the enhancement algorithm may include Gaussian perturbation.

[0073] In some embodiments, the loss function associated with the second network 230 L Based on the first loss function L 1 and second loss functions L 2 is determined as described by the following formula (2): L = L 1+ L twenty two) The following section will describe the first loss function in conjunction with his three descriptions. L The process of determining 1, and will be combined with Figure 4 Describe the second loss function L The process of determining 2.

[0074] Figure 3 This is a flowchart of the first process of training a second intelligent agent according to an embodiment of the present invention.

[0075] In some embodiments, a third eye image 222 and a fourth eye image 224 associated with different viewpoints may be generated based on the second eye image 220.

[0076] In some embodiments, portions of the third eye image 222 and the fourth eye image 224 may be cropped to obtain a first set of local images 242 and a second set of local images 244 (which together constitute a set of local images).

[0077] In some embodiments, the first network 210 may generate a first set of predicted probabilities 282 based on the third eye image 222 and the fourth eye image 224.

[0078] In some embodiments, the second network 230 may generate a second set of predicted probabilities 284 based on a set of local images.

[0079] In some embodiments, a first loss function associated with the second network 230 may be determined based on a first set of predicted probabilities 282 and a second set of predicted probabilities 284. L 1.

[0080] In some embodiments, the first loss function may be determined based on the cross-entropy loss between the first predicted probability set 282 and the second predicted probability set 284. L1.

[0081] As an example, the first loss function can be determined by the following formula (3). L 1: (3) in, P t It is the first predicted probability set 282 output by the first network 210. P s’ It is the second predicted probability set 284 output by the second network 230.

[0082] Some implementations incorporate both a set of prediction probabilities reflecting the global aspects of the image and a set of prediction probabilities reflecting the local aspects of the image into the calculation of the loss function, enabling the model to learn local-global correlations, thereby improving the model's generalization and robustness.

[0083] Figure 4 This is a flowchart of the second process for training a second intelligent agent according to an embodiment of the present invention.

[0084] and Figure 2 and Figure 3 Similarly, in some embodiments, a third eye image 222 and a fourth eye image 224 may be generated based on the second eye image 220.

[0085] In some embodiments, a first patch primitive set 262 may be generated based on the third eye image 222 and the fourth eye image 224.

[0086] In some embodiments, the first patch primitive set 262 may be randomly masked to form the second patch primitive set 264.

[0087] In some embodiments, the first network 210 may generate a third prediction probability set based on the first patch primitive set 262. In some embodiments, the prediction probabilities at positions corresponding to the masked patch primitive positions in the third prediction probability set may be retained to obtain a subset 286 of the third prediction probability set.

[0088] In some embodiments, the second network 230 may generate a fourth prediction probability set based on the second patch primitive set 264. In some embodiments, the prediction probabilities at positions corresponding to the masked patch primitive positions in the fourth prediction probability set may be retained to obtain a subset 288 of the fourth prediction probability set.

[0089] In some embodiments, the second loss function associated with the second network 230 may be determined based on a subset 286 of the third prediction probability set and a subset 288 of the fourth prediction probability set. L 2.

[0090] In some embodiments, the second loss function may be determined based on the cross-entropy loss between a subset 286 of the third predicted probability set and a subset 288 of the fourth predicted probability set. L 2.

[0091] As an example, the first loss function can be determined by the following formula (4). L 2: (4) in, It is a subset of the third prediction probability set, 286. It is a subset of the fourth prediction probability set, 288.

[0092] Some implementations incorporate feature calculation at the patch primitive level into the loss function calculation, enabling the model to focus on finer details in local image areas, thereby enhancing the model's ability to recognize image details. This is crucial for image recognition-based ophthalmological auxiliary diagnosis and treatment.

[0093] Table 1 below shows the results for the second loss function. L Results of the ablation experiment of 2

[0094] Table 1 Ablation Experiment Results

[0095] In some embodiments, the loss function associated with the second network 230 L It can also be based on a third loss function L 3 is determined. As an example, the loss function... L It can be determined based on the following formula (5).

[0096] L = L 1+ L 2+ aL 3 (5)

[0097] in, a It is a predetermined constant. As an example, a =0.5.

[0098] In some embodiments, the third loss function L 3 can be determined based on Koleo regularization. Koleo regularization encourages a more uniform distribution of features within a batch. As an example, L 3 can be determined by the following formula (6): (6) in, ,in, , It is the feature vector corresponding to a batch of samples.

[0099] Some implementations incorporate a third loss function based on Koleo regularization into the loss function calculation, further enhancing the model's ability to represent subtle textures and structures in the eye.

[0100] Figure 5 This is a schematic diagram of the operation process of the first intelligent agent according to an embodiment of the present invention. Figure 5 The process can be made by Figure 1 The first intelligent agent 130 in the process executes.

[0101] In some embodiments, the first agent may be configured to generate multiple bounding boxes 520 including eyes from a face image 120 and confidence scores indicating that each bounding box includes the left eye and the right eye.

[0102] In some embodiments, the first agent may be configured to divide the plurality of bounding boxes 520 into a set 530 of left-eye bounding boxes and a set 540 of right-eye bounding boxes based on the center position of the coordinates of each bounding box and the central axis position of the face image 120.

[0103] In some embodiments, the first agent may be configured to select candidate left-eye bounding boxes 562 from the set of left-eye bounding boxes 530 and candidate right-eye bounding boxes 564 from the set of right-eye bounding boxes 540 based on confidence scores and the intersection over union (IOU) ratio of the left-eye bounding boxes to the right-eye bounding boxes.

[0104] As an example, the first agent can be configured to select the current left-eye bounding box 532 (i.e., the left-eye bounding box with the highest current confidence score) from the set of left-eye bounding boxes 530 in descending order of confidence score, and select the current right-eye bounding box 542 (i.e., the right-eye bounding box with the highest current confidence score) from the set of right-eye bounding boxes 540. The first agent can be configured to perform a judgment 550: determine whether the IOU between the current left-eye bounding box 532 and the current right-eye bounding box 542 is zero. If so, the current left-eye bounding box 532 can be determined as a candidate left-eye bounding box 562, and the current right-eye bounding box 542 can be determined as a candidate right-eye bounding box 564. If the IOU is not zero, the next left-eye bounding box can be selected from the set of left-eye bounding boxes 530 and / or the next right-eye bounding box can be selected from the set of right-eye bounding boxes 540, and the judgment 550 can continue.

[0105] In some embodiments, the first agent may be configured to generate a first eye image 140 based on a candidate left eye bounding box 562 and a candidate right eye bounding box 564. As an example, the first eye image 140 may be determined as the minimum bounding rectangle of the candidate left eye bounding box 562 and the candidate right eye bounding box 564.

[0106] Figure 6 This is a flowchart of the process for determining the missing axial length of both eyes in a case according to an embodiment of the present invention.

[0107] In some embodiments, the process may include step 610: obtaining the baseline monocular axial length associated with the left or right eye for a non-myopic person of a predetermined age group and a specified gender.

[0108] As an example, refer to the distribution range table of axial length of non-myopic boys and girls aged 3-18. Independent uniformly distributed sampling can be performed within each range to serve as the baseline axial length for the left and right eyes, respectively.

[0109] In some embodiments, the process may include step 620: for one of the left or right eyes, determining a first monocular axial length associated with that eye based on the baseline axial length associated with that eye, according to a predetermined relationship between monocular axial length and refractive power.

[0110] Based on clinical experience, the axial length of the left and right eyes is related to the degree of myopia: if the refractive power of the left and right eyes is not much different, the axial length of the left and right eyes will be similar (for example, within 0.3 mm). For every 100 degrees increase in refractive power, the axial length will increase by about 0.3 mm.

[0111] As an example, the calculation starts with the left eye. The axial length can be increased by the refractive power based on the basic axial length of the left eye determined in step 610, and for every 100 degrees of increase, a Gaussian noise following a distribution of N(0.3, 0.1^2) can be independently superimposed.

[0112] In some embodiments, the process may include step 630: for the other eye in the left or right eye, determining the axial length difference based on the difference in refractive power between the other eye and the first eye.

[0113] As an example, for the right eye, if the difference in refractive power between the right and left eyes is less than a predetermined threshold (e.g., 100 degrees), the axial length difference corresponding to the refractive power difference can first be added to the axial length of the left eye, and then physiological noise following a Uniform (-0.15, 0.15) (mm) distribution can be added. If the difference in refractive power between the right and left eyes is greater than the predetermined threshold, then to maintain the consistency of basic development in both eyes, the basic axial length of the right eye should be close to that of the left eye. As an example, a small random perturbation conforming to a uniform (-0.1, 0.1) distribution can be added to the basic axial length of the left eye to serve as the basic axial length of the right eye.

[0114] In some embodiments, the process may include step 640: For the other eye of the left eye or the right eye, determine a second monocular axial length associated with the other eye according to a pre-determined relationship between the monocular axial length and the refractive power and based on the axial length difference.

[0115] As an example, for the right eye, on the basis of the right eye's basic axial length, for each additional 100 degrees, a Gaussian noise obeying the N(0.3, 0.1^2) distribution is independently superimposed.

[0116] In some embodiments, the process may include step 650: Determine the binocular axial length based on the first monocular axial length and the second monocular axial length.

[0117] Figure 7 It is a flowchart of a process for determining the corneal curvature of missing binoculars in a case according to an embodiment of the present invention. The corneal curvature may include a flat curvature K1 and a steep curvature K2.

[0118] In some embodiments, the process may include step 710: Obtain the basic flat curvature associated with the left eye or the right eye of a person of a predetermined age group and a specified gender.

[0119] As an example, the corneal curvature percentile table by age group from 6 to 15 years old can be referred to, from which the flat curvature K1 and the steep curvature K2 corresponding to the respective ages can be obtained.

[0120] In some embodiments, the process may include step 720: For one eye of the left eye or the right eye, determine the steep curvature associated with the eye based on the basic flat curvature associated with the eye according to the difference constraint between the flat curvature associated with the eye and the steep curvature associated with the eye.

[0121] As an example, for the left eye, taking the flat curvature K1 obtained in step 710 as an example, since the normal range of the difference K1 - K2 between the flat curvature K1 and the steep curvature K2 is 0.25D - 1.75D (generally less than 1D) (D is the refractive power). In addition, the constraint K1 < K2 needs to be satisfied. Therefore, the steep curvature K2 of the left eye can be determined based on the flat curvature K1 of the left eye.

[0122] In some embodiments, the process may include step 730: For the other eye of the left eye or the right eye, determine the flat curvature associated with the other eye based on the basic flat curvature associated with the one eye according to the difference constraint between the flat curvature associated with the one eye and the flat curvature associated with the other eye. [[ID=二十九]]

[0123] As an example, for the left eye, taking the flat curvature K1 obtained at step 710 as an example, since the difference between the flat curvature K1 of the left eye and the flat curvature K1 of the right eye is generally within 0.3D, the flat curvature K1 of the right eye can be determined based on the flat curvature K1 of the left eye.

[0124] In some embodiments, the process may include step 740: for the other eye in the left or right eye, determining the steep curvature associated with the other eye based on the steep curvature associated with the first eye, according to a constraint on the difference between the steep curvature associated with the first eye and the steep curvature associated with the other eye.

[0125] As an example, for the right eye, taking the steep curvature K2 obtained at step 720 as an example, since the difference between the steep curvature K2 of the left eye and the steep curvature K2 of the right eye is generally within 0.3D, the steep curvature K2 of the right eye can be determined based on the steep curvature K2 of the left eye.

[0126] The following describes specific examples of users using computing devices according to some embodiments for ophthalmic-assisted diagnosis and treatment.

[0127] User action: The user uploaded a front-facing photo of themselves (this could be...) Figure 1 (Example of face image 120), and enter in the information bar: "Recently, my vision for distant objects has been a little blurry. I have no known history of eye disease." (This could be a...) Figure 1 Additional inputs to the third agent 170 in the system).

[0128] System Response: Upon receiving the request, the computing device initiates the diagnostic process. It first invokes the first intelligent agent (e.g., Figure 1 The first intelligent agent in the middle (130).

[0129] Eye localization: The model successfully identified the eyes in Zhang San's photo and cropped out high-resolution images of the left eye, right eye, and both eyes.

[0130] Disease screening: The cropped image is fed into a second intelligent agent (e.g., Figure 1 (Second agent 150 in the model). After analysis, the model outputs the following results: {"Myopia": "Positive", "Myopia Degree": "Low", "Ptosis": "Negative", "Strabismus": "Negative"}. This result is stored in the memory bank.

[0131] Suggested generation: The computing device integrates the screening results and the user's complaint of "blurred vision" and sends a request to a third-party intelligent agent (e.g., Figure 1 The third intelligent agent (170) issues instructions.

[0132] Output Report: The computing device ultimately generates and displays the following formatted report to the user: 1) Prevention: None; 2) Screening: None; 3) Treatment: Wear defocus glasses, 1 drop of 0.05% low-concentration atropine eye drops every night, refrigerate, 1 bottle per month, replace with a second bottle if not finished, continue for 3 months (mild near vision blur and photophobia are normal phenomena, and photophobia can be relieved by wearing sunglasses outdoors). 4) Rehabilitation: None; 5) Follow-up: Three months later, outpatient follow-up will be conducted to measure computer-assisted refraction, axial length, fundus photography, uncorrected visual acuity, and corrected visual acuity; 6) Lifestyle recommendations: Increase daily outdoor activity time by 2 hours, reduce daily close-range eye use time, reduce daily screen time, and reduce the frequency of eating sweets.

[0133] Embodiments of the invention have been described with reference to the accompanying drawings. These embodiments are illustrative and not restrictive.

Claims

1. A computing device for auxiliary diagnosis and treatment in ophthalmology, characterized in that, include: Computing resources; The first intelligent agent is configured to invoke the computing resources to: Obtain the user's facial image; A first eye image, including the left and right eyes, is generated based on the facial image; The second intelligent agent is configured to invoke the computing resources to: The ophthalmological screening results are determined based on the first eye image. The second agent is trained based on the following steps: Provide a third eye image and a fourth eye image associated with different perspectives of the second eye image, the second eye image being obtained by cropping an image including the eyes from a face image in an open-source face dataset; A partial image of the third eye and a partial image of the fourth eye are cropped to generate a set of partial images; A first patch primitive set is generated based on the third eye image and the fourth eye image; The first patch primitive set is randomly masked to form the second patch primitive set; The first network generates a first set of predicted probabilities based on the third eye image and the fourth eye image; The second network generates a second set of prediction probabilities based on the local image set; The first network generates a third prediction probability set based on the first patch primitive set; The predicted probabilities at the positions corresponding to the masked patch primitives in the third prediction probability set are retained to obtain a subset of the third prediction probability set; The second network generates a fourth prediction probability set based on the second patch primitive set; The predicted probabilities at the positions corresponding to the masked patch primitives in the fourth prediction probability set are retained to obtain a subset of the fourth prediction probability set; The loss function associated with the second network is determined based on the first set of predicted probabilities, the second set of predicted probabilities, a subset of the third set of predicted probabilities, and a subset of the fourth set of predicted probabilities; and The parameters associated with the second network are iteratively updated based on the loss function to train the second network, wherein the trained second network is used as the second agent; and A third intelligent agent is configured to invoke the computing resources to: Based on the ophthalmology screening results, ophthalmology treatment recommendations are generated.

2. The computing device as described in claim 1, characterized in that, The first agent is configured as follows: Detect multiple bounding boxes including eyes from the face image and confidence scores indicating that each bounding box includes the left and right eyes; Based on the center position of the coordinates of each bounding box and the position of the central axis of the face image, the multiple bounding boxes are divided into a set of left eye bounding boxes and a set of right eye bounding boxes. as well as Based on the confidence score and the intersection-union ratio of the left-eye bounding box and the right-eye bounding box, candidate left-eye bounding boxes are selected from the set of left-eye bounding boxes, and candidate right-eye bounding boxes are selected from the set of right-eye bounding boxes. as well as The first eye image is generated based on the candidate left eye bounding box and the candidate right eye bounding box.

3. The computing device as described in claim 2, characterized in that, The first agent is configured as follows: The left-eye bounding box is selected iteratively from the set of left-eye bounding boxes and the right-eye bounding box is selected from the set of right-eye bounding boxes in descending order of confidence score, until the intersection-union ratio of the selected left-eye bounding boxes and right-eye bounding boxes is zero.

4. The computing device as claimed in claim 1, characterized in that, The third and fourth eye images are obtained by performing an enhancement algorithm on the second eye image. The enhancement algorithm includes at least one of the following algorithms: random cropping, horizontal flipping, color perturbation, grayscale conversion, and Gaussian perturbation.

5. The computing device as claimed in claim 1, characterized in that, The loss function is determined based on a first loss function and a second loss function, the first loss function is determined based on a first set of predicted probabilities and a second set of predicted probabilities, and the second loss function is determined based on a subset of the third set of predicted probabilities and a subset of the fourth set of predicted probabilities.

6. The computing device as described in claim 5, characterized in that, The loss function is further determined based on a third loss function, which is determined based on Koleo regularization.

7. The computing device as claimed in claim 1, characterized in that, The second agent is trained using a first training dataset, which includes a fifth eye image. The fifth eye image includes at least one of the following labels: a label indicating whether the person is nearsighted, a label indicating the degree of nearsightedness, a label indicating the refractive power, and a label indicating an eye condition.

8. The computing device as claimed in claim 1, characterized in that, The third agent is trained using the second training dataset. The second training dataset includes task instructions, input information, and diagnostic suggestion text.

9. The computing device as claimed in claim 8, characterized in that, The input information includes ophthalmological screening results, personal information, auxiliary examination results, and biometric data.

10. The computing device as claimed in claim 9, characterized in that, The input information is obtained from the patient's medical record, wherein missing biometric data in the medical record is determined based on the personal information and the results of the auxiliary examinations.

11. The computing device as claimed in claim 10, characterized in that, The biometric data includes the axial length of both eyes.

12. The computing device as claimed in claim 11, characterized in that, The missing axial lengths of both eyes in the medical records were determined through the following steps: Obtain the baseline monocular axial length associated with the left or right eye in a non-myopic person of a predetermined age and specified gender; For one of the left or right eyes, the first monocular axial length associated with the eye is determined based on the baseline axial length associated with the eye, according to a predetermined relationship between monocular axial length and refractive power. For the other eye in the left or right eye, the difference in axial length is determined based on the difference in refractive power between the other eye and the first eye; For the other eye in the left or right eye, a second monocular axial length associated with the other eye is determined based on the predetermined relationship between monocular axial length and refractive power and the difference in axial length. as well as The axial length of both eyes is determined based on the axial length of the first monocular eye and the axial length of the second monocular eye.

13. The computing device as claimed in claim 10, characterized in that, The biometric data includes corneal curvature of both eyes, with each eye's corneal curvature including flat curvature and steep curvature.

14. The computing device as claimed in claim 13, characterized in that, The corneal curvature of the two missing eyes in the medical records was determined through the following steps: Obtain the baseline flat curvature associated with the left or right eye for individuals of a predetermined age and gender; For either the left or right eye, the steep curvature associated with the eye is determined based on the underlying flat curvature associated with the eye, according to the difference constraint between the flat curvature and the steep curvature associated with the eye. For the other eye in the left or right eye, the flat curvature associated with the other eye is determined based on the underlying flat curvature associated with the first eye, according to the difference constraint between the flat curvature associated with the first eye and the flat curvature associated with the other eye; For the other eye in either the left or right eye, the steep curvature associated with the other eye is determined based on the steep curvature associated with the first eye, according to the difference constraint between the steep curvature associated with the first eye and the steep curvature associated with the other eye.

15. The computing device as claimed in claim 1, characterized in that, The third agent is also configured to: Receive user input, and The ophthalmology treatment recommendations are also generated based on the user input.

16. The computing device as claimed in claim 1, characterized in that, The first intelligent agent is also configured to: After providing the first eye image to the second agent, the face image and the first eye image are deleted, and The second agent is configured as follows: After the ophthalmological screening results are confirmed, the first eye image is deleted.

17. The computing device as claimed in claim 16, characterized in that, The first intelligent agent is also configured to: Before collecting the facial image from the user, the user is prompted, and The first intelligent agent is only allowed to obtain the facial image with the user's authorization.

Images