Visual field inspection method and device based on fundus OCT and fundus color photo and medium

Abstract

The invention provides a visual field examination method and device based on fundus OCT and fundus color photograph and a medium, and belongs to the technical field of medical image processing, and the method comprises the steps: obtaining fundus optical coherence tomography (OCT) image data of a patient, and obtaining fundus color photograph data of the patient; performing visual field defect prediction processing on the OCT image data and the fundus color photo data based on a preset deep learning model to obtain a visual field defect prediction result of the patient; and performing data post-processing on the visual field defect prediction result based on a preset patient physiological condition regression model to obtain a final visual field defect inspection result of the patient. By adopting the technical scheme of the invention, the second-level visual field inspection test can be realized, so that the visual field inspection efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a method, device and medium for visual field examination based on fundus OCT and fundus color photography. Background Technology

[0002] Visual field testing is mainly used to assess the integrity of a patient's visual function. By detecting the extent and shape of visual field defects, it can help diagnose glaucoma, optic neuropathy, retinal diseases, and nervous system diseases, and provide objective evidence for treatment and disease monitoring.

[0003] In related technologies, the main method for assessing visual field testing is through perimeter testing. This involves presenting the patient with a series of bright spots or patterns at different locations using a perimeter, and having the patient react upon noticing them (e.g., by pressing a button or giving a verbal response). This process maps the boundaries of the patient's perceptible visual range, thereby assessing the integrity and sensitivity of their visual field. However, perimeter-based visual field testing typically takes a relatively long time to complete. This is because the perimeter testing process requires presenting stimuli at multiple locations and waiting for the patient's response, which can last for several minutes or even longer. This results in low overall efficiency for visual field testing. Summary of the Invention

[0004] The main objective of this application is to propose a method, device, and medium for visual field examination based on fundus OCT and fundus color photography, aiming to achieve visual field examination tests at the second level, thereby greatly improving the efficiency of visual field examination.

[0005] To achieve the above objectives, a first aspect of this application proposes a visual field examination method based on fundus OCT and fundus color photography, the method comprising:

[0006] Acquire optical coherence tomography (OCT) image data of the patient's fundus, and acquire color fundus photograph data of the patient;

[0007] Based on a preset deep learning model, visual field defect prediction processing is performed on the OCT image data and the fundus color photograph data to obtain the visual field defect prediction results of the patient.

[0008] The visual field defect prediction results are post-processed based on a pre-defined regression model of the patient's physiological condition to obtain the patient's final visual field defect examination results.

[0009] In some embodiments, the visual field defect prediction processing based on a preset deep learning model on the OCT image data and the fundus color photograph data to obtain the visual field defect prediction result for the patient includes:

[0010] The feature vectors of the OCT image data and the fundus color photograph data are input into a preset deep learning model; the feature vectors are used to characterize the fundus structural features of the patient.

[0011] Based on the deep learning model, structural feature information is extracted from the feature vector to perform visual field defect prediction processing, thereby obtaining the predicted visual field defect location information and visual field defect degree information; the visual field defect prediction result of the patient includes the visual field defect location information and the visual field defect degree information.

[0012] In some embodiments, inputting the feature vectors of the OCT image data and the fundus color photograph data into a preset deep learning model includes:

[0013] The feature vector of the fundus color image data is used as a category token to embed the feature vector of the OCT image data to obtain the feature vector after embedding the category token.

[0014] The positional encoding of the category token is added to the feature vector after embedding the category token to obtain the feature vector to be processed;

[0015] The feature vector to be processed is input into a preset deep learning model.

[0016] In some embodiments, the post-processing of the visual field defect prediction results based on a preset patient physiological regression model to obtain the patient's final visual field defect examination results includes:

[0017] The visual field defect prediction results are input into a preset patient physiological regression model;

[0018] Based on the patient's physiological condition regression model, the deviation value of the visual field defect prediction result is corrected to obtain the corrected visual field defect pattern map and the corrected visual field defect degree value output by the patient's physiological condition regression model.

[0019] The corrected visual field defect pattern and the corrected visual field defect severity value are used as the final visual field defect examination results for the patient.

[0020] In some embodiments, the patient physiological regression model uses the patient's corresponding demographic characteristics and ocular structural and functional related characteristics as independent variables;

[0021] The deviation correction of the visual field defect prediction results based on the regression model of the patient's physiological condition includes:

[0022] The visual function sensitivity is corrected based on the regression model of the patient's physiological condition, and the parameters of the visual field defect prediction results are corrected based on the Naive Bayes model based on the regression model of the patient's physiological condition.

[0023] In some embodiments, acquiring the patient's fundus optical coherence tomography (OCT) image data includes:

[0024] OCT image data of patients are acquired through the fundus OCT image acquisition unit of medical imaging equipment;

[0025] The acquisition of the patient's fundus color imaging data includes:

[0026] The patient's fundus color image data is acquired through the fundus color image acquisition unit of the medical imaging acquisition device.

[0027] In some embodiments, the method further includes any one of the following;

[0028] The deep learning model and / or the patient physiological regression model are deployed independently of the medical image acquisition device;

[0029] The deep learning model and / or the patient's physiological regression model are integrated and deployed on the medical image acquisition device.

[0030] To achieve the above objectives, a second aspect of this application provides a visual field examination system based on fundus OCT and fundus color photography, the system comprising:

[0031] The data acquisition module is used to acquire optical coherence tomography (OCT) image data of the patient's fundus, and to acquire color photographic data of the patient's fundus;

[0032] The model prediction module is used to perform visual field defect prediction processing on the OCT image data and the fundus color photograph data based on a preset deep learning model, so as to obtain the visual field defect prediction result of the patient.

[0033] The data post-processing module is used to perform data post-processing on the visual field defect prediction results based on a preset patient physiological regression model to obtain the patient's final visual field defect examination results.

[0034] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0035] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0036] To achieve the above objectives, a fifth aspect of the present application provides a computer program product storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0037] The visual field examination method, system, computer device, computer-readable storage medium, and computer program product proposed in this application are based on fundus OCT and fundus color photography. The method acquires optical coherence tomography (OCT) image data of the patient's fundus and fundus color photography data. It then performs visual field defect prediction processing on the OCT image data and fundus color photography data based on a preset deep learning model to obtain the patient's visual field defect prediction result. Finally, it performs post-processing on the visual field defect prediction result based on a preset patient physiological regression model to obtain the patient's final visual field defect examination result.

[0038] Therefore, compared to the traditional method of visual field examination using a perimeter, this embodiment of the application acquires the patient's fundus optical coherence tomography (OCT) image data and fundus color photograph data. Then, a deep learning model is used to first perform visual field defect prediction processing based on the OCT image data and fundus color photograph data to obtain the patient's visual field defect prediction result. Then, the visual field defect prediction result is post-processed based on the patient's physiological condition regression model to obtain the patient's final visual field defect examination result. In this way, by using the model to perform visual field examination on the patient, the testing time for visual field examination can be compressed to the second level, that is, to achieve second-level visual field examination testing, thereby greatly improving the efficiency of visual field examination.

[0039] Furthermore, in this embodiment of the application, the patient's fundus OCT image data and fundus color photograph data are obtained. For the patient, the light stimulation they feel is relatively weak compared to the light stimulation provided by the perimeter. This can greatly reduce the burden on the eyes of patients with low vision when undergoing visual field testing, thereby avoiding causing eye discomfort.

[0040] Furthermore, the embodiments of this application use a large-scale trained deep learning model to perform visual field defect prediction processing, which can effectively extract the corresponding information of the patient's fundus structure and visual function from the patient's fundus OCT image data and fundus color photography data, thereby accurately estimating the patient's visual field range and providing the patient with a more user-friendly and accurate visual field examination solution. Attached Figure Description

[0041] Figure 1 A schematic flowchart of the steps in some embodiments of the visual field examination method based on fundus OCT and fundus color photography provided in this application.

[0042] Figure 2 for Figure 1 A detailed flowchart of step S102;

[0043] Figure 3 for Figure 1 A detailed flowchart of step S103;

[0044] Figure 4 A schematic diagram of the structure of a patient visual field detection system involved in a complete embodiment of the visual field examination method based on fundus OCT and fundus color photography provided in the embodiments of this application;

[0045] Figure 5 A schematic diagram of the examination report and system input data involved in a complete embodiment of the visual field examination method based on fundus OCT and fundus color photography provided in the embodiments of this application;

[0046] Figure 6 A schematic diagram of the structure of the visual field examination system based on fundus OCT and fundus color photography provided in the embodiments of this application;

[0047] Figure 7 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0051] First, the overall concept of the embodiments of this application will be explained.

[0052] Visual field testing is mainly used to assess the integrity of a patient's visual function. By detecting the extent and shape of visual field defects, it can help diagnose glaucoma, optic neuropathy, retinal diseases, and nervous system diseases, and provide objective evidence for treatment and disease monitoring.

[0053] In related technologies, the main method for assessing visual field testing is through perimeter testing. This involves presenting the patient with a series of bright spots or patterns at different locations using a perimeter, and having the patient react upon noticing them (e.g., by pressing a button or giving a verbal response). This process maps the boundaries of the patient's perceptible visual range, thereby assessing the integrity and sensitivity of their visual field. However, perimeter-based visual field testing typically takes a relatively long time to complete. This is because the perimeter testing process requires presenting stimuli at multiple locations and waiting for the patient's response, which can last for several minutes or even longer. This results in low overall efficiency for visual field testing.

[0054] In addition, because the perimeter test takes a long time, it may aggravate visual fatigue and cause eye discomfort for patients with poor eye health.

[0055] Furthermore, the light stimulation used in perimetry testing is usually quite strong, which can be a burden for patients with photophobia. These patients are often more sensitive to light, and excessive light stimulation can trigger discomfort such as eye pain, dryness, or tearing. Therefore, perimetry testing is too burdensome for these patients, making it difficult for them to persist. Thus, developing a gentler and more efficient alternative for visual field estimation imaging is particularly urgent.

[0056] Based on this, embodiments of this application propose a method, system, computer device, computer-readable storage medium, and computer program product for visual field examination based on fundus OCT and fundus color photography. The method involves acquiring optical coherence tomography (OCT) image data of the patient's fundus and fundus color photography data; performing visual field defect prediction processing on the OCT image data and fundus color photography data based on a preset deep learning model to obtain the patient's visual field defect prediction result; and performing post-processing on the visual field defect prediction result based on a preset patient physiological regression model to obtain the patient's final visual field defect examination result.

[0057] Therefore, compared to the traditional method of visual field examination using a perimeter, this embodiment of the application acquires the patient's fundus optical coherence tomography (OCT) image data and fundus color photograph data. Then, a deep learning model is used to first perform visual field defect prediction processing based on the OCT image data and fundus color photograph data to obtain the patient's visual field defect prediction result. Then, the visual field defect prediction result is post-processed based on the patient's physiological condition regression model to obtain the patient's final visual field defect examination result. In this way, by using the model to perform visual field examination on the patient, the testing time for visual field examination can be compressed to the second level, that is, to achieve second-level visual field examination testing, thereby greatly improving the efficiency of visual field examination.

[0058] Furthermore, in this embodiment of the application, the patient's fundus OCT image data and fundus color photograph data are obtained. For the patient, the light stimulation they feel is relatively weak compared to the light stimulation provided by the perimeter. This can greatly reduce the burden on the eyes of patients with low vision when undergoing visual field testing, thereby avoiding causing eye discomfort.

[0059] Furthermore, the embodiments of this application use a large-scale trained deep learning model to perform visual field defect prediction processing, which can effectively extract the corresponding information of the patient's fundus structure and visual function from the patient's fundus OCT image data and fundus color photography data, thereby accurately estimating the patient's visual field range and providing the patient with a more user-friendly and accurate visual field examination solution.

[0060] Based on the overall concept of the embodiments of this application described above, specific embodiments of the visual field examination method, apparatus, computer device, computer-readable storage medium, and computer program product based on fundus OCT and fundus color photography provided in the embodiments of this application are proposed. First, the specific embodiments of the visual field examination method based on fundus OCT and fundus color photography in the embodiments of this application are described in detail.

[0061] It should be noted that the embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0062] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0063] Furthermore, in various specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Moreover, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. Additionally, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after explicitly obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of these embodiments acquired.

[0064] Furthermore, the visual field examination method based on fundus OCT and fundus color photography provided in this application embodiment can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a medical imaging device (e.g., a medical image acquisition device), a control and management device for a medical imaging device, a smartphone, tablet computer, laptop computer, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the visual field examination method based on fundus OCT and fundus color photography, etc., but is not limited to the above forms.

[0065] Alternatively, the visual field examination method based on fundus OCT and fundus color photography provided in this application embodiment can also be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0066] For ease of understanding and explanation, the following description will use the visual field examination method based on fundus OCT and fundus color photography provided in the embodiments of this application as an example. The implementation of any of the above-described subject matter using the visual field examination method based on fundus OCT and fundus color photography provided in the embodiments of this application can refer to the process described below for applying the visual field examination method based on fundus OCT and fundus color photography to a terminal device.

[0067] Please refer to Figure 1 , Figure 1 The flowchart illustrates the steps of the visual field examination method based on fundus OCT and fundus color photography provided in some embodiments of this application. It should be understood that, although... Figure 1 The figure shows the execution order of some method steps, but based on different design needs of practical applications, the visual field examination method based on fundus OCT and fundus color photography provided in this application embodiment can of course adopt a different execution order of method steps than that shown in the figure. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the visual field examination method based on fundus OCT and fundus color photography provided in the embodiments of this application. Any other method based on… Figure 1 Reasonable changes to the sequence of steps shown should be included within the protection scope of the visual field examination method based on fundus OCT and fundus color photography provided in the embodiments of this application.

[0068] like Figure 1 As shown, in some embodiments, the visual field examination method based on fundus OCT and fundus color photography provided in this application may include, but is not limited to, steps S101 to S103.

[0069] Step S101: Acquire optical coherence tomography (OCT) image data of the patient's fundus, and acquire color fundus photograph data of the patient.

[0070] Before conducting a formal visual field examination, the terminal device first performs a visual field examination test on the patient to obtain the patient's fundus optical coherence tomography (OCT) image data, as well as the patient's fundus color photograph data.

[0071] In some embodiments, the terminal device can use a medical image acquisition device to perform fundus OCT and fundus color photography examinations on the patient to obtain the corresponding examination results. Then, the terminal device can use the examination results obtained from the fundus OCT examination as the patient's OCT image data, and the examination results obtained from the fundus color photography examination as the patient's fundus color photography data.

[0072] In some embodiments, the terminal device can obtain corresponding examination results by performing fundus OCT and fundus color photography on the patient through medical image acquisition equipment, further preprocess these examination results (e.g., cropping and removing regions unrelated to visual function, etc.), and then use these processed examination results as the patient's OCT image data and fundus color photography data.

[0073] Step S102: Based on a preset deep learning model, perform visual field defect prediction processing on the OCT image data and the fundus color photograph data to obtain the visual field defect prediction result of the patient.

[0074] It should be noted that the preset deep learning model can be obtained by training the terminal device with large-scale clinical data in advance.

[0075] After acquiring the patient's OCT image data and fundus color photograph data, the terminal device can formally conduct a visual field examination on the patient. That is, the terminal device uses a deep learning model that has been trained on a large scale in advance to perform visual field defect prediction processing on the OCT image data and fundus color photograph data, thereby obtaining the visual field defect prediction result for the patient.

[0076] In some embodiments, the terminal device can extract corresponding information about the patient's fundus structure and visual function from the image data and the fundus color photograph data through a large-scale trained deep learning model, thereby accurately estimating the patient's visual field range. Then, the terminal device can use the patient's visual field range estimated by the model as the prediction result of the patient's visual field defect.

[0077] Step S103: Based on the preset patient physiological regression model, perform data post-processing on the visual field defect prediction results to obtain the patient's final visual field defect examination results.

[0078] It should be noted that the preset patient physiological regression model can also be obtained by training the terminal device with large-scale clinical data.

[0079] After obtaining the visual field defect prediction result based on the deep learning model, the terminal device further performs data post-processing on the visual field defect prediction result through a preset patient physiological regression model, thereby correcting the deviation that exists (or may exist) in the visual field defect prediction result. Then, the terminal device can use the corrected visual field defect prediction result as the patient's final visual field defect examination result.

[0080] In this embodiment, a visual field test is first performed on the patient using a terminal device to obtain optical coherence tomography (OCT) images and color fundus photographs. Then, a pre-trained deep learning model is used on the terminal device to predict visual field defects in the OCT images and fundus photographs, resulting in a predicted visual field defect outcome. Finally, a pre-defined regression model based on the patient's physiological condition is used on the terminal device to perform post-processing on the predicted visual field defect outcome, correcting any biases. The corrected predicted visual field defect outcome is then used as the patient's final visual field defect examination result.

[0081] Therefore, compared to the traditional method of visual field examination using a perimeter, this embodiment of the application acquires the patient's fundus optical coherence tomography (OCT) image data and fundus color photograph data. Then, a deep learning model is used to first perform visual field defect prediction processing based on the OCT image data and fundus color photograph data to obtain the patient's visual field defect prediction result. Then, the visual field defect prediction result is post-processed based on the patient's physiological condition regression model to obtain the patient's final visual field defect examination result. In this way, by using the model to perform visual field examination on the patient, the testing time for visual field examination can be compressed to the second level, that is, to achieve second-level visual field examination testing, thereby greatly improving the efficiency of visual field examination.

[0082] Furthermore, in this embodiment of the application, the patient's fundus OCT image data and fundus color photograph data are obtained. For the patient, the light stimulation they feel is relatively weak compared to the light stimulation provided by the perimeter. This can greatly reduce the burden on the eyes of patients with low vision when undergoing visual field testing, thereby avoiding causing eye discomfort.

[0083] Furthermore, the embodiments of this application use a large-scale trained deep learning model to perform visual field defect prediction processing, which can effectively extract the corresponding information of the patient's fundus structure and visual function from the patient's fundus OCT image data and fundus color photography data, thereby accurately estimating the patient's visual field range and providing the patient with a more user-friendly and accurate visual field examination solution.

[0084] In some embodiments, step S101 described above, "acquiring the patient's fundus optical coherence tomography (OCT) image data," may include the following steps:

[0085] OCT image data of patients are acquired through the fundus OCT image acquisition unit of medical imaging equipment.

[0086] When a terminal device performs a fundus OCT examination on a patient using a medical imaging acquisition device, it can use the fundus OCT image acquisition unit of the medical imaging acquisition device to adopt an array scanning mode. By continuously acquiring laser reflection information in a parallel direction, multiple scan lines are formed to display the internal structure of the eye tissue, thereby obtaining the patient's OCT image data.

[0087] In some embodiments, step S101 above, "acquiring the patient's fundus color imaging data", may include the following steps:

[0088] The patient's fundus color image data is acquired through the fundus color image acquisition unit of the medical imaging acquisition device.

[0089] When a terminal device examines a patient's fundus in color using a medical imaging acquisition device, it can synthesize a color fundus image by emitting multiple wavelengths of light toward the eye and capturing the reflected light through the fundus in color image acquisition unit of the medical imaging acquisition device. This color fundus image is the patient's fundus in color image data.

[0090] In some embodiments, before step S102 above: performing visual field defect prediction processing on the OCT image data and the fundus color photograph data based on a preset deep learning model to obtain the visual field defect prediction result of the patient, the visual field examination method based on fundus OCT and fundus color photograph provided in this application embodiment can also deploy the deep learning model and patient physiological condition regression model to be used in the formal visual field examination stage of the patient in any of the following ways.

[0091] Method 1: Deploy the deep learning model and / or the patient physiological regression model independently of the medical image acquisition device.

[0092] The terminal device can be deployed via a web interface, independently of the medical imaging acquisition equipment, to implement deep learning models and / or patient physiological regression models. In this way, through the web application interface, users can connect to the medical imaging acquisition equipment via the network to obtain raw data of fundus OCT images and fundus color photographs. Then, based on this OCT image data and fundus color photograph data, the deep learning model and / or patient physiological regression model can be used to perform a formal visual field examination on the patient.

[0093] Method 2: Integrate and deploy the deep learning model and / or the patient physiological regression model on the medical image acquisition device.

[0094] The terminal device can also choose to directly integrate and deploy deep learning models and / or patient physiological regression models on the medical imaging acquisition device. In this way, when the terminal device performs fundus OCT and fundus color imaging examinations on a patient through the medical imaging acquisition device, it can directly store the patient's OCT image data and fundus color imaging data acquired by the medical imaging acquisition device on the local disk of the medical imaging acquisition device. Subsequently, during the formal visual field examination of the patient, the terminal device can directly upload the OCT image data and fundus color imaging data from the local disk through the medical imaging acquisition device and perform archiving and analysis. That is, it is possible to perform a formal visual field examination on the patient locally on the medical imaging acquisition device based on the OCT image data and fundus color imaging data, using deep learning models and / or patient physiological regression models.

[0095] Please refer to Figure 2 , Figure 2 for Figure 1 A detailed flowchart of step S102.

[0096] like Figure 2 As shown, in some embodiments, step S102 above, which involves performing visual field defect prediction processing on the OCT image data and the fundus color photograph data based on a preset deep learning model to obtain the visual field defect prediction result of the patient, may include steps S201 and S202 as shown below.

[0097] Step S201: Input the feature vectors of the OCT image data and the fundus color photograph data into a preset deep learning model; the feature vectors are used to characterize the fundus structural features of the patient.

[0098] During the formal visual field examination based on the patient's OCT image data and fundus color photography data, when using a deep learning model to predict visual field defects in the OCT image data and fundus color photography data, the terminal device first performs a series of linear transformations and discretization processes on the OCT image data and fundus color photography data to obtain the feature vectors of the OCT image data and fundus color photography data respectively. These feature vectors can represent the fundus structural features of the patient in this region (the respective region where the OCT image data or fundus color photography data is located).

[0099] Then, the terminal device inputs the feature vectors of the OCT image data and fundus color photograph data into the deep learning model.

[0100] In some embodiments, the terminal device can process the input OCT image data sequence and fundus color photograph data sequence through a series of linear transformations and discretization processes to prepare them for input into a deep learning model. For example, for an input fundus OCT image data or fundus color photograph data (i,j), the terminal device can use a two-dimensional convolution kernel K(m,n) to perform a convolution operation to obtain the feature matrix O(i,j):

[0101] O(i,j)=(i*K)(i,j)=∑ m ∑ n I(im,in)·K(m,n).

[0102] Next, the terminal device divides the feature matrix O(i,j) into several small blocks according to the spatial location (i,j) and flattens them into a one-dimensional feature vector, which is used to represent the fundus structure features in this region. This one-dimensional feature vector can then be used as input into the deep learning model for subsequent processing.

[0103] Step S202: Based on the deep learning model, structural feature information is extracted from the feature vector to perform visual field defect prediction processing, and the predicted visual field defect location information and visual field defect degree information are obtained; the visual field defect prediction result of the patient includes the visual field defect location information and the visual field defect degree information.

[0104] When a terminal device uses a deep learning model to predict visual field defects using OCT image data and fundus color photography data, after inputting the feature vectors of the OCT image data and fundus color photography data into the deep learning model, the model can extract structural feature information from the input feature vectors to perform visual field defect prediction. This yields the model's predicted location and severity information for the visual field defect, which is then output. In this way, the terminal device can use this location and severity information as the visual field defect prediction result obtained by the deep learning model for predicting visual field defects in the patient.

[0105] In some embodiments, the terminal device can perform a series of linear transformations and discretization processes on OCT image data and fundus color image data to obtain one-dimensional feature vectors for each of the OCT image data and fundus color image data. Then, the feature vector of the fundus color image data can be used as a class token and embedded into the feature dependency learning of the fundus OCT image data, thereby realizing global feature aggregation of fundus OCT and fundus images.

[0106] It's important to note that the positional encoding of the class token is fixed, meaning that the positional encoding of the class token will not change regardless of how many image patches the input image is segmented into. This helps the model maintain consistent performance when processing input images of different sizes.

[0107] Based on this, step S201 above: inputting the feature vectors of the OCT image data and the fundus color photograph data into a preset deep learning model may include the following steps:

[0108] The feature vector of the fundus color image data is used as a category token to embed the feature vector of the OCT image data to obtain the feature vector after embedding the category token.

[0109] The positional encoding of the category token is added to the feature vector after embedding the category token to obtain the feature vector to be processed;

[0110] The feature vector to be processed is input into a preset deep learning model.

[0111] The terminal device performs a series of linear transformations and discretization processes on OCT image data and fundus color photography data to obtain a one-dimensional feature vector of the OCT image data as X = [x1, x2, ..., x]. N ], where N is the number of sampling regions, and the feature vector of the fundus color image data is obtained as in, This represents the pre-trained deep learning module. Thus, the terminal device embeds the feature vector from the fundus color photograph data as a category token into the feature vector of the OCT image data, resulting in the feature vector after embedding the category token:

[0112]

[0113] Subsequently, the terminal device adds the positional encoding of the category token to the feature vector X′ after embedding the category token, thus obtaining the feature vector to be processed as Z = X. ′ +P, where P is defined at position k as:

[0114]

[0115] Subsequently, the terminal device can input the feature vector Z into the deep learning model, and after multiple iterations, obtain the final prediction information, namely the model's prediction result of the patient's visual field defect.

[0116] In some embodiments, the terminal device can input feature vectors representing the structural features of each local region of the fundus into the attention layer through a deep learning model to extract structural feature information. This information will accurately reflect key information about the health of the fundus in multiple rounds of iterative optimization. In this way, the terminal device can obtain the visual field defect prediction results of the patient from the deep learning model. For example, given an input sequence... Where n is the sequence length and d is the feature dimension, the deep learning model first obtains Query(Q), Key(K), and Value(V) through linear transformation:

[0117] Q = XW Q K = XW K V = XW V .

[0118] in, It is a learnable parameter matrix, d k It is the projection dimension.

[0119] Then, the deep learning model calculates the attention score using dot products and scales it to obtain the output projection matrix:

[0120]

[0121] In some embodiments, when the size and volume of training data are large, the terminal device can further simplify the self-attention model into a selective state space model (SSM) using a deep learning model for iterative extraction of effective interdependencies. For example, the selective state space model (SSM) can be modeled as follows:

[0122]

[0123] Where h(t) represents the hidden dependency vector, x(t) represents the input feature vector, y(t) represents the output feature vector, and A, B, C, D represent the learnable parameters.

[0124] In some embodiments, the terminal device can perform iterations in a discrete system such as a neural network in the following manner:

[0125] h[n+1]=A d h[n]+B d x[n]y[n]=C d h[n]+D d x[n].

[0126] Through each iteration, the dependency vector can be learned by fitting the output and labels of the neural network.

[0127] Please refer to Figure 3 , Figure 3 for Figure 1 A detailed flowchart of step S103.

[0128] like Figure 3 As shown, in some embodiments, step S103 above: performing data post-processing on the visual field defect prediction results based on a preset patient physiological regression model to obtain the patient's final visual field defect examination results may include steps S301 to S303 as shown below.

[0129] Step S301: Input the visual field defect prediction results into a preset patient physiological regression model.

[0130] After receiving the visual field defect prediction results from the deep learning model, the terminal device further inputs these prediction results into the patient's physiological condition regression model. At this point, since the visual field defect prediction results include the location and severity information of the visual field defect predicted by the deep learning model, the terminal device can input this location and severity information into the patient's physiological condition regression model for subsequent bias correction.

[0131] Step S302: Correct the deviation value of the visual field defect prediction result based on the patient's physiological condition regression model to obtain the corrected visual field defect pattern map and the corrected visual field defect degree value output by the patient's physiological condition regression model.

[0132] After the terminal device inputs the visual field defect prediction results into the patient's physiological condition regression model, it corrects the deviation value of the visual field defect prediction results based on the patient's physiological condition regression model. After the deviation value correction, the patient's physiological condition regression model outputs the corrected visual field defect prediction results. Thus, when the input visual field defect prediction results include both the location and severity information of the visual field defect, the terminal device can obtain the corrected visual field defect pattern and the corrected visual field defect severity value output by the patient's physiological condition regression model after correcting the deviation values ​​for both the location and severity information.

[0133] Step S303: The corrected visual field defect pattern and the corrected visual field defect severity value are used as the final visual field defect examination results for the patient.

[0134] After obtaining the corrected visual field defect pattern map and the corrected visual field defect degree value output by the patient's physiological regression model, the terminal device can directly use the corrected visual field defect pattern map and the corrected visual field defect degree value as the final visual field defect examination result obtained from the current visual field examination of the patient.

[0135] In some embodiments, the above-described regression model of patient physiological conditions can use the patient's corresponding demographic characteristics (such as race, gender, and age) and basic ocular structural and functional characteristics (such as left / right eye, axial length, intraocular pressure, etc.) as independent variables.

[0136] Based on this, the step S302 above, "correcting the deviation value of the visual field defect prediction result based on the regression model of the patient's physiological condition," can include the following steps:

[0137] The visual function sensitivity is corrected based on the regression model of the patient's physiological condition, and the parameters of the visual field defect prediction results are corrected based on the Naive Bayes model based on the regression model of the patient's physiological condition.

[0138] When the terminal device corrects the bias of visual field defect prediction results based on the patient's physiological condition regression model, the terminal device can perform learnable visual function sensitivity correction based on the structural design of the patient's physiological condition regression model. For example, for each local region sensitivity y(i,j) predicted by the model in the visual field defect prediction results, the patient's physiological condition regression model can set two sets of correction parameters H. k and H s The gradient and bias term used to correct the sensitivity are used to obtain the final predicted sensitivity.

[0139]

[0140] Among them, W k b k W s b s All of these are learnable parameters that are optimized along with the neural network.

[0141] Furthermore, when correcting the bias of visual field defect prediction results based on a patient physiological regression model, the terminal device, based on the structural design of the patient physiological regression model, can also perform parameter correction based on a Naive Bayes model for the visual field defect prediction results. For example, for a set of discrete parameters of patient physiological conditions x = [x1, x2, ..., x...], ... nThe patient's physiological condition regression model can regress a parameter of the degree of visual impairment based on the Naive Bayes principle.

[0142]

[0143] Next, a complete embodiment of the visual field examination method based on fundus OCT and fundus color photography provided in this application is presented.

[0144] In a complete embodiment of the visual field examination method based on fundus OCT and fundus color photography provided in this application, the terminal device can perform a more user-friendly and accurate visual field examination scheme for patients through the patient visual field detection system compared to visual field examination using a perimeter.

[0145] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the patient visual field detection system involved in a complete embodiment of the visual field examination method based on fundus OCT and fundus color photography provided in the embodiments of this application.

[0146] like Figure 4 As shown, the patient visual field detection system includes three modules: data collection and preprocessing, model prediction, and data postprocessing.

[0147] In the data collection and preprocessing module, the patient visual field detection system reads the patient's fundus OCT and fundus color imaging results and removes areas unrelated to visual function by cropping.

[0148] In the model prediction module, the patient visual field detection system inputs the processed patient fundus-related images (OCT image data and fundus color photography data) into the deep learning model that has been trained on a large scale to obtain the location information and degree information of the visual field defect predicted by the model.

[0149] In the data post-processing module, the patient visual field detection system inputs the location and degree information of visual field defects predicted by the deep learning model into the patient's physiological regression model for bias correction, and finally outputs the patient's visual field defect pattern map and visual field defect degree value.

[0150] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the examination report and system input data involved in a complete embodiment of the visual field examination method based on fundus OCT and fundus color photography provided in the embodiments of this application. Figure 5 The visual field examination report data and photograph data shown are from the Internet. The visual field examination method based on fundus OCT and fundus color photography provided in this application does not limit the specific types and values ​​of the visual field examination report data and photograph data.

[0151] like Figure 5 As shown, Figure 5 The left side shows the visual field examination report obtained based on a visual field test using a perimeter. Figure 5 The blue box on the right (the patient's fundus OCT image sequence and fundus color photograph) represents the input to the visual field testing system, while the red box on the left represents the output. This demonstrates that the visual field testing system can functionally replace a conventional perimeter using a deep learning model.

[0152] Compared to perimeter-based visual field testing, patient visual field testing systems offer significant advantages in terms of testing speed and light stimulation. For example, a patient visual field testing system can reduce the testing time to less than 10 seconds, greatly improving efficiency. Simultaneously, during fundus OCT and fundus photography examinations, the light stimulation experienced by the patient is relatively weak, significantly reducing the eye strain for patients with low vision. Furthermore, through a large-scale trained deep learning model, the patient visual field testing system can effectively extract corresponding information about fundus structure and visual function from fundus OCT and fundus photography, accurately estimating the patient's visual field range and providing a more patient-friendly visual field testing solution.

[0153] Based on the same technical concept as the above-mentioned visual field examination method based on fundus OCT and fundus color photography, this application embodiment also provides a visual field examination system based on fundus OCT and fundus color photography, which can implement the above-mentioned visual field examination method based on fundus OCT and fundus color photography.

[0154] Please see Figure 6 The visual field examination system based on fundus OCT and fundus color photography provided in this application embodiment may include:

[0155] The data acquisition module is used to acquire optical coherence tomography (OCT) image data of the patient's fundus, and to acquire color photographic data of the patient's fundus;

[0156] The model prediction module is used to perform visual field defect prediction processing on the OCT image data and the fundus color photograph data based on a preset deep learning model, so as to obtain the visual field defect prediction result of the patient.

[0157] The data post-processing module is used to perform data post-processing on the visual field defect prediction results based on a preset patient physiological regression model to obtain the patient's final visual field defect examination results.

[0158] In some embodiments, the model prediction module is further configured to input the feature vectors of the OCT image data and the fundus color photograph data into a preset deep learning model; the feature vectors are used to characterize the fundus structural features of the patient; and, based on the deep learning model, extract structural feature information from the feature vectors to perform visual field defect prediction processing to obtain predicted visual field defect location information and visual field defect degree information; the patient's visual field defect prediction result includes the visual field defect location information and the visual field defect degree information.

[0159] In some embodiments, the model prediction module is further configured to embed the feature vector of the fundus color photograph data as a category token into the feature vector of the OCT image data to obtain a feature vector after embedding the category token; add the position encoding of the category token to the feature vector after embedding the category token to obtain a feature vector to be processed; and input the feature vector to be processed into a preset deep learning model.

[0160] In some embodiments, the data post-processing module is further configured to input the visual field defect prediction result into a preset patient physiological condition regression model; correct the deviation value of the visual field defect prediction result based on the patient physiological condition regression model to obtain the corrected visual field defect pattern map and the corrected visual field defect degree value output by the patient physiological condition regression model; and use the corrected visual field defect pattern map and the corrected visual field defect degree value as the patient's final visual field defect examination result.

[0161] In some embodiments, the patient physiological regression model uses the patient's corresponding demographic characteristics and ocular basic structure and function-related characteristics as independent variables; the data post-processing module is further used to perform visual function sensitivity correction on the visual field defect prediction results based on the patient physiological regression model, and to perform parameter correction on the visual field defect prediction results based on the Naive Bayes model based on the patient physiological regression model.

[0162] In some embodiments, the data acquisition module is further configured to acquire the patient's OCT image data through the fundus OCT image acquisition unit of the medical imaging acquisition device; and to acquire the patient's fundus color image data through the fundus color image acquisition unit of the medical imaging acquisition device.

[0163] In some embodiments, the visual field examination system based on fundus OCT and fundus color photography provided in this application may further include:

[0164] The model deployment module is used to deploy the deep learning model and / or the patient physiological regression model independently of the medical image acquisition device; or, to integrate and deploy the deep learning model and / or the patient physiological regression model on the medical image acquisition device.

[0165] It should be noted that the specific implementation of the visual field examination system based on fundus OCT and fundus color photography provided in this application is basically the same as the specific implementation of the visual field examination method based on fundus OCT and fundus color photography described above, and will not be repeated here.

[0166] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described visual field examination method based on fundus OCT and fundus color photography. This computer device can be a medical imaging device, a control and management device for medical imaging devices, a smartphone, tablet computer, laptop computer, desktop computer, or other terminal device.

[0167] Please see Figure 7 , Figure 7 This illustration shows the hardware structure of a computer device according to one embodiment. The computer device includes:

[0168] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0169] The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 to execute the visual field examination method based on fundus OCT and fundus color photography according to the embodiments of this application.

[0170] The input / output interface 703 is used to implement information input and output;

[0171] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0172] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0173] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0174] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described visual field examination method based on fundus OCT and fundus color photography.

[0175] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0176] This application also provides a computer program product that stores a computer program that, when executed by a processor, implements the above-described visual field examination method based on fundus OCT and fundus color photography.

[0177] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0178] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0181] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0182] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0184] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0186] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A visual field test method based on fundus OCT and fundus color photography, characterized by, The method comprises: obtaining fundus optical coherence tomography (OCT) image data of a patient, and obtaining fundus color photograph data of the patient; performing visual field defect prediction processing on the OCT image data and the fundus color photograph data based on a preset deep learning model to obtain a visual field defect prediction result of the patient; performing data post-processing on the visual field defect prediction result based on a preset patient physiological condition regression model to obtain a final visual field defect examination result of the patient.

2. The method of claim 1, wherein, The method comprises: inputting feature vectors of the OCT image data and the fundus color photograph data into a preset deep learning model; the feature vectors are used to represent fundus structure features of the patient; extracting structure feature information from the feature vectors based on the deep learning model to perform visual field defect prediction processing, to obtain predicted visual field defect position information and visual field defect degree information; the visual field defect prediction result of the patient comprises the visual field defect position information and the visual field defect degree information.

3. The method of claim 2, wherein, The method comprises: embedding a feature vector of the fundus color photograph data into a feature vector of the OCT image data as a category token, to obtain a feature vector after embedding the category token; adding position encoding of the category token to the feature vector after embedding the category token, to obtain a to-be-processed feature vector; inputting the to-be-processed feature vector into the preset deep learning model.

4. The method of claim 1, wherein, The method comprises: inputting the visual field defect prediction result into a preset patient physiological condition regression model; performing bias value correction on the visual field defect prediction result based on the patient physiological condition regression model, to obtain a corrected visual field defect mode chart and a corrected visual field defect degree value output by the patient physiological condition regression model; taking the corrected visual field defect mode chart and the corrected visual field defect degree value as the final visual field defect examination result of the patient.

5. The method of claim 4, wherein, The patient physiological condition regression model takes demographic characteristics and eye basic structure and function related features of the patient as independent variables. The method comprises: performing visual function sensitivity correction on the visual field defect prediction result based on the patient physiological condition regression model, and performing parameter correction on the visual field defect prediction result based on a Naive Bayes model based on the patient physiological condition regression model.

6. The method according to any one of claims 1 to 5, characterized in that, The method comprises: obtaining OCT image data of a patient through an OCT image acquisition unit of a medical image acquisition device; The method comprises: The fundus photograph data of the patient is acquired by a fundus photograph image acquisition unit of a medical image acquisition device.

7. The method of claim 6, wherein, The method further comprises any one of the following: The deep learning model and / or the patient physiological condition regression model are deployed independently of the medical image acquisition device; The deep learning model and / or the patient physiological condition regression model are integrated and deployed on the medical image acquisition device.

8. A visual field examination system based on fundus OCT and fundus color photography, characterized by, The system comprises: A data acquisition module is configured to acquire fundus optical coherence tomography (OCT) image data of a patient and fundus photograph data of the patient; A model prediction module is configured to perform visual field defect prediction processing on the OCT image data and the fundus photograph data based on a preset deep learning model to obtain a visual field defect prediction result of the patient; A data post-processing module is configured to perform data post-processing on the visual field defect prediction result based on a preset patient physiological condition regression model to obtain a final visual field defect examination result of the patient.

9. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the fundus OCT and fundus photograph based visual field examination method in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to implement the fundus OCT and fundus photograph based visual field examination method in any one of claims 1 to 7.

Images