Anti-VEGF curative effect prediction method and device, computer equipment and storage medium

Through multimodal data fusion, combined with fundus color imaging and OCT images, the problem of single data and multitasking association neglect in anti-VEGF efficacy prediction is solved, improving the accuracy and comprehensiveness of the prediction, and achieving more accurate vision change prediction.

CN119090852AActive Publication Date: 2024-12-06SHENZHEN UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411225780.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-12-06
Estimated Expiration
2044-09-03

AI Technical Summary

Technical Problem

The existing anti-VEGF efficacy prediction methods have problems such as single data, ignoring multitasking association and insufficient multimodal data fusion, resulting in inaccurate prediction results.

Method used

The multimodal data fusion method is used to obtain the original fundus color images and OCT images, and input them into the prediction model for prediction and splicing operations. Combined with the vision change prediction model, multi-dimensional information of the eye health status is captured.

Benefits of technology

It significantly improves the comprehensiveness and accuracy of eye health assessment and improves the accuracy of vision change prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090852A_ABST
    Figure CN119090852A_ABST
Patent Text Reader

Abstract

The embodiment of the invention belongs to the technical field of image processing, and relates to an anti-VEGF curative effect prediction method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a to-be-predicted original fundus color photo and an original OCT image; inputting the original fundus photograph into a fundus photograph prediction model to carry out fundus photograph prediction operation to obtain a predicted fundus photograph; performing fundus color photo splicing operation on the original fundus color photo and the predicted fundus color photo to obtain a spliced fundus color photo; inputting the original OCT image into an OCT image prediction model to carry out OCT image prediction operation to obtain a predicted OCT image; performing OCT image splicing operation on the original OCT image and the predicted OCT image to obtain a spliced OCT image; and inputting the spliced fundus color photo and the spliced OCT image into a vision change prediction model for vision change prediction operation to obtain a vision change prediction result. According to the method, the comprehensiveness and accuracy of eye health condition evaluation are remarkably improved, and then the precision of vision change prediction is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, computer equipment and storage medium for predicting anti-VEGF therapeutic efficacy. Background Art

[0002] In the process of treating age-related macular degeneration (AMD), predicting the efficacy of individual treatment has always been an important and challenging issue. The most effective treatment method currently is through the injection of anti-vascular endothelial growth factor (VEGF), however, the effect of this treatment varies significantly between different patients. Some patients' condition improves after treatment, while others fail to improve effectively or even worsen. Due to the high cost of anti-VEGF treatment, this treatment method is obviously inappropriate for those patients who cannot obtain effective results. In addition, patients will experience changes in vision during treatment, and the degree of change in the condition and the magnitude of change in vision after treatment are difficult to accurately assess, which poses a challenge to clinicians in formulating effective treatment plans.

[0003] Existing methods for predicting the efficacy of anti-VEGF mainly rely on single-modality data, such as OCT images or fundus color images. These methods usually use traditional machine learning algorithms or simple deep learning networks. Although they have achieved certain results in some cases, they have significant limitations when dealing with complex and diverse clinical data. First, single-modality data cannot fully reflect the changes in the patient's condition. The prediction of anti-VEGF efficacy requires comprehensive consideration of multiple factors. Using only OCT images or fundus color images cannot fully describe the treatment effect, and single-modality data is difficult to accurately predict the treatment effect. Secondly, existing multi-task learning methods are rarely used in the prediction of anti-VEGF efficacy. Most methods only focus on the optimization of a single task and ignore the potential correlation between multiple tasks. This leads to the limitation of the generalization ability and prediction accuracy of the model when dealing with complex tasks.

[0004] It can be seen that the traditional anti-VEGF efficacy prediction method has problems such as single data, neglect of multi-task association and insufficient multimodal data fusion. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose an anti-VEGF efficacy prediction method, device, computer equipment and storage medium to solve the problems of traditional anti-VEGF efficacy prediction methods, such as single data, neglect of multi-task association and insufficient multi-modal data fusion.

[0006] In order to solve the above technical problems, the present application embodiment provides a method for predicting the efficacy of anti-VEGF, which adopts the following technical solution:

[0007] Obtaining the original fundus color photograph and original OCT image to be predicted;

[0008] Inputting the original fundus color photograph into a fundus color photograph prediction model to perform a fundus color photograph prediction operation to obtain a predicted fundus color photograph;

[0009] Performing a fundus color photo splicing operation on the original fundus color photo and the predicted fundus color photo to obtain a spliced ​​fundus color photo;

[0010] Inputting the original OCT image into the OCT image prediction model to perform an OCT image prediction operation to obtain a predicted OCT image;

[0011] Perform an OCT image stitching operation on the original OCT image and the predicted OCT image to obtain a stitched OCT image;

[0012] The spliced ​​fundus color photograph and the spliced ​​OCT image are input into a vision change prediction model to perform a vision change prediction operation to obtain a vision change prediction result.

[0013] Furthermore, the fundus color photograph prediction model is composed of a fundus color photograph residual module, a blood vessel extraction module and a fundus color photograph global feature extraction module.

[0014] Further, the predicted fundus color photograph I pre-Fundus It is expressed as:

[0015] I pre-Fundus =CA(F Fundus +ReLu(Conv(Cat(V Fundus , I Fundus )))+G Fundus )

[0016] G Fundus =G global (F Fundus )

[0017] V Fundus =H(I Fundus )

[0018] F Fundus =ResBlock5(...(ResBlock(ReLu(Conv(Cat(V Fundus , I Fundus ))))

[0019] Among them, CA represents channel attention, F Fundus represents the feature map after the fundus color photo residual module is processed, V Fundus represents the high-frequency feature map extracted by the blood vessel extraction module, G Fundusrepresents the global feature map output by the fundus color photo global feature extraction module, ReLu represents the activation function, Conv represents the convolution operation, and Cat represents the splicing along the channel dimension.

[0020] Furthermore, the blood vessel extraction module H(I) is expressed as:

[0021] H(I)=II*gaussion(r,σ)

[0022] Wherein, H represents the blood vessel extraction module, I represents the input image, and gaussion(r,σ) represents a low-pass Gaussian filter with a radius of r and a spatial constant of σ.

[0023] Furthermore, the OCT image prediction model consists of a downsampling module, an OCT image residual module, an upsampling module and an OCT image global feature extraction module.

[0024] Furthermore, the predicted OCT image I pre-OCT It is expressed as:

[0025] I pre-OCT =ReLu(Conv(ReflectionPad Cat(G OCT ,U OCT )))

[0026] G OCT =G global (F OCT )

[0027] U OCT =UpSanple3(UpSample2(UpSample1(F OCT )))

[0028] F OCT =ResBlock5(...(ResBlock(D OCT )))

[0029] D OCT

[0030] =DownSanple3(DownSample2(DownSample1(ReLu(InstanceNorm(Conv(ReflectionPad(I OCT )))))))

[0031] Among them, D OCT Represents the OCT image after the downsampling module, DownSample i represents the downsampling operation, F OCTRepresents the OCT feature map after the OCT image residual module is processed, UpSanple i represents the upsampling operation, U OCT The OCT image after the upsampling module, ReflectionPad represents reflection padding, G OCT Represents global feature processing of OCT images.

[0032] Furthermore, the fundus color photo global feature extraction module or the OCT image global feature extraction module is expressed as:

[0033] GFB(I)=Sigmoid(Conv(Cat(Max(I),Mean(I))))*I

[0034] Among them, GFB represents the global feature processing module, I represents the input feature image, Sigmoid represents the nonlinear activation layer, Conv represents the convolution operation, Cat represents the splicing operation along the channel dimension, Max represents the maximum pooling, and Mean represents the average pooling.

[0035] In order to solve the above technical problems, the present application also provides an anti-VEGF efficacy prediction device, which adopts the following technical solution:

[0036] A raw data acquisition module is used to acquire the raw fundus color photograph and raw OCT image to be predicted;

[0037] A fundus color photo prediction module, used for inputting the original fundus color photo into the fundus color photo prediction model to perform a fundus color photo prediction operation to obtain a predicted fundus color photo;

[0038] A fundus color photo stitching module, used for performing a fundus color photo stitching operation on the original fundus color photo and the predicted fundus color photo to obtain a stitched fundus color photo;

[0039] An OCT image prediction module, used for inputting the original OCT image into the OCT image prediction model to perform an OCT image prediction operation to obtain a predicted OCT image;

[0040] An OCT image stitching module, used for performing an OCT image stitching operation on the original OCT image and the predicted OCT image to obtain a stitched OCT image;

[0041] The vision change prediction module is used to input the spliced ​​fundus color photograph and the spliced ​​OCT image into the vision change prediction model to perform a vision change prediction operation to obtain a vision change prediction result.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0043] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the anti-VEGF efficacy prediction method as described above when executing the computer-readable instructions.

[0044] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0045] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the anti-VEGF efficacy prediction method described above are implemented.

[0046] The present application provides an anti-VEGF efficacy prediction method, including: obtaining the original fundus color photograph and the original OCT image to be predicted; inputting the original fundus color photograph into the fundus color photograph prediction model to perform the fundus color photograph prediction operation to obtain the predicted fundus color photograph; performing the fundus color photograph splicing operation on the original fundus color photograph and the predicted fundus color photograph to obtain the spliced ​​fundus color photograph; inputting the original OCT image into the OCT image prediction model to perform the OCT image prediction operation to obtain the predicted OCT image; performing the OCT image splicing operation on the original OCT image and the predicted OCT image to obtain the spliced ​​OCT image; inputting the spliced ​​fundus color photograph and the spliced ​​OCT image into the vision change prediction model to perform the vision change prediction operation to obtain the vision change prediction result. Compared with the prior art, the present application combines two different types of medical image data, namely fundus color photograph and OCT (optical coherence tomography) image, so that the system can capture information on the health status of the eye from multiple dimensions. Fundus color photography mainly reflects the macroscopic structure and color changes of the retina, while OCT images provide microscopic tomographic structural information of the retina and deeper tissues. The fusion of this multimodal data significantly improves the comprehensiveness and accuracy of the assessment of eye health, thereby improving the accuracy of the prediction of vision changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0049] Figure 2 is a flowchart of the method for predicting the anti-VEGF efficacy provided in Example 1 of the present application;

[0050] Figure 3 It is a structural schematic diagram of a specific implementation of the multi-modal and multi-task anti-VEGF efficacy prediction network provided in Example 1 of the present application;

[0051] Figure 4 is a schematic diagram of the structure of the anti-VEGF therapeutic efficacy prediction device provided in Example 2 of the present application;

[0052] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0054] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0057] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0058] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0059] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0060] It should be noted that the anti-VEGF efficacy prediction method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the anti-VEGF efficacy prediction device is generally set in the server / terminal device.

[0061] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0062] Embodiment 1

[0063] Continue to refer Figure 2 , shows a flow chart of an embodiment of the anti-VEGF efficacy prediction method according to the present application. The anti-VEGF efficacy prediction method comprises: step S201, step S202, step S203, step S204, step S205 and step S206.

[0064] In step S201, the original fundus color photograph and the original OCT image to be predicted are obtained;

[0065] In step S202, the original fundus color photograph is input into the fundus color photograph prediction model to perform a fundus color photograph prediction operation to obtain a predicted fundus color photograph;

[0066] In step S203, the original fundus color photograph and the predicted fundus color photograph are spliced ​​to obtain a spliced ​​fundus color photograph;

[0067] In step S204, the original OCT image is input into the OCT image prediction model to perform an OCT image prediction operation to obtain a predicted OCT image;

[0068] In step S205, an OCT image stitching operation is performed on the original OCT image and the predicted OCT image to obtain a stitched OCT image;

[0069] In step S206, the stitched fundus color photograph and the stitched OCT image are input into the vision change prediction model to perform a vision change prediction operation to obtain a vision change prediction result.

[0070] In the embodiments of this application, see Figure 3 The multi-modal and multi-task anti-VEGF efficacy prediction network shown in the figure consists of three parts: OCT image prediction branch, fundus color photo prediction branch, and vision change prediction branch. The two image prediction branches also design corresponding discriminators for adversarial training, where:

[0071] The main body of the fundus color photo prediction branch is composed of five identical residual blocks. In addition, a blood vessel extraction module is designed to achieve accurate prediction of blood vessel morphology after treatment. At the same time, a global feature extraction module is designed for feature learning of medium and low frequency lesions.

[0072] The OCT image prediction branch does not use the high-frequency blood vessel extraction module. Instead, it performs three downsamplings before inputting multiple residual blocks to convert the original image size (1, 256, 256) to (256, 64, 64), extracts features through multiple residual blocks, and then performs three upsamplings to restore the feature map to (1, 256, 256). The OCT prediction branch also uses the designed global feature extraction module to learn medium and low-frequency features;

[0073] The vision prediction branch consists of two parallel classification blocks, which receive the input of fundus color photos and OCT images respectively, and then extract and classify them through the classification block, and finally fuse the two classification results to obtain the final vision prediction result. It is worth noting that the images input to the classification block are feature maps obtained by splicing the pre-treatment image with the corresponding predicted post-treatment image.

[0074] In an embodiment of the present application, a method for predicting the efficacy of anti-VEGF is provided, including: obtaining the original fundus color photograph and the original OCT image to be predicted; inputting the original fundus color photograph into the fundus color photograph prediction model to perform the fundus color photograph prediction operation to obtain the predicted fundus color photograph; performing the fundus color photograph splicing operation on the original fundus color photograph and the predicted fundus color photograph to obtain the spliced ​​fundus color photograph; inputting the original OCT image into the OCT image prediction model to perform the OCT image prediction operation to obtain the predicted OCT image; performing the OCT image splicing operation on the original OCT image and the predicted OCT image to obtain the spliced ​​OCT image; inputting the spliced ​​fundus color photograph and the spliced ​​OCT image into the vision change prediction model to perform the vision change prediction operation to obtain the vision change prediction result. Compared with the prior art, the present application combines two different types of medical image data, namely, fundus color photograph and OCT (optical coherence tomography) image, so that the system can capture information on the health status of the eye from multiple dimensions. Fundus color photography mainly reflects the macroscopic structure and color changes of the retina, while OCT images provide microscopic tomographic structural information of the retina and deeper tissues. The fusion of this multimodal data significantly improves the comprehensiveness and accuracy of the assessment of eye health, thereby improving the accuracy of the prediction of vision changes.

[0075] In some optional implementations of the embodiments of the present application, the above-mentioned predicted fundus color photograph I pre-Fundus It is expressed as:

[0076] I pre-Fundus =CA(F Fundus +ReLu(Conv(Cat(V Fundus , I Fundus )))+G Fundus )

[0077] G Fundus =G global (F Fundus )

[0078] V Fundus =H(I Fundus )

[0079] F Fundus =ResBlock5(...(ResBlock(ReLu(Conv(Cat(V Fundus , I Fundus ))))

[0080] Among them, CA represents channel attention, F Fundus V represents the feature map after the fundus color photo residual module is processed. Fundus represents the high-frequency feature map extracted by the blood vessel extraction module, G FundusIt represents the global feature map output by the global feature extraction module of fundus color photos, ReLu represents the activation function, Conv represents the convolution operation, and Cat represents the concatenation along the channel dimension.

[0081] In some optional implementations of the embodiments of the present application, the blood vessel extraction module H(I) is expressed as:

[0082] H(I)=II*gaussion(r,σ)

[0083] Among them, H represents the blood vessel extraction module, I represents the input image, and gaussion(r,σ) represents a low-pass Gaussian filter with a radius of r and a spatial constant of σ.

[0084] In the embodiment of the present application, the high-frequency blood vessel extraction module uses a Gaussian filter to filter out high-frequency details to obtain a low-frequency component, and then subtracts the low-frequency component pixel by pixel from the original image to finally obtain an image with high-frequency blood vessel details. This module is applied to the input pre-treatment image and the output post-treatment image at the same time to obtain the pre-treatment blood vessel features and the post-treatment blood vessel features. In the training process of the model, the average absolute value is used to calculate the time difference between the treatment blood vessel features and the post-treatment blood vessel features, so that the predicted post-treatment blood vessels can be more realistic.

[0085] H(I)=II*gaussion(r,σ)

[0086] Where H refers to the blood vessel extraction module, I refers to the input image, and gaussion(r,σ) refers to a low-pass Gaussian filter with a radius of r and a spatial constant of σ. During model training, the radius r and the spatial constant σ are set to 26 and 9, respectively.

[0087] In some optional implementations of the embodiments of the present application, the above-mentioned predicted OCT image I pre-OCT It is expressed as:

[0088] I pre-OCT =ReLu(Conv(ReflectionPad Cat(G OCT ,U OCT )))

[0089] G OCT =G global (F OCT )

[0090] U OCT =UpSanple3(UpSample2(UpSample1(F OCT )))

[0091] F OCT=ResBlock5(...(ResBlock(D OCT )))

[0092] D OCT

[0093] =DownSanple3(DownSample2(DownSample1(ReLu(InstanceNorm(Conv(ReflectionPad(I OCT )))))))

[0094] Among them, D OCT Represents the OCT image after the downsampling module, DownSample i represents the downsampling operation, F OCT Represents the OCT feature map after OCT image residual module processing, UpSanple i represents the upsampling operation, U OCT OCT image after upsampling module, ReflectionPad represents reflection padding, G OCT Represents global feature processing of OCT images.

[0095] In some optional implementations of the embodiments of the present application, the above fundus color photo global feature extraction module or OCT image global feature extraction module is expressed as:

[0096] GFB(I)=Sigmoid(Conv(Cat(Max(I),Mean(I))))*I

[0097] Among them, GFB represents the global feature processing module, I represents the input feature image, Sigmoid represents the nonlinear activation layer, Conv represents the convolution operation, Cat represents the splicing operation along the channel dimension, Max represents the maximum pooling, and Mean represents the average pooling.

[0098] In the embodiment of the present application, the global feature processing module can effectively extract global information and predict the changes of lesions before and after treatment. This module splices the feature maps obtained by maximum pooling and average pooling in the channel dimension, and then extracts features through a convolution layer and an activation layer to obtain global features, and then multiplies this global feature with the original image pixel by pixel to obtain the final global feature.

[0099] GFB(I)=Sigmoid(Conv(Cat(Max(I),Mean(I))))*I

[0100] Among them, GFB represents the global feature processing module, I represents the input feature image, Sigmoid represents the nonlinear activation layer, Conv represents the convolution operation, Cat represents the splicing operation along the channel dimension, Max represents the maximum pooling, and Mean represents the average pooling.

[0101] In some optional implementations of the embodiments of the present application, the network accepts OCT images and fundus color photographs before treatment as input, and uses OCT and fundus color photographs after treatment and vision changes as labels. The vision change prediction branch inputs OCT images and fundus color photographs before and after treatment to predict the changes in vision after treatment, and calculates the loss between the predicted changes and the actual changes to implement constraints on the model. The OCT image prediction branch and the fundus color photograph prediction branch output predicted OCT images and fundus color photographs after treatment, respectively, and then these predicted images and the actual post-treatment images are input into the following three networks in pairs: on the one hand, the input discriminator calculates the adversarial loss to ensure the authenticity of the predicted image; on the other hand, the predicted image and the actual post-treatment image are mapped to the VGG feature space through the pre-trained VGG19 network, and the content loss between the features is calculated to ensure the content accuracy of the predicted image. In addition, the predicted image and the actual image will also be input into the registration network to calculate the registration loss between the two to maintain the consistency of the image structure. Through these steps, the network is able to accurately predict the efficacy of anti-VEGF treatment under the synergy of multi-task and multi-modal data, and ensure the visual and structural consistency of the prediction results.

[0102] The loss calculation in the above three training processes can be expressed as:

[0103]

[0104] in, Represents resistance to loss, Indicates content loss, represents the registration loss, I post represents the real image after treatment, I pred represents the predicted post-treatment image, D() represents the discriminator network, VGG() represents the pre-trained VGG19 model, and Registration() represents the registration network.

[0105] The main evaluation indicators are:

[0106] (1) Mean Square Error (MAE): It measures the average value of the absolute difference between each corresponding pixel in the predicted image and the actual image. The smaller the MAE, the smaller the difference between the predicted image and the actual image, that is, the predicted image of the model is closer to the actual image.

[0107]

[0108] Where n is the number of pixels. i and i are the pixel values ​​at the same position in the predicted image and the actual image respectively.

[0109] (2) Peak Signal-to-Noise Ratio (PSNR): This measures the mean square error (MSE) between the predicted image and the actual image and converts it to a logarithmic scale. The higher the PSNR, the smaller the mean square error between the predicted image and the actual image, that is, the higher the quality of the predicted image.

[0110]

[0111] max is the maximum possible value of the pixel value (for example, for an 8-bit grayscale image, max = 255), and MSE is the mean square error. The calculation method is:

[0112]

[0113] Where n is the number of pixels. i and i are the pixel values ​​at the same position in the predicted image and the actual image respectively.

[0114] (3) Structural Similarity Index (SSIM): It considers the similarity of brightness, contrast, and structural information between the predicted image and the actual image. SSIM comprehensively considers the structural information of the predicted image, not just the similarity of pixel values. The SSIM value is in the range of [-1,1]. The closer it is to 1, the more similar the predicted image is to the actual image.

[0115]

[0116] Among them, μ x and μ y It is the average brightness of the predicted image x and the actual image y, which measures the brightness of the image; and It is the pixel value variance between the predicted image x and the actual image y, which measures the distribution of pixel values. A larger variance means that the image contains more details. xy It is the covariance of the pixel values ​​of the predicted image x and the actual image y, which measures the linear relationship between the pixels of the predicted image and the actual image. A positive covariance indicates a positive correlation between the two, and a negative covariance indicates a negative correlation. C1 and C2 are constants used to stabilize the calculation.

[0117] In practical applications, the anti-VEGF efficacy prediction model proposed in the embodiment of the present application has better evaluation indicators than the existing model, and the predicted results after treatment have good details, especially in the detail generation of fundus color photos and the prediction of lesion changes in OCT images.

[0118]

[0119] Table 1 Comparison of test results

[0120] Through qualitative and quantitative analysis, it can be seen that the added modules can effectively improve the quality of images generated by the model, especially the addition of high-frequency feature processing modules, which can make the generated fundus color photos have more vascular details. The added global feature module can significantly improve the relevant performance indicators of OCT images during training.

[0121] Through comparative analysis, we can conclude that adding global features, registration loss, and blood vessel extraction modules can significantly improve the performance of the model. Specifically, although the global feature module has limited effect in improving image quality, it can significantly improve the prediction accuracy; the registration loss module performs well in improving image quality, especially in reducing the mean absolute error and improving the peak signal-to-noise ratio and structural similarity coefficient, but has a slight negative impact on the prediction accuracy; and the blood vessel extraction module has achieved the best performance in reducing prediction error, improving image quality, and prediction accuracy. Comprehensively considered, the introduction of these three modules can effectively improve the overall performance of the model, among which the blood vessel extraction module is particularly outstanding in improving image quality and prediction accuracy, while the global feature module plays a positive role in improving prediction accuracy.

[0122] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0123] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0125] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0126] Embodiment 2

[0127] Further references Figure 4 , as a response to the above Figure 2 The present application provides an embodiment of an anti-VEGF efficacy prediction device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0128] like Figure 4 As shown, the anti-VEGF therapeutic efficacy prediction device 200 of the embodiment of the present application includes:

[0129] The original data acquisition module 210 is used to acquire the original fundus color photograph and the original OCT image to be predicted;

[0130] The fundus color photo prediction module 220 is used to input the original fundus color photo into the fundus color photo prediction model to perform fundus color photo prediction operation to obtain a predicted fundus color photo;

[0131] The fundus color photo stitching module 230 is used to perform a fundus color photo stitching operation on the original fundus color photo and the predicted fundus color photo to obtain a stitched fundus color photo;

[0132] The OCT image prediction module 240 is used to input the original OCT image into the OCT image prediction model to perform an OCT image prediction operation to obtain a predicted OCT image;

[0133] An OCT image stitching module 250 is used to perform an OCT image stitching operation on the original OCT image and the predicted OCT image to obtain a stitched OCT image;

[0134] The vision change prediction module 260 is used to input the spliced ​​fundus color photographs and the spliced ​​OCT images into the vision change prediction model to perform a vision change prediction operation and obtain a vision change prediction result.

[0135] In the embodiment of the present application, an anti-VEGF efficacy prediction device 200 is provided, comprising: an original data acquisition module 210, used to acquire the original fundus color photo and the original OCT image to be predicted; a fundus color photo prediction module 220, used to input the original fundus color photo into the fundus color photo prediction model to perform fundus color photo prediction operation to obtain the predicted fundus color photo; a fundus color photo splicing module 230, used to perform fundus color photo splicing operation on the original fundus color photo and the predicted fundus color photo to obtain the spliced ​​fundus color photo; an OCT image prediction module 240, used to input the original OCT image into the OCT image prediction model to perform OCT image prediction operation to obtain the predicted OCT image; an OCT image splicing module 250, used to perform OCT image splicing operation on the original OCT image and the predicted OCT image to obtain the spliced ​​OCT image; and a vision change prediction module 260, used to input the spliced ​​fundus color photo and the spliced ​​OCT image into the vision change prediction model to perform vision change prediction operation to obtain the vision change prediction result. . Compared with the prior art, this application combines two different types of medical image data, fundus color photography and OCT (optical coherence tomography) images, so that the system can capture information about the health status of the eyes from multiple dimensions. Fundus color photography mainly reflects the macroscopic structure and color changes of the retina, while OCT images provide microscopic tomographic structure information of the retina and deeper tissues. This fusion of multimodal data significantly improves the comprehensiveness and accuracy of the assessment of eye health, thereby improving the accuracy of predicting vision changes.

[0136] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0137] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0138] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0139] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the anti-VEGF efficacy prediction method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0140] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the anti-VEGF efficacy prediction method.

[0141] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0142] The computer device provided in this application combines two different types of medical image data, fundus color photography and OCT (optical coherence tomography) images, so that the system can capture information about the health status of the eyes from multiple dimensions. Fundus color photography mainly reflects the macroscopic structure and color changes of the retina, while OCT images provide microscopic tomographic structure information of the retina and deeper tissues. This fusion of multimodal data significantly improves the comprehensiveness and accuracy of the assessment of eye health, thereby improving the accuracy of predicting vision changes.

[0143] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the anti-VEGF efficacy prediction method as described above.

[0144] The computer-readable storage medium provided by this application combines two different types of medical image data, fundus color photography and OCT (optical coherence tomography) images, so that the system can capture information about the health status of the eyes from multiple dimensions. Fundus color photography mainly reflects the macroscopic structure and color changes of the retina, while OCT images provide microscopic tomographic structure information of the retina and deeper tissues. This fusion of multimodal data significantly improves the comprehensiveness and accuracy of the assessment of eye health, thereby improving the accuracy of predicting vision changes.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0146] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for predicting the efficacy of anti-VEGF, characterized in that: The steps include: Obtaining the original fundus color photograph and original OCT image to be predicted; Inputting the original fundus color photograph into a fundus color photograph prediction model to perform a fundus color photograph prediction operation to obtain a predicted fundus color photograph; Performing a fundus color photo splicing operation on the original fundus color photo and the predicted fundus color photo to obtain a spliced ​​fundus color photo; Inputting the original OCT image into the OCT image prediction model to perform an OCT image prediction operation to obtain a predicted OCT image; Perform an OCT image stitching operation on the original OCT image and the predicted OCT image to obtain a stitched OCT image; The spliced ​​fundus color photograph and the spliced ​​OCT image are input into a vision change prediction model to perform a vision change prediction operation to obtain a vision change prediction result.

2. The method for predicting the anti-VEGF efficacy according to claim 1, characterized in that: The fundus color photo prediction model consists of a fundus color photo residual module, a blood vessel extraction module and a fundus color photo global feature extraction module.

3. The method for predicting the anti-VEGF therapeutic effect according to claim 2, characterized in that: The predicted fundus color photo I pre-Fundus It is expressed as: and pre-Fundus =CA(F Fundus +ReLu(Conv(Cat(V Fundus ,I Fundus )))+G Fundus ) G Fundus =G global (F Fundus ) V Fundus =H(I Fundus ) F Fundus =ResBlock5(...(ResBlock(ReLu(Conv(Cat(V Fundus ,I Fundus )))) Among them, CA represents channel attention, F Fundus represents the feature map after the fundus color photo residual module is processed, V Fundus represents the high-frequency feature map extracted by the blood vessel extraction module, G Fundus represents the global feature map output by the fundus color photo global feature extraction module, ReLu represents the activation function, Conv represents the convolution operation, and Cat represents the splicing along the channel dimension.

4. The method for predicting the anti-VEGF efficacy according to claim 2, characterized in that: The blood vessel extraction module H(I) is expressed as: H(I)=II*gaussion(r,σ) Wherein, H represents the blood vessel extraction module, I represents the input image, and gaussion(r,σ) represents a low-pass Gaussian filter with a radius of r and a spatial constant of σ.

5. The method for predicting the anti-VEGF efficacy according to claim 1, characterized in that: The OCT image prediction model consists of a downsampling module, an OCT image residual module, an upsampling module and an OCT image global feature extraction module.

6. The method for predicting the anti-VEGF therapeutic effect according to claim 5, characterized in that: The predicted OCT image I pre-OCT It is expressed as: and pre-OCT =ReLu(Conv(ReflectionPad Cat(G OCT ,U OCT ))) G OCT =G global (F OCT ) IN OCT =UpSample3(UpSample2(UpSample1(F OCT ))) F OCT =ResBlock5(...(ResBlock(D OCT ))) D OCT =DownSample3(DownSample2(DownSample1(ReLu(InstanceNorm(Conv(ReflectionPad(I OCT ))))))) Among them, D OCT Represents the OCT image after the downsampling module, DownSample i represents the downsampling operation, F OCT Represents the OCT feature map after the OCT image residual module is processed, UpSanple i represents the upsampling operation, U OCT The OCT image after the upsampling module, ReflectionPad represents reflection padding, G OCT Represents global feature processing of OCT images.

7. The method for predicting the anti-VEGF therapeutic effect according to claim 2 or 5, characterized in that: The fundus color photo global feature extraction module or the OCT image global feature extraction module is expressed as: GFB(I)=Sigmoid(Conv(Cat(Max(I),Mean(I))))*I Among them, GFB represents the global feature processing module, I represents the input feature image, Sigmoid represents the nonlinear activation layer, Conv represents the convolution operation, Cat represents the splicing operation along the channel dimension, Max represents the maximum pooling, and Mean represents the average pooling.

8. An anti-VEGF therapeutic effect prediction device, characterized in that: include: The original data acquisition module is used to obtain the original fundus color photograph and original OCT image to be predicted; A fundus color photo prediction module, used for inputting the original fundus color photo into the fundus color photo prediction model to perform a fundus color photo prediction operation to obtain a predicted fundus color photo; A fundus color photo stitching module, used for performing a fundus color photo stitching operation on the original fundus color photo and the predicted fundus color photo to obtain a stitched fundus color photo; An OCT image prediction module, used for inputting the original OCT image into the OCT image prediction model to perform an OCT image prediction operation to obtain a predicted OCT image; An OCT image stitching module, used for performing an OCT image stitching operation on the original OCT image and the predicted OCT image to obtain a stitched OCT image; The vision change prediction module is used to input the spliced ​​fundus color photograph and the spliced ​​OCT image into the vision change prediction model to perform a vision change prediction operation to obtain a vision change prediction result.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the anti-VEGF efficacy prediction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the anti-VEGF efficacy prediction method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Anti-vascular endothelial growth factor VEGF curative effect prediction device and method thereof

    CN111920375A