Methods, devices, electronic devices and readable storage media for identifying fundus lesions

Abstract

A method and apparatus for identifying fundus lesions are provided. The method includes: acquiring OCT fundus images of a specific patient; using a first neural network model to determine the boundary information of retinal tissue in the OCT fundus images, and segmenting the OCT fundus images into multiple image blocks based on the boundary information of the retinal tissue; providing the multiple image blocks to a second neural network model to obtain the fundus lesion type of the specific patient; the second neural network model calculating the feature vector of each image block and the attention weight of each image block for the specified fundus lesion type, and obtaining the fused feature vector through weighted calculation to obtain the identification result for the specified fundus lesion type. The second neural network model combines spatial saliency and scanning position saliency to enhance important information in the image block group and filter out unimportant information, thereby achieving low-cost model training and application.

Description

Technical Field

[0001] This disclosure relates to the technical field of combining artificial intelligence with medical image processing, and in particular to a method, apparatus, electronic device and readable storage medium for identifying fundus lesions. Background Technology

[0002] Optical coherence tomography (OCT) imaging technology can perform detailed scans of the layered structure of the retina, obtaining OCT fundus images that reflect the shape, thickness, and other morphological structural information of each layer of the retina. Based on these OCT fundus images, intelligent image processing can assist doctors in diagnosing fundus lesions. This not only improves the efficiency of disease diagnosis to a certain extent but also significantly enhances the accuracy of diagnosis.

[0003] In recent years, with the rise of deep learning, researchers have begun to apply neural network models to the medical field, and deep learning models have been widely used in the detection and diagnosis of fundus lesions. Compared with traditional rule-based disease diagnosis, deep learning models have the advantages of stronger generalization ability and less reliance on rule design or feature engineering.

[0004] However, current methods for detecting and diagnosing fundus lesions based on OCT fundus images combined with deep learning models require detailed annotation of lesion locations on the OCT fundus images (target bounding boxes or dense pixel-level annotations). Clinically available OCT fundus images mostly only have diagnostic-level annotations (i.e., the OCT fundus image of a particular patient only indicates the type of fundus lesion, without detailed target bounding boxes). Furthermore, when performing CT scans, cross-sectional images of the fundus are acquired at regular intervals. Each patient actually receives a set of images, and the lesion may only exist in one or a few of these images.

[0005] Therefore, given the situation where a single patient corresponds to a set of OCT images and there is only weak supervision signal, a more suitable model structure needs to be designed in order to achieve lower annotation costs. Summary of the Invention

[0006] In view of this, the present disclosure aims to provide a method, apparatus, electronic device and readable storage medium for identifying fundus lesions based on OCT fundus images.

[0007] According to a first aspect of this disclosure, a method for identifying fundus lesions is provided, comprising:

[0008] Obtain OCT fundus images of specific patients;

[0009] The boundary information of the retinal tissue in the OCT fundus image is determined using a first neural network model, and the OCT fundus image is divided into multiple image blocks based on the boundary information of the retinal tissue.

[0010] The multiple image blocks are provided to the second neural network model to obtain the fundus lesion identification result of the specific patient. The second neural network model calculates the feature vector of the multiple image blocks and the attention weight of each image block for the set fundus lesion type. The fused feature vector is obtained by weighted calculation, and the identification result for the set fundus lesion type is obtained based on the fused feature vector.

[0011] In some embodiments, there are multiple types of fundus lesions, so the second neural network model obtains multiple fused feature vectors for the multiple types of fundus lesions, and obtains multiple recognition results based on the multiple fused feature vectors.

[0012] In some embodiments, dividing the OCT fundus image into multiple image blocks includes: uniformly dividing the OCT fundus image with overlapping edges based on the boundary information of the retinal tissue to obtain multiple image blocks of equal size.

[0013] In some embodiments, the first neural network model is a U-Net model.

[0014] In some embodiments, the second neural network model includes: a feature extraction layer, an attention layer, a feature fusion layer, and a classification layer.

[0015] In the feature extraction layer, the feature vector of each image patch is extracted;

[0016] In the attention layer, the attention weight of each feature vector for the specified fundus lesion type is calculated;

[0017] In the feature fusion layer, the fused feature vector is obtained through weighted calculation;

[0018] In the classification layer, the identification result for the set fundus lesion type is obtained based on the fused feature vector.

[0019] In some embodiments, the feature extraction layer includes: a plurality of convolutional layers, at least one pooling layer, and at least one first fully connected layer, wherein the convolutional layers are used for convolution calculation, the pooling layer is used for outputting a feature map, and the first fully connected layer is used for outputting a one-dimensional feature vector based on the feature map.

[0020] In some embodiments, the attention layer includes an attention module for learning corresponding attention weights for the specified fundus lesion type.

[0021] In some embodiments, the feature fusion layer employs a second fully connected layer to achieve the fused feature vector through weighted calculation.

[0022] In some embodiments, the classification layer uses a normalization operation to convert the fused feature vector into a probability value for the specified fundus lesion type.

[0023] According to a second aspect of this disclosure, a fundus lesion identification device is provided, comprising:

[0024] The image segmentation module is used to acquire OCT fundus images of specific patients, use a first neural network model to determine the boundary information of the retinal tissue in the OCT fundus image, and segment the OCT fundus image into multiple image blocks according to the boundary information of the retinal tissue.

[0025] An image recognition module is used to provide the multiple image blocks to a second neural network model to obtain the fundus lesion recognition result of the specific patient. The second neural network model calculates the feature vectors of the multiple image blocks and the attention weight of each image block for a set fundus lesion type. The fused feature vector is obtained by weighted calculation, and the recognition result for the set fundus lesion type is obtained based on the fused feature vector.

[0026] According to a third aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program executable on the processor, wherein the processor executes the program to perform the fundus lesion identification method described in any of the preceding claims.

[0027] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the fundus lesion identification method described in any of the preceding claims.

[0028] This disclosure uses a second neural network model to identify fundus lesions based on image patch groups. The second neural network model utilizes an attention mechanism to learn the attention weights of each image patch within an image patch group belonging to the same patient. This allows for the enhancement of important information within the image patch group by combining spatial saliency and scan position saliency, while filtering out unimportant information, thus achieving low-cost model training and application. Attached Figure Description

[0029] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0030] Figure 1A structural diagram illustrating the model training and application scenarios of this disclosure;

[0031] Figure 2 This is a schematic diagram of the structure of an exemplary electronic device;

[0032] Figure 3 This is a flowchart of the fundus lesion identification method provided in the embodiments of this disclosure;

[0033] Figure 4 yes Figure 3 A flowchart of an embodiment of step 302 in the process;

[0034] Figure 5 This is a schematic diagram of a U-shaped network structure;

[0035] Figure 6 This is a schematic diagram of the structure of a second neural network model used in an embodiment of this disclosure;

[0036] Figure 7 This is a more specific structural diagram of the second neural network model used in an embodiment of this disclosure;

[0037] Figure 8 This is a schematic diagram of the structure of a fundus lesion identification device provided in an embodiment of this disclosure. Detailed Implementation

[0038] The present disclosure is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present disclosure below, certain specific details are described in detail. Those skilled in the art will fully understand the present disclosure even without these details. To avoid obscuring the substance of the present disclosure, well-known methods, processes, and procedures are not described in detail. Furthermore, the accompanying drawings are not necessarily drawn to scale.

[0039] Before providing a detailed description of the various embodiments of this disclosure, the knowledge used therein will be explained as follows.

[0040] The retinal tissue has a layered structure consisting of: internal limiting membrane, nerve fiber layer, ganglion cell layer, internal lamina, internal granular layer, external lamina, external granular layer, inner segment, outer segment, and retinal pigment epithelium. In terms of light and dark perception, the nerve fiber layer, the area between the inner and outer segments, and the retinal pigment epithelium are significantly brighter, while the other layers are relatively darker.

[0041] OCT imaging technology works by detecting light traveling through the refractive media of the examined eye to the retina, and then reconstructing an image based on the tissue thickness and distance information provided by reflections at different tissue interfaces. Specifically, in normal retinal tissue, highly reflective layers include the nerve fiber layer (RNFL), the intra- and extra-segmental junction layer (IS / OS), and the retinal pigment epithelium (RPE); moderately reflective layers are mainly the plexiform layer; and weakly reflective layers include the bipolar cell layer and other nuclear layers and photoreceptor cell layers. However, abnormal retinal tissue exhibits abnormal reflections and morphological changes due to pathological alterations. Examples include strong reflections from choroidal neovascularization (CNV), and foveal bulging or deformation due to edema or traction (macular thickening). These abnormal changes are reflected in pathological OCT fundus images. Therefore, deep learning can be performed based on normal and pathological OCT fundus images to identify various types of fundus lesions. It should be understood that the types of fundus lesions discussed in this article refer to physiological lesions occurring in the fundus tissues. These lesions may or may not develop into a disease. Therefore, fundus lesions are not equivalent to diseases.

[0042] In the medical classification of OCT images, pathological images are currently mainly classified into four types of lesions: IRF (intraretinal fluid), SRF (subretinal fluid), PED (pigment epithelial detachment), and HRF (hyperreflective foci).

[0043] Application scenarios and system architecture

[0044] Figure 1 A structural diagram illustrating the model training and application scenarios of this disclosure. The diagram includes a training device 110 and an execution device 120. The training device 110 and the execution device 120 communicate via a network 101. The network 101 is one or a combination of various communication technologies based on switched signals, including but not limited to wired technologies using electrical and / or optical cables, and wireless technologies using infrared, radio frequency, and / or other forms. In different application scenarios, the network 101 can be the Internet, a wide area network (WAN), or a local area network (LAN), such as a company's private network. The network 101 can also be a wired network or a wireless network. The training device 110 and the execution device 120 can have, for example, […]. Figure 2 The hardware structure shown.

[0045] like Figure 1As shown, the training device 110 stores a model 111 to be trained, which is trained by the model training device 112. The training process can be described as follows: a large number of training samples 114 are provided to the model 111 to be trained, and the model 111 to be trained is repeatedly executed. The model 111 to be trained continuously generates result data. Based on the loss between the expected result data and the actual result data, the weight parameters of the model 111 to be trained are continuously adjusted until the loss between the expected result data and the actual result data meets the requirements. Here, the model training device 112 is a virtual device, which can be understood as a script program used to execute the model 111 to be trained and provide training samples to the model 111 to be trained. The model obtained after training stops is called the trained model 113. The training device 110 provides the trained model 113 to the model application system 121 running on the execution device 120 for corresponding processing.

[0046] Specifically, in the embodiments of this disclosure, since the fundus lesion recognition system (or device) of this disclosure uses two neural network models: a first neural network model for segmenting OCT fundus images into image block groups and a second neural network model for identifying lesion types based on image block groups, the model to be trained 111 here refers to both the first neural network model and the second neural network model to be trained. That is, the first neural network model and the second neural network model to be trained are trained through model training to obtain the trained first neural network model and the second neural network model, and then applied to the fundus lesion recognition system.

[0047] Figure 2 This is a schematic diagram of an exemplary electronic device 200. As shown in the figure, the server 200 includes a scheduler 201, a storage unit 203, an I / O interface 204, and multiple model acceleration units 202, all coupled via a bus 205.

[0048] Storage unit 203 may include readable media in the form of volatile storage units, such as random access memory (RAM) and / or cache memory units. Storage unit 203 may also include readable media in the form of non-volatile storage units, such as read-only memory (ROM), flash memory, and various disk drives.

[0049] Storage unit 203 can store various program modules and data, including operating systems and application programs that provide functions such as text processing, video playback, software editing, and compilation. The executable code of these application programs is read from storage unit 203 and executed by scheduler 201 to implement the predetermined operations of these program modules. Scheduler 201 is generally a processor (CPU).

[0050] Bus 205 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0051] Server 200 can communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth device, etc.), and also with one or more devices that enable users to interact with server 200, and / or with any device that enables server 200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 204. Furthermore, server 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via a network adapter (not shown). For example, via a network adapter, Figure 1 Terminal 103 can access server 200. It should be understood that, although not shown in the figure, server 200 may use other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0052] As shown in the figure, server 200 includes multiple model acceleration units 202. Traditional processor architectures are highly efficient in logic control but inefficient in large-scale parallel computing, making them inefficient for model computation. To address this, model acceleration units have been developed, with different models adaptable to different units. For example, a neural network acceleration unit (NPU) can be used. The NPU employs a data-driven parallel computing architecture to handle the large number of operations (such as convolution and pooling) of each neural network node. Alternatively, a graphics processing unit (GPU) can be used, dedicated to image and graphics-related computations. Because the GPU utilizes a large number of dedicated computing units for graphics calculations, it reduces the GPU's dependence on the CPU, taking over some of the computationally intensive graphics and image processing tasks previously handled by the CPU, thus significantly improving the efficiency of image data processing. Multiple model acceleration units 202 are controlled by a scheduler 201, enabling them to work collaboratively.

[0053] Combination Figure 1 As shown, the training device 110 can adopt the following... Figure 2In this hardware structure, the scheduler 201 controls the training process, distributing the execution of model 111 to multiple model acceleration units 202 and aggregating the execution results of these units. Data interaction between the multiple model acceleration units 202 can be achieved through the scheduler 201 or directly through the units themselves. Similarly, the execution device 120 can also adopt a similar hardware structure. Figure 2 The hardware structure of the model application system is such that the execution of the system is completed collaboratively by the scheduler 201 and multiple model acceleration units 202.

[0054] The present invention discloses a method and device for identifying fundus lesions.

[0055] Figure 3 This is a flowchart of a method for identifying fundus lesions according to an embodiment of the present disclosure. As shown in the figure, it includes the following steps.

[0056] In step S301, OCT fundus images of a specific patient are acquired.

[0057] In step S302, the first neural network model is used to determine the boundary information of the retinal tissue in the OCT fundus image, and the OCT fundus image is divided into multiple image blocks according to the boundary information of the retinal tissue.

[0058] In step S303, multiple image blocks are provided to the second neural network model to obtain the fundus lesion identification results for the specific patient.

[0059] In this embodiment, the second neural network model calculates the feature vectors of each image block of the specific patient and the attention weights of each image block for the set fundus lesion type, and obtains the fused feature vector by weighted calculation to obtain the recognition result of the patient for the set fundus lesion type.

[0060] One or more types of fundus lesions can be defined. For example, if the fundus lesion type is defined as IRF, then each image patch belonging to that patient will generate an attention weight for IRF, and a fused feature vector will be obtained through weighted calculation. The IRF identification result for that patient will then be determined based on the fused feature vector, indicating whether the patient has IRF or not. Correspondingly, when the four fundus lesion types are defined as IRF, SRF, PED, and HRF, each image patch of that patient will calculate an attention weight for IRF, SRF, PED, and HRF respectively. Then, a fused feature vector for IRF, SRF, PED, and HRF will be obtained through weighted calculation, and the identification results for IRF, SRF, PED, and HRF for that patient will be obtained based on the four fused feature vectors. It should be noted that multiple predefined fundus lesion types are not mutually exclusive on OCT fundus images. Therefore, it is possible to determine from a patient's OCT fundus images that the patient has multiple predefined fundus lesions, such as the patient having both IRF and SRF.

[0061] It should be understood that, due to the spatial importance differences in raw OCT fundus images, retinal tissue is the location of lesions, while noise and holes above and below the retina are useless or even detrimental to classification results. Directly using raw OCT fundus images without a strategy will hardly achieve ideal classification results. Therefore, this embodiment uses a deep learning model to learn an attention weight for each image patch to represent the spatial importance of the OCT fundus image. Then, the feature vector obtained by weighted summation is used for fundus lesion classification, thereby improving classification accuracy.

[0062] In some embodiments, step S302 may employ methods such as Figure 4 The implementation method includes sub-steps S3021 and S3022. Sub-step S3021 uses a U-shaped network structure (U-Net model) to obtain the boundary information of retinal tissue in the OCT fundus image. Sub-step S3022 divides the OCT fundus image into multiple image blocks in an overlapping manner (edge ​​overlap) based on the boundary information.

[0063] Preferably, the U-net neural network model is increased to five layers, which expands the receptive field and improves image feature extraction. This U-net neural network model includes an Encoder path and a Decoder path. The Encoder path is responsible for feature extraction, extracting relevant features from the region of interest. The Decoder path is responsible for restoring the extracted features to correspond with the original image. (Reference) Figure 5The left side shows the Endcoder encoding path, and the right side shows the Decoder decoding path. The OCT fundus image has a resolution of 600*600*3. This image first undergoes five downsampling layers in the Endcoder encoding path. With each downsampling layer, the resolution is reduced to half of the input, and the number of channels gradually increases, thus obtaining the features. Figure 1 ,feature Figure 2 ,feature Figure 3 ,feature Figure 4 ,feature Figure 5 Then, after five upsampling layers in the decoder path, the resolution of the feature map increases to twice that of the input with each upsampling layer, and the number of channels gradually decreases, thus obtaining the feature map. Figure 6 ,feature Figure 7 ,feature Figure 8 ,feature Figure 8 The weight parameters in each sample are updated by calculating the loss between the original image and the feature map. It should be noted that both upsampling and downsampling in this network structure can be implemented based on the convolutional blocks of a convolutional neural network (CNN). The convolutional blocks of a CNN can include functional units such as convolution, pooling, and activation, and the corresponding functional units can be arbitrarily combined according to actual needs.

[0064] In some embodiments, such as Figure 6 As shown, the second neural network model includes a feature extraction layer 601, an attention layer 602, a feature fusion layer 603, and a classification layer 604.

[0065] In the feature extraction layer 602, K image patches forming an image patch group 601 are input to the feature extraction layer 602, resulting in K feature vectors h corresponding to the K image patches. Each feature vector has a dimension of 1×n. In some embodiments, the feature extraction layer includes multiple convolutional layers, at least one pooling layer, and at least one fully connected layer. In this structure, convolutional operations are used to extract different features of the image. Multiple convolutional layers can sequentially extract low-level and high-level image features. The pooling layer downsamples the image features (i.e., low-level and high-level image features), thereby compressing the data and parameters of the image features while maintaining the invariance of the image features. Each node of the fully connected layer is connected to all nodes of the previous layer, used to synthesize the final features extracted by the previous layer (i.e., the downsampled image features) to obtain and output a one-dimensional feature vector.

[0066] At the attention layer, attention weights for each image patch are calculated for each fundus lesion type. Specifically, if there are four fundus lesion types: IRF, SRF, PED, and HRF, then for each of IRF, SRF, PED, and HRF, attention weights are calculated using two learnable parameter matrices V∈R. l×n and U∈R l×nThe attention weights for each image patch are calculated, where two learnable parameter matrices serve as data dimensionality reduction and a mask. After transforming the image patch into a feature vector of dimension l×1, nonlinear transformations are performed using the tanh and sigmoid functions, respectively. The resulting two new feature vectors are then multiplied element-wise and fed into a fully connected layer (parameter w∈R). l×1 Finally, the attention weights for each of the K image blocks in the image group will be obtained through the Softmax operation. See formula (1) for a specific example.

[0067]

[0068] Where K represents the number of image patches, and k is the index of the current image patch. and For matrix transpose, h k and h j It is a feature vector of dimension l×1.

[0069] In some embodiments, the attention layer includes an attention module equal to the number of fundus lesion types. Each attention module calculates its attention weight using formula (1). Referring to formula (1), and still setting four fundus lesion types—IRF, SRF, PED, and HRF—then it is necessary to learn, through training, the two learnable parameter matrices V∈R in each of the four formulas (1) of the four attention modules. l×n and U∈R l×n That is, the parameter matrix V∈R has a total of 8 elements. l×n and U∈R l×n This needs to be learned. At the beginning of training, two of the four formulas in the four attention modules have learnable parameter matrices V∈R. l×n and U∈R l×n They can be the same, but during and after training, the parameter matrix V∈R l×n and U∈R l×n The differences become apparent. In other words, when the model is actually applied, different attention weights are calculated using different parameter matrices for different types of fundus lesions, even for the same group of image patches.

[0070] In the feature fusion layer, the feature vectors of each image block and the attention weights are weighted and calculated to obtain the fused feature vector. Equation (2) shows the calculation formula for obtaining the fused feature vector by weighted summation.

[0071]

[0072] Among them, a k The attention weights for the k image patches output by formula (1) are given by k and j, where k and j are indices.

[0073] In other embodiments, the feature fusion layer can also calculate the fused feature vector by weighting and then finding the maximum value or by weighting and then averaging.

[0074] At the classification layer, the final classification can be performed based on the fused feature vectors. Through the feature fusion layer, assuming there are four fundus lesion types—IRF, SRF, PED, and HRF—four fused feature vectors corresponding to these four fundus lesion types are obtained based on a patient's image patch set. These four fused feature vectors are then converted into identification results to determine whether the patient has the corresponding fundus lesion type. These four identification results are then superimposed as the final result for the patient.

[0075] When training the second neural network model, multiple image patch groups from multiple patients are used as training samples. Since each patient only has a label for the diagnosis level of that patient (i.e., the OCT image of that patient only indicates what kind of fundus lesion the patient has), each training sample also includes the diagnosis result corresponding to the image patch group. The classification layer uses cross-entropy as the loss function and continuously updates the weight parameters according to the loss function.

[0076] Figure 7 This is a more specific structural diagram of the second neural network model used in an embodiment of this disclosure. Referring to the figure, the input to the feature extraction layer 602 is an image patch group 601 obtained by segmentation based on the first neural network model. Each image patch in the image patch group 601 needs to pass through the feature extraction layer 602, that is, through convolutional layer 1, convolutional layer 2 to convolutional layer n for convolutional operation, and then through pooling layer for dimensionality reduction operation to output a feature map. Then the feature map passes through the first fully connected layer to output a feature vector. It should be noted that if the image patch group 601 has K image patches, then K feature vectors corresponding to each of the K image patches should be output. The attention layer 603 adopts an attention mechanism to output the attention weight of each image patch for a specific type of fundus lesion. It should be noted that if the image patch group 601 has K image patches, then K attention weights corresponding to each of the K image patches should be output. Then the K attention weights and K feature vectors are provided to the second fully connected layer, and the fused feature vector is obtained through feature fusion. Feature fusion can be multiplication, addition or concatenation. Finally, the fused feature vectors are used to provide a normalization layer, which yields the probability of a specific eye disease. The normalization layer maps the output of the previous layer (the second fully connected layer) to the (0, 1) interval, thus obtaining the probability that the fused features are classified into a specific fundus lesion type. Based on the probability of that specific fundus lesion type, the identification result for that fundus lesion type is obtained and output. This process continues; if there are N such fundus lesion types, then after... Figure 7The process shown yields corresponding recognition results. The final recognition result for the corresponding patient is obtained by superimposing the N eye disease recognition results.

[0077] It should be pointed out that, Figure 7 The model structure shown is an improvement on the CNN model. Existing techniques directly feed the entire OCT fundus image into the CNN model to obtain fundus lesion identification results, and... Figure 7 Compared to the previous method, the existing processing method requires higher computational resources or has lower performance with the same computational resources because it processes the entire image each time. Moreover, since each image is provided to the CNN model as a whole, it cannot distinguish the differences in spatial importance and scanning position of different parts of the image.

[0078] Accordingly, one embodiment of this disclosure provides a fundus lesion identification device, including as follows: Figure 8 The image segmentation module 801 and the image recognition module 802 are shown.

[0079] The image segmentation module 801 is used to acquire OCT fundus images of specific patients, determine the boundary information of retinal tissue in the OCT fundus images based on the first neural network model, and segment the OCT fundus images into multiple image blocks according to the boundary information of retinal tissue.

[0080] The image recognition module 802 provides multiple image blocks to the second neural network model to obtain the fundus lesion recognition result for a specific patient. The second neural network model calculates the feature vectors of the multiple image blocks and the attention weight of each image block for a set fundus lesion type. The fused feature vector is obtained by weighted calculation, and the recognition result for the set fundus lesion type is obtained based on the fused feature vector.

[0081] In some embodiments, there are multiple types of fundus lesions, and the second neural network model outputs multiple recognition results of fundus lesion types, thereby superimposing the recognition results of multiple types of fundus lesion types as the final recognition result for the specific patient.

[0082] In some embodiments, the image segmentation module 801 performs uniform, edge-overlapping segmentation of the OCT fundus image based on the boundary information of the retinal tissue to obtain multiple image blocks of equal size.

[0083] In some embodiments, the fundus lesion recognition device further includes an image preprocessing module. The image preprocessing module may perform operations such as noise reduction and grayscale conversion.

[0084] In some embodiments, the second neural network model includes: a feature extraction layer, an attention layer, a feature fusion layer, and a classification layer. In the feature extraction layer, feature vectors of each image patch are extracted. In the attention layer, attention weights for each feature vector are calculated for a given type of fundus lesion. In the feature fusion layer, the feature vectors and attention weights of each image patch are weighted and summed to obtain a fused feature vector for a specific patient. In the classification layer, the fundus lesion identification result for a specific patient is obtained based on the fused feature vector.

[0085] In summary, the embodiments of this disclosure use a second neural network model for fundus lesion identification. The second neural network model utilizes an attention mechanism to learn the attention weights of each image patch in an image patch group belonging to the same patient. This allows for the enhancement of important information in the image patch group by combining spatial saliency and scan position saliency, while filtering out unimportant information, thus achieving low-cost model training and application.

[0086] In addition, embodiments of this disclosure also provide a computer-readable medium for storing computer-readable instructions for implementing the above-described fundus recognition method.

[0087] The disclosed commercial value

[0088] This disclosure provides a method for identifying fundus lesions. Compared to existing technologies, the second neural network model in this method utilizes an attention mechanism to learn the attention weights of each image patch in an image patch group belonging to the same patient. This allows for the enhancement of important information in the image patch group by combining spatial saliency and scan position saliency, while filtering out unimportant information. Therefore, this method has practical application and economic value for fundus lesion identification and diagnosis.

[0089] It should be understood that the above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. For those skilled in the art, there are many variations of the embodiments in this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

[0090] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0091] It should be understood that the foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0092] It should be understood that the use of a singular form to describe an element or to show only one element in the accompanying drawings does not imply that the number of such element is limited to one. Furthermore, modules or elements described or shown as separate herein may be combined into a single module or element, and modules or elements described or shown as single herein may be broken down into multiple modules or elements.

[0093] It should also be understood that the terminology and expressions used herein are for descriptive purposes only, and one or more embodiments described herein should not be limited to these terms and expressions. The use of these terms and expressions does not exclude any illustrative and descriptive equivalent features (or parts thereof), and it should be recognized that various modifications that may exist should also be included within the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be considered to cover all such equivalents.

Claims

1. An eye fundus lesion identification method, comprising: obtaining an OCT eye fundus image of a specific patient; determining boundary information of retinal tissue of the OCT eye fundus image by using a first neural network model, and performing uniform and edge-overlapping segmentation on the OCT eye fundus image according to the boundary information of the retinal tissue to obtain a plurality of image blocks of equal size; providing the plurality of image blocks to a second neural network model to obtain an eye fundus lesion identification result of the specific patient, the second neural network model calculating a feature vector of each of the plurality of image blocks and an attention weight of each image block for a set eye fundus lesion type, obtaining a fused feature vector by weighted calculation, and obtaining an identification result for the set eye fundus lesion type according to the fused feature vector, the set eye fundus lesion type being multiple, and each image block corresponding to an attention weight for multiple set eye fundus lesion types.

2. The method of identifying ocular fundus lesions according to claim 1, wherein, The second neural network model obtains a plurality of fused feature vectors for a plurality of set eye fundus lesion types, and obtains a plurality of identification results according to the plurality of fused feature vectors.

3. The method of identifying ocular fundus lesions according to claim 1, wherein, The first neural network model is a U-Net model.

4. The method of identifying ocular pathology according to claim 1, wherein, The second neural network model comprises a feature extraction layer, an attention layer, a feature fusion layer, and a classification layer, In the feature extraction layer, a feature vector of each image block is extracted; In the attention layer, an attention weight of each feature vector for the set eye fundus lesion type is calculated; In the feature fusion layer, the fused feature vector is obtained by weighted calculation; In the classification layer, an identification result for the set eye fundus lesion type is obtained according to the fused feature vector.

5. The method of identifying ocular pathology according to claim 4, wherein, The feature extraction layer comprises a plurality of convolution layers, at least one pooling layer, and at least one first fully connected layer, the convolution layers are used for convolution calculation, the pooling layer is used for outputting a feature map, and the first fully connected layer is used for outputting a one-dimensional feature vector based on the feature map.

6. The method of identifying ocular fundus lesions according to claim 4, wherein, The attention layer comprises an attention module to learn corresponding attention weights for the set eye fundus lesion types in the attention module.

7. The method of identifying ocular fundus lesions according to claim 4, wherein, The feature fusion layer adopts a second fully connected layer to realize the fused feature vector obtained by weighted calculation.

8. The method of identifying ocular fundus lesions according to claim 4, wherein, The classification layer adopts a normalization operation to convert the fused feature vector into a probability value for the set eye fundus lesion type.

9. An eye fundus lesion identification device, comprising: An image segmentation module is used to obtain an OCT eye fundus image of a specific patient, determine the boundary information of the retinal tissue of the OCT eye fundus image by using a first neural network model, and perform uniform and edge-overlapping segmentation on the OCT eye fundus image according to the boundary information of the retinal tissue to obtain a plurality of image blocks of equal size; An image recognition module is configured to provide the plurality of image blocks to a second neural network model to obtain an eye fundus lesion recognition result of the specific patient. The second neural network model is configured to calculate a feature vector of each of the plurality of image blocks and an attention weight of each image block for a set eye fundus lesion type. The second neural network model is further configured to obtain a fused feature vector by weighted calculation and obtain the recognition result for the set eye fundus lesion type according to the fused feature vector. The set eye fundus lesion type has a plurality of types, and each image block corresponds to an attention weight for each of the plurality of set eye fundus lesion types. 10.An electronic device comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor implements the eye fundus lesion recognition method according to any one of claims 1 to 8 when executing the program. 11.A computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the computer program is capable of implementing the eye fundus lesion recognition method according to any one of claims 1 to 8 when executed by a processor.

Images