A multi-lesion retinal segmentation method based on implicit cues

Abstract

The present invention claims protection for a multi-lesion retinal segmentation method based on implicit prompts, which mainly includes three-color domain feature enhancement, modulation module, implicit prompt module and prompt combination module. The method realizes data enhancement by converting the color space and selecting components therein for combination and combining with Gaussian blur. In order to reduce the impact of lesions on small blood vessels, the modulation module adopts adjacent jump connections with small contextual semantic differences to realize bottom-up modulation and realize feature fusion from coarse to fine. In order to effectively segment the lesions, the implicit prompt module uses prompt information combined with the encoded middle-level features to adaptively learn the lesion type, and the prompt combination module combines the prompt information with the jump connection in the decoding stage to realize reliable segmentation of various lesions. The implementation of the above measures significantly improves the segmentation accuracy of retinal blood vessels, strengthens the ability of the network to process blood vessels with large morphological differences in the context of lesions, and improves the accuracy of terminal capillary segmentation.

Description

Technical Field

[0001] The invention belongs to the technical field of retinal image feature extraction and segmentation methods, and in particular is directed to a multi-lesion retinal segmentation method based on implicit prompts. Background Art

[0002] Fully convolutional neural network (FCN) has become the mainstream in image segmentation, especially in the field of semantic segmentation. It has the characteristics of accepting input images of any size, high computing speed, and being able to achieve pixel-level image segmentation effects.

[0003] Since FCN was first proposed in 2015, a lot of research and experiments have been conducted. On this basis, a number of classic models have emerged, such as UNet and UNet-like structures. These models are based on the FCN structure and improve the accuracy of retinal vessel segmentation by increasing the number of network layers. However, for diseased retinas, vascular breakpoints, missing bifurcations, and structural misprediction are still difficult problems to solve. The current segmentation model cannot be well applied to the segmentation of diseased retinas, which is usually caused by the following reasons: ① The training samples are limited, especially the number of diseased retinal samples is small, which makes the segmentation model often have breakpoints and missing when facing diseased samples. ② The encoding of shallow skip connections is usually affected by severe lesions, which in turn affects the segmentation of small blood vessels, while the encoding of deep features often loses the semantics of small blood vessels and cannot supplement the contextual information of small blood vessels. ③ The decoding cannot perform corresponding segmentation according to the specific lesion type, especially when the samples are unbalanced, the model will be more inclined to segment non-lesion types.

[0004] Patent publication number CN116205870A adds deformable convolution, multi-scale adaptive detail feature fusion module and dual-path upsampling module to the U-Net network to achieve high-quality segmentation. This method has the following defects:

[0005] (1) This method uses deformable convolutions frequently, which results in many parameters, large amount of computation, and long training time, increasing the training cost and difficulty, making it difficult to deploy.

[0006] (2) This method uses a multi-scale adaptive detail feature fusion module, which ignores the differences in contextual semantics at different scales. Simultaneous operations at multiple scales will lead to feature aliasing, affecting the segmentation results.

[0007] (3) This method uses deconvolution and PixelShuffle to jointly perform upsampling, which requires additional parameters to unify the channels. Deconvolution is used at different step sizes, and when the convolution kernel is learned unevenly, there is a high probability of a grid effect.

[0008] The starting point of the present invention is completely different from CN116205870A. First, the present method enriches samples by fusing three color spaces, further artificially adds degradation information, and alleviates sample imbalance. Secondly, the network is guided adaptively to various lesions by implicit prompts, and the segmentation of the lesion retina is completed. Structurally, the present method mainly utilizes the small semantic difference of adjacent skip connections to gradually reduce the impact of lesions on shallow skip connections from the bottom up.

[0009] This method benefits from the following design:

[0010] (1) By introducing three-color space fusion for sample expansion, the clarity and contrast of the image are enhanced, data redundancy is reduced, the image details and color balance are better maintained, and sample imbalance is avoided.

[0011] (2) Feature modulation is performed by combining skip connections at adjacent scales to achieve coarse-to-fine feature fusion, and sub-pixel convolution is used to improve the receptive field of upsampling, which is beneficial to feature information at different levels, reduces interference from lesions, and improves segmentation accuracy and robustness.

[0012] (3) The adjacent modulation block uses GhostNet linear mapping to enrich samples and prompts the fusion module to split channels to improve the expressiveness of features, which is similar to group convolution and helps reduce network parameters for fast network reasoning and deployment. Summary of the invention

[0013] The present invention aims to solve the problems of the above prior art, including the segmentation accuracy of retinal blood vessels, the accuracy of terminal capillary segmentation, etc.

[0014] The present invention adopts a multi-lesion retinal segmentation method based on implicit prompts, expands samples by introducing three-color domain feature enhancement, artificially adds shadows and spots to increase the number of lesion samples, and alleviates sample imbalance; secondly, in order to alleviate the influence of lesions on the shallow features of the coding, a bottom-up modulation method is used to achieve coarse-to-fine fusion of contextual semantics of different scales; and by fusing the features of the middle scale, the weighting of the prompt information is completed, the distinction of different lesions is achieved, and prompt information is provided to act on the decoding stage, so that the network can adaptively achieve effective segmentation of different lesions. In terms of structure, the encoding-decoding architecture is used to encode the degraded image into the latent space. In the modulation process, a linear mapping similar to GhostNet and a decoding fusion prompt are used. The channel split structure is adopted to reduce network parameters and improve segmentation speed, making actual deployment possible.

[0015] In view of this, the technical solution adopted by the present invention is: a multi-lesion retinal segmentation method based on implicit prompts, comprising the following steps:

[0016] Step 1: After the data set is fused in three color spaces, lesion simulation, data enhancement and data set division operations are performed;

[0017] Step 2: Construct a segmentation network with an encoding-decoding architecture, in which the residual structure is used in the encoding stage to extract features and change the channels and sizes of the features, and the implicit hint module is used to extract features from the input lesion image and act on the decoding stage. At the same time, the skip connections of different scales are modulated to achieve feature fusion from coarse to fine. In the decoding stage, the hint information and the skip connection are combined to achieve reliable segmentation of various lesions.

[0018] Step 3: For the encoding-decoding architecture in step 2, use dice_loss to calculate the similarity between the network prediction value y and the label y' to evaluate the segmentation effect of the network on the validation set;

[0019] Step 4: Load the test image data in step 1 into the network and quickly test to complete the image segmentation.

[0020] Specifically, the simulation of lesions, data enhancement and data set division operations in step 1 include: randomly adding shadows and spots to the image to simulate lesions, further performing data enhancement by flipping and rotating, and dividing the enhanced image into patches to increase the number of training sets, and dividing the training set, validation set and test set in a ratio of 8:1:1.

[0021] Furthermore, the three-color space fusion of the data set includes converting the RGB image into YCrCb, YUV, and LAB spaces, and randomly selecting a spatial component therefrom and combining it with a component of the RGB space to realize the color space fusion, and performing a weighted average algorithm on each component, and its expression is:

[0022] F Blend =α×T 1 (M) m +β×T 2 (M) n +γ×M k

[0023] Where M is the input feature, m,n,k∈[1,2,3] is the component index of the corresponding color space, α+β+γ=1, T 1 ,T 2 is the transformation operation of the color space, T(M) m Select a component of the color space with index m from it, M k Represents the component with index k in the RGB space, F Blend It represents the characteristics after the three color domains are fused.

[0024] Furthermore, the simulated lesion specifically includes: randomly generating strip and / or circular lesion areas, smoothing the edges of the areas in combination with a Gaussian model, applying a Gaussian kernel function to each pixel in the lesion area, and taking a weighted average of the surrounding pixels to obtain an image with blurred edges, the expression of which is as follows:

[0025]

[0026] Among them, R Lesion represents the lesion area, (x, y) is the pixel coordinate of the image, (i, j) is the center pixel coordinate of the Gaussian kernel, and F blur (x, y) represents the image after the lesion edge is blurred;

[0027]

[0028] Where (m,n) is the offset from the kernel center and σ is the standard deviation of the Gaussian kernel.

[0029] Further, the patch division of the enhanced image includes randomly extracting patches from the training set and sequentially extracting patch maps from the test set, which is specifically:

[0030] For the training set, set the width size patch_w and height size patch_h of the patch image, and randomly select a pixel as the center of the patch. The coordinates of the pixel must meet the following conditions:

[0031]

[0032] Among them, x is the horizontal coordinate, y is the vertical coordinate, img_w is the width of the original image, and img_h is the height of the original image. Finally, with the pixel point as the center, a training image patch with a width of patch_w and a height of patch_h is intercepted from the original image;

[0033] For the test set, set patch_w and patch_h, and then determine whether the size of the original image is proportional to the size of the set patch image. If not, fill the original image with pixels. Finally, take the upper left corner of the input original image as the starting point and extract the patch image cyclically according to the principle of "from top to bottom and from left to right".

[0034] Furthermore, in the encoding stage, each stage is performed with two 3×3 convolution modules for feature extraction, and the features are selected into the skip connection. The downsampling operation with a step size of 2 follows the convolution. After each downsampling, the size of the feature map is reduced by one time, and the number of feature channels is doubled. The implicit prompt module unifies the scale of the medium-sized features by maximum pooling, and stacks them on the channel at the same time, and then compresses them into a one-dimensional weight vector through a compression operation. The self-learning implicit prompt is realized by weighting the specific degradation prompt vector, which is expressed as:

[0035]

[0036]

[0037] Where P Max represents the maximum pooling, P GAP represents global average pooling, || represents feature channel stacking, F represents input features, represents full connection, σ represents Sigmoid activation function, FS represents FS. The channel shuffle is used to strengthen the interaction between channels. The global average pooling is used to encode the spatial feature value of each pixel into a global feature to achieve flattening. The full connection is further used to reduce the dimension to the number of channels with implicit information. w It represents the weighted feature obtained by encoding features, and X represents the features of the input FS.

[0038] The implicit hint of self-learning is further realized by weighting the specific degradation hint vector, which is expressed as:

[0039] I prompt =F w ·F prompt

[0040] I prompt Represents the hint feature weighted by the weight feature. prompt Represents self-learned suggestive features that characterize lesions.

[0041] At the same time, due to the large scale difference of multi-scale jump connections, the modulation module uses jump connections of adjacent sizes for bottom-up modulation to achieve coarse-to-fine feature fusion to avoid excessive influence of lesion information encoding shallow features on the segmentation results, which can be expressed as:

[0042]

[0043] in represents the large-scale skip connection feature, is the scaling factor obtained by convolution, and λ is the translation factor obtained by convolution, which comes from the Ghost-Subpixel feature The calculation is as follows:

[0044]

[0045] Conv 1 、Conv 2 Respectively represent the convolution operation without sharing weights, features The skip connection from the next layer,

[0046]

[0047] in represents the skip connection feature with 2C channels, G represents Ghost linear mapping, Represents the characteristics The features after Ghost linear mapping, UP[X,k] means upsampling the input X by k times through sub-pixel, where k is the upsampling multiple, and here k is 2.

[0048] Furthermore, a hint combining module is added to the decoding stage, which generates a mask while reducing parameters through channel splitting, which is expressed as:

[0049]

[0050] Where M is the input feature, Conv k represents a convolution operation with a convolution kernel of k×k, M 1 ,M 2 The sub-features after splitting the channel, Split represents the channel splitting operation, and Mask represents the sub-features after splitting the channel. 2 The mask obtained by convolution.

[0051] Then, the prompt information and mask are dynamically fused and the channels are restored to the original number of channels. The 3×3 convolution is combined to further integrate the prompt and context information, which is expressed as:

[0052]

[0053]

[0054] Where || represents the stacking of feature channels, Conv k represents a convolution operation with a convolution kernel of k×k, I prompt It represents the hint feature weighted by the weight feature. Finally, after decoding, the channel is adjusted to a single-channel output, and then the loss is calculated with the label.

[0055] Furthermore, the dice_loss calculates the similarity between the network prediction value y and the label y', and the formula expression is:

[0056]

[0057] Where y∩y' represents the intersection between the predicted value y and the label y', which is usually approximated as the dot product y y' between the label and the predicted value, |y|+|y'| represents the element-by-element sum, and ε is a minimum value to avoid the denominator being 0.

[0058] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned implicit cue-based multi-lesion retinal segmentation method when executing the computer program.

[0059] The present invention finally provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned multi-lesion retinal segmentation method based on implicit cues are implemented.

[0060] The advantages and beneficial effects of the present invention are as follows:

[0061] Existing segmentation methods often face the following challenges when dealing with diseased retinal segmentation:

[0062] 1. Limited training samples: In particular, the number of diseased retinal samples is small, which makes the segmentation model prone to breakpoints and missing samples.

[0063] 2. The shallow skip connection of the encoding layer is easily affected by lesions: it may affect the segmentation of small blood vessels. At the same time, the deep features of the encoding layer may lose the semantics of small blood vessels and fail to supplement the contextual information of small blood vessels.

[0064] 3. The decoding cannot perform corresponding segmentation according to the specific lesion type: Especially in the case of unbalanced samples, the model will be more inclined to segment non-lesion types, resulting in poor segmentation of diseased retina.

[0065] In view of the above problems, the innovation of the present invention lies in the combination of step 1 and step 2. Specifically, the method achieves the effect of data enhancement by converting the color space and selecting the components therein for combination. This ingenious design avoids additional calculations while enhancing the clarity and contrast of the image. In addition, the present invention also uses Gaussian blur technology to provide smooth shadows and light spots for the image, thereby increasing the network's attention to the type of lesions. This combination method not only improves the image processing effect, but also reduces the complexity of the calculation, and has important practical value and broad application prospects.

[0066] Furthermore, in order to reduce the impact of lesions on small blood vessels, the present invention proposes a modulation module that uses adjacent skip connections with small context semantic differences to achieve bottom-up modulation. Through this modulation method, coarse-to-fine feature fusion can be achieved, thereby avoiding excessive feature aliasing and redundancy caused by multi-scale features, and more effectively extracting and processing small blood vessel information in the image, improving the accuracy and robustness of segmentation.

[0067] Furthermore, in order to effectively segment the lesions, the present invention proposes an implicit hint module. The module uses hint information combined with the encoded middle-level features to adaptively learn the lesion type. In the decoding stage, the hint combination module combines the mask features and hint information in parallel to achieve high-quality segmentation of the lesion retina, so that the model can better capture and utilize lesion information and improve the accuracy and stability of segmentation.

[0068] The implementation of the above improvement measures enables the present invention to significantly improve the segmentation accuracy of retinal blood vessels, strengthen the network's ability to process blood vessels with large morphological differences in the context of lesions, and improve the accuracy of terminal capillary segmentation. In addition, through the use of linear mapping enriched features and channel splitting technology, the present invention successfully optimizes model parameters, reduces the complexity of the model, and improves the speed of inference. Therefore, it is possible to further promote the application of this model in the field of medical image segmentation applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of the multi-lesion retinal segmentation method based on implicit hints provided by the present invention;

[0070] Figure 2 This is a model architecture diagram of a multi-lesion retinal segmentation method based on implicit cues provided by the present invention;

[0071] Figure 3 It is a structural diagram of an implicit prompt module of a multi-lesion retinal segmentation method based on implicit prompts provided by the present invention;

[0072] Figure 4 It is a structural diagram of a modulation module of a multi-lesion retinal segmentation method based on implicit cues provided by the present invention;

[0073] Figure 5 This is a hint-combined module structure diagram of the implicit hint-based multi-lesion retinal segmentation method provided by the present invention. DETAILED DESCRIPTION

[0074] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0075] like Figure 2 As shown, the model architecture of the multi-lesion retinal segmentation method based on implicit cues mainly includes a three-color domain feature fusion enhancement module, a modulation module, an implicit cue module and a cue combination module. The three-color domain feature fusion enhancement module achieves data enhancement by converting the color space and selecting components therein for combination and combining with Gaussian blur. In order to reduce the impact of lesions on small blood vessels, the modulation module uses adjacent jump connections with small contextual semantic differences to achieve bottom-up modulation and achieve feature fusion from coarse to fine. In order to effectively segment lesions, the implicit cue module uses cue information combined with encoded middle-level features to adaptively learn lesion types, and the cue combination module combines cue information with jump connections in the decoding stage to achieve reliable segmentation of various lesions.

[0076] like Figure 1 As shown, the multi-lesion retinal segmentation method based on implicit prompts provided in this embodiment includes the following steps:

[0077] Step 1: After the three color spaces are fused for the data set, shadows and light spots are randomly added to the image to simulate lesions. Data enhancement is further performed by flipping, rotating, etc., and the enhanced image is divided into patches to increase the number of training sets. The ratio of the training set, validation set, and test set is 8:1:1. Specifically, lesion retinal images are mostly manifested as color differences. The color space fusion is achieved by converting the RGB image to YCrCb, YUV, and LAB space, and randomly selecting a spatial component from it and combining it with a component of the RGB space. The algorithm for weighted averaging of each component is expressed as:

[0078] F Blend =α×T 1 (M) m +β×T 2 (M) n +γ×M k

[0079] Where M is the input feature, m,n,k∈[1,2,3] is the component index of the corresponding color space. α+β+γ=1,T 1 ,T 2 is the transformation operation of the color space, T(M) m Select a component of the color space with index m from it, M k Represents the component with index k in the RGB space. Blend Represents the fused features.

[0080] The fusion between color space components can enhance the clarity and contrast of the image and reduce data redundancy: better maintain the details and color balance of the image.

[0081] Then, strip and circular lesion areas are randomly generated, and the edges of the smoothed areas are smoothed with the Gaussian model to achieve a natural transition, simulating the light spots and shadows caused by the lesions. The Gaussian kernel function is applied to each pixel in the lesion area, and the weighted average of the surrounding pixels is taken to obtain an image with blurred edges. The expression is as follows:

[0082]

[0083] Among them, R Lesion represents the lesion area, (x, y) is the pixel coordinate on the image, and (i, j) is the center pixel coordinate of the Gaussian kernel. blur (x,y) represents the image after the lesion edge is blurred.

[0084]

[0085] Where (m,n) is the offset from the kernel center and σ is the standard deviation of the Gaussian kernel.

[0086] In order to effectively avoid overfitting during deep network training, the number of data sets is increased by randomly extracting patches from the training set and sequentially extracting patches from the test set. Specifically:

[0087] For the training set, set the width size patch_w and height size patch_h of the patch image, and randomly select a pixel as the center of the patch. The coordinates of the pixel must meet the following conditions:

[0088]

[0089] Among them, x is the horizontal coordinate, y is the vertical coordinate, img_w is the width of the original image, and img_h is the height of the original image. Finally, with the pixel point as the center, a training image patch with a width of patch_w and a height of patch_h is intercepted from the original image.

[0090] For the test set, set patch_w and patch_h. Then determine whether the size of the original image is proportional to the size of the set patch. If not, fill the original image with pixels. Finally, starting from the upper left corner of the input original image, extract the patch image cyclically according to the principle of "from top to bottom and from left to right". The test set is done to restore the image in the training results. Finally, in order to avoid negative, too large, and nan loss values ​​during network training, the pixel values ​​on the image are normalized and mapped to between 0 and 1.

[0091] Step 2: Construct a segmentation network with an encoding-decoding architecture, in which the encoding stage mainly uses the residual structure to extract features and change the channel and size of the features. In addition, the implicit prompt module is used to extract features from the input lesion image and act on the decoding stage. At the same time, the modulation module modulates the features of jump connections of different scales to achieve feature fusion from coarse to fine. In the decoding stage, the prompt information and jump connections are combined to achieve reliable segmentation of various lesions. Specifically:

[0092] Each stage of encoding is performed by two 3×3 convolution modules for feature extraction, which are selected into skip connections, and a downsampling operation with a stride of 2 follows the convolution. After each downsampling, the size of the feature map is reduced by half and the number of feature channels is doubled. Figure 3 As shown in Figure 1, the implicit prompt module unifies the scale of the medium-sized features with the maximum pooling, stacks them on the channel, and then compresses them into a one-dimensional weight vector through a compression operation. The implicit prompt that is self-learned by weighting a specific degraded prompt vector can be expressed as:

[0093]

[0094]

[0095] Where P Max represents the maximum pooling, P GAP represents global average pooling, || represents feature channel stacking, F represents input features, represents full connection, σ represents Sigmoid activation function, FS stands for Flattening and Squeeze. It strengthens the interaction between channels through channel shuffle, and encodes each pixel spatial feature value into a global feature through global average pooling to achieve flattening. Further, full connection is used to reduce the dimension to the number of channels with implicit information. w It represents the weighted feature obtained by encoding features, and X represents the features of the input FS.

[0096] The implicit hint of self-learning is further realized by weighting the specific degradation hint vector, which is expressed as:

[0097] I prompt =F w ·F prompt

[0098] I prompt Represents the hint feature weighted by the weight feature. prompt Represents self-learned suggestive features that characterize lesions.

[0099] At the same time, due to the large scale difference of multi-scale jump connections, such as Figure 4As shown in the figure, the modulation module uses the jump connection of adjacent sizes to perform bottom-up modulation, realizing the feature fusion from coarse to fine to avoid the lesion information encoding shallow features from excessively affecting the segmentation results, which is expressed as:

[0100]

[0101] in represents the large-scale skip connection feature, is the scaling factor obtained by convolution, and λ is the translation factor obtained by convolution, which comes from the Ghost-Subpixel feature The calculation is as follows:

[0102]

[0103] Furthermore, Conv 1 、Conv 2 Respectively represent the convolution operation without sharing weights, features The skip connection from the next layer uses Ghost linear mapping to achieve feature amplification to meet the channel requirements of sub-pixel upsampling.

[0104]

[0105] in represents the skip connection feature with 2C channels, G represents Ghost linear mapping, Represents the characteristics The features after Ghost linear mapping, UP[X,k] means that the input X is upsampled k times through sub-pixel, k is the upsampling multiple, here k is 2, and the channel will drop by 2 after sampling 2 times.

[0106] In the decoding stage, in addition to the conventional upsampling and convolution structure for channel and size adjustment, a hint combination module (such as Figure 5 As shown), channel splitting is used to achieve grouping and reduce parameters while generating masks, which can be expressed as:

[0107]

[0108] Where M is the input feature, Conv k M represents a convolution operation with a convolution kernel of k×k. 1 ,M 2 It is the sub-feature after splitting the channel. Split represents the channel splitting operation, and Mask represents the sub-feature after splitting the channel. 2 The mask obtained by convolution.

[0109] Then, the prompt information and mask are dynamically fused and the channels are restored to their original number. The 3×3 convolution is combined to further integrate the prompt and context information, so that specific lesions can be segmented, which can be expressed as:

[0110]

[0111]

[0112] Where || represents the stacking of feature channels, Conv k Indicates a convolution operation with a convolution kernel of k×k. Finally, after decoding, the channel is adjusted to a single-channel output, and then the loss is calculated with the label. prompt Represents the hint feature weighted by the weight feature.

[0113] Step 3: For the encoding-decoding architecture in step 2, use dice_loss to calculate the similarity between the network prediction value y and the label y' to evaluate the segmentation effect of the network on the validation set. Specifically:

[0114] dice_loss evaluates the similarity between two samples, as shown in the formula expression:

[0115]

[0116] Where y∩y' represents the intersection between the predicted value y and the label y', which is usually approximated as the dot product y·y' between the label and the predicted value, |y|+|y'| represents the element-by-element addition, and ε is a minimum value to avoid the denominator being 0. For binary classification, since the values ​​on the prediction map are only 0 and 1, the y·y' operation can effectively clear the pixels that are not activated in the manual segmentation map, and for those activated pixels, the corresponding confidence can be obtained through the dot product. During the training process, the network will severely punish points with low confidence and give higher dice to high confidence pixels.

[0117] Step 4: Finally, load the test image data in step 1 into the network and quickly test to complete the image segmentation.

[0118] The methods described in the above embodiments can be implemented by a computer chip or entity, or by a product with a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0119] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0120] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A multi-lesion retinal segmentation method based on implicit cues, characterized in that: The following steps are involved: Step 1: After the data set is fused in three color spaces, lesion simulation, data enhancement and data set division operations are performed; Step 2: Construct a segmentation network with an encoding-decoding architecture, in which the residual structure is used in the encoding stage to extract features and change the channels and sizes of the features, and the implicit hint module is used to extract features from the input lesion image and act on the decoding stage. At the same time, the skip connections of different scales are modulated to achieve feature fusion from coarse to fine. In the decoding stage, the hint information and the skip connection are combined to achieve reliable segmentation of various lesions. In the encoding stage, each stage is extracted by two 3×3 convolution modules, and the features are selected into the skip connection. The downsampling operation with a step size of 2 follows the convolution. After each downsampling, the size of the feature map is reduced by one time, and the number of feature channels is doubled. The implicit prompt module unifies the scale of the medium-sized features by maximum pooling, and stacks them on the channel at the same time, and then compresses them into a one-dimensional weight vector through a compression operation. The self-learning implicit prompt is realized by weighting the specific degradation prompt vector, which is expressed as: Where P Max represents the maximum pooling, P GAP represents global average pooling, || represents feature channel stacking, F represents input features, represents full connection, σ represents Sigmoid activation function, FS represents FS. The channel shuffle is used to strengthen the interaction between channels. The global average pooling is used to encode the spatial feature value of each pixel into a global feature to achieve flattening. The full connection is further used to reduce the dimension to the number of channels with implicit information. w represents the weighted feature obtained by encoding features, and X represents the feature of the input FS; The implicit hint of self-learning is further realized by weighting the specific degradation hint vector, which is expressed as: I prompt =F w ·F prompt I prompt represents the hint feature weighted by the weight feature, F prompt Represents self-learned cue features that characterize lesions; At the same time, due to the large scale difference of multi-scale jump connections, the modulation module uses jump connections of adjacent sizes for bottom-up modulation to achieve coarse-to-fine feature fusion to avoid excessive influence of lesion information encoding shallow features on the segmentation results, which can be expressed as: in represents the large-scale skip connection feature, is the scaling factor obtained by convolution, and λ is the translation factor obtained by convolution, which comes from the Ghost-Subpixel feature The calculation is as follows: Conv1 and Conv2 represent convolution operations without sharing weights. The skip connection from the next layer, in represents the skip connection feature with 2C channels, G represents Ghost linear mapping, Represents the characteristics The features after Ghost linear mapping, UP[X,k] means upsampling the input X by k times through sub-pixel, where k is the upsampling multiple, and here k is 2; The decoding stage adds a hint combination module, which generates a mask while reducing parameters through channel splitting, which is expressed as: Where M is the input feature, Conv k Represents a convolution operation with a convolution kernel of k×k, M1 and M2 are sub-features after channel splitting, Split represents a channel splitting operation, and Mask represents the mask obtained by the convolution of M2; Then, the prompt information and mask are dynamically fused and the channels are restored to the original number of channels. The 3×3 convolution is combined to further integrate the prompt and context information, which is expressed as: Where || represents the stacking of feature channels, Conv k represents a convolution operation with a convolution kernel of k×k, I prompt It represents the hint feature weighted by the weight feature. Finally, after decoding, the channel is adjusted to a single-channel output, and then the loss is calculated with the label. Step 3: For the encoding-decoding architecture in step 2, use dice_loss to calculate the similarity between the network prediction value y and the label y' to evaluate the segmentation effect of the network on the validation set; Step 4: Load the test image data in step 1 into the network and quickly test to complete the image segmentation.

2. The method for segmenting retinal lesions based on implicit cues according to claim 1, characterized in that: The simulation of lesions, data enhancement and data set division operations in step 1 include: randomly adding shadows and spots to the image to simulate lesions, further performing data enhancement by flipping and rotating, and dividing the enhanced image into patches to increase the number of training sets, and dividing the training set, validation set and test set in a ratio of 8:1:

1.

3. A multi-lesion retinal segmentation method based on implicit cues according to claim 1 or 2, characterized in that: The three-color space fusion of the data set includes converting the RGB image into YCrCb, YUV, and LAB spaces, and randomly selecting a spatial component from them to combine with a component of the RGB space to achieve color space fusion, and performing a weighted average algorithm on each component, and its expression is: F Blend =α×T1(M) m +β×T2(M) n +γ×M k Where M is the input feature, m,n,k∈[1,2,3] is the component index of the corresponding color space, α+β+γ=1, T1,T2 is the transformation operation of the color space, T(M) m Select a component of the color space with index m from it, M k Represents the component with index k in the RGB space, F Blend Represents the fused features.

4. A multi-lesion retinal segmentation method based on implicit cues according to claim 1 or 2, characterized in that: The simulated lesion specifically includes: randomly generating strip and / or circular lesion areas, smoothing the edge of the area with a Gaussian model, applying a Gaussian kernel function to each pixel in the lesion area, and taking the weighted average of the surrounding pixels to obtain an image with blurred edges, and the expression is as follows: Among them, R Lesion represents the lesion area, (x, y) is the pixel coordinate on the image, (i, j) is the center pixel coordinate of the Gaussian kernel, and F blur (x, y) represents the image after the lesion edge is blurred; Where (m,n) is the offset from the kernel center and σ is the standard deviation of the Gaussian kernel.

5. The method for segmenting retinal lesions based on implicit cues according to claim 2, characterized in that: The patch division of the enhanced image includes randomly extracting patches from the training set and sequentially extracting patch maps from the test set, which is specifically as follows: For the training set, set the width size patch_w and height size patch_h of the patch image, and randomly select a pixel as the center of the patch. The coordinates of the pixel must meet the following conditions: Among them, x is the horizontal coordinate, y is the vertical coordinate, img_w is the width of the original image, and img_h is the height of the original image. Finally, with the pixel point as the center, a training image patch with a width of patch_w and a height of patch_h is intercepted from the original image; For the test set, set patch_w and patch_h, and then determine whether the size of the original image is proportional to the size of the set patch image. If not, fill the original image with pixels. Finally, take the upper left corner of the input original image as the starting point and extract the patch image cyclically according to the principle of "from top to bottom and from left to right".

6. The method for segmenting retina with multiple lesions based on implicit cues according to claim 1, characterized in that: The dice_loss calculates the similarity between the network prediction value y and the label y', and the formula is: Where y∩y' represents the intersection between the predicted value y and the label y', which is usually approximated as the dot product y y' between the label and the predicted value, |y|+|y'| represents the element-by-element sum, and ε is a minimum value to avoid the denominator being 0.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the implicit cue-based multi-lesion retinal segmentation method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the implicit cue-based multi-lesion retinal segmentation method according to any one of claims 1 to 6 are implemented.

Images