Self-supervised multimodal segmentation method, system, device and medium for retinal blood vessels

Abstract

The present invention discloses a self-supervised multimodal segmentation method, system, device, and medium for retinal blood vessels, comprising: training a CycleGAN generative network using a multimodal retinal image dataset to learn the mapping relationship between color retinal angiography and fluorescein angiography, and synthesizing the retinal disease dataset into a second fluorescein angiography image; preprocessing the color retinal angiography image in the retinal disease dataset; inputting the second fluorescein angiography image into a ViT to generate an encoding vector, which is reconstructed into a filter; using the filter to perform correlation filtering on the preprocessed color retinal angiography to obtain a multimodal fusion feature map; performing instance-level feature learning on the multimodal fusion feature map to obtain feature parameters; and migrating the feature parameters to a downstream blood vessel segmentation network to obtain retinal blood vessel segmentation results. The present invention provides clues for downstream blood vessel segmentation tasks through transfer learning, accelerating model convergence.

Description

Technical Field

[0001] The present invention relates to an image segmentation technology, and in particular to a self-supervised multimodal segmentation method, system, device and medium for retinal blood vessels. Background Art

[0002] The eye is the most common organ humans use to capture information. Most of the knowledge and memory we possess is derived from external information perceived through the eye, which is then fed back to the cerebral cortex and stored in the brain. This demonstrates its importance. As the only vascular structure in the human fundus that can be directly observed non-destructively, accurate segmentation of retinal vessels in clinical practice allows medical professionals to more intuitively observe changes in their morphology and structure, enabling preventive measures or further diagnosis of related diseases. Fluorescein angiography is a common method for visualizing retinal vessels in medical examinations. It involves injecting yellow fluorescein into a vein. The fluorescein in the bloodstream clearly displays the vessels, which are then captured in detail using a camera. However, side effects can cause allergic reactions or even death in some patients. Leakage of fluorescein sodium can cause local soft tissue necrosis, phlebitis, toxic neuritis, and other conditions. Therefore, research on high-precision retinal color angiography vessel segmentation methods can avoid the many side effects of invasive examinations and possesses significant clinical significance and practical value.

[0003] Existing retinal vessel segmentation methods can be roughly divided into traditional supervised methods and unsupervised methods. Traditional supervised methods perform segmentation based on manual features. Machine learning methods convolve the Laplacian operator with a Gaussian filter to obtain a Laplacian Gaussian filter, which is applied to multiple scales to obtain multiple single-scale LoG response maps and merge them to obtain a multi-scale Laplacian map, in which blood vessels of different widths are enhanced. The convolutional network based on manual annotation uses the known image features extracted and designed to segment the pixels of the vascular area through a deep network, which greatly improves the performance. However, in fact, relying on manual segmentation of retinal vessels not only requires extremely high professional quality of doctors, but is also seriously affected by subjective factors. The fine details of low-contrast image areas make manual annotation tasks difficult, tedious and time-consuming.

[0004] Unsupervised methods (such as unsupervised, semi-supervised, remotely supervised, and self-supervised) alleviate the problem of lack of annotated data available for training through some auxiliary tasks such as local annotation, instance differentiation, dataset updating, and adversarial learning. However, labeled data for auxiliary tasks is still very scarce, making it difficult to effectively learn relevant domain patterns from unlabeled data, and the importance of prior knowledge is not recognized. Due to the limitations of fundus image acquisition technology, there is often a large amount of noise in the image, and the structure of retinal blood vessels themselves is complex and changeable, making the segmentation of retinal blood vessels difficult. Only using single-modal data for training, the data source is too single, and the objective influence of the acquisition equipment and the patient's status is ignored. To this end, the present invention proposes a retinal blood vessel segmentation method based on a self-supervised multimodal pre-training framework to study how to use complementary multimodal images to more comprehensively and accurately learn domain knowledge and mine domain information. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a self-supervised multimodal segmentation method for retinal blood vessels. By introducing multimodal data into the self-supervised pre-training framework, the retinal structure and the appearance of the lesions are imaged from multiple angles to better learn domain features.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In one aspect, the present invention provides a self-supervised multimodal segmentation method for retinal blood vessels, comprising the following steps:

[0008] A pre-established CycleGAN generative network is trained using a paired multimodal retinal image dataset to obtain a trained CycleGAN generative network; the trained CycleGAN generative network is used to learn a mapping relationship between a first color retinal angiography image and a first fluorescein angiography image; and a corresponding second fluorescein angiography image is synthesized for the pre-trained retinal lesion dataset;

[0009] performing grayscale conversion, normalization, histogram equalization, and gamma correction preprocessing operations on the second color retinal angiography image in the retinal lesion dataset to obtain a preprocessed third color retinal angiography image;

[0010] Inputting the second fluorescein angiography image into a Vision Transformer encoder to generate a fluorescein angiography encoding vector, and reconstructing the vector into a filter, i.e., a convolution filter for extracting vector features from the image; performing correlation filtering on the third color retinal angiography image using the filter to obtain a multimodal fusion feature map;

[0011] Using an unsupervised deep mutual information loss function to perform instance-level feature learning on the multimodal fusion feature map to obtain feature parameters;

[0012] The characteristic parameters are transferred to the downstream blood vessel segmentation network as initialization parameters of the segmentation network model, and the segmentation network is trained on this basis to obtain the retinal blood vessel segmentation results.

[0013] As a preferred technical solution, the multimodal retinal angiography image dataset includes multiple pairs of unannotated first color retinal angiography images and first fluorescein angiography images, each pair of images is from the same patient;

[0014] The method of training a pre-established CycleGAN generative network using the paired multimodal retinal image dataset is to use the first color retinal angiography image as an input to the CycleGAN generative network and the corresponding first fluorescein angiography image as a label for training;

[0015] The trained CycleGAN generation network learns the mapping relationship between the first color retinal angiography image and the first fluorescein angiography image through the process of generating the first fluorescein angiography image from the first color retinal angiography image.

[0016] As a preferred technical solution, the second color retinal angiography in the retinopathy dataset is sequentially subjected to grayscale conversion, normalization, histogram equalization, and gamma correction preprocessing operations, specifically:

[0017] First, the multi-channel second color retinal angiography images are proportionally fused into a single-channel grayscale image. Next, image normalization is performed to ensure the image's affine invariance and improve computational accuracy. The vascular system is then enhanced through contrast-limited adaptive histogram equalization. Finally, local adaptive gamma correction is used to enhance image brightness information and reduce artifact interference. The gamma correction has two adjustable parameters: the compensation coefficient and the gamma coefficient. Different values of the gamma coefficient selectively enhance the contrast of low-grayscale areas or high-grayscale areas.

[0018] As a preferred technical solution, the unsupervised deep mutual information loss function is used to perform instance-level feature learning on the multimodal fusion feature map to obtain feature parameters; specifically:

[0019] Let x∈X represent the patient's multimodal fusion feature map, z∈Z represent the encoding vector, and define the mutual information I as follows:

[0020]

[0021] Among them, mutual information I represents the correlation between X and Z. represents the distribution of the original data, and p(z) represents the distribution of the entire Z after p(z|x) is given, that is:

[0022]

[0023] In order to obtain a good encoder, the mutual information should be larger, that is:

[0024]

[0025] In order to make the encoding space more regular, decouple features, and facilitate subsequent learning, we set q(z) to be a standard normal distribution and minimize the KL divergence between p(z) and the prior distribution q(z):

[0026]

[0027] KL divergence is also called relative entropy, which is equivalent to the difference in information entropy between two probability distributions. At this time, the total objective function can be minimized by weighted mixing:

[0028]

[0029] Among them, x and its corresponding z are regarded as a positive sample pair, x and a randomly selected z are regarded as a negative sample pair, λ is a weighting coefficient, which takes into account the correlation between the entire image. However, the recognition and classification of images is a process from local to global, so it is necessary to introduce a local mutual information to describe the correlation between the intermediate layer features and z, and the intermediate layer features are recorded as {C ij (x)|i=1,2…,h;j=1,2,…,w}, that is, the collection of hw vectors, the complete deep mutual information loss function is expressed as:

[0030]

[0031]

[0032] Among them, T1 and T2 are the network parameters of the encoder, σ is the activation function, σ(T1(x,z)) and σ(T2(x,z)) are instance-level discriminant networks; α, β, and γ represent the proportions of global mutual information, local mutual information, and prior distribution, respectively. Adding an appropriate cross-entropy loss on top of it is the final total loss function of the self-supervised feature learning model for the purpose of instance differentiation.

[0033] As a preferred technical solution, the characteristic parameters are transferred to the downstream vascular segmentation network as initialization parameters of the segmentation network model, and the segmentation network is trained on this basis to obtain the retinal vascular segmentation results, which are specifically as follows:

[0034] U-Net was selected as the downstream vascular segmentation network, and the U-Net was initialized using feature parameters; however, the difference between vascular features and non-vascular features in retinal vascular images was not obvious, especially in vascular areas with small edges; by introducing a spatial attention module on the fourth-layer skip connection of U-Net, the spatial relationship between image pixels was used to generate a spatial attention map, thereby enhancing important features and suppressing unimportant features, thereby improving the representation ability of the network; specifically, the input features were max-pooled and average-pooled through channels to obtain two feature maps respectively, and then the two feature maps were connected through a convolution operation to generate an effective cascade feature descriptor; then the Sigmod activation function was used on the cascade feature descriptor to generate a feature map with a spatial attention mechanism.

[0035] In another aspect, the present invention provides a self-supervised multimodal segmentation system for retinal vessels, which is applied to the self-supervised multimodal segmentation method for retinal vessels, and includes a training generation network module, a preprocessing module, a multimodal fusion feature map module, a feature learning module, and a segmentation module;

[0036] The training generative network module is used to train a pre-established CycleGAN generative network using a paired multimodal retinal image dataset to obtain a trained CycleGAN generative network; the trained CycleGAN generative network is used to learn a mapping relationship between a first color retinal angiography image and a first fluorescein angiography image, and synthesize a corresponding second fluorescein angiography image for the pre-trained retinal lesion dataset;

[0037] The preprocessing module is used to perform grayscale conversion, normalization, histogram equalization and gamma correction preprocessing operations on the second color retinal angiography image in the retinopathy data set in sequence to obtain a preprocessed third color retinal angiography image;

[0038] The multimodal fusion feature map module is configured to input the second fluorescein angiography image into a Vision Transformer encoder to generate a coding vector for the fluorescein angiography, and reconstruct the coding vector into a filter, i.e., a convolution filter for extracting vector features from the image; and perform correlation filtering on the third color retinal angiography using the filter to obtain a multimodal fusion feature map.

[0039] The feature learning module is used to perform instance-level feature learning on the multimodal fusion feature map using an unsupervised deep mutual information loss function to obtain feature parameters;

[0040] The segmentation module is used to transfer the characteristic parameters to the downstream blood vessel segmentation network as initialization parameters of the segmentation network model, and train the segmentation network on this basis to obtain the retinal blood vessel segmentation results.

[0041] Another aspect of the present invention provides an electronic device, characterized in that the electronic device includes:

[0042] at least one processor; and,

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the self-supervised multimodal segmentation method for retinal blood vessels.

[0045] In another aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the self-supervised multimodal segmentation method for retinal blood vessels.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] 1. The self-supervised pre-training framework introduces multimodal data to image the retinal structure and lesion appearance from multiple angles, thereby better learning domain features.

[0048] 2. Correlation filtering allows the model to focus on the vascular pixel area, reducing the objective noise of surrounding tissues or lesions;

[0049] 3. The introduction of global and local deep mutual information loss functions enables the selection of channels with real representation capabilities;

[0050] 4. It eliminates the need for large-scale annotated data in deep learning while maintaining or even improving the original performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 Schematic diagram of the self-supervised multimodal segmentation method for retinal blood vessels proposed in the present invention;

[0053] Figure 2 It is the network structure diagram of the downstream task;

[0054] Figure 3 This is a block diagram of a self-supervised multimodal segmentation system for retinal vessels according to an embodiment of the present invention.

[0055] Figure 4 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0057] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0058] See also Figure 1 The present invention provides a self-supervised multimodal segmentation method for retinal blood vessels, comprising the following steps:

[0059] S1. Using a paired multimodal retinal image dataset to train a pre-established CycleGAN generative network to obtain a trained CycleGAN generative network; the trained CycleGAN generative network is used to learn the mapping relationship between a first color retinal angiography image and a first fluorescein angiography image, and to synthesize a corresponding second fluorescein angiography image for the pre-trained retinal lesion dataset.

[0060] Furthermore, the multimodal retinal angiography image dataset comprises 59 pairs of unannotated first color retinal angiography images and first fluorescein angiography images, each pair of images being from the same patient;

[0061] The method of training a pre-established CycleGAN generative network using the paired multimodal retinal image dataset is to use the first color retinal angiography image as an input to the CycleGAN generative network and the corresponding first fluorescein angiography image as a label for training;

[0062] The trained CycleGAN generation network learns the mapping relationship between the first color retinal angiography image and the first fluorescein angiography image through the process of generating the first fluorescein angiography image from the first color retinal angiography image.

[0063] Furthermore, since fluorescein angiography is invasive and difficult to acquire at many clinical sites, the present invention uses CycleGAN to synthesize a second fluorescein angiography image, so that self-supervised pre-training can still be performed when only retinal color angiography is present. The loss function of CycleGAN is adversarial loss and cycle consistency loss. It is trained for 500 epochs at a learning rate of 0.0001, and then the corresponding fluorescein angiography modality is synthesized in the pre-training dataset Ichallenge-AMD to complete the self-supervised feature learning process.

[0064] S2. Performing grayscale conversion, normalization, histogram equalization, and gamma correction preprocessing operations on the second color retinal angiography image in the retinal lesion dataset in sequence to obtain a preprocessed third color retinal angiography image.

[0065] Furthermore, first, the 3-channel second color retinal angiography image is proportionally fused into a 1-channel grayscale image; then image normalization is performed to ensure the affine invariance of the image and improve the calculation accuracy; then the vascular system is enhanced by limiting the contrast adaptive histogram equalization; finally, local adaptive gamma correction is used to enhance the image brightness information and reduce the interference of artifacts; among them, the gamma transform has two adjustable parameters, the compensation coefficient and the gamma coefficient, which can selectively enhance the contrast of low grayscale areas or high grayscale areas according to different values of the gamma coefficient.

[0066] S3. Input the synthesized corresponding second fluorescein angiography image into Vision Transformer encoder to generate a coding vector of fluorescein angiography, and reconstruct it into a filter, that is, a convolution filter for extracting vector features in the image; use the filter to perform correlation filtering on the third color retinal angiography image to obtain a multimodal fusion feature map.

[0067] Furthermore, this method introduces the Vision Transformer (ViT) to act on the second fluorescein angiography image to obtain a one-dimensional coding vector containing vascular information. The one-dimensional coding vector has a length of 2700 and is reconstructed into three 30×30 filters. The filter is then used to generate a multimodal fusion feature map of the preprocessed third color retinal angiography image with a size of 3×224×224. The third color retinal angiography image generated by the four-step preprocessing is complementary to the information in the second fluorescein angiography image, among which the vascular information in the fluorescein angiography is more obvious, and the response value of the pixel where the blood vessel is located should be higher than that of the non-vascular area. This is a rough segmentation result, which not only contains the color information in the retinal color angiography, but also highlights the vascular texture.

[0068] S4. Use an unsupervised deep mutual information loss function to perform instance-level feature learning on the multimodal fusion feature map to obtain feature parameters for initializing downstream tasks.

[0069] The present invention uses deep mutual information to measure the information extracted by the encoder in the pre-training framework. Let x∈X represent the patient's multimodal fusion feature map, z∈Z represent the encoding vector, and define the mutual information I as follows:

[0070]

[0071] Among them, mutual information I represents the correlation between X and Z. represents the distribution of the original data, and p(z) represents the distribution of the entire Z after p(z|x is given, that is:

[0072]

[0073] In order to obtain a good encoder, the mutual information should be larger, that is:

[0074]

[0075] In order to make the encoding space more regular, decouple features, and facilitate subsequent learning, we set q(z) to be a standard normal distribution and minimize the KL divergence between p(z) and the prior distribution q(z):

[0076]

[0077] KL divergence is also called relative entropy, which is equivalent to the difference in information entropy between two probability distributions. At this time, the total objective function can be minimized by weighted mixing:

[0078]

[0079] Among them, x and its corresponding z are regarded as a positive sample pair, x and a randomly selected z are regarded as a negative sample pair, and λ is a weighting coefficient; the association between the entire image is taken into account, but the recognition and classification of the image is a process from local to global, so it is necessary to introduce a local mutual information to describe the association between the intermediate layer features and z, and the intermediate layer features are recorded as

[0080] {C ij (x)|i=1,2…,h;j=1,2,…,w}, that is, the collection of hw vectors; the complete deep mutual information loss function is expressed as:

[0081]

[0082] Among them, T1 and T2 are the network parameters of the encoder, σ is the activation function, σ(T19x,z)) and σ(T2(x,z)) are instance-level discriminant networks; α, β, and γ represent the proportions of global mutual information, local mutual information, and prior distribution, respectively. Adding appropriate cross-entropy loss on top of it is the final total loss function of the self-supervised feature learning model for the purpose of instance differentiation.

[0083] S5. Migrating the initialized feature parameters to the downstream blood vessel segmentation network as the initialization parameters of the segmentation network model, and training the segmentation network on this basis to obtain the retinal blood vessel segmentation result.

[0084] See also Figure 2 , U-Net is selected as the downstream vascular segmentation network, and the feature learning network parameters after pre-training are initialized to U-Net; however, the difference between vascular features and non-vascular features in retinal vascular images is not obvious, especially in the vascular area with small edges; by introducing a spatial attention module on the fourth-layer skip connection of U-Net, the spatial relationship between image pixels is used to generate a spatial attention map, thereby enhancing important features and suppressing unimportant features, thereby improving the representation ability of the network; specifically, the input features are maximum pooled and average pooled in a channel manner to obtain two feature maps respectively, and then the two feature maps are connected by a convolution operation to generate an effective cascade feature descriptor; then the Sigmod activation function is used on the cascade feature descriptor to generate a feature map with a spatial attention mechanism.

[0085] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0086] Based on the same principles as the self-supervised multimodal segmentation method for retinal vessels described in the aforementioned embodiment, the present invention further provides a self-supervised multimodal segmentation system for retinal vessels, which can be used to implement the aforementioned self-supervised multimodal segmentation method for retinal vessels. For ease of illustration, the schematic diagram of the embodiment of the self-supervised multimodal segmentation system for retinal vessels only shows portions relevant to the present invention. Those skilled in the art will appreciate that the illustrated structure does not limit the device, and the device may include more or fewer components than shown, or combine certain components, or have a different component arrangement.

[0087] See also Figure 3 In another embodiment of the present application, a self-supervised multimodal segmentation system 100 for retinal blood vessels is provided, the system comprising a training generation network module 101, a preprocessing module 102, a multimodal fusion feature map module 103, a feature learning module 104, and a segmentation module 105;

[0088] The training generative network module 101 is configured to train a pre-established CycleGAN generative network using a paired multimodal retinal image dataset to obtain a trained CycleGAN generative network; the trained CycleGAN generative network is configured to learn a mapping relationship between a first color retinal angiography image and a first fluorescein angiography image, and to synthesize a corresponding second fluorescein angiography image for the pre-trained retinal lesion dataset;

[0089] The preprocessing module 102 is configured to sequentially perform grayscale conversion, normalization, histogram equalization, and gamma correction preprocessing operations on the second color retinal angiography image in the retinal lesion dataset to obtain a preprocessed third color retinal angiography image;

[0090] The multimodal fusion feature map module 103 is configured to input the second fluorescein angiography image into a Vision Transformer encoder to generate a fluorescein angiography encoding vector, and reconstruct the vector into a filter, i.e., a convolution filter for extracting vector features from the image; and perform correlation filtering on the third color retinal angiography image using the filter to obtain a multimodal fusion feature map.

[0091] The feature learning module 104 is used to perform instance-level feature learning on the multimodal fusion feature map using an unsupervised deep mutual information loss function to obtain feature parameters;

[0092] The segmentation module 105 is used to transfer the characteristic parameters to the downstream blood vessel segmentation network as initialization parameters of the segmentation network model, and train the network on this basis to obtain retinal blood vessel segmentation results.

[0093] It should be noted that the self-supervised multimodal segmentation system for retinal vessels of the present invention corresponds one-to-one to the self-supervised multimodal segmentation method for retinal vessels of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the self-supervised multimodal segmentation method for retinal vessels are applicable to the embodiment of the self-supervised multimodal segmentation system for retinal vessels. For specific details, please refer to the description in the embodiment of the method of the present invention, and will not be repeated here. This is hereby stated.

[0094] In addition, in the implementation of the self-supervised multimodal segmentation system for retinal vessels in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the self-supervised multimodal segmentation system for retinal vessels can be divided into different program modules to complete all or part of the functions described above.

[0095] See also Figure 4 In one embodiment, an electronic device for implementing a self-supervised multimodal segmentation method for retinal blood vessels is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a self-supervised multimodal segmentation program 203 for retinal blood vessels.

[0096] The first memory 202 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 may include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can be used not only to store application software and various data installed in the electronic device 200, such as the code of the self-supervised multimodal segmentation program 203 for retinal vessels, but also to temporarily store data that has been output or is about to be output.

[0097] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0098] Figure 4 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0099] The self-supervised multimodal segmentation program 203 for retinal blood vessels stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When executed in the first processor 201, the following can be achieved:

[0100] Using the paired multimodal retinal image dataset to train a pre-established CycleGAN generative network to obtain a trained CycleGAN generative network; the trained CycleGAN generative network is used to learn a mapping relationship between a first color retinal angiography image and a first fluorescein angiography image, and to synthesize a corresponding second fluorescein angiography image for the pre-trained retinal lesion dataset;

[0101] performing grayscale conversion, normalization, histogram equalization, and gamma correction preprocessing operations on the second color retinal angiography image in the retinal lesion dataset to obtain a preprocessed third color retinal angiography image;

[0102] Inputting the second fluorescein angiography image into a Vision Transformer encoder to generate a fluorescein angiography encoding vector, and reconstructing the vector into a filter, i.e., a convolution filter for extracting vector features from the image; performing correlation filtering on the third color retinal angiography image using the filter to obtain a multimodal fusion feature map;

[0103] Using an unsupervised deep mutual information loss function to perform instance-level feature learning on the multimodal fusion feature map to obtain feature parameters;

[0104] The characteristic parameters are transferred to the downstream blood vessel segmentation network as initialization parameters of the segmentation network model, and the network is trained on this basis to obtain the retinal blood vessel segmentation results.

[0105] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0106] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0107] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A self-supervised multimodal segmentation method for retinal blood vessels, characterized by: The steps include: A pre-established CycleGAN generative network is trained using a paired multimodal retinal image dataset to obtain a trained CycleGAN generative network; the trained CycleGAN generative network is used to learn a mapping relationship between a first color retinal angiography image and a first fluorescein angiography image; and a corresponding second fluorescein angiography image is synthesized for the pre-trained retinal lesion dataset; performing grayscale conversion, normalization, histogram equalization, and gamma correction preprocessing operations on the second color retinal angiography image in the retinal lesion dataset to obtain a preprocessed third color retinal angiography image; Inputting the second fluorescein angiography image into a Vision Transformer encoder to generate a fluorescein angiography encoding vector, and reconstructing the vector into a filter, i.e., a convolution filter for extracting vector features from the image; performing correlation filtering on the third color retinal angiography image using the filter to obtain a multimodal fusion feature map; Using an unsupervised deep mutual information loss function to perform instance-level feature learning on the multimodal fusion feature map to obtain feature parameters; The characteristic parameters are transferred to the downstream vascular segmentation network as initialization parameters of the segmentation network model, and the segmentation network is trained on this basis to obtain the retinal vascular segmentation results; specifically: U-Net is selected as the downstream blood vessel segmentation network, and the U-Net is initialized using feature parameters; by introducing a spatial attention module on the fourth-layer skip connection of U-Net, the spatial relationship between image pixels is used to generate a spatial attention map, thereby enhancing important features and suppressing unimportant features, thereby improving the representation ability of the network; specifically, the input features are max-pooled and average-pooled through channels to obtain two feature maps respectively, and then the two feature maps are connected through a convolution operation to generate an effective cascade feature descriptor; then the Sigmund activation function is used on the cascade feature descriptor to generate a feature map with a spatial attention mechanism.

2. The self-supervised multimodal segmentation method for retinal vessels according to claim 1, characterized in that: The multimodal retinal image dataset comprises a plurality of pairs of unannotated first color retinal angiography images and first fluorescein angiography images, each pair of images being from the same patient; The method of training a pre-established CycleGAN generative network using the paired multimodal retinal image dataset is to use the first color retinal angiography image as an input to the CycleGAN generative network and the corresponding first fluorescein angiography image as a label for training; The trained CycleGAN generation network learns the mapping relationship between the first color retinal angiography image and the first fluorescein angiography image through the process of generating the first fluorescein angiography image from the first color retinal angiography image.

3. The self-supervised multimodal segmentation method for retinal blood vessels according to claim 1, characterized in that: The second color retinal angiography in the retinal lesion data set is sequentially subjected to grayscale conversion, normalization, histogram equalization, and gamma correction preprocessing operations, specifically: First, the multi-channel second color retinal angiography images are proportionally fused into a single-channel grayscale image. Next, image normalization is performed to ensure the affine invariance of the image. Then, the vascular system is enhanced by contrast-limited adaptive histogram equalization. Finally, local adaptive gamma correction is used to enhance image brightness information and reduce artifact interference. Among them, the gamma correction has two adjustable parameters: the compensation coefficient and the gamma coefficient. Different values of the gamma coefficient selectively enhance the contrast of low-grayscale areas or the contrast of high-grayscale areas.

4. A self-supervised multimodal segmentation system for retinal vessels, characterized by: A self-supervised multimodal segmentation method for retinal blood vessels applied to any one of claims 1-3, comprising a training generation network module, a preprocessing module, a multimodal fusion feature map module, a feature learning module, and a segmentation module; The training generative network module is used to train a pre-established CycleGAN generative network using a paired multimodal retinal image dataset to obtain a trained CycleGAN generative network; the trained CycleGAN generative network is used to learn a mapping relationship between a first color retinal angiography image and a first fluorescein angiography image, and synthesize a corresponding second fluorescein angiography image for the pre-trained retinal lesion dataset; The preprocessing module is used to perform grayscale conversion, normalization, histogram equalization and gamma correction preprocessing operations on the second color retinal angiography image in the retinopathy data set in sequence to obtain a preprocessed third color retinal angiography image; The multimodal fusion feature map module is configured to input the second fluorescein angiography image into a VisionTransformer encoder to generate a fluorescein angiography encoding vector, and reconstruct the encoding vector into a filter, i.e., a convolution filter for extracting vector features from the image; and perform correlation filtering on the third color retinal angiography using the filter to obtain a multimodal fusion feature map. The feature learning module is used to perform instance-level feature learning on the multimodal fusion feature map using an unsupervised deep mutual information loss function to obtain feature parameters; The segmentation module is used to transfer the characteristic parameters to the downstream blood vessel segmentation network as initialization parameters of the segmentation network model, and train the segmentation network on this basis to obtain the retinal blood vessel segmentation results.

5. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can perform the self-supervised multimodal segmentation method for retinal blood vessels as described in any one of claims 1-3.

6. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the self-supervised multimodal segmentation method for retinal blood vessels according to any one of claims 1 to 3 is implemented.

Images