Transformer-based unsupervised cross-domain oct segmentation method and system,and device

Abstract

A Transformer-based Neural Network for Optical Coherence Tomography segmentation is trained with source images and labels. A target image is segmented using a teacher network and self-training to obtain pseudo-labelled target image data. The NN is optimised using the pseudo labelled target image data. The unsupervised cross-domain OCT segmentation trains a segmentation NN with a source domain image and labelling by end-to-end learning combined with space scale and linear warm-up of a learning rate. Pseudo-labels are generated for a target domain image by semantic segmentation by a self-training strategy and using a teacher network. The segmentation model is optimised using a cross entropy loss function for weighting based on proportions of different categories of pixels. An adaptive cross-domain (i.e. capable of handling images from different devices) segmentation model is therefore realised.

Description

[0001] The present disclosure relates to the technical field of image processing, and in particular, to a Transformer-based unsupervised cross-domain optical coherence tomography (OCT) segmentation method and system, and a device. BACKGROUND

[0002] With the development of the artificial intelligence technology, deep learning has become a new method of intelligent OCT image segmentation. The accurate segmentation of an OCT image can quantify geometric features between layers of retina, realize comprehensive analysis of a thickness between the layers of retina, improve the accuracy and speed of retinal disease diagnosis, and reduce the dependence of a doctor on subjective experience.

[0003] There are a wide variety of OCT devices, and there is a significant difference between image data acquired by different OCT devices. Existing deep learning models are all designed to learn and be trained based on images obtained by a specific OCT device, and can only have a good segmentation effect for images obtained by a single device. When the models are applied to images obtained by other OCT devices, the performance of the models will decrease significantly. Therefore, a cross-domain segmentation method for an OCT image has gradually become a research hotspot. Transfer learning is widely used in cross-domain segmentation issues. A network is pre-trained on a large universal database, and then fine-tuned on a small specific dataset. However, a lot of manually annotated data will still be needed, and a model needs to be retrained. A generative adversarial network may realize cross-domain segmentation between different OCT images through feature alignment, but its effect is related to a device. Therefore, it is impossible to have a good cross-domain effect for any OCT device. SUMMARY

[0004] In order to solve the problems in the prior art, the present disclosure provides a Transformer-based unsupervised cross-domain OCT segmentation method and system, and a device, to realize accurate cross-domain segmentation of an image obtained by any OCT device and improve the generalization performance of a cross-domain segmentation model.

[0005] The technical solutions adopted by the present disclosure are as follows: a Transformer-based unsupervised cross-domain OCT segmentation method includes:

[0006] training a segmentation network ge with a source domain image Xs = ]x~ [ and a t 'i=l t labelling Xs = ]y$ f , and transforming an original image to a scale space to serve as a model t 'i=l input, thereby reducing influences caused by inconsistent sizes of images obtained by different devices;

[0007] optionally, for source domain image data, adopting a categorical cross entropy loss function for weighting based on proportions of different categories of pixels:

[0008] I® =

[0009] where wW'c) = N is a total number of pixels of a category to which the pixel belongs; and less pixels of the category correspond to a larger weight such that a segmentation effect for the category of less pixels is optimized;

[0010] optionally, using a learning rate warm-up strategy, and linearly warming-up a learning rate at the beginning of training:

[0011] ^Ibase ‘ ^ / ^warm

[0012] optionally, performing semantic segmentation on a target domain image by a self-training strategy and using a teacher network to generate a pseudo-labelling for target domain image data, with an updating formula thereof given below:

[0013] 0t+1 = a0t + (1 - a)0t

[0014] where 0t is a parameter corresponding to a pseudo-labelling generation network; 0t represents a parameter corresponding to the segmentation network at step t; and a represents the learning rate;

[0015] optionally, for the target domain image data, using a categorical cross entropy loss function, and calculating the loss function by replacing y with the pseudo-labelling gp(x): W X«' <7 :::: - ' log PT..... ™ [e arg maxq ; and optionally, implementing and optimizing a cross-domain segmentation model, with an updating formula thereof given below:

[0020] £ = ylogg0(x)

[0016]

[0017]

[0018]

[0019] [00211 where y represents a labelling corresponding to a pixel x; and g0(x) represents a probability that the pixel x belongs to different categories, as predicted by the segmentation network.

[0022] In a second aspect, the present disclosure provides a Transformer-based unsupervised cross-domain OCT segmentation system, including:

[0023] a source domain image training module configured to put a source domain image in one-to-one correspondence with a labelling and input the source domain image and the labelling to a network for training, and extract an image feature with a domain invariant, thereby obtaining a segmentation network model;

[0024] a pseudo-labelling generation module for a target domain image connected with the source domain image training module and configured to:

[0025] perform semantic segmentation on a target domain image by a self-training strategy and using a teacher network to obtain a pseudo-labelling of the target domain image; and

[0026] a cross-domain segmentation model optimization module connected with the pseudo-labelling generation module for a target domain image and configured to calculate a probability that a pixel x belongs to different categories, as predicted by the segmentation network, and update the network.

[0027] In a third aspect, the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and runnable on the processor, where the processor is configured to, when executing the computer program, implement the Transformer-based unsupervised cross-domain OCT segmentation method.

[0028] The present disclosure has the beneficial effects: the present disclosure provides a Transformer-based unsupervised cross-domain OCT segmentation method and system, and a device. Based on a Transformer architecture, a segmentation network is trained with a source domain image and a labelling by end-to-end learning combined with a learning rate warm-up method; semantic segmentation is performed on a target domain image by a self-training strategy and using a teacher network to obtain a pseudo-labelling of target domain image data; and the segmentation method is optimized using the categorical cross entropy loss function, thereby realizing an adaptive cross-domain segmentation model for images obtained by any OCT device and improving the generalization performance of the cross-domain segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG. 1 is a flowchart of the present disclosure; and

[0030] FIG. 2 is a diagram illustrating a cross-domain segmentation effect of an example of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] The following clearly and completely describes the technical solutions in the examples of the present disclosure with reference to the accompanying drawings in the examples of the present disclosure. Apparently, the described examples are merely a part rather than all of the examples of the present disclosure. All other embodiments derived from the embodiments in the present disclosure by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0032] The present disclosure provides a Transformer-based unsupervised cross-domain OCT segmentation method and system, and a device. Based on a Transformer, adaptive cross-domain segmentation of an OCT image is realized without a lot of manually annotated data. The OCT image can still be segmented completely and accurately even in case of low video quality or serious noise.

[0033] To make the above objective, features, and advantages of the present disclosure clearer and more comprehensible, the present disclosure will be further described in detail below with reference to the accompanying drawings and the specific examples.

[0034] FIG. 1 illustrates an exemplary flow of a Transformer-based unsupervised cross-domain OCT segmentation method. Steps are described in detail below. f (A)Ws

[0035] In step 1, a source domain image Xs = ]Xg 1 is put in one-to-one correspondence t Ji=l f with a labelling Ys = jy~ 1 ; an original image is then transformed to a scale space and input to a network for training; and an image feature with a domain invariant is extracted, thereby obtaining a segmentation network model. A loss function of the segmentation network model is a categorical cross entropy function for weighting based on proportions of different categories of pixels: [00361 1® =

[0037] where wW'c) = N is a total number of pixels of a category to which the pixel belongs; and less pixels of the category correspond to a larger weight such that a segmentation effect for the category of less pixels is optimized.

[0038] Using a learning rate warm-up strategy, a linearly warm-up learning rate at the beginning of training is as follows:

[0039] T]t — ribase ‘ t / twarm

[0040] In step 2, semantic segmentation is performed on a target domain image by a self-training strategy and using a teacher network to obtain a pseudo-labelling of the target domain image, with an updating formula thereof given below:

[0041] 0t+1 = a0t + (1 - a)0t

[0042] For target domain image data, a categorical cross entropy loss function is used, and the loss function is calculated by replacing y with the pseudo-labelling g©(x): HxW C ™ ■■■

[0043] ; [e—

[0044] >' ' ; , { t >- X $ / .5 ----------------------------

[0045] n n

[0046] In step 3: a cross-domain segmentation model is optimized, which is configured to calculate a probability that a pixel x belongs to different categories, as predicted by the segmentation network, and update the network, with an updating formula thereof given below:

[0047] £ = ylogge(x).

[0048] In conclusion, in the embodiments of the present disclosure, based on a Transformer architecture, a segmentation network is trained with a source domain image and a labelling by end-to-end learning combined with a learning rate warm-up strategy, and an image feature with a domain invariant is extracted. Semantic segmentation is then performed on a target domain image by a self-training strategy and using a teacher network to obtain a pseudo-labelling of target domain image data. Finally, the segmentation model is optimized by the categorical cross entropy loss function, and an adaptive cross-domain segmentation model for images obtained by any OCT device is realized. Therefore, there is no need for a lot of manually annotated data. An OCT image can still be segmented completely and accurately even in case of low video quality or serious noise. The generalization performance of the cross-domain segmentation model is effectively improved.

[0049] Further, the present disclosure further provides an electronic device. The electronic device may include: a processor, a communication interface, a memory and a communication bus. The processor, the communication interface and the memory communicate with one another by means of the communication bus. The processor can all a computer program in the memory to implement the Transformer-based unsupervised cross-domain OCT segmentation method when executing the computer program.

[0050] In addition, the computer program in the above memory may be stored in a computer-readable storage medium when the computer program is implemented in a form of a software function unit and is sold or used as an independent product. On the basis of such understanding, the technical solutions of the present disclosure essentially or the part contributing to the prior art may be embodied in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for enabling a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or some steps of the methods described in the embodiments of the present disclosure. The above storage medium includes any medium that may store program codes, such as a USB flash drive, a removable hard disk, a read-only memory, a random access memory, a magnetic disk and an optical disc.

[0051] Further, the present disclosure further provides a non-transitory computer-readable storage medium, storing a computer program. The computer program may implement the Transformer-based unsupervised cross-domain OCT segmentation method when executed.

[0052] It should be understood by those skilled in the art that although the present disclosure has been described in terms of the above specific examples, however, the inventive concept of the present disclosure is not limited thereto.

[0053] The foregoing descriptions are only preferred implementations of the present disclosure, and the scope of the present disclosure is not limited to the foregoing embodiments. All technical solutions based on the idea of the present disclosure fall within the protection scope of the present disclosure. It should be noted that those of ordinary skill in the art can make several improvements and modifications without departing from the principles of the present disclosure. These improvements and modifications should also be considered as falling within the protection scope of the present disclosure.

Claims

1. A Transformer-based unsupervised cross-domain optical coherence tomography (OCT) segmentation method, comprising the following steps:(1) based on a Transformer architecture, training a segmentation network with a source domain image and a labelling by end-to-end learning combined with a scale space and linear warm-up of a learning rate;(2) performing semantic segmentation on a target domain image by a self-training strategy and using a teacher network to obtain a pseudo-labelling of target domain image data; and(3) optimizing a segmentation model using a cross entropy loss function for weighting based on proportions of different categories of pixels, thereby realizing an adaptive cross-domain segmentation model for images obtained by any OCT device.

2. The Transformer-based unsupervised cross-domain OCT segmentation method according to claim 1, wherein the training a segmentation network with a source domain image and a labelling in step (1) specifically comprises: training the segmentation network gQ with theXg 1 and the labelling Ys = Wg 1 , and transforming anJi=i t 'i=ioriginal image to the scale space to serve as a model input, thereby reducing influences caused by inconsistent sizes of images obtained by different devices.

3. The Transformer-based unsupervised cross-domain OCT segmentation method according to claim 2, wherein a categorical cross entropy loss function UP for weighting based on proportions of different categories of pixels is adopted for image data of the source domainV f 0)1^image Xs = 1 1 :*i=lwherein = ——, N is a total number of pixels of a category to which the pixelbelongs; and less pixels of the category correspond to a larger weight such that a segmentation effect for the category of less pixels is optimized.

4. The Transformer-based unsupervised cross-domain OCT segmentation method according to claim 1, wherein the linear warm-up of a learning rate in step (1) is specifically as follows: using a learning rate warm-up strategy, and linearly wanning up the learning rate at the beginning of training: i / t = Vbase ‘ t / twarm5. The Transformer-based unsupervised cross-domain OCT segmentation method according to claim 1, wherein an updating formula for the performing semantic segmentation on a target domain image by a self-training strategy and using a teacher network to obtain a pseudo-labelling of target domain image data in step (2) is as follows: 0t+1 = a0t + (1-wherein 0t is a parameter corresponding to a pseudo-labelling generation network; 9t represents a parameter corresponding to the segmentation network at step t; and a represents the learning rate.

6. The Transformer-based unsupervised cross-domain OCT segmentation method according to claim 1, wherein for the target domain image data in step (2), the categorical cross entropy loss function is used, and the loss function is calculated by replacing y with the pseudo-labelling g0(x):w x w c ;= [c arg max’ ........................................................................

7. The Transformer-based unsupervised cross-domain OCT segmentation method according to claim 1, wherein an updating formula for the optimizing the segmentation model using a cross entropy loss function for weighting based on proportions of different categories of pixels in step (3) is as follows:£ = ylogge(x);wherein y represents a labelling corresponding to a pixel x; and g0(x) represents a probability that the pixel x belongs to different categories, as predicted by the segmentation network.

8. A Transformer-based unsupervised cross-domain OCT segmentation system, comprising:a source domain image training module, configured to put a source domain image in one-to-one correspondence with a labelling and input the source domain image and the labelling to a network for training, and extract an image feature with a domain invariant, thereby obtaining a segmentation network model;a pseudo-labelling generation module for a target domain image, connected with the source domain image training module and configured to perform semantic segmentation on a target domain image by a self-training strategy and using a teacher network to obtain apseudo-labelling of the target domain image; anda cross-domain segmentation model optimization module, connected with the pseudo-labelling generation module for a target domain image and configured to calculate a probability that a pixel x belongs to different categories, as predicted by the segmentation network, and update the network .

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the processor is configured to, when executing the computer program, implement the Transformer-based unsupervised cross-domain OCT segmentation method according to claim 1.

Images