Pathological image processing method and electronic device based on deep learning segmentation model

Abstract

This application discloses a pathological image processing method and an electronic device based on a deep learning segmentation model. The method includes: collecting image data of a pathological section, processing the image data of the pathological section to obtain a training data set; building a deep learning segmentation model, where the deep learning segmentation model includes a blood vessel deep learning segmentation model and a ganglion cell deep learning segmentation model; combining the training data set and using a semi-supervised technique based on the pseudo-label loss weight coefficient W d to train the deep learning segmentation model to obtain a trained deep learning segmentation model; identifying blood vessels and ganglion cells in a to-be-tested picture through the trained deep learning segmentation model. Based on the semi-supervised technique of the pseudo-label loss weight coefficient W d , the training processes of two stages are directly merged, avoiding a large number of errors in pseudo-labels caused by poor quality of labeled data, and controlling the periodic convergence of the deep learning segmentation model through W d .

Description

Technical Field

[0001] The present disclosure generally relates to the technical field of computer-aided pathological analysis and diagnosis, and more particularly to a pathological image processing method and electronic device based on a deep learning segmentation model. Background Art

[0002] In pathological image analysis, due to the differences in the perspectives of different observers, there is often a certain amount of noise data in the labeled data. Therefore, the lack of training data sets with high-quality annotations has become the main limitation of the application of supervised deep learning in pathological image tasks.

[0003] In the traditional semi-supervised training process, a two-stage approach is often used, that is, in the first stage, two models with different initializations are trained using real labeled data, and in the second stage, a semi-supervised method is used to pseudo-label each other for further training, and finally the better model is selected as the final result. Since the traditional semi-supervised process uses a two-stage approach, the overall training time is relatively long, and the effect of the second stage is greatly affected by the first stage model. If the accuracy of the first stage model is low, there will be a lot of erroneous information in the pseudo-label, resulting in poor semi-supervised results. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desired to provide a pathological image processing method and electronic device based on a deep learning segmentation model.

[0005] In a first aspect, a pathological image processing method based on a deep learning segmentation model is provided, comprising:

[0006] Collecting image data of pathological sections, and processing the image data of the pathological sections to obtain a training data set;

[0007] Building a deep learning segmentation model, wherein the deep learning segmentation model includes a blood vessel deep learning segmentation model and a ganglion cell deep learning segmentation model;

[0008] Combined with the training data set, the pseudo-label loss weight coefficient W is used d Training the deep learning segmentation model to obtain a trained deep learning segmentation model;

[0009] The trained deep learning segmentation model is used to identify blood vessels and ganglion cells in the image to be tested.

[0010] In a second aspect, an electronic device is provided, including:

[0011] one or more processors;

[0012] a memory for storing one or more programs,

[0013] When one or more programs are executed by one or more processors, the one or more processors execute the pathological image processing method based on the deep learning segmentation model as provided in the embodiments of the present application.

[0014] In a third aspect, a computer-readable storage medium storing a computer program is provided, which, when executed by a processor, implements a pathological image processing method based on a deep learning segmentation model as provided in the embodiments of the present application.

[0015] According to the technical solution provided in the embodiment of the present application, in order to solve the problems existing in the traditional semi-supervised technology, the present application provides a method based on the pseudo-label loss weight coefficient W. d The semi-supervised technology uses model consistency constraints to initialize two deep learning segmentation models with the same structure but different weight parameters, and the pseudo-label loss weight coefficient W is increased by polynomial growth. d , changing the training process from two stages to end-to-end, while avoiding the problem that the model is difficult to converge to the optimal state due to a large number of errors in pseudo-labels caused by poor quality of labeled data.

[0016] Furthermore, according to the technical solution provided in the embodiment of the present application, the target to be measured is segmented through a deep learning segmentation model, thereby improving the stability and computational efficiency of the calculation of pathological indicators.

[0017] Furthermore, according to certain embodiments of the present application, the accuracy of the target located by the deep learning segmentation model is optimized through an image segmentation optimization unit, and finally the required indicators are calculated and displayed, and the measurement targets are fully automatically detected and calculated, which greatly improves the efficiency of indicator calculation and reduces the complexity of indicator calculation. The measurement indicators are objectively quantified, overcoming the subjectivity and time shift caused by current manual pathology calculations, and providing an efficient auxiliary system for pathology projects that require artificial intelligence-assisted diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1 An exemplary flowchart of a pathological image processing method based on a deep learning segmentation model provided in an embodiment of the present application;

[0020] Figure 2 The pseudo-label loss weight coefficient W provided in the embodiment of the present application is d A diagram of the structure of a deep learning segmentation model trained using semi-supervised techniques;

[0021] Figure 3The pseudo-label loss weight coefficient W provided in the embodiment of the present application d Growth curve chart;

[0022] Figure 4 An exemplary flowchart of a preferred implementation of a pathological image processing method based on a deep learning segmentation model provided in an embodiment of the present application;

[0023] Figure 5 A diagram of the architecture of a blood vessel deep learning segmentation model provided in an embodiment of the present application;

[0024] Figure 6 for Figure 5 The architecture diagram of the encoding module, decoding module and output module in the deep learning segmentation model of blood vessels; Figure (a) is the architecture diagram of the encoding module; Figure (b) is the architecture diagram of the decoding module; Figure (c) is the architecture diagram of the output module;

[0025] Figure 7 for Figure 6 The architecture diagram of the encoding module in the decoder module and the feature extraction module in the decoder module;

[0026] Figure 8 A diagram of the architecture of the ganglion cell deep learning segmentation model provided in an embodiment of the present application;

[0027] Fig. 9 for Figure 8 The architecture diagram of the encoding module, decoding module and output module in the deep learning segmentation model of ganglion cells; Figure (a) is the architecture diagram of the encoding module; Figure (b) is the architecture diagram of the decoding module; Figure (c) is the architecture diagram of the output module;

[0028] Fig.10 A diagram of an adaptive threshold selection method for fluorescent stain screening provided in an embodiment of the present application;

[0029] Fig.11 The effect diagrams before and after the image segmentation optimization unit is used in the embodiment of the present application; wherein, Figure (a) is the effect diagram before the processing; Figure (b) is the effect diagram after the processing;

[0030] Fig.12 A schematic diagram of the indicator calculation unit provided in the embodiment of the present application calculating various measurement indicators; wherein, Figure (a) is the original image; Figure (b) is the region of interest;

[0031] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] Please refer to Figure 1 , shows an exemplary flow chart of a pathological image processing method based on a deep learning segmentation model provided in an embodiment of the present application.

[0035] like Figure 1 As shown, in this embodiment, the pathological image processing method based on the deep learning segmentation model provided by the present invention includes:

[0036] S110: collecting image data of pathological sections, and processing the image data of the pathological sections to obtain a training data set;

[0037] S120: Building a deep learning segmentation model, wherein the deep learning segmentation model includes a blood vessel deep learning segmentation model and a ganglion cell deep learning segmentation model;

[0038] S130: Using the pseudo-label loss weight coefficient W in combination with the training data set d The deep learning segmentation model is trained using a semi-supervised technology to obtain a trained deep learning segmentation model;

[0039] S140: Using the trained deep learning segmentation model to identify blood vessels and ganglion cells in the image to be tested.

[0040] Specifically, traditional semi-supervised techniques often adopt a two-stage approach, while this application directly combines the two-stage training process and proposes a polynomially growing pseudo-label loss weight coefficient W d The calculation method avoids a large number of errors in pseudo-labels due to poor quality of labeled data, and uses the pseudo-label loss weight coefficient W d , control the periodic convergence of the deep learning segmentation model so that the deep learning segmentation model converges to the optimal.

[0041] In one embodiment, step S130 includes:

[0042] Inputting the real annotated training data set into two deep learning segmentation models for parallel training respectively, the two deep learning segmentation models are respectively a first deep learning segmentation model and a second deep learning segmentation model with the same structure and different weight parameters, obtaining the trained first deep learning segmentation model and the second deep learning segmentation model, as well as a first supervisory signal corresponding to the first deep learning segmentation model and a second supervisory signal corresponding to the second deep learning segmentation model;

[0043] Using the first supervisory signal to update the weight parameters of the first deep learning segmentation model, and retraining the first deep learning segmentation model; using the second supervisory signal to update the weight parameters of the second deep learning segmentation model, and retraining the second deep learning segmentation model;

[0044] The first pseudo label of the unlabeled training data set is calibrated using the trained first deep learning segmentation model, and the loss Loss generated by the first pseudo label data training is calculated. F1 ; Use the trained second deep learning segmentation model to calibrate the second pseudo label of the unlabeled training data set, and calculate the loss Loss generated by the second pseudo label data training F2 ;

[0045] The loss Loss generated by the second pseudo-label data training F2 Configure the pseudo label loss weight coefficient W d , configure the pseudo label loss weight coefficient W d The second pseudo label is used as a supervisory signal of the first deep learning segmentation model, weight parameters of the first deep learning segmentation model are updated, and the first deep learning segmentation model is retrained;

[0046] The loss Loss generated by the first pseudo-label data training F1 Configure the pseudo label loss weight coefficient W d , configure the pseudo label loss weight coefficient W d The first pseudo label is used as a supervisory signal of a second deep learning segmentation model, a weight parameter of the second deep learning segmentation model is updated, and the second deep learning segmentation model is retrained;

[0047] A model with better model convergence is selected from the first deep learning segmentation model and the second deep learning segmentation model as the trained deep learning segmentation model.

[0048] Specifically, the deep learning segmentation model is trained to enable it to have the ability to identify specified targets, such as training the blood vessel deep learning segmentation model to enable it to have the ability to identify blood vessels; training the ganglion cell deep learning segmentation model to enable it to have the ability to identify fundus RGC ganglion cells. The present invention trains two deep learning segmentation models in parallel to perform cross-model collaborative optimization in an end-to-end manner (such as Figure 2 ). The two deep learning segmentation models are respectively a first deep learning segmentation model and a second deep learning segmentation model with the same structure but different initial weight parameters. The embodiment of the present application takes the Res-UNet model as an example for illustrative explanation. Specifically, the weight parameters are often initialized by certain strategies before training, such as: Xavier initialization, He initialization, random initialization, etc. The present application obtains two deep learning segmentation models through different initialization methods.

[0049] like Figure 2 As shown in Figure 2, in each training train_loop, each deep learning model updates its weight parameters twice. The first time, the deep learning segmentation model is trained using the real labeled training data set, and the loss of the corresponding deep learning segmentation model is calculated. T1 、Loss T2 ), using the loss T1 、Loss T2 ) as the supervisory signal, and reversely update the weight parameters of the corresponding deep learning segmentation model; the second time, the pseudo labels obtained by the forward propagation of the two deep learning segmentation models are used as the loss Loss generated by the first pseudo label data training of the first deep learning segmentation model F1 Configure the pseudo label loss weight coefficient W d , and serves as the supervisory signal for the second deep learning segmentation model, and is the loss Loss generated by the second pseudo-label data training of the second deep learning segmentation model F2 Configure the pseudo label loss weight coefficient W d , and as a supervision signal for the first deep learning segmentation model, by using the pseudo-label loss weight coefficient W that shows a decreasing periodic growth d , control the contribution of pseudo labels when back-propagating to update the parameters of the deep learning segmentation model, update the weight parameters of the two deep learning segmentation models, and use the consistency constraints of the models to achieve the expectation of consistent predictions from different models.

[0050] Specifically, by Figure 3 As shown, in the early stage of deep learning segmentation model training, the pseudo label loss weight coefficient W d It starts from 0 and grows slowly. At this time, the main supervision information of the deep learning segmentation model comes from the real labeled data. At the same time, in order to avoid the deviation of the model convergence direction caused by the quality problem of pseudo-label data, the pseudo-label loss weight coefficient Wd It shows a decreasing cyclical growth.

[0051] The deep learning segmentation model includes: the selection and improvement of the deep learning segmentation model architecture, the preparation of data for model training, and the method of model training and optimal model preservation. The trained deep learning segmentation model can segment the target of interest through model reasoning. The input size of the deep learning segmentation model is set to 256×256; the ratio of the validation set to the training set is set to 0.15; the batch size is set to 16; the Adam optimizer is used, and the learning rate is set to a constant of 0.001; the total number of rounds of deep learning segmentation model training is set to 120 rounds (preset number threshold), and training is stopped after completion; during the training process, when the loss of the deep learning segmentation model on the validation set decreases, the model update and preservation is triggered, and the model obtained after 120 rounds of model training is the optimal model.

[0052] It should be noted that the model parameters set in the above deep learning segmentation model training process are only exemplary. In actual application scenarios, other parameter values ​​may also be selected.

[0053] In one embodiment, Figure 2 As shown, the method for determining the first supervisory signal includes:

[0054] The loss Loss generated by the first real label data training is calculated based on the first real prediction result P1 of the real labeled training data set by the first deep learning segmentation model and the real labeled training data set. T1 , the loss Loss generated by the first real label data training T1 As the first supervisory signal;

[0055] The method for determining the second supervisory signal includes:

[0056] The loss Loss generated by the second real label data training is calculated based on the second real prediction result P2 of the real labeled training data set by the second deep learning segmentation model and the real labeled training data set. T2 , the loss Loss generated by training the second real label data T2 as the second supervisory signal.

[0057] Specifically, the loss Loss generated by training the first real label data T1 Update the weight parameters of the first deep learning segmentation model, using the loss Loss generated by the second real label data training T2 Update the weight parameters of the second deep learning segmentation model.

[0058] In one embodiment, the first pseudo label of the unlabeled training data set is calibrated using the trained first deep learning segmentation model, and the loss Loss generated by the training of the first pseudo label data is calculated. F1 include:

[0059] The prediction result of the trained first deep learning segmentation model on the unlabeled training data set is used as the first pseudo label, and the loss Loss generated by the first pseudo label data training is calculated according to the first pseudo label and the prediction result of the second deep learning segmentation model. F1 .

[0060] The second pseudo label of the unlabeled training data set is calibrated by using the trained second deep learning segmentation model, and the loss Loss generated by the second pseudo label data training is calculated. F2 include:

[0061] The prediction result of the trained second deep learning segmentation model on the unlabeled training data set is used as the second pseudo label, and the loss Loss generated by the second pseudo label data training is calculated according to the second pseudo label and the prediction result of the first deep learning segmentation model. F2 .

[0062] Specifically, it is the loss Loss generated by the first pseudo-label data training F1 Configure the pseudo label loss weight coefficient W d , and serves as the supervisory signal of the second deep learning segmentation model to update the weight parameters of the second deep learning segmentation model; Loss generated by the second pseudo-label data training F2 Configure the pseudo label loss weight coefficient W d , and as the supervision signal of the first deep learning segmentation model, update the weight parameters of the first deep learning segmentation model by using the pseudo label loss weight coefficient W d , controls the contribution of pseudo labels when back-propagating to update model weight parameters.

[0063] It should be noted that P1 is the prediction result of the first deep learning segmentation model. When a real labeled training data set is input into the first deep learning segmentation model, P1 is the first real prediction result of the real labeled training data set; when an unlabeled training data set is input into the first deep learning segmentation model, P1 is the first pseudo prediction result of the unlabeled training data set. Similarly, P2 is the prediction result of the second deep learning segmentation model.

[0064] In one embodiment, the loss generated after the first deep learning segmentation model is trained is loss1, and the calculation formula is as follows:

[0065] loss1=loss T1 +Wd *loss F2

[0066] The loss generated after training the second deep learning segmentation model is loss2, and the calculation formula is as follows:

[0067] loss2 = loss T2 +W d *loss F1

[0068] Among them, Loss T1 is the loss generated by the first real label data training, Loss F2 is the loss generated by the second pseudo-label data training, Loss T2 The loss generated by training the second real label data, Loss F1 is the loss generated by the first pseudo-label data training; W d is the pseudo-label loss weight coefficient, and its expression is as follows:

[0069]

[0070] S c =S c +max[(α-θ),0.5]*S d

[0071]

[0072] Among them, S g is the current training step number, S c It is used to control the minimum value of each weight cycle and the decreasing cycle of the weight coefficient; floor is the rounding function, S d is the weight growth cycle, α is the weight cycle control coefficient, variable θ∈[0,α], the decreasing cycle starts from αS d , (α-1)S d Decrease to S d .

[0073] In one embodiment, Figure 2 As shown, the method of using the prediction result of the trained first deep learning segmentation model on the unlabeled training data set as the first pseudo label includes:

[0074] The prediction results of the trained first deep learning segmentation model on the unlabeled training data set are binarized and data enhanced, and the labels after the data enhancement processing are used as the first pseudo labels of the unlabeled training data set.

[0075] The step of using the prediction result of the trained second deep learning segmentation model on the unlabeled training data set as the second pseudo label includes:

[0076] The prediction results of the trained second deep learning segmentation model on the unlabeled training data set are binarized and data enhanced, and the labels after the data enhancement processing are used as the second pseudo labels of the unlabeled training data set.

[0077] Specifically, the prediction result P1 of the first deep learning segmentation model is binarized to obtain the first pseudo label Y1, and then the first pseudo label Y1 is subjected to data augmentation (Aug). Similarly, the prediction result P2 of the second deep learning segmentation model is binarized to obtain the second pseudo label Y2, and then the second pseudo label Y2 is subjected to data augmentation (Aug). The loss Loss generated by the first pseudo label data training is calculated based on the first pseudo label Y1 and the prediction result P2 of the second deep learning segmentation model. F1 ; Calculate the loss Loss generated by the second pseudo label data training based on the second pseudo label Y2 and the prediction result P1 of the first deep learning segmentation model F2 .

[0078] In one embodiment, the unlabeled training data set is image data of newly acquired pathological sections or data from a truly labeled training data set with label information removed.

[0079] In one embodiment, in step S110, the image data of the pathological slice is processed to perform at least one of the following operations: data cropping, data scaling, generating a true mask, and data enhancement.

[0080] Specifically, the collected image data of pathological sections are processed to make them suitable for deep learning segmentation models. Data processing includes: raw data acquisition, data screening, data cropping or scaling, generating real masks and data enhancement (rotation, flipping, blurring, adding noise, color jittering, etc.); data enhancement can avoid overfitting.

[0081] For example, the collected retinal pathology images and the corresponding pathology annotation information are first preprocessed, that is, the retinal pathology images are cropped or scaled to an image size suitable for the input of the deep learning segmentation model, and the true contour mask and true area mask of the cropped image are generated according to the pathology annotation information. The processed data is then subjected to multiple data enhancement. The training data already contains retinal images of various situations, which is sufficient to train the deep learning segmentation model to complete the cell segmentation task.

[0082] In one embodiment, in step S120, the blood vessel deep learning segmentation model adopts a standard U-Net model; the ganglion cell deep learning segmentation model adopts an improved U-Net model, and the improved U-Net model adopts a residual module to replace the encoding module in the standard U-Net model.

[0083] Specifically, the deep learning segmentation model is used to locate the target that needs to be measured in the pathological image. The deep learning segmentation model includes a standard U-Net model and an improved U-Net model; the standard U-Net model is used to segment the blood vessels in the retinal image, and the improved U-Net model is used to segment the ganglion cells (RGC, retinal ganglion cells) in the retinal image. The standard U-Net model and the improved U-Net model can obtain a high-accuracy model with less training data and shorter training time, and locate the ganglion cells and blood vessels and calculate indicators at a faster running speed.

[0084] Among them, the standard U-Net model is one of the classic network structures for deep learning networks to complete segmentation tasks. It can quickly perform image segmentation using a small amount of training data. By using a symmetrical encoding and decoding structure, a unique skip connection, an innovative elastic deformation data enhancement strategy, and an improved loss function, the network can obtain highly accurate segmentation results on medical images with a small amount of annotations.

[0085] The improved U-Net model is the Res-UNet model. The improved U-Net model uses a residual module to replace the encoding module in the standard U-Net model. The residual module draws on the cross-layer connection idea of ​​the high-speed network. This residual jumping structure breaks the traditional neural network n-1 layer output can only give the n layer as input convention, so that the output of a certain layer can directly skip several layers as the input of a certain layer behind. Adding a residual module can not only prevent the network gradient from disappearing but also increase the connection between deep and shallow networks.

[0086] In one embodiment, in step S120, the standard U-Net model includes at least one encoding module, at least one decoding module and an output module, and the encoding module is symmetrical to the decoding module.

[0087] Specifically, Figure 5As shown in the figure, the standard U-Net model includes a five-level encoding module (Encoding Module, i.e. EB1, EB2, EB3, EB4, EB5 in the figure), a five-level decoding module (Decoding Module, i.e. DB1, DB2, DB3, DB4, DB5 in the figure) and an output module (Output Module). Among them, the encoding module is mainly responsible for downsampling encoding, which is used to obtain rich context information and classify cells and backgrounds; the decoder is mainly responsible for upsampling decoding, which is used to achieve accurate positioning of cells; upsampling uses deconvolution to enlarge the feature map, which means that it can be consistent with the size of the shallow feature map and connected, and the shallow feature map is spliced ​​with the deep feature map through jump connection to obtain the final feature map, combining the fine-grained layer information and the coarse-grained layer information to obtain more accurate context information and more cell detail information. The output module is used to integrate the output of the deep learning segmentation model of the blood vessel into a blood vessel area mask, and perform binary cross entropy calculation loss with the true value. The U-Net network can be trained with a small number of training sets to obtain better segmentation results.

[0088] In one embodiment, each encoding module includes a feature extraction module and a two-dimensional maximum pooling layer; each decoding module includes a two-dimensional deconvolution layer, a feature fusion layer and a feature extraction module; the output module includes three two-dimensional convolution layers, wherein the two two-dimensional convolution layers close to the input end of the output module respectively have ReLU activation functions, and the two-dimensional convolution layer close to the output end has a Sigmoid activation function.

[0089] Specifically, Figure 6 As shown in (a), each encoding module includes a feature extraction block (FEB) and a 2D max pooling layer (2DMax Pooling Layer) arranged from the outside to the inside to complete the downsampling encoding work; Figure 6 As shown in (b), each decoding module includes a 2D transpose convolution layer, a feature fusion layer, and a feature extraction block, which are arranged from the outside to the inside to complete the upsampling decoding work; the output feature map of the 2D maximum pooling layer in the encoding module is input to the 2D deconvolution layer in the corresponding decoding module; the output feature map of the feature extraction module in the encoding module is input to the feature fusion layer of the corresponding decoding module to complete the fusion of feature scales, improve the problem of insufficient information during upsampling, and improve segmentation accuracy. Figure 6As shown in (c), the output module includes three 2D convolution layers, where the two 2D convolution layers at the input of the output module are respectively activated with ReLU functions, and the 2D convolution layer at the output of the output module is activated with Sigmoid functions. The output module integrates the output of the deep learning segmentation model of blood vessels into a blood vessel region mask, and performs binary cross entropy calculation loss with the true value.

[0090] In one embodiment, in step S120, the feature extraction module includes two two-dimensional convolutional layers and one dropout layer, and each two-dimensional convolutional layer has a ReLU activation function.

[0091] Specifically, Figure 7 As shown in the figure, the feature extraction module includes a two-dimensional convolution layer (Conv2D), a ReLU activation function, a discard layer (Dropout), Conv 2D, and a ReLU activation function, which are arranged from the outside to the inside. Dropout can solve the overfitting problem caused by too few model training samples.

[0092] Specifically, the ganglion cell deep learning segmentation model uses an improved U-Net model structure. Based on the standard U-Net model, the encoding module is replaced by a residual block, and the binary cross entropy loss of the contour mask and the region mask is calculated and then added as the final model loss value. The improved U-Net model can greatly reduce the information loss in the encoding and decoding process, while reducing the probability of model overfitting, and obtain a segmentation model with higher robustness.

[0093] The overall architecture of the improved U-Net model is as follows: Figure 8 As shown in the figure, the improved U-Net model consists of a five-level encoding module (Encoding Module, namely EB1, EB2, EB3, EB4, EB5), a five-level decoding module (Decoding Module, namely DB1, DB2, DB3, DB4, DB5 in the figure) and an output module (Output Module). Fig. 9 As shown in (a), each encoding module consists of a convolution residual block and an identity residual block arranged in sequence from the outside to the inside; Fig. 9As shown in (b), each decoding module is composed of a 2D transpose convolution layer, a feature fusion layer, and a feature extraction block to complete upsampling decoding; Fig. 9 As shown in (c), in the model output stage, three layers of convolution and Sigmoid activation function are used to integrate the model output into a cell contour mask and a cell area mask, and the binary cross entropy is performed with the true values ​​of the two masks and then added as the model loss.

[0094] In one embodiment, the identical residual module includes a residual branch, a bridge connection branch, an addition module and a ReLU activation function layer; the feature map output by the convolution residual module is respectively input into the residual branch and the bridge connection branch for processing;

[0095] The processed feature maps output by the residual branch and the bridge connection branch are added through the addition module, and the added result is input into the ReLU activation function layer for processing to obtain the input feature map of the next encoding module.

[0096] Specifically, Fig. 9 As shown in (a), the identical residual module in the encoding module is used to add the input image of the residual branch and the output image of the residual branch and use them as the input feature map of the next encoding module after being processed by the ReLU activation function.

[0097] In one embodiment, the residual branch in the identical residual module includes a two-dimensional convolutional layer, a batch normalization layer (Batch Normalization, BN), a ReLU activation function layer, a two-dimensional convolutional layer and a batch normalization layer arranged in sequence from the outside to the inside.

[0098] In one embodiment, the convolution residual module includes a residual branch, a bridge connection branch, an addition module and a ReLU activation function layer; the feature map to be processed is respectively input into the residual branch and the bridge connection branch for processing; the bridge connection branch includes a two-dimensional convolution layer and a batch normalization layer arranged in sequence from the outside to the inside;

[0099] The processed feature maps output by the residual branch and the bridge connection branch are added through the addition module, and the added result is input into the ReLU activation function layer for processing to obtain the input feature map of the identical residual module.

[0100] Specifically, Fig. 9As shown in (a), the residual branch structure of the convolutional residual module is the same as that of the identical residual module. In the convolutional residual module, the feature map is input into the bridge connection branch and the residual branch for processing; in the bridge connection branch, the feature map is processed and output by the two-dimensional convolution layer and the batch normalization layer in turn; in the residual branch, the feature map is processed by multiple two-dimensional convolution layers and batch normalization layers and the activation function ReLU and then output; the processed feature maps output from the bridge connection branch and the residual branch are added by the addition module and then processed by the activation function ReLU to obtain the input feature map of the identical residual module.

[0101] Therefore, the residual module is introduced into the U-Net network to shorten the path between deep abstract features and shallow features and enhance the feature propagation capability, so that more cell detail information can be extracted, avoiding the gradient vanishing phenomenon, while reducing the probability of model overfitting and obtaining a more robust segmentation model.

[0102] It should be noted that the standard U-Net and Res-UNet models are used as examples in this application for illustrative purposes. The deep learning segmentation network can also use a fully connected network (FCN), a nested U-Net network (U-Net++, U 2 -Net) or other deep learning segmentation models, or Mask-RCNN or other target detection networks, can also achieve the effect of the embodiment of the present invention. And the pseudo-label loss weight coefficient W provided in the embodiment of the present application d Semi-supervised techniques are also applicable to training these deep learning segmentation models.

[0103] Specifically, in step S140, the trained deep learning segmentation model is used to perform target segmentation on the pathological image to be analyzed. For example, the trained blood vessel deep learning segmentation model and ganglion cell deep learning segmentation model can predict all blood vessels and ganglion cells in a retinal fluorescence image, but may contain some false positive targets due to segmentation errors.

[0104] In a preferred embodiment, if Figure 4 As shown, after step S140, the following steps are also included:

[0105] S150: Using an image segmentation optimization unit to optimize the accuracy of the blood vessels and ganglion cells identified by the deep learning segmentation model to remove false positive targets.

[0106] Specifically, the image segmentation optimization unit is used to optimize the accuracy of the target located by the deep learning segmentation model and remove false positive targets.

[0107] In a preferred embodiment, in S150, the image segmentation optimization unit includes at least one of the following items: cell size screening, fluorescent stain mis-segmentation target screening, and background mis-segmentation target screening.

[0108] Specifically, the image segmentation optimization unit checks the features of the targets segmented by the deep learning segmentation model one by one, thereby deleting the targets with segmentation errors to achieve the purpose of optimizing the segmentation results.

[0109] In a preferred embodiment, the cell size screening includes: presetting a cell size threshold and screening out targets smaller than the preset cell size threshold.

[0110] According to the characteristic that the size of retinal ganglion cells in medical images is not less than 50 pixels (preset cell size threshold), the sizes of all predicted targets are first checked, and all targets with a size less than 50 pixels are screened out. The preset cell size threshold can also be set to other parameter values ​​according to actual needs.

[0111] In a preferred embodiment, a pixel intensity adaptive threshold is set, the average pixel intensity value of each target is checked and predicted, and targets with an intensity value higher than the pixel intensity adaptive threshold are screened out; wherein the pixel intensity adaptive threshold is is the mean intensity of all predicted targets, σ ​​is the standard deviation of the intensity of all predicted targets, and a is a random constant.

[0112] Specifically, according to the characteristic that the brightness of fluorescent stains is much higher than the brightness of cells, the average pixel intensity of all predicted targets is checked, and an adaptive threshold is set to filter out all predicted targets whose pixel intensity is higher than the adaptive threshold. The pixel intensity adaptive threshold S selection mechanism is as follows: Fig.10 As shown, firstly, the average pixel intensity of each predicted target is calculated, and the mean and standard deviation of the pixel intensity of all predicted targets are calculated. The mean plus three times the standard deviation is selected as the threshold, and the target above the adaptive threshold S is an outlier. Here, a is 3 for exemplary purposes and can also be set to other parameter values.

[0113] In one embodiment, the mean of the background pixel intensity is checked, the predicted mean pixel intensity value of each target is checked, and the targets whose difference between the mean pixel intensity value of the target and the mean pixel intensity of the background is less than a preset threshold are screened out.

[0114] Specifically, according to the characteristic that the brightness of the background mis-segmented target is similar to that of the background, the difference between the average pixel intensity value of all predicted targets and the average background pixel intensity value is checked, and all targets with too small brightness difference from the background are screened out, and the preset threshold is set to 15. After the above three optimization methods are processed, Fig.11 As shown in the figure, the targets that are not eliminated are determined to be target node cells, and the cell centroid is marked with a dot. Fig.11 It can be seen that the pathological image processing method based on the deep learning segmentation model provided in the embodiment of the present application can effectively identify and count all ganglion cells in retinal pathological images.

[0115] In a preferred embodiment, if Figure 4 As shown, after step S150, the following steps are further included:

[0116] S160: Calculate corresponding measurement indicators according to the target optimized by the image segmentation optimization unit.

[0117] Specifically, by performing a series of statistical calculations on the targets optimized by the image segmentation optimization unit, indicators that are helpful for pathological analysis and diagnosis are finally output to assist pathologists in analysis and diagnosis. The area of ​​the region of interest, the area of ​​blood vessels, the number of ganglion cells, and the density of ganglion cells in the fluorescently stained retinal pathological image can be calculated based on the targets optimized by the image segmentation optimization unit. The area of ​​the region of interest is defined as the total area of ​​all regions except blood vessels in the entire retinal fluorescent image, such as Fig.12 As shown in , the specific calculation formula is as follows:

[0118] Area of ​​ROI = (total number of image pixels - total number of blood vessel area pixels) × image resolution 2

[0119] In the formula, the image resolution factor is 0.66mpp (micron per pixel) in this embodiment, so the calculated area of ​​the region of interest is in μm 2 .

[0120] The target number of ganglion cells predicted by the image segmentation optimization unit can be checked, and the ganglion cell density can be calculated based on the predicted number of ganglion cells. The density is obtained by dividing the number of ganglion cells by the area of ​​the region of interest, and the unit is / μm 2 , the specific calculation formula is as follows:

[0121] Ganglion cell density = (number of ganglion cells ÷ area of ​​interest)

[0122] It should be noted that in the image segmentation optimization unit of the present invention, optimization methods such as outlier threshold screening, target size threshold segmentation, chromaticity threshold segmentation, watershed, clustering-based outlier detection, etc. can be used to achieve the effect of the embodiment of the present invention.

[0123] The present invention provides a pathological image processing method based on a deep learning segmentation model. The method is simple to operate and can intuitively display pathological index information. The target to be measured is segmented by the deep learning segmentation model, and the segmented results are optimized by image processing, and finally the required indicators are calculated and displayed. For projects that require pathologists to manually count specific targets under a microscope, it can not only reduce the workload of pathologists, but also improve the stability and efficiency of pathological index calculation.

[0124] It should be noted that although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. For example, step S120 can be performed first and then step S110.

[0125] Fig.13 A schematic structural diagram of an electronic device provided according to an embodiment of the present application is shown.

[0126] like Fig.13 As shown, as another aspect, the present application also provides an electronic device 200, including one or more central processing units (CPU) 201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 202 or the program loaded from the storage part 208 to the random access memory (RAM) 203. In the RAM 203, various programs and data required for system operation are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0127] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0128] In particular, according to the embodiments of the present disclosure, the above reference Figure 1-2The described process can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes program code for executing a pathology image processing method based on a deep learning segmentation model. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 209, and / or installed from a removable medium 211.

[0129] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] As another aspect, the present application also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the pathological image processing method based on the deep learning segmentation model described in the present application.

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs a specified function or operation, or can be realized by a combination of dedicated hardware and computer instructions.

[0132] The units or modules involved in the embodiments described in the present application may be implemented by software or by hardware. The units or modules described may also be arranged in a processor. For example, each of the units may be a software program arranged in a computer or a mobile smart device, or may be a separately configured hardware device. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0133] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. A pathological image processing method based on a deep learning segmentation model, characterized in that: include: Collecting image data of pathological sections, and processing the image data of the pathological sections to obtain a training data set; Building a deep learning segmentation model, wherein the deep learning segmentation model includes a blood vessel deep learning segmentation model and a ganglion cell deep learning segmentation model; Combined with the training data set, a pseudo-label loss weight coefficient is used The deep learning segmentation model is trained by using a semi-supervised technology to obtain a trained deep learning segmentation model, specifically comprising: Inputting the real annotated training data set into two deep learning segmentation models for parallel training respectively, the two deep learning segmentation models are respectively a first deep learning segmentation model and a second deep learning segmentation model with the same structure and different weight parameters, obtaining the trained first deep learning segmentation model and the second deep learning segmentation model, as well as a first supervisory signal corresponding to the first deep learning segmentation model and a second supervisory signal corresponding to the second deep learning segmentation model; Using the first supervisory signal to update the weight parameters of the first deep learning segmentation model, and retraining the first deep learning segmentation model; using the second supervisory signal to update the weight parameters of the second deep learning segmentation model, and retraining the second deep learning segmentation model; The first pseudo label of the unlabeled training data set is calibrated using the trained first deep learning segmentation model, and the loss Loss generated by the first pseudo label data training is calculated. F1 ; Use the trained second deep learning segmentation model to calibrate the second pseudo label of the unlabeled training data set, and calculate the loss Loss generated by the second pseudo label data training F2 ; The loss Loss generated by the second pseudo-label data training F2 Configure pseudo label loss weight coefficient , the pseudo label loss weight coefficient will be configured The second pseudo label is used as a supervisory signal of the first deep learning segmentation model, weight parameters of the first deep learning segmentation model are updated, and the first deep learning segmentation model is retrained; The loss Loss generated by the first pseudo-label data training F1 Configure pseudo label loss weight coefficient , the pseudo label loss weight coefficient will be configured The first pseudo label is used as a supervisory signal of a second deep learning segmentation model, a weight parameter of the second deep learning segmentation model is updated, and the second deep learning segmentation model is retrained; Selecting a model with better model convergence from the first deep learning segmentation model and the second deep learning segmentation model as a trained deep learning segmentation model; Using the trained deep learning segmentation model to identify blood vessels and ganglion cells in the image to be tested; The loss generated after training the first deep learning segmentation model is , the calculation formula is as follows: The loss generated after training the second deep learning segmentation model is , the calculation formula is as follows: Among them, Loss T1 is the loss generated by the first real label data training, Loss F2 is the loss generated by the second pseudo-label data training, Loss T2 The loss generated by training the second real label data, Loss F1 The loss generated by the first pseudo-label data training; pseudo-label loss weight coefficient The expression is as follows: in, is the current training step number, Used to control the minimum value of each weight cycle and to control the decreasing cycle of the weight coefficient; is the weight growth cycle, α is the weight cycle control coefficient, and the variable , is the floor function.

2. The pathological image processing method based on the deep learning segmentation model according to claim 1, characterized in that: The method for determining the first supervisory signal includes: Calculate the loss Loss generated by the first real label data training according to the first real prediction result of the first deep learning segmentation model on the real labeled training data set and the real labeled training data set T1 , the loss Loss generated by the first real label data training T1 As the first supervisory signal; The method for determining the second supervisory signal includes: Calculate the loss Loss generated by the second real label data training according to the second real prediction result of the second deep learning segmentation model on the real labeled training data set and the real labeled training data set T2 , the loss Loss generated by training the second real label data T2 as the second supervisory signal.

3. The pathological image processing method based on the deep learning segmentation model according to claim 1, characterized in that: The first pseudo label of the unlabeled training data set is calibrated by using the trained first deep learning segmentation model, and the loss Loss generated by the training of the first pseudo label data is calculated. F1 include: The prediction result of the trained first deep learning segmentation model on the unlabeled training data set is used as the first pseudo label, and the loss Loss generated by the first pseudo label data training is calculated according to the first pseudo label and the prediction result of the second deep learning segmentation model. F1 ; The second pseudo label of the unlabeled training data set is calibrated by using the trained second deep learning segmentation model, and the loss Loss generated by the second pseudo label data training is calculated. F2 include: The prediction result of the trained second deep learning segmentation model on the unlabeled training data set is used as the second pseudo label, and the loss Loss generated by the second pseudo label data training is calculated according to the second pseudo label and the prediction result of the first deep learning segmentation model. F2 .

4. The pathological image processing method based on the deep learning segmentation model according to claim 3 is characterized in that: The step of using the prediction result of the trained first deep learning segmentation model on the unlabeled training data set as the first pseudo label includes: Binarizing and data enhancing the prediction results of the trained first deep learning segmentation model on the unlabeled training data set, and using the label after the data enhancement process as the first pseudo label of the unlabeled training data set; The step of using the prediction result of the trained second deep learning segmentation model on the unlabeled training data set as the second pseudo label includes: The prediction results of the trained second deep learning segmentation model for the unlabeled training data set are binarized and data enhanced, and the labels after the data enhancement processing are used as second pseudo labels of the unlabeled training data set.

5. The pathological image processing method based on the deep learning segmentation model according to claim 1, characterized in that: The unlabeled training data set is the image data of newly acquired pathological sections or the data of the truly labeled training data set with the label information removed.

6. The pathological image processing method based on the deep learning segmentation model according to claim 1 is characterized in that: The blood vessel deep learning segmentation model adopts a standard U-Net model; the ganglion cell deep learning segmentation model adopts an improved U-Net model, and the improved U-Net model adopts a residual module to replace the encoding module in the standard U-Net model.

7. The pathological image processing method based on the deep learning segmentation model according to claim 1 is characterized in that: After the trained deep learning segmentation model is used to identify blood vessels and ganglion cells in the image to be tested, the method further includes: An image segmentation optimization unit is used to optimize the accuracy of blood vessels and ganglion cells identified by the deep learning segmentation model to remove false positive targets.

8. The pathological image processing method based on the deep learning segmentation model according to claim 7 is characterized in that: The method of using an image segmentation optimization unit to optimize the accuracy of blood vessels and ganglion cells identified by the deep learning segmentation model, and removing false positive targets also includes: The corresponding measurement index is calculated according to the target optimized by the image segmentation optimization unit.

9. The pathological image processing method based on the deep learning segmentation model according to claim 8, characterized in that: The image segmentation optimization unit optimizes the accuracy of the blood vessels and ganglion cells identified by the deep learning segmentation model, including at least one of the following operations: outlier threshold screening, target size threshold segmentation, chromaticity threshold segmentation, watershed, and clustering-based outlier detection.

10. The pathological image processing method based on the deep learning segmentation model according to any one of claims 1 to 9, characterized in that: The image data of the pathological slice is processed to perform at least one of the following operations: data cropping, data scaling, generating a true mask, and data enhancement.

11. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the pathological image processing method based on the deep learning segmentation model as described in any one of claims 1-10.

Images