An organoid segmentation method and system based on improved U-Net network

Abstract

The present invention relates to an organoid segmentation method and system based on an improved U-Net network, and relates to the field of image segmentation. The method comprises: inputting an organoid image to be segmented into an organoid segmentation model, and outputting an organoid segmentation map; the organoid segmentation model is obtained by training the improved U-Net based on an organoid dataset; an encoder comprises N sequentially connected encoding layers and a feature enhancement unit connected to the Nth encoding layer, each encoding layer comprises a dynamic convolution module and a downsampling module, the feature enhancement unit comprises a dynamic convolution module and a coordinated attention module, and the coordinated attention module is used to refine and enhance input feature information; a decoder comprises N sequentially connected decoding layers, each decoding layer comprises an upsampling module and a dynamic convolution module connected in sequence; each encoding layer and the decoding layer corresponding to each encoding layer are connected using an attention gate jump connection; and the downsampling module adopts a convolution downsampling operation. The present invention improves image segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation, and in particular to an organoid segmentation method and system based on an improved U-Net network. Background Art

[0002] Organoids are three-dimensional models that better reflect individual conditions than cell-based models. Currently, organoids can simulate organs such as the brain, retina, and intestine. Tumor organoids, derived directly from patients, can mimic the in vivo tumor microenvironment, allowing in vitro cultured tumor cells to be placed in a more realistic environment, improving the accuracy of oncology drug screening research applications.

[0003] With the continuous development of artificial intelligence in recent years, deep convolutional neural networks (DCNNs), represented by SegNet, VGGNet, GAN, and ResNet, have been widely used in the field of computer vision. Powered by convolutional networks, medical image acquisition and analysis have achieved significant progress, such as in fundus retinal lesion detection and classification, tumor target detection, and lung nodule classification. In organoid drug screening research, researchers typically observe and screen organoid images with the naked eye. This approach is labor-intensive and inefficient, and the enormous workload can also lead to reduced accuracy. In 2015, Shelhamer et al. employed fully convolutional networks (FCNs) to perform pixel-level, end-to-end image segmentation. FCNs are considered the pioneering work in network models for image segmentation. Subsequently, Ronneberger et al. proposed the U-Net, building on the FCN. This model's skip connections effectively compensate for the information loss inherent in the FCN's downsampling process. Due to its simple structure, small number of parameters, and strong flexibility, this model has been widely used in various image segmentation tasks, achieving particularly remarkable results in medical image segmentation. However, the segmentation effect of some detailed parts in medical images is still lacking. For example, in the retinal blood vessel segmentation task, the segmentation of small terminal blood vessels may be broken or missing. Later researchers improved the U-Net framework. For example, Quan et al. combined the U-Net segmentation network with the first residual structure, while improving the skip connection and constructing a deeper model for segmentation. Chen et al. combined the Transformer model with U-Net to construct TransUNet, which was used on the multi-organ segmentation dataset Synapse (including 8 abdominal organs: aorta, gallbladder, spleen, kidney, etc.) and achieved good segmentation results.

[0004] At present, due to problems such as uneven imaging brightness, blurred organoid boundary contours, organoid ghosting and adhesion, and low contrast with the background during optical imaging, these problems affect the accuracy of organoid segmentation, and the accuracy of organoid segmentation still needs to be improved. Summary of the Invention

[0005] The purpose of the present invention is to provide an organoid segmentation method and system based on an improved U-Net network, thereby improving image segmentation accuracy.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] An organoid segmentation method based on an improved U-Net network, comprising:

[0008] Obtain the organoid image to be segmented;

[0009] Inputting the organoid image to be segmented into an organoid segmentation model and outputting an organoid segmentation map; the organoid segmentation model is obtained by training an improved U-Net based on the organoid dataset;

[0010] The encoder of the improved U-Net includes N sequentially connected encoding layers and a feature enhancement unit connected to the Nth encoding layer, each encoding layer includes a dynamic convolution module and a downsampling module, the feature enhancement unit includes a dynamic convolution module and a coordinated attention module, and the coordinated attention module is used to refine and enhance the input feature information; the decoder includes N sequentially connected decoding layers, each decoding layer includes an upsampling module and the dynamic convolution module connected in sequence; each encoding layer and the decoding layer corresponding to each encoding layer adopts an attention gate jump connection; the downsampling module adopts a convolution downsampling operation.

[0011] Optionally, it also includes:

[0012] Binarizing the organoid segmentation map;

[0013] Extract the outlines of various organs in the binarized organoid segmentation image;

[0014] Compressing the contour points of each extracted contour and storing the compressed contour point coordinates according to the organoid;

[0015] Write the stored contour point coordinates of various organs into a fixed-format JSON file;

[0016] Open the JSON file with annotation software to obtain sample images with labels;

[0017] Expanding the organoid dataset based on a set number of labeled sample images corresponding to the organoid segmentation map to obtain an expanded organoid dataset;

[0018] The improved U-Net was trained using the expanded organoid dataset to obtain an updated organoid segmentation model;

[0019] The updated organoid segmentation model is used to perform organoid segmentation on the organoid images to be segmented.

[0020] Optionally, the first coding layer is used to input the organoid image to be segmented, the first coding layer to the N-1th coding layer all include the dynamic convolution module and the downsampling module connected in sequence, the Nth coding layer includes the dynamic convolution module, the coordinated attention module and the downsampling module connected in sequence, and the feature enhancement unit includes the dynamic convolution module and the coordinated attention module connected in sequence.

[0021] Optionally, each of the dynamic convolution modules is sequentially connected to a first residual structure, a first batch normalization layer, a first dynamic convolution layer, a second batch normalization layer, a second dynamic convolution layer and a third batch normalization layer, the first batch normalization layer, the second batch normalization layer and the third batch normalization layer all include a ReLu activation function, the first dynamic convolution layer and the second dynamic convolution layer have different hole rates, the dynamic convolution module also includes a first convolution layer, the output of the first residual structure is connected to the input of the first convolution layer, the output of the first convolution layer is subtracted from the output of the third batch normalization layer as the output of the dynamic convolution module, and the first convolution layer includes a convolution operation with a convolution kernel of 1×1.

[0022] Optionally, the convolution kernel of the first dynamic convolution layer is 3×3, and the dilation ratio is 1; the convolution kernel of the second dynamic convolution layer is 3×3, and the dilation ratio is 2.

[0023] Optionally, the decoder further includes a fourth convolutional layer connected to each decoding layer and a fifth convolutional layer connected to the feature enhancement unit, and the feature maps output by each of the fourth convolutional layer and the fifth convolutional layer are feature fused to output the organoid segmentation map; the fourth convolutional layer and the fifth convolutional layer both include a convolution operation with a convolution kernel of 1×1.

[0024] Optionally, the organoid image to be segmented is a bladder cancer organoid image.

[0025] The present invention also discloses an organoid segmentation system based on an improved U-Net network, comprising:

[0026] An organoid image acquisition module to be segmented, used to acquire images of the organoid to be segmented;

[0027] An organoid segmentation module is configured to input the organoid image to be segmented into an organoid segmentation model and output an organoid segmentation map; the organoid segmentation model is obtained by training an improved U-Net based on the organoid dataset;

[0028] The encoder of the improved U-Net includes N sequentially connected encoding layers and a feature enhancement unit connected to the Nth encoding layer, each encoding layer includes a dynamic convolution module and a downsampling module, the feature enhancement unit includes a dynamic convolution module and a coordinated attention module, and the coordinated attention module is used to refine and enhance the input feature information; the decoder includes N sequentially connected decoding layers, each decoding layer includes an upsampling module and the dynamic convolution module connected in sequence; each encoding layer and the decoding layer corresponding to each encoding layer adopts an attention gate jump connection; the downsampling module adopts a convolution downsampling operation.

[0029] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0030] The present invention adopts a dynamic convolution module to improve the feature extraction capability, and the downsampling module adopts a convolution downsampling operation to reduce information loss. By adding a coordinated attention module at the bottom layer of the encoder, the extracted features are refined and enhanced, thereby improving the image segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 This is a flow chart of an organoid segmentation method based on an improved U-Net network according to the present invention;

[0033] Figure 2 is the original image of the sample of the present invention;

[0034] Figure 3 is the label corresponding to the original image of the sample of the present invention;

[0035] Figure 4 The label image corresponding to the original sample image of the present invention;

[0036] Figure 5 This is a schematic diagram of the improved U-Net network model training process of the present invention;

[0037] Figure 6This is a schematic diagram of the improved U-Net network structure of the present invention;

[0038] Figure 7 This is a schematic diagram of the structure of the dynamic convolution module of the present invention;

[0039] Figure 8 This is a schematic diagram of the structure of the coordinated attention module of the present invention;

[0040] Figure 9 Note the schematic diagram of the door structure of the present invention;

[0041] Figure 10 Schematic diagram for comparing segmentation performance indicators of different models;

[0042] Figure 11 Schematic diagram for comparing segmentation results of different models;

[0043] Figure 12 Schematic diagram of a drug screening method based on an improved U-Net network-based organoid segmentation method of the present invention;

[0044] Figure 13 Schematic diagram of the automatic annotation process of the organoid segmentation map of the present invention;

[0045] Figure 14 This is a schematic diagram of the coordinate information of the organoid contour points of the present invention;

[0046] Figure 15 for Figure 12 Flowchart of the process of model segmentation of organoids and drawing violin plots in the drug screening method flow;

[0047] Figure 16 This is a statistical diagram of the organoid area of the present invention;

[0048] Figure 17 A schematic diagram of a violin reflecting organoid growth according to the present invention;

[0049] Figure 18 This is a schematic diagram of the structure of an organoid segmentation system based on an improved U-Net network in the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] The purpose of the present invention is to provide an organoid segmentation method and system based on an improved U-Net network, thereby improving image segmentation accuracy.

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Figure 1 This is a flow chart of an organoid segmentation method based on an improved U-Net network of the present invention. Figure 1 As shown, an organoid segmentation method based on an improved U-Net network includes:

[0054] Step 101: Obtain an image of the organoid to be segmented.

[0055] The organoid image to be segmented is a bladder cancer organoid image.

[0056] Step 102: Input the organoid image to be segmented into an organoid segmentation model, and output an organoid segmentation map; the organoid segmentation model is obtained by training an improved U-Net network based on the organoid dataset.

[0057] Improve the U-Net network such as Figure 6 As shown, the improved U-Net network includes an encoder and a decoder. Figure 6 The left side of the diagram shows the encoder, and the right side shows the decoder. The input image enters the encoder, where it undergoes feature extraction through a dynamic convolution module and downsampling. The decoder then upsamples the extracted feature map to the original input image size. The decoder outputs from each layer are then fused, and finally, a one-dimensional convolution is used to compress the number of channels and output the organoid segmentation map.

[0058] Figure 6 The CA (CoordinateAttention) module and AG (Attention Gate) module are two attention mechanisms.

[0059] The encoder of the improved U-Net network includes N sequentially connected encoding layers and a feature enhancement unit connected to the Nth encoding layer, each encoding layer includes a dynamic convolution module and a downsampling module, the feature enhancement unit includes a dynamic convolution module and a coordinated attention module, and the coordinated attention module is used to refine and enhance the input feature information; the decoder includes N sequentially connected decoding layers, each decoding layer includes an upsampling module and the dynamic convolution module connected in sequence; each encoding layer and the decoding layer corresponding to each encoding layer adopts an attention gate jump connection; the downsampling module adopts a convolution downsampling operation.

[0060] As a specific implementation, the encoder of the improved U-Net network of the present invention includes 4 encoding layers connected in sequence and 4 decoding layers connected in sequence.

[0061] The downsampling module is a convolution operation with a convolution kernel of 5×5 (Conv5×5).

[0062] Count the area of organoids in the organoid segmentation map.

[0063] Different drug-induced growth environments are constructed for organoids. By comparing the statistical areas of organoids corresponding to the images of organoids to be segmented under different growth environments, the growth conditions of organoids under different growth environments are reflected, thereby realizing drug screening.

[0064] The first coding layer is used to input the organoid image to be segmented, the first coding layer to the N-1th coding layer all include the dynamic convolution module and the downsampling module connected in sequence, the Nth coding layer includes the dynamic convolution module, the coordinated attention module and the downsampling module connected in sequence, and the feature enhancement unit includes the dynamic convolution module and the coordinated attention module connected in sequence.

[0065] The first input end of the i-th attention gate is connected to the output end of the dynamic convolution module in the i-th encoding layer, the second input end of the i-th attention gate is connected to the output end of the upsampling module in the N+1-i-th decoding layer, and the output end of the i-th attention gate is connected to the dynamic convolution module in the N+1-i-th decoding layer.

[0066] Note that the door structure is as follows Figure 9 As shown. The i-th attention gate is used to adjust the size of the output feature map of the dynamic convolution module in the i-th encoding layer and the output feature map of the upsampling module in the N+1-i-th decoding layer to the same through a one-dimensional convolution operation, and adjust the number of channels to the same, to obtain a first feature map and a second feature map, to add the first feature map and the second feature map point by point, to obtain a third feature map, to pass the third feature map through the ReLU activation function and the Sigmoid function, to obtain a weight coefficient matrix, and to resample the weight coefficient matrix to restore it to the same size as the output feature map of the dynamic convolution module in the i-th encoding layer, and then multiply it with the output feature map of the dynamic convolution module in the i-th encoding layer to obtain the output of the i-th attention gate. Figure 9 Middle W g 、W x and ψ both represent one-dimensional convolution operations.

[0067] The upsampling module is Figure 6 Transposed Conv in.

[0068] The AG module (attention gate) is designed to address the semantic gap at the jump connection. Specifically, a gating signal is added at the junction of the encoder and decoder features to control the importance of features at different spatial locations in the feature map, filtering out and suppressing feature responses in irrelevant areas. The decoder's feature map and the previous encoder's feature map are used as module inputs. Since the inputs come from different layers of the model, the size and number of channels of the input feature maps need to be adjusted to the same through one-dimensional convolution. This process then adds the input x to the original image. l Same as g, the region of interest is enhanced, x l Represents the output of the encoding layer, and g represents the output of the upsampling module. After the ReLU activation function is applied, these areas of interest are strengthened and irrelevant areas are suppressed. Finally, the weight coefficients are calculated by the Sigmoid function, and the weight coefficient matrix is restored to the input x by Resample. l Multiply the same size to get the output result The AG module adopts the idea of Soft-Attention, and the attention weight coefficient can be continuously adjusted as the model is trained.

[0069] like Figure 7 As shown, each of the dynamic convolution modules (Res-double Dynamic Conv modules) is sequentially connected to the first residual structure ( Figure 7 Residual), a first batch normalization layer (BN, ReLu), a first dynamic convolution layer (Dy Conv3×3, rate=1), a second batch normalization layer (BN, ReLu), a second dynamic convolution layer (Dy Conv3×3, rate=2) and a third batch normalization layer (BN, ReLu), the first batch normalization layer, the second batch normalization layer and the third batch normalization layer all include a ReLu activation function, the first dynamic convolution layer and the second dynamic convolution layer have different hole rates, the dynamic convolution module also includes a first convolution layer, the output of the first residual structure is connected to the input of the first convolution layer, the output of the first convolution layer is subtracted from the output of the third batch normalization layer as the output of the dynamic convolution module, and the first convolution layer includes a convolution operation with a convolution kernel of 1×1.

[0070] The dynamic convolution module not only enhances model extraction capabilities but also prevents exploding and vanishing gradients that can occur during deep network training. Furthermore, by adding different dilation rates to different convolutional layers, we can leverage multi-scale information without increasing computational overhead, further enhancing the encoder's feature extraction capabilities.

[0071] The convolution kernel of the first dynamic convolution layer is 3×3, and the dilation ratio is 1; the convolution kernel of the second dynamic convolution layer is 3×3, and the dilation ratio is 2. After the dilation ratio increases, the convolution kernel size is expanded to 5×5.

[0072] Standard convolution is expressed as y = g(W T x+b). Where y represents the convolution result, x represents the input, WT represents the weight matrix, g() represents the activation function, and b represents the bias.

[0073] The dynamic convolution operation formula of the dynamic convolution module is expressed as:

[0074]

[0075]

[0076] Wherein, y represents the output of the dynamic convolution module, x represents the input of the dynamic convolution module, and π k (x) is the kth function The attention weight, Represents the K convolution kernels aggregated

[0077] weight, Represents the bias after K convolution kernels are aggregated, represents the weight of the kth convolution kernel, represents the bias of the kth convolution kernel, and K represents the number of convolution kernels.

[0078] π k The dynamic convolution module changes with each input x, representing the optimal aggregation of the linear model for a given input. In addition, the aggregated model is a nonlinear function, so dynamic convolution has more feature expression capabilities than traditional convolution.

[0079] Convolutional networks typically use downsampling to increase the receptive field without increasing computational complexity. While downsampling can increase the receptive field, it inevitably reduces resolution, hindering segmentation accuracy. Incorporating dilation into the dynamic convolution module increases the kernel size and receptive field without increasing computational complexity, while maintaining spatial resolution and facilitating accurate object localization. Furthermore, combining convolutional modules with different dilation rates can enhance the model's ability to learn multi-scale features.

[0080] dilated kernel=kernel+(kernel-1)(rate-1);

[0081] Where dilated kernel is the size of the convolution kernel after adding the dilation rate, kernel is the original convolution kernel size, and rate is the dilation rate. When rate is 1, it is a standard 3×3 convolution, and when rate is 2, the convolution kernel size is expanded to 5×5.

[0082] is the receptive field calculation formula, where r n is the receptive field of this layer, r n-1 is the receptive field of the previous layer, s i is the convolution or pooling stride of the i-th layer, and k is the convolution kernel size. According to the above formula, we can calculate that the receptive field size of the first layer of the model after the original two-layer convolution module is 5×5, while the receptive field after the convolution with dilation rate is 7×7.

[0083] like Figure 8 As shown, the coordinated attention module (CA module) includes a second residual structure ( Figure 8 Residual), the first global average pooling layer (XAvgPool), the second global average pooling layer (YAvg Pool), the concatenation layer (Concat+Conv), the normalization layer (BatchNorm+Non-liner), the second convolutional layer, the third convolutional layer, the first Sigmoid function and the second Sigmoid function.

[0084] Figure 8 Here C represents the number of channels of the image, H represents the height of the image, W represents the width of the image, and r represents the channel scaling ratio.

[0085] The second residual structure is respectively connected to the first global average pooling layer and the second global average pooling layer, the first global average pooling layer is used to perform global average pooling in the height direction of the feature map output by the second residual structure, and the second global average pooling layer is used to perform global average pooling in the width direction of the feature map output by the second residual structure. The splicing layer is used to splice and reduce the channel dimension of the output of the first global average pooling layer and the output of the second global average pooling layer, and the normalization layer is used to normalize the output of the splicing layer and split the normalized feature map into a feature map in the height direction and a feature map in the width direction along the spatial dimension. The normalization processing layer is connected to the second convolution layer and the third convolution layer respectively, the second convolution layer is used to perform a convolution operation on the feature map in the height direction, and the third convolution layer is used to perform a convolution operation on the feature map in the width direction. The second convolution layer is connected to the first Sigmoid function, and the third convolution layer is connected to the second Sigmoid function. The first Sigmoid function is used to output a first weight coefficient, and the second Sigmoid function is used to output a second weight coefficient; the feature map output by the second residual structure is multiplied by the first weight coefficient and the second weight coefficient in sequence as the output of the coordinated attention module.

[0086] The CA module first performs global average pooling on the input feature map in both height and width directions, obtaining one-dimensional feature maps in both directions. This enables the attention module to capture long-range spatial interactions with precise positional information. These two features are then concatenated, and the number of channels is reduced and normalized. The feature map is then split along the spatial dimension into height and width, and the number of channels is increased through one-dimensional convolution. Finally, the weight coefficients are calculated using a sigmoid function and multiplied by the initial input feature map. The CA module is used at the bottom layer of the encoder, primarily to strengthen and refine the high-level features extracted by the encoder. Due to the limitations of convolution, convolutional networks struggle to capture long-range dependencies, meaning they cannot effectively utilize global information. Therefore, the CA module is introduced to improve this. The CA module not only captures cross-channel information but also the spatial positional information of the feature map.

[0087] The decoder further includes a fourth convolution layer connected to each decoding layer and a fifth convolution layer connected to the feature enhancement unit, and the feature maps output by each of the fourth convolution layer and the fifth convolution layer are subjected to feature fusion ( Figure 6 After the above steps, the organoid segmentation map is output through Conv1×1 (convolution operation with a convolution kernel of 1×1).

[0088] The fourth convolutional layer and the fifth convolutional layer both include convolution operations with a convolution kernel of 1×1.

[0089] First, the present invention changes the original double-layer convolution module into a dynamic convolution module with a combination of residual connection and void rate, which enhances the encoder's feature extraction capability and the utilization of multi-scale information; secondly, the addition of an attention module not only solves the semantic gap problem caused by jump connections, but also enhances the model's anti-interference ability, and can achieve excellent segmentation effects even in the case of complex image background information; finally, multi-scale feature splicing is added to the decoder part to enhance feature recovery capabilities. Through these three improvements, the organoid segmentation model improves the segmentation accuracy of organoid images. The subsequent work is to automatically perform area statistics on the organoids in the segmented image by a computer, and to reflect their growth status by performing area statistics on the organoids under different growth environments, thereby achieving the purpose of drug screening and evaluation.

[0090] The specific improvements of the present invention include: (1) replacing the standard convolution with a dynamic convolution module with a residual structure and a void rate, which can significantly improve the feature extraction capability of the model while slightly increasing the amount of computation. (2) changing the original maximum pooling downsampling to 5×5 convolution downsampling to further reduce information loss during the downsampling process. (3) adding a CA (CoordinateAttention) module to the bottom layer of the encoder to refine and strengthen the extracted features. (4) adding an Attention Gate at the jump connection to overcome the semantic gap generated when high-level features are fused with low-level features. (5) splicing the output of each layer of the decoder, making full use of feature information of different scales through feature fusion, and enhancing the feature recovery capability of the decoder. (6) the computer automatically counts and analyzes the area of the organoid after segmentation and draws violin plots. (7) developing computer automatic annotation, which can greatly reduce the time consumed by manual annotation when expanding the data set.

[0091] The improved U-Net network (Res-double Dynamic conv Attention U-Net, RDAU-Net) of the present invention can also achieve good segmentation accuracy when the data set is small.

[0092] The RDAU-Net model was built using the PyTorch framework in Python 3.8. The experimental environment was Ubuntu 20.04, with an i9-7900 CPU and an NVIDIA GTX3090 GPU with 24GB of video memory. The dataset contained 200 organoid images aged 1 to 7 days, 70% of which were used for training, 15% for validation, and 15% for testing.

[0093] During the project, the dataset will be expanded regularly to improve the generalization ability and segmentation accuracy of the organoid segmentation model. However, a bladder cancer cell organoid image contains 100 to 200 organoids that need to be labeled. Traditional manual labeling is time-consuming, labor-intensive, and inefficient. Figure 13 As shown, the present invention also discloses a method for automatically annotating organoid segmentation images, the key point of which is to extract the organoid coordinate information in the segmented binary organoid image. Each organoid outline in the image is searched, and the coordinates of each point on the outline are recorded. Then, a Python program is written to generate a JSON file in a specified format with these coordinate information. Then, the JSON file corresponding to the image is opened in the annotation software, and the existing annotations can be modified and saved directly based on it. After experimental comparison, it was found that batch annotation using this method can reduce the annotation time by more than 70%. Specifically, the following steps are included.

[0094] The organoid segmentation map is binarized.

[0095] The RETR_EXTERNAL function is used to extract the contours of various organs in the binarized organoid segmentation image.

[0096] The CHAIN_APPROX_TC89_L1 function is used to compress the contour points of each extracted contour using the The-Chini approximation algorithm, and the coordinates of the compressed contour points are stored according to the organoid. The storage format is as follows Figure 14 shown.

[0097] The stored contour point coordinates of various organs are written into a fixed-format JSON file.

[0098] Open the JSON file with annotation software to obtain sample images with labels.

[0099] The organoid dataset is expanded based on a set number of sample images carrying labels corresponding to the organoid segmentation images to obtain an expanded organoid dataset.

[0100] The improved U-Net was trained using the expanded organoid dataset to obtain an updated organoid segmentation model.

[0101] The updated organoid segmentation model is used to perform organoid segmentation on the organoid images to be segmented.

[0102] The RETR_EXTERNAL function and the CHAIN_APPROX_TC89_L1 function are both functions in the OpenVC library.

[0103] Figure 14To extract the coordinates of the outline points of a particular organoid, I wrote a Python program to write the extracted coordinates of the outline points into a fixed-format JSON file. By opening the JSON file in the annotation software, I could adjust the outline of each organoid and generate new labels.

[0104] Figure 12 FIG. 1 is a flow chart of a drug screening method based on an improved U-Net network-based organoid segmentation method of the present invention. Figure 12 As shown, the present invention automatically completes the last three steps through a computer, which can save at least 70% of the time compared to manual completion, greatly accelerating the drug screening research process.

[0105] The dataset for improving U-Net network training is provided by the Life Science Center of the School of Life Sciences of Yunnan University. The dataset consists of images of bladder cancer cell organoids after 1 to 7 days of treatment with different drugs. There are three scales: 5X, 10X, and 20X. Among them, the 5X image is used for area statistics and is also the dataset used in the present invention. At this stage, the dataset contains 200 images, of which the image sizes are 1944×2592 pixels and 768×1024 pixels respectively. Since image segmentation based on deep learning requires experts to manually draw labels for model learning. The present invention uses Labelme software to make labels. First, the organoids to be segmented are marked, and finally regenerated into black and white binary images. Figure 2 This is an image of an organoid on the third day. You can see a lot of organoid ghosting and bubbles in the image, which greatly increases the difficulty of model segmentation. Figure 3 The white area is the target organoid, and the black area is the background; Figure 4 The marked area is the organoid that the model needs to identify. The present invention uses the non-local means filter (NL-Means) method to preprocess the image and enhance the contrast between the target organoid and the background.

[0106] Improve the training process of U-Net network as follows Figure 5 As shown in the figure, the main function is first run to read the configuration files and data required by the improved U-Net network and build the improved U-Net network. The required organoid image dataset is read and trained on the improved U-Net network. After the improved U-Net network is trained, the organoid segmentation model is obtained, i.e., the target segmentation accuracy is achieved. The training weights are obtained, and the subsequent organoid image segmentation can be performed with the trained weights.

[0107] The present invention uses three indicators to evaluate the segmentation performance of the model, namely precision, recall and Dice Similarity Coefficient (DSC). Precision indicates the proportion of correctly classified organoid pixels to the total pixels predicted to be organoids; recall indicates the proportion of correctly classified organoid pixels to the actual total organoid pixels; and the Dice coefficient measures the similarity between the segmentation result and the label. The following compares the segmentation indicators of five models, namely U-Net, AttentionU-Net, U-Net++ and MultiResU-Net, as well as the RDAU-Net model of the present invention. Figure 10 , the segmentation result comparison is Figure 11 . Figure 10 The three indicators corresponding to each model are precision, recall and Dice coefficient.

[0108] like Figure 11 As shown in the figure, it was found that the U-Net basic model had problems such as misidentification of organoid ghosting and missing boundaries of segmented organoids. The improved RDAU-Net model can reduce the occurrence of misidentification, and at the same time, it can also reduce the phenomenon of missed identification of target organoid areas and incomplete boundaries. Figure 11 Here, OR represents the original image and GT represents the label image.

[0109] The flow chart of the organoid segmentation model and the process of drawing violin plots is as follows Figure 15 As shown, the prediction process first requires running the main function, which reads the required configuration files and data and then loads the network model. Starting with the image to be predicted, the dataset image size is adjusted. After network prediction, a segmented binary image is generated, and the organoid area within the image is counted. This organoid area data is presented as a violin plot for subsequent drug evaluation.

[0110] The organoid area statistics are implemented using the Opencv library function in Python. A program is written to search for connected domains in the segmented organoid binary image and record their areas, as follows Figure 16 As shown in the figure, organoid images from days 1 to 7 were processed in batches, and the organoid area data within the images was stored in Excel for easy access later. Violin plots were also created using a third-party Python library. By presenting the data stored in Excel graphically, it allows for more intuitive and convenient comparison of the growth of bladder cancer cell organoids under different drug environments. Figure 17This is a violin plot, which reflects the growth status of bladder cancer cell organoids under three environments from 1 to 7 days. The CTR group represents the control group without drug treatment, and RA and 14 represent the abbreviations of the drug names, respectively. Figure 17 The horizontal axis represents the number of days, and the vertical axis represents the area. Figure 17 The CTR group, RA group, and 14 group are divided into three groups, with CTR, 14, and RA in each group. For a more intuitive and convenient comparison, an quartile distribution diagram is added to the violin plot, and the dotted line in the middle represents the median of the data in this group. As can be seen from Figure 15, days 1 to 3 are the initial growth period of organoids. There is no obvious growth difference between drug-treated and non-drug-treated organoids. After the fifth day, the difference gradually becomes apparent. The CTR group was not treated with drugs, and its peak and median organoid area were higher than those of the RA group and the 14 group. This shows that these two drugs have a certain inhibitory effect on the growth of bladder cancer cell line organoids. To ensure the rigor of the experiment, the phenomenon reflected in this figure must be confirmed by professionals in the relevant field to be correct and reliable.

[0111] Experimental comparisons show that the present invention only takes 15 minutes to segment 30 organoid images and draw violin plots reflecting their growth patterns. Traditional manual observation and extraction methods require 4 to 6 hours. This suggests that the present invention can accelerate drug development and has broad application prospects in organoid anticancer drug screening research. Similarly, the present invention can be used to screen and test a wider range of drugs in this way, promoting the development of anticancer drugs.

[0112] Figure 18 This is a schematic diagram of the structure of an organoid segmentation system based on an improved U-Net network in the present invention. Figure 18 As shown, an organoid segmentation system based on an improved U-Net network includes:

[0113] The organoid image acquisition module 201 is used to acquire the organoid image to be segmented.

[0114] The organoid segmentation module 202 is used to input the organoid image to be segmented into an organoid segmentation model and output an organoid segmentation map; the organoid segmentation model is obtained by training an improved U-Net based on the organoid dataset.

[0115] The encoder of the improved U-Net includes N sequentially connected encoding layers and a feature enhancement unit connected to the Nth encoding layer, each encoding layer includes a dynamic convolution module and a downsampling module, the feature enhancement unit includes a dynamic convolution module and a coordinated attention module, and the coordinated attention module is used to refine and enhance the input feature information; the decoder includes N sequentially connected decoding layers, each decoding layer includes an upsampling module and the dynamic convolution module connected in sequence; each encoding layer and the decoding layer corresponding to each encoding layer are connected using an attention gate jump connection.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0117] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. An organoid segmentation method based on an improved U-Net network, characterized in that: include: Obtain the organoid image to be segmented; Inputting the organoid image to be segmented into an organoid segmentation model and outputting an organoid segmentation map; the organoid segmentation model is obtained by training an improved U-Net based on the organoid dataset; The improved U-Net encoder includes N sequentially connected encoding layers and a feature enhancement unit connected to the Nth encoding layer, each encoding layer includes a dynamic convolution module and a downsampling module, the feature enhancement unit includes a dynamic convolution module and a coordinated attention module, and the coordinated attention module is used to refine and enhance the input feature information; the decoder includes N sequentially connected decoding layers, each decoding layer includes an upsampling module and the dynamic convolution module connected in sequence; each encoding layer and the decoding layer corresponding to each encoding layer adopt an attention gate jump connection; the downsampling module adopts a convolution downsampling operation; The first coding layer is used to input the organoid image to be segmented, the first coding layer to the N-1th coding layer all include the dynamic convolution module and the downsampling module connected in sequence, the Nth coding layer includes the dynamic convolution module, the coordinated attention module and the downsampling module connected in sequence, and the feature enhancement unit includes the dynamic convolution module and the coordinated attention module connected in sequence; Each of the dynamic convolution modules includes a first residual structure, a first batch normalization layer, a first dynamic convolution layer, a second batch normalization layer, a second dynamic convolution layer and a third batch normalization layer connected in sequence, wherein the first batch normalization layer, the second batch normalization layer and the third batch normalization layer all include a ReLu activation function, and the first dynamic convolution layer and the second dynamic convolution layer have different hole rates. The dynamic convolution module also includes a first convolution layer, the output of the first residual structure is connected to the input of the first convolution layer, and the output of the first convolution layer is subtracted from the output of the third batch normalization layer as the output of the dynamic convolution module, and the first convolution layer includes a convolution operation with a convolution kernel of 1×1.

2. The organoid segmentation method based on the improved U-Net network according to claim 1, characterized in that Also includes: Binarizing the organoid segmentation map; Extract the outlines of various organs in the binarized organoid segmentation image; Compressing the contour points of each extracted contour and storing the compressed contour point coordinates according to the organoid; Write the stored contour point coordinates of various organs into a fixed-format JSON file; Open the JSON file with annotation software to obtain sample images with labels; Expanding the organoid dataset based on a set number of labeled sample images corresponding to the organoid segmentation map to obtain an expanded organoid dataset; The improved U-Net was trained using the expanded organoid dataset to obtain an updated organoid segmentation model; The updated organoid segmentation model is used to perform organoid segmentation on the organoid images to be segmented.

3. The organoid segmentation method based on the improved U-Net network according to claim 1, characterized in that The convolution kernel of the first dynamic convolution layer is 3×3, and the dilation ratio is 1; the convolution kernel of the second dynamic convolution layer is 3×3, and the dilation ratio is 2.

4. The organoid segmentation method based on the improved U-Net network according to claim 1, characterized in that The decoder also includes a fourth convolutional layer connected to each decoding layer and a fifth convolutional layer connected to the feature enhancement unit. The feature maps output by each of the fourth convolutional layer and the fifth convolutional layer are feature fused to output the organoid segmentation map; the fourth convolutional layer and the fifth convolutional layer both include a convolution operation with a convolution kernel of 1×1.

5. The organoid segmentation method based on the improved U-Net network according to claim 1, characterized in that The organoid image to be segmented is a bladder cancer organoid image.

6. An organoid segmentation system based on an improved U-Net network, characterized in that: include: An organoid image acquisition module to be segmented, used to acquire images of the organoid to be segmented; An organoid segmentation module is configured to input the organoid image to be segmented into an organoid segmentation model and output an organoid segmentation map; the organoid segmentation model is obtained by training an improved U-Net based on the organoid dataset; The improved U-Net encoder includes N sequentially connected encoding layers and a feature enhancement unit connected to the Nth encoding layer, each encoding layer includes a dynamic convolution module and a downsampling module, the feature enhancement unit includes a dynamic convolution module and a coordinated attention module, and the coordinated attention module is used to refine and enhance the input feature information; the decoder includes N sequentially connected decoding layers, each decoding layer includes an upsampling module and the dynamic convolution module connected in sequence; each encoding layer and the decoding layer corresponding to each encoding layer adopt an attention gate jump connection; the downsampling module adopts a convolution downsampling operation; The first coding layer is used to input the organoid image to be segmented, the first coding layer to the N-1th coding layer all include the dynamic convolution module and the downsampling module connected in sequence, the Nth coding layer includes the dynamic convolution module, the coordinated attention module and the downsampling module connected in sequence, and the feature enhancement unit includes the dynamic convolution module and the coordinated attention module connected in sequence; Each of the dynamic convolution modules includes a first residual structure, a first batch normalization layer, a first dynamic convolution layer, a second batch normalization layer, a second dynamic convolution layer and a third batch normalization layer connected in sequence, wherein the first batch normalization layer, the second batch normalization layer and the third batch normalization layer all include a ReLu activation function, and the first dynamic convolution layer and the second dynamic convolution layer have different hole rates. The dynamic convolution module also includes a first convolution layer, the output of the first residual structure is connected to the input of the first convolution layer, and the output of the first convolution layer is subtracted from the output of the third batch normalization layer as the output of the dynamic convolution module, and the first convolution layer includes a convolution operation with a convolution kernel of 1×1.

Images