Automatic labeling method for organoids based on global context-aware two-stream deep learning
Through the global context-aware dual-stream deep learning method, the dual-stream feature encoder and relational graph convolution neural network are used to build an organoid labeling model, solving the problem of low labeling accuracy of organoid bright field images, and achieving fast and accurate automatic labeling result.
Patent Information
- Application Number
- CN202410233736.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-01
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-03-01
AI Technical Summary
The prior art is difficult to achieve efficient and accurate annotation of organoid brightfield images, especially due to low contrast and large differences in organ size and shape, deep learning algorithms lack global prior information and multi-task assisted learning capabilities, resulting in low accuracy of labeling results.
Using a dual-stream deep learning method based on global context perception, the organoid labeling model is constructed through a dual-stream feature encoder, a relational graph convolutional neural network and a multi-task feature decoder, local and global features are extracted, and the features are optimized using multi-task training loss function. Consistency of extraction and annotation results.
It realizes fast, accurate, and high-throughput automatic labeling of organoid microscopy images, improves the labeling accuracy, robustness and generalization capabilities of the model, and solves the problem of low organoid labeling accuracy.
Smart Images

Figure CN118279240B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an automatic organoid labeling method based on global context-aware dual-stream deep learning, and belongs to the technical field of computer vision and deep learning. Background Art
[0002] In recent years, organoids have become a research hotspot because they can simulate the in vivo tissue structure and function to the greatest extent and can be cultured stably for a long time. Organoids are cell cultures differentiated and cultured in vitro. They can reproduce some of the functions of the corresponding organs through similar spatial organization to the corresponding organs, thus providing a highly physiologically relevant system. Organoid technology has great potential in many aspects, including developmental biology, disease pathology, cell biology, regeneration mechanisms, precision medicine, and drug toxicity and efficacy testing. Among them, labeling organoids in bright field images of organoids is a very important step in quality control testing. However, existing methods generally use manual labeling, which is time-consuming and labor-intensive, and there is subjectivity, making it difficult to ensure efficient and accurate labeling of organoids.
[0003] Recently, deep learning algorithms have achieved great success in medical image annotation, and a large amount of research has been devoted to tasks such as lesion annotation, cell annotation, and organ annotation. However, organoid annotation algorithms for brightfield images are in urgent need of development. At the same time, due to the low contrast of organoid brightfield images and the huge differences in size, shape, and density of organoids, existing deep learning algorithms lack the ability to extract global prior information and the ability to optimize feature extraction from multi-task assisted learning, which limits the accuracy of organoid annotation results. Therefore, it is a challenging problem to propose a deep learning organoid annotation framework based on global context perception. Summary of the invention
[0004] In order to improve the accuracy of organoid annotation results, the present invention provides an automatic organoid annotation method based on global context-aware dual-stream deep learning, and the technical solution is as follows:
[0005] The first object of the present invention is to provide an organoid automatic labeling method, comprising:
[0006] Step 1: Acquire bright field images of organoids;
[0007] Step 2: Processing the organoid bright field image using the organoid annotation model to obtain an organoid annotation mask, thereby completing automatic annotation of the organoid;
[0008] The organoid annotation model includes: a dual-stream feature encoder, a relational graph convolutional neural network, a feature fusion module and a multi-task feature decoder;
[0009] The dual-stream feature encoder includes a whole-image feature encoder and a patch feature encoder, wherein the whole-image feature encoder is used to extract local features of the organoid bright field image, and the patch feature encoder is used to extract block features of the organoid bright field image block;
[0010] The relationship graph convolutional neural network is used to mine the global prior dependency between the organoid bright field image blocks, so as to obtain the global features of the organoid bright field image;
[0011] The feature fusion module is used to fuse the local features and global features of the organoid bright field image;
[0012] The multi-task feature decoder includes: a labeling decoder, a reconstruction decoder and a Patch decoder, wherein the labeling decoder predicts the organoid labeling result, the reconstruction decoder is used to predict the organoid bright field image reconstruction result, and the Patch decoder is used to predict the organoid block labeling result;
[0013] The whole-image feature encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the annotation decoder; the patch encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the patch decoder.
[0014] Optionally, the training process of the organoid annotation model includes:
[0015] Combined with the fluorescent protein staining image, labelme is used to annotate the organoids in the input bright field image;
[0016] Construct training sample sets and test sample sets;
[0017] The organoid annotation model is trained, and the multi-task training loss function includes: organoid annotation main loss, multi-task auxiliary loss and spatial consistency loss, wherein the organoid annotation main loss is used to guide network annotation learning, the multi-task auxiliary loss is used to optimize the high-level semantic information and edge detail information of the extracted features, and the spatial consistency loss is used to constrain the consistency of the annotation results in different spaces;
[0018] The calculation formula for the loss of the organoid annotation subject is:
[0019]
[0020] The calculation formula for the multi-task auxiliary loss of organoid annotation is:
[0021]
[0022]
[0023]
[0024] in, is the reconstruction loss, is the content constraint loss, is the structural constraint loss, is the block annotation loss, η is the weight factor to balance the weights between reconstruction and block annotation tasks;
[0025] The calculation formula of the spatial consistency loss is:
[0026]
[0027] Among them, frac is the fragmentation operation, p s is the annotation prediction result, p p Label the results for Patch;
[0028] Therefore, the optimized multi-task training loss function is:
[0029]
[0030] Among them, λ1 and λ2 are weight factors to balance the relationship between losses.
[0031] Optionally, the whole-image feature encoder and the patch feature encoder each include four sub-modules, each sub-module including two convolutional layers connected in sequence, a ReLU activation function, a batch normalization layer, a maximum pooling layer and a residual connection layer.
[0032] Optionally, the relational graph convolutional neural network includes: two graph convolutional layers, the number of graph nodes is defined as N, the extracted block feature vector is x, and the relationship metric between the i-th block and the j-th block can be expressed as:
[0033]
[0034] After obtaining the relationship between block features, the adjacency matrix A = [w(i,j)] is obtained by calculating the weights of the edges:
[0035]
[0036] Among them, l is a hyperparameter to determine the degree of decay of the relationship weight, and the expression of graph convolution is:
[0037]
[0038] H (l) is the input feature, H (l+1) is the output feature, for The degree matrix of a matrix is defined as σ is a nonlinear activation function.
[0039] Optionally, the feature fusion module includes: a concatenation operation, three convolution layers, a batch normalization layer, a ReLU activation function and a residual connection layer, which is specifically expressed as follows:
[0040]
[0041] Among them, W f1 , W f2 and W f3 is the convolution layer, BN is the batch normalization layer, is the Concatenation operation, φ is the ReLU activation function, is the residual addition operation.
[0042] Optionally, the labeling decoder, reconstruction decoder and Patch decoder each include four sub-modules, and each sub-module includes: a deconvolution layer, two convolution layers, a ReLU activation function, a batch normalization layer and a residual connection layer connected in sequence.
[0043] Optionally, using Python opencv to extract organoid bright field image information and fluorescent protein staining image information, obtain slices of two-dimensional or three-dimensional bright field images and fluorescent protein staining images, and generate a two-dimensional matrix or a three-dimensional matrix from the slices;
[0044] The fluorescent protein staining image matrix is superimposed on the bright field image matrix to obtain the bright field superimposed image matrix, and labelme is used to annotate the organoid region of interest on the bright field superimposed image matrix;
[0045] The values of all pixels within the region of interest are set to 1, and the values of all pixels outside the region of interest are set to 0, to obtain a binary bright field mask image.
[0046] A second object of the present invention is to provide an organoid automatic annotation system, comprising:
[0047] An organoid bright field image acquisition module, used to acquire the organoid bright field image to be labeled;
[0048] An organoid automatic labeling module, used to process the organoid bright field image using the organoid labeling model to obtain an organoid labeling mask and complete the automatic labeling of the organoid;
[0049] The organoid annotation model includes: a dual-stream feature encoder, a relational graph convolutional neural network, a feature fusion module and a multi-task feature decoder;
[0050] The dual-stream feature encoder includes a whole-image feature encoder and a patch feature encoder, wherein the whole-image feature encoder is used to extract local features of the organoid bright field image, and the patch feature encoder is used to extract block features of the organoid bright field image block;
[0051] The relationship graph convolutional neural network is used to mine the global prior dependency between the organoid bright field image blocks, so as to obtain the global features of the organoid bright field image;
[0052] The feature fusion module is used to fuse the local features and global features of the organoid bright field image;
[0053] The multi-task feature decoder includes: a labeling decoder, a reconstruction decoder and a Patch decoder, wherein the labeling decoder predicts the organoid labeling result, the reconstruction decoder is used to predict the organoid bright field image reconstruction result, and the Patch decoder is used to predict the organoid block labeling result;
[0054] The whole-image feature encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the annotation decoder; the patch encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the patch decoder.
[0055] A third object of the present invention is to provide an electronic device, comprising a memory and a processor;
[0056] The memory is used to store computer programs;
[0057] The processor is used to implement the automatic organoid labeling method as described in any one of the above items when executing the computer program.
[0058] The third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the automatic organoid labeling method as described in any one of the above items is implemented.
[0059] The beneficial effects of the present invention are:
[0060] The present invention can batch process organoid microscopic images, accurately annotate them in the original images, and efficiently annotate organoids in microscopic images; the present invention utilizes a dual-stream feature encoder, a relational graph convolutional neural network, and a feature fusion module to construct an annotation model, which effectively mines local and global prior information in organoid microscopic images and solves the problem of low organoid annotation accuracy; the present invention utilizes a multi-task feature decoder to fully learn the collaborative interaction relationship between multiple tasks to further improve the robustness and generalization ability of the organoid annotation model, and can achieve fast, accurate, and high-throughput organoid annotation. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0062] Figure 1 A schematic diagram of the basic flow of a method for automatic organoid labeling based on global context-aware dual-stream deep learning provided by an embodiment of the present invention.
[0063] Figure 2 A structural diagram of the encoder submodule of a method for automatic organoid labeling based on global context-aware dual-stream deep learning provided in one embodiment of the present invention.
[0064] Figure 3 A schematic diagram of a contextual two-stream deep learning architecture of an automatic organoid labeling method based on global context-aware two-stream deep learning provided for an embodiment of the present invention.
[0065] Figure 4 A schematic diagram of the structure of a relational graph convolution module of an automatic organoid labeling method based on global context-aware dual-stream deep learning provided in one embodiment of the present invention.
[0066] Figure 5 A schematic diagram of the structure of the local and global feature fusion modules of an automatic organoid labeling method based on global context-aware dual-stream deep learning provided in one embodiment of the present invention.
[0067] Figure 6 A structural diagram of a decoder submodule of an automatic organoid labeling method based on global context-aware dual-stream deep learning provided in one embodiment of the present invention.
[0068] Figure 7 Annotated original image of a method for automatic organoid labeling based on global context-aware dual-stream deep learning provided by an embodiment of the present invention.
[0069] Figure 8 Annotated organoid image of an automatic organoid labeling method based on global context-aware dual-stream deep learning provided by an embodiment of the present invention.
[0070] Fig. 9 A predicted mask image of an automatic organoid labeling method based on global context-aware dual-stream deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0071] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0072] Embodiment 1:
[0073] This embodiment provides an organoid automatic labeling method, including:
[0074] Step 1: Acquire bright field images of organoids;
[0075] Step 2: Use the organoid annotation model to process the organoid bright field image to obtain the organoid annotation mask and complete the automatic annotation of the organoid;
[0076] The organoid annotation model includes: a two-stream feature encoder, a relational graph convolutional neural network, a feature fusion module, and a multi-task feature decoder;
[0077] The dual-stream feature encoder includes a whole-image feature encoder and a patch feature encoder. The whole-image feature encoder is used to extract local features of the organoid bright field image, and the patch feature encoder is used to extract block features of the organoid bright field image block.
[0078] The relational graph convolutional neural network is used to mine the global prior dependencies between organoid bright field image patches, thereby obtaining the global features of organoid bright field images;
[0079] The feature fusion module is used to fuse the local features and global features of the organoid bright field image;
[0080] The multi-task feature decoder includes: an annotation decoder, a reconstruction decoder and a patch decoder. The annotation decoder predicts the organoid annotation results, the reconstruction decoder is used to predict the organoid bright field image reconstruction results, and the patch decoder is used to predict the organoid block annotation results.
[0081] The whole-image feature encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the annotation decoder; the patch encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the patch decoder.
[0082] Embodiment 2:
[0083] This embodiment provides an organoid automatic labeling method based on global context-aware dual-stream deep learning, including:
[0084] S1: Combined with the fluorescent protein staining image, labelme is used to annotate the organoids in the input bright field image.
[0085] It should be noted that the input of organoid microscopic images includes:
[0086] Use Python opencv to extract organoid bright field image information and fluorescent protein staining image information, obtain slices of two-dimensional or three-dimensional bright field images and fluorescent protein staining images, and generate two-dimensional matrices f(x,y) and fp(x,y) or three-dimensional matrices f(x,y,z) and fp(x,y,z) from the slices.
[0087] Wherein, the binarization results of the organoid microscopic image include:
[0088] The fluorescent protein staining image fp(x,y,z) is superimposed on the bright field image f(x,y,z) to obtain the bright field superimposed image ffp(x,y,z), and labelme is used to annotate the organoid region of interest on the bright field superimposed image ffp(x,y,z);
[0089] The values of all pixels within the region of interest are set to 1, and the values of all pixels outside the region of interest are set to 0;
[0090] Get the binary bright field mask image binf_mask(x,y,z):
[0091]
[0092] S2: Build an organoid annotation model based on global context-aware dual-stream deep learning and train the annotated data to obtain a trained network model;
[0093] It should be noted that the organoid annotation model includes: a two-stream feature encoder, a patch-wise relation graph convolution module (PRGCM), a local-global semantic fusion module (LGFFM), and a multi-task feature decoder.
[0094] Specifically, refer to Figure 3 The dual-stream feature encoder consists of a whole-image encoder (E w) and the block feature encoder (Patch encoder, E p ), each encoder includes four submodules, refer to Figure 2 , including two n×n×n convolution operations connected in sequence, a ReLU activation function, a batch normalization layer, a maximum pooling operation and a residual connection, where n is not limited in size and is generally an odd number, 3 in this example. The organoid bright field images in the training set are converted into arrays, which are then normalized and input into the two-stream feature encoder to extract local features and block features respectively.
[0095] Further, see Figure 4 , the relational graph convolutional neural network includes two graph convolutional layers, defining the number of graph nodes as N, and the extracted block feature vector as x. The relationship metric between the i-th block and the j-th block can be expressed as:
[0096]
[0097] Where k represents the number of bits per pixel and is required to be a positive integer. After obtaining the relationship between block features, the adjacency matrix A = [w(i,j)] is obtained by calculating the edge weights:
[0098]
[0099] Among them, l is a hyperparameter that determines the degree of decay of the relationship weight. The detailed expression of graph convolution is:
[0100]
[0101] H (l) is the input feature, H (l+1) is the output feature, for The degree matrix of a matrix is defined as σ is a nonlinear activation function, represents the modified adjacency matrix, where A is the original adjacency matrix, I is the identity matrix, and W (l) Represents the weight matrix, and the global prior dependencies between image blocks are mined through relational graph convolution operations to obtain the global features of the organoid bright field image.
[0102] Further, see Figure 5 The feature fusion module includes a concatenation operation, two 1×1×1 convolutions, one n×n×n convolution, a batch normalization layer, a ReLU activation function, and a residual connection. The size of n is not limited and is generally an odd number. In this example, it is 3. It can be specifically expressed as:
[0103]
[0104] Among them, f l represents the local features, f g represents the global feature, f lg represents the fused features, W f1 , W f2 and W f3 is the convolution layer, BN is the batch normalization layer, is the Concatenation operation, φ is the ReLU activation function, The local features and global features are fused through the feature fusion module to mine more discriminative deep representations.
[0105] Furthermore, the multi-task feature decoder includes a label decoder D s , reconstruction decoder D r and Patch decoder D p , each decoder consists of four submodules, refer to Figure 6 , define the deconvolution of each submodule as t×t×t, two f×f×f convolution operations, a ReLU activation function, a batch normalization layer and a residual connection, where the size of t and f is not restricted and is generally an odd number, 5 and 3 respectively in this example. By labeling the decoder D s Predict the organoid annotation results by reconstructing the decoder D r Predict the reconstruction results of the organoid bright field image, through the Patch decoder D p Predict the organoid block labeling results. Wherein, the connection between the encoder and the decoder network includes: Whole-image encoder E w and the annotation decoder D s The features before the maximum pooling and after the deconvolution operation are connected by skip paths. Patch encoder E p and Patch decoder D p Feature skip path connections before max pooling and after deconvolution operations.
[0106] Furthermore, a multi-task training loss function is used for model training, including organoid annotation main loss, multi-task auxiliary loss and spatial consistency loss. The organoid annotation main loss is used to guide network annotation learning, the multi-task auxiliary loss is used to optimize the high-level semantic information and edge detail information of the extracted features, and the spatial consistency loss is used to constrain the consistency of the annotation results in different spaces.
[0107] Calculate the organoid annotation subject loss:
[0108]
[0109] Calculate the multi-task auxiliary loss for organoid annotation:
[0110]
[0111]
[0112]
[0113] in, is the reconstruction loss, is the content constraint loss, is the structural constraint loss, is the block annotation loss, and η is the weight factor to balance the weights between reconstruction and block annotation tasks.
[0114] Calculate spatial consistency loss:
[0115]
[0116] Among them, frac is the fragmentation operation, p s is the annotation prediction result, p p Patch labeling result. Therefore, the optimized multi-task training loss function is:
[0117]
[0118] Among them, λ1 and λ2 are weight factors to balance the relationship between losses.
[0119] S4: Use the trained global context-aware two-stream deep learning network model to annotate an organoid bright field in the test set, obtain the organoid annotation mask, and complete the automatic and accurate annotation of the organoid.
[0120] Specifically, the unlabeled original organoid bright field images in the test set are converted into array form; these arrays are normalized; the arrays are input into the network model trained in step S3, prediction is performed, and the organoid image array predicted by the model is output, thereby obtaining the organoid labeling result.
[0121] Embodiment 3
[0122] Reference Figure 7-9 As another example of the present invention, in order to verify and illustrate the technical effect achieved by the present method, this embodiment selects the traditional technical solution organoid labeling method and the manual labeling method to compare with the method of the present invention to verify the real effect of the present method.
[0123] The cultured organoids were photographed using a live cell imaging system, with a 10x objective magnification and sampling every 3-5 microns. The size of the bright field image obtained was 1536×2048, and each organoid bright field individual was stored in a folder; the bright field image information was extracted, the slices of the organoid image were obtained, and the slices were superimposed in sequence to generate a three-dimensional matrix f(x, y, z), and a total of 500 organoid bodies were obtained. The bright field images were annotated using labelme (a deep learning image annotation tool); for the 500 organoid bright field volume images, 400 (about 80%) were used as training samples and the remaining 100 (about 20%) were used as test samples.
[0124] A training model is constructed and trained according to step S1. In this embodiment, two 3×3×3 convolution operations, a ReLU activation function, a 3×3×3 maximum pooling operation and a residual connection are used with a step size of 1 to convert the organoid bright field images in the training set into arrays, which are then normalized and input into a dual-stream feature encoder to extract local features and block features, respectively.
[0125] According to S3, a multi-task training loss function is used to train the model, including the organoid annotation main loss, the multi-task auxiliary loss and the spatial consistency loss. The organoid annotation main loss is used to guide the network annotation learning, the multi-task auxiliary loss is used to optimize the high-level semantic information and edge detail information of the extracted features, and the spatial consistency loss is used to constrain the consistency of the annotation results in different spaces.
[0126] The comparison between the traditional solution and the traditional manual labeling solution is shown in Table 1 below:
[0127] Table 1: Comparison results between the present invention and the traditional solution
[0128] Comparison Object Manual cutting method Traditional annotation method Method of the present invention Accuracy / % 86 94 98 Marking accuracy / % 83 96 99 Scope of application wide Specific images wide Time / s 128.6 0.43 0.25 efficiency Low Higher high
[0129] From the above comparison data, it can be seen that the method of the present invention is superior to the traditional scheme in accuracy, labeling precision and efficiency. It can achieve rapid removal in a shorter time than the traditional scheme. The traditional organoid labeling method is only applicable to specific images and cannot be promoted on a large scale. The manual cutting method and the method of the present invention have a wide range of applications, but the traditional scheme is inferior to the method of the present invention in other data. The comparison results show that the method of the present invention can quickly, efficiently, accurately and automatically label the organoid part in the bright field microscopy image.
[0130] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.
[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for automatic labeling of organoids, characterized in that: The method comprises: Step 1: Acquire bright field images of organoids; Step 2: Processing the organoid bright field image using the organoid annotation model to obtain an organoid annotation mask, thereby completing automatic annotation of the organoid; The organoid annotation model includes: a dual-stream feature encoder, a relational graph convolutional neural network, a feature fusion module and a multi-task feature decoder; The dual-stream feature encoder includes a whole-image feature encoder and a patch feature encoder, wherein the whole-image feature encoder is used to extract local features of the organoid bright field image, and the patch feature encoder is used to extract block features of the organoid bright field image block; the whole-image feature encoder and the patch feature encoder each include four submodules, each submodule includes two convolutional layers connected in sequence, a ReLU activation function, a batch normalization layer, a maximum pooling layer, and a residual connection layer; The relationship graph convolutional neural network is used to mine the global prior dependency between the organoid bright field image blocks, so as to obtain the global features of the organoid bright field image; The feature fusion module is used to fuse the local features and global features of the organoid bright field image; The multi-task feature decoder includes: a labeling decoder, a reconstruction decoder and a Patch decoder, the labeling decoder predicts the organoid labeling result, the reconstruction decoder is used to predict the organoid bright field image reconstruction result, and the Patch decoder is used to predict the organoid block labeling result; the labeling decoder, the reconstruction decoder and the Patch decoder each include four submodules, each submodule includes: a deconvolution layer, two convolution layers, a ReLU activation function, a batch normalization layer and a residual connection layer connected in sequence; The whole-image feature encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the annotation decoder; the patch feature encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the patch decoder.
2. The method for automatic organoid labeling according to claim 1, characterized in that: The training process of the organoid annotation model includes: Combined with the fluorescent protein staining image, labelme is used to annotate the organoids in the input bright field image; Construct training sample set and test sample set; The organoid annotation model is trained, and the multi-task training loss function includes: organoid annotation main loss, multi-task auxiliary loss and spatial consistency loss, wherein the organoid annotation main loss is used to guide network annotation learning, the multi-task auxiliary loss is used to optimize the high-level semantic information and edge detail information of the extracted features, and the spatial consistency loss is used to constrain the consistency of the annotation results in different spaces; The calculation formula for the loss of the organoid annotation subject is: The calculation formula for the multi-task auxiliary loss of organoid annotation is: in, is the reconstruction loss, is the content constraint loss, is the structural constraint loss, is the block annotation loss, η is the weight factor to balance the weights between reconstruction and block annotation tasks; The calculation formula of the spatial consistency loss is: Among them, frac is the fragmentation operation, p s is the annotation prediction result, p p Label the results for Patch; Therefore, the optimized multi-task training loss function is: Among them, λ1 and λ2 are weight factors to balance the relationship between losses.
3. The method for automatic organoid labeling according to claim 1, characterized in that: The relationship graph convolutional neural network includes: two graph convolution layers, defining the number of graph nodes as N, and the extracted block feature vector as x, then the relationship metric between the i-th block and the j-th block is expressed as: After obtaining the relationship between block features, the adjacency matrix A = [w(i,j)] is obtained by calculating the weights of the edges: Among them, l is a hyperparameter to determine the degree of decay of the relationship weight, and the expression of graph convolution is: H (l) is the input feature, H (l+1) is the output feature, for The degree matrix of a matrix is defined as σ is a nonlinear activation function.
4. The method for automatic organoid labeling according to claim 1, characterized in that: The feature fusion module includes: a concatenation operation, three convolutional layers, a batch normalization layer, a ReLU activation function and a residual connection layer, which is specifically described as follows: Among them, W f1 , W f2 and W f3 is the convolution layer, BN is the batch normalization layer, is the Concatenation operation, φ is the ReLU activation function, is the residual addition operation, f l represents the local features, f g represents the global feature, f lg Represents the fused features.
5. The method for automatic organoid labeling according to claim 2, characterized in that: Use Python opencv to extract organoid bright field image information and fluorescent protein staining image information, obtain slices of two-dimensional or three-dimensional bright field images and fluorescent protein staining images, and generate two-dimensional or three-dimensional matrices from the slices; The fluorescent protein staining image matrix is superimposed on the bright field image matrix to obtain the bright field superimposed image matrix, and labelme is used to annotate the organoid region of interest on the bright field superimposed image matrix; The values of all pixels within the region of interest are set to 1, and the values of all pixels outside the region of interest are set to 0, to obtain a binary bright field mask image.
6. An automatic organoid labeling system, characterized in that: The system comprises: An organoid bright field image acquisition module, used to acquire the organoid bright field image to be labeled; An organoid automatic labeling module, used to process the organoid bright field image using the organoid labeling model to obtain an organoid labeling mask and complete the automatic labeling of the organoid; The organoid annotation model includes: a dual-stream feature encoder, a relational graph convolutional neural network, a feature fusion module and a multi-task feature decoder; The dual-stream feature encoder includes a whole-image feature encoder and a patch feature encoder, wherein the whole-image feature encoder is used to extract local features of the organoid bright field image, and the patch feature encoder is used to extract block features of the organoid bright field image block; the whole-image feature encoder and the patch feature encoder each include four submodules, each submodule includes two convolutional layers connected in sequence, a ReLU activation function, a batch normalization layer, a maximum pooling layer, and a residual connection layer; The relationship graph convolutional neural network is used to mine the global prior dependency between the organoid bright field image blocks, so as to obtain the global features of the organoid bright field image; The feature fusion module is used to fuse the local features and global features of the organoid bright field image; The multi-task feature decoder includes: a labeling decoder, a reconstruction decoder and a Patch decoder, the labeling decoder predicts the organoid labeling result, the reconstruction decoder is used to predict the organoid bright field image reconstruction result, and the Patch decoder is used to predict the organoid block labeling result; the labeling decoder, the reconstruction decoder and the Patch decoder each include four submodules, each submodule includes: a deconvolution layer, two convolution layers, a ReLU activation function, a batch normalization layer and a residual connection layer connected in sequence; The whole-image feature encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the annotation decoder; the patch feature encoder is connected to the feature skip path before the maximum pooling and after the deconvolution operation in the patch decoder.
7. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the organoid automatic labeling method according to any one of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method for automatic organoid labeling according to any one of claims 1 to 5 is implemented.
