An intelligent recognition method and device for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids

Abstract

The present application discloses a method and device for intelligent identification of third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids, relating to the technical field of lung adenocarcinoma organoid identification. The method includes: obtaining a training set; obtaining an OncoAGMS-U-Net++ model; training the OncoAGMS-U-Net++ model with the training set to obtain a trained OncoAGMS-U-Net++ model; obtaining image data of a sample to be identified, where the image data of the sample to be identified includes bright-field image data to be identified, fluorescence image data to be identified, and spatial probability map data of gene mutations to be identified; inputting the image data of the sample to be identified into the OncoAGMS-U-Net++ model to obtain a drug resistance region probability map, a volume change rate, an apoptosis resistance intensity, and a drug resistance type probability output by the OncoAGMS-U-Net++ model. The OncoAGMS-U-Net++ model created in the present application focuses on the drug resistance of lung adenocarcinoma organoids from four dimensions, and can more accurately model the complex phenotypes of third-generation EGFR-TKIs-resistant lung adenocarcinoma through multi-modal data fusion and drug resistance feature enhancement strategies.

Description

Technical Field

[0001] The present application relates to the technical field of lung adenocarcinoma organoid recognition, and particularly to an intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids and an intelligent recognition device for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids. Background Art

[0002] Third-generation EGFR-TKIs are targeted drugs designed specifically for sensitive mutations of the EGFR gene (such as exon 19 deletion or L8B58R mutation) and the T790M mutation that appears after the resistance of first-generation / second-generation TKIs. Third-generation EGFR-TKIs can specifically bind to the EGFR T790M mutant, while reducing the inhibition of wild-type EGFR, thereby reducing the risk of adverse reactions while ensuring the efficacy. However, despite the significant efficacy of third-generation EGFR-TKIs in the treatment of patients with EGFR-mutant NSCLC, the problem of drug resistance is still inevitable.

[0003] Through the organoid technology, a three-dimensional structure with in vivo tumor characteristics can be cultured in vitro, so as to more accurately simulate the tumor microenvironment and drug response. By constructing third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids, researchers can deeply study the drug resistance mechanism and provide an experimental basis for the development of new treatment strategies.

[0004] However, the existing technology does not have a technical solution for drug resistance recognition through in vitro cultured organoids.

[0005] Therefore, it is desirable to have a technical solution to overcome or at least mitigate at least one of the above defects of the existing technology. Summary of the Invention

[0006] The purpose of the present application is to provide an intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids to overcome or at least mitigate at least one of the above defects of the existing technology.

[0007] To achieve the above purpose, the present application provides an intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids, and the intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids includes:

[0008] Obtaining a training set; the training set includes gene mutation spatial probability map data and corresponding label information, a sensitive group dual-channel 2D image set and corresponding label information, and a drug-resistant group dual-channel 2D image set and corresponding label information;

[0009] Obtaining an OncoAGMS-U-Net++ model;

[0010] Train the OncoAGMS-U-Net++ model with the training set to obtain the trained OncoAGMS-U-Net++ model;

[0011] Obtain the sample image data to be recognized, which includes the bright-field image data to be recognized, the fluorescence image data to be recognized, and the gene mutation spatial probability map data to be recognized;

[0012] Input the sample image data to be recognized into the OncoAGMS-U-Net++ model to obtain the drug-resistant region probability map, volume change rate, apoptosis resistance intensity, and drug-resistant type probability output by the OncoAGMS-U-Net++ model.

[0013] Optionally, the sensitive group dual-channel 2D image set includes sensitive group bright-field image data and sensitive group fluorescence image data;

[0014] The sensitive group dual-channel 2D image set is obtained by the following method:

[0015] Register the sensitive group bright-field image data and sensitive group fluorescence image data of the same type of organ sample to obtain the registered sensitive group bright-field image data and sensitive group fluorescence image data;

[0016] Fuse each group of registered sensitive group bright-field image data and sensitive group fluorescence image data to form a sensitive group dual-channel 2D image, and each sensitive group dual-channel 2D image constitutes the sensitive group dual-channel 2D image set.

[0017] Optionally, the drug-resistant group dual-channel 2D image set includes drug-resistant group bright-field image data and drug-resistant group fluorescence image data;

[0018] The drug-resistant group dual-channel 2D image set is obtained by the following method:

[0019] Register the drug-resistant group bright-field image data and drug-resistant group fluorescence image data of the same type of organ sample to obtain the registered drug-resistant group bright-field image data and drug-resistant group fluorescence image data;

[0020] Fuse each group of registered drug-resistant group bright-field image data and drug-resistant group fluorescence image data to form a drug-resistant group dual-channel 2D image, and each drug-resistant group dual-channel 2D image constitutes the drug-resistant group dual-channel 2D image set.

[0021] Optionally, the OncoAGMS-U-Net++ model includes:

[0022] An input layer, which includes a dual-channel image input layer and a parallel image input layer;

[0023] An encoder, the encoder including a four-stage depthwise separable convolution block;

[0024] An attention guidance module, the attention guidance module including a spatial attention module, a channel attention module, and a dual attention fusion module;

[0025] A decoder, the decoder including an upsampling layer, nested skip connections, and a feature fusion unit;

[0026] An output layer, the output layer including a drug resistance region segmentation branch, a volume change rate prediction branch, an apoptosis resistance intensity prediction branch, and a drug resistance type classification branch;

[0027] Inputting the to-be-recognized sample image data into the OncoAGMS-U-Net++ model to obtain a drug resistance region probability map, a volume change rate, an apoptosis resistance intensity, and a drug resistance type probability includes:

[0028] The dual-channel image input layer is used to receive the to-be-recognized bright-field image data and the to-be-recognized fluorescence image data after registration and transmit them to the encoder;

[0029] The parallel image input layer is used to receive the to-be-recognized gene mutation spatial probability map data and transmit it to the encoder;

[0030] The encoder is used to generate a multi-scale feature map according to the to-be-recognized bright-field image data, the to-be-recognized fluorescence image data, and the to-be-recognized gene mutation spatial probability map data;

[0031] The spatial attention module is used to generate a spatial weight map according to the multi-scale feature map;

[0032] The channel attention module is used to generate a channel weight map according to the multi-scale feature map;

[0033] The dual attention fusion module is used to fuse the spatial weight map and the channel weight map into a calibrated feature map;

[0034] The upsampling layer is used to generate a decoder feature map according to the multi-scale feature map and the calibrated feature map;

[0035] The feature fusion unit is used to obtain a final fused feature according to each decoder feature map;

[0036] The drug resistance region segmentation branch is used to obtain a drug resistance region probability map according to the obtained final fused feature;

[0037] The volume change rate prediction branch is used to obtain a volume change rate according to the obtained final fused feature;

[0038] The apoptosis resistance intensity prediction branch is used to obtain the apoptosis resistance intensity according to the acquired final fusion features;

[0039] The drug resistance type classification branch is used to obtain the probability distribution of drug resistance types according to the acquired final fusion features.

[0040] Optionally, training the OncoAGMS-U-Net++ model with the training set to obtain the trained OncoAGMS-U-Net++ model includes:

[0041] Obtain the OncoAGMS-U-Net++ model;

[0042] Initialize the OncoAGMS-U-Net++ model;

[0043] Define the loss function of the OncoAGMS-U-Net++ model;

[0044] Define the optimizer of the OncoAGMS-U-Net++ model;

[0045] Perform cyclic training on the OncoAGMS-U-Net++ model until the number of cycles is reached, and then save the information of the model weight parameters with the best performance.

[0046] Optionally, the loss function of the OncoAGMS-U-Net++ model includes a drug resistance region segmentation loss function, a volume change rate prediction loss function, an apoptosis resistance intensity prediction loss function, and a drug resistance type classification loss function; among them,

[0047] The drug resistance region segmentation loss function adopts the following formula:

[0048] ;

[0049] Where, is the drug resistance region segmentation loss function, A is the binary image of the drug resistance region predicted by the model, B is the binary image of the true drug resistance region, is the number of pixels in the intersection region of the prediction result and the true label, is the number of pixels in the drug resistance region in the prediction result, is the number of pixels in the drug resistance region in the true label, is the first smoothing coefficient, is the number of pixels on the boundary of the drug resistance region, is the boundary pixel index of the drug resistance region, is the value of the prediction result at the boundary pixel j, is the value of the true label at the boundary pixel j, and λ is the weight of the fixed boundary loss term;

[0050] The volume change rate prediction loss function adopts the following formula:

[0051] ;

[0052] wherein, is the volume change rate prediction loss function, N is the number of volume change rate samples, is the th predicted value of the volume change rate of the th sample, is the true value of the volume change rate of the th sample;

[0053] The apoptosis resistance intensity prediction loss function adopts the following formula:

[0054] ;

[0055] wherein, is the apoptosis resistance intensity prediction loss function, M is the number of apoptosis resistance intensity samples, is the th predicted probability of the apoptosis resistance intensity of the th sample, is the smoothed value of the true label of the apoptosis resistance intensity of the

[0056] The drug resistance type classification loss function adopts the following formula:

[0057] ;

[0058] wherein, is the drug resistance type classification loss function, N is the number of drug resistance type samples, C is the number of categories of drug resistance types, is the th predicted probability that the c th sample belongs to the th drug resistance type, is the c th true label that the T th sample belongs to the th drug resistance type, is the temperature scaling factor,

[0059] Optionally, the total loss function is obtained through the following formula:

[0060] ;

[0061] Among them, w D is the reciprocal of the gradient norm of the drug resistance region segmentation loss, w M is the reciprocal of the gradient norm of the volume change rate prediction loss, w B is the reciprocal of the gradient norm of the apoptosis resistance intensity prediction loss, w C is the reciprocal of the gradient norm of the drug resistance type classification loss, L D represents the drug resistance region segmentation loss function, L M is the volume change rate prediction loss function, L B is the apoptosis resistance intensity prediction loss, L C is the drug resistance type classification loss.

[0062] Optionally, the intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids further includes:

[0063] Obtain molecular markers;

[0064] Overlay the obtained drug resistance region probability map and molecular markers to obtain spatial correlation;

[0065] Obtain the drug resistance progression speed according to the volume change rate.

[0066] This application also provides an intelligent recognition device for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids. The intelligent recognition device for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids includes:

[0067] A training set acquisition module, which is used to acquire a training set; the training set includes gene mutation spatial probability map data and corresponding label information, a sensitive group dual-channel 2D image set and corresponding label information, and a drug resistance group dual-channel 2D image set and corresponding label information;

[0068] A model acquisition module, which is used to acquire the OncoAGMS-U-Net++ model;

[0069] A training module, which is used to train the OncoAGMS-U-Net++ model through the training set to obtain a trained OncoAGMS-U-Net++ model;

[0070] A module for obtaining sample image data to be recognized, which is used to obtain sample image data to be recognized, and the sample image data to be recognized includes bright-field image data to be recognized, fluorescence image data to be recognized, and spatial probability map data of gene mutations to be recognized;

[0071] An output module, which is used to input the sample image data to be recognized into the OncoAGMS-U-Net++ model, so as to obtain the probability map of drug-resistant regions, volume change rate, apoptosis resistance intensity, and drug-resistant type probability output by the OncoAGMS-U-Net++ model.

[0072] The OncoAGMS-U-Net++ model created by the intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids in this application focuses on the drug resistance of lung adenocarcinoma organoids from four dimensions. Through multi-modal data fusion and drug resistance feature enhancement strategies, it can more accurately model the complex phenotypes of third-generation EGFR-TKIs-resistant lung adenocarcinoma, so as to more accurately predict the probability map of drug-resistant regions, volume change rate, apoptosis resistance intensity, and drug-resistant type probability. Brief Description of the Drawings

[0073] Figure 1 It is a schematic flowchart of an intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids according to an embodiment of the present application.

[0074] Figure 2 It is a comparison schematic diagram of comparing the OncoAGMS-U-Net++ model of the present application with other models. Detailed Embodiments

[0075] To make the purpose, technical solutions, and advantages of the implementation of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the embodiments of the present application. In the drawings, the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. The described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation to the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0076] In the description of this application, it should be understood that the orientation or positional relationships indicated by terms such as "center", "longitudinal", "transverse", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the drawings. These are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation on the protection scope of this application.

[0077] As Figure 1 shown, the intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids includes:

[0078] Step 1: Obtain a training set; the training set includes gene mutation spatial probability map data and corresponding label information, a sensitive group dual-channel 2D image set and corresponding label information, and a resistant group dual-channel 2D image set and corresponding label information;

[0079] Step 2: Obtain the OncoAGMS-U-Net++ model;

[0080] Step 3: Train the OncoAGMS-U-Net++ model with the training set to obtain the trained OncoAGMS-U-Net++ model;

[0081] Step 4: Obtain the sample image data to be recognized, which includes the bright-field image data to be recognized, the fluorescence image data to be recognized, and the gene mutation spatial probability map data to be recognized;

[0082] Step 5: Input the sample image data to be recognized into the OncoAGMS-U-Net++ model to obtain the resistant region probability map, volume change rate, apoptosis resistance intensity, and resistant type probability output by the OncoAGMS-U-Net++ model.

[0083] The OncoAGMS-U-Net++ model created in this application focuses on the drug resistance of lung adenocarcinoma organoids from four dimensions. Through multi-modal data fusion and drug resistance feature enhancement strategies, it can more accurately model the complex phenotypes of third-generation EGFR-TKIs-resistant lung adenocarcinoma, thereby being able to more accurately predict the resistant region probability map, volume change rate, apoptosis resistance intensity, and resistant type probability.

[0084] In this embodiment, the sensitive group dual-channel 2D image set includes sensitive group bright-field image data and sensitive group fluorescence image data;

[0085] The sensitive group dual-channel 2D image set is obtained through the following method:

[0086] Register the bright-field image data and fluorescence image data of the sensitive group of the same type of organ samples to obtain the registered bright-field image data and fluorescence image data of the sensitive group;

[0087] Fuse the registered bright-field image data and fluorescence image data of each group to form a two-channel 2D image of the sensitive group, and the two-channel 2D images of each sensitive group form the two-channel 2D image set of the sensitive group.

[0088] In this embodiment, the two-channel 2D image set of the drug-resistant group includes bright-field image data and fluorescence image data of the drug-resistant group;

[0089] The two-channel 2D image set of the drug-resistant group is obtained by the following method:

[0090] Register the bright-field image data and fluorescence image data of the drug-resistant group of the same type of organ samples to obtain the registered bright-field image data and fluorescence image data of the drug-resistant group;

[0091] Fuse the registered bright-field image data and fluorescence image data of each group to form a two-channel 2D image of the drug-resistant group, and the two-channel 2D images of each drug-resistant group form the two-channel 2D image set of the drug-resistant group.

[0092] In this embodiment, the OncoAGMS-U-Net++ model includes:

[0093] An input layer, which includes a two-channel image input layer and a parallel image input layer;

[0094] An encoder, which includes four levels of depthwise separable convolution blocks;

[0095] An attention guidance module, which includes a spatial attention module, a channel attention module, and a dual attention fusion module;

[0096] A decoder, which includes an upsampling layer, nested skip connections, and a feature fusion unit;

[0097] An output layer, which includes a drug-resistant area segmentation branch, a volume change rate prediction branch, an apoptosis resistance intensity prediction branch, and a drug-resistant type classification branch;

[0098] Inputting the image data of the sample to be recognized into the OncoAGMS-U-Net++ model to obtain a drug-resistant area probability map, a volume change rate, an apoptosis resistance intensity, and a drug-resistant type probability includes:

[0099] The dual-channel image input layer is used to receive the registered bright-field image data to be recognized and the fluorescence image data to be recognized and transmit them to the encoder;

[0100] The parallel image input layer is used to receive the data of the spatial probability map of the gene mutation to be recognized and transmit it to the encoder;

[0101] The encoder is used to generate a multi-scale feature map according to the bright-field image data to be recognized, the fluorescence image data to be recognized, and the data of the spatial probability map of the gene mutation to be recognized;

[0102] The spatial attention module is used to generate a spatial weight map according to the multi-scale feature map;

[0103] The channel attention module is used to generate a channel weight map according to the multi-scale feature map;

[0104] The dual attention fusion module is used to fuse the spatial weight map and the channel weight map into a calibrated feature map;

[0105] The upsampling layer is used to generate a decoder feature map according to the multi-scale feature map and the calibrated feature map;

[0106] The feature fusion unit is used to obtain the final fusion feature according to each decoder feature map;

[0107] The drug-resistant region segmentation branch is used to obtain the drug-resistant region probability map according to the obtained final fusion feature;

[0108] The volume change rate prediction branch is used to obtain the volume change rate according to the obtained final fusion feature;

[0109] The apoptosis resistance intensity prediction branch is used to obtain the apoptosis resistance intensity according to the obtained final fusion feature;

[0110] The drug-resistant type classification branch is used to obtain the drug-resistant type probability distribution according to the obtained final fusion feature.

[0111] Next, each module of the OncoAGMS-U-Net++ model of the present application will be further elaborated in detail.

[0112] In this embodiment, the reception size of the dual-channel image input layer is 1024×1024×2 (including two channels for bright-field image data and fluorescence image data), and the data type is floating-point type (range: 0.0 to 1.0, after normalization processing).

[0113] In this embodiment, the reception size of the parallel image input layer: 1024×1024×1 (spatial probability map of gene mutation), and the data type is floating-point type (range: 0.0 to 1.0, representing probability).

[0114] In this embodiment, the encoder includes four levels of depthwise separable convolution blocks, for example, which can be respectively called the first convolution block, the second convolution block, the third convolution block, and the fourth convolution block; among them, the first-level convolution block includes:

[0115] Depthwise separable convolution layer module: 32 convolutional kernels, filter size 3×3

[0116] Normalization module: Batch Normalization;

[0117] ReLU activation layer;

[0118] Output feature map size: 1024×1024×32.

[0119] The second convolution block includes:

[0120] Depthwise separable convolution layer module: 64 convolutional kernels, filter size 3×3;

[0121] Normalization module: Batch Normalization;

[0122] ReLU activation layer;

[0123] Output feature map size: 512×512×64 (downsampling is achieved through convolution with a stride of 2).

[0124] The third convolution block includes:

[0125] Depthwise separable convolution layer module: 128 convolutional kernels, filter size 3×3;

[0126] Normalization module: Batch Normalization;

[0127] ReLU activation layer;

[0128] Output feature map size: 256×256×128 (downsampling is achieved through convolution with a stride of 2).

[0129] The fourth convolution block includes:

[0130] Depthwise separable convolution layer module: 256 convolutional kernels, filter size 3×3;

[0131] Normalization module: Batch Normalization;

[0132] ReLU activation layer;

[0133] Output feature map size: 128×128×256 (downsampling is achieved through convolution with a stride of 2).

[0134] In this embodiment, the input of the spatial attention module comes from the feature maps of each layer of the encoder (the size varies according to the level, such as 128×128×256);

[0135] The structure of the spatial attention module is as follows:

[0136] Two convolutional layers (filter size 3×3, with 64 and 1 convolutional kernels respectively);

[0137] The Sigmoid activation function generates a spatial attention map.

[0138] In this embodiment, the input of the channel attention module comes from the feature maps of each layer of the encoder;

[0139] The structure of the channel attention module is as follows:

[0140] Global average pooling;

[0141] Two fully connected layers (with 256 and the number of channels of the feature map neurons respectively);

[0142] The Sigmoid activation function generates a channel attention map.

[0143] In this embodiment, the input of the dual attention fusion module is the spatial attention map and the channel attention map, and its structure is as follows:

[0144] Multiply the spatial attention map and the channel attention map element by element, and adjust the number of channels through 1×1 convolution; the output is the fused attention feature map.

[0145] In this embodiment, the number of upsampling layers is the same as the number of depthwise separable convolution blocks, which is also four levels. Among them, the input of the first-level upsampling layer: the output of the fourth-level encoder (128×128×256) and the output of the attention guidance module, and its structure is as follows:

[0146] Transposed convolution layer (filter size 2×2, stride 2, output channels 128);

[0147] Batch normalization;

[0148] ReLU activation layer;

[0149] Output feature map size: 256×256×128;

[0150] Connection with the encoder: Through nested skip connections, splice the output of the third-level encoder (256×256×128) with the upsampled feature map.

[0151] In this embodiment, the input of the second-level upsampling layer is the output of the first-level upsampling layer (256×256×128) and the spliced feature map. Its structure is as follows:

[0152] Transposed convolution layer (filter size 2×2, stride 2, output channels 64);

[0153] Batch normalization;

[0154] ReLU activation layer;

[0155] Output feature map size: 512×512×64;

[0156] Connection with the encoder: Through nested skip connections, the output of the second - level encoder (512×512×64) is concatenated with the upsampled feature map.

[0157] In this embodiment, the input of the third - level upsampling layer is the output of the second - level upsampling layer (512×512×64) and the concatenated feature map, and its structure is as follows:

[0158] Transposed convolution layer (filter size 2×2, stride 2, number of output channels 32);

[0159] Batch normalization;

[0160] ReLU activation layer;

[0161] Output feature map size: 1024×1024×32;

[0162] Connection with the encoder: Through nested skip connections, the output of the first - level encoder (1024×1024×32) is concatenated with the upsampled feature map.

[0163] In this embodiment, the input of the fourth - level upsampling layer is the output of the third - level upsampling layer (1024×1024×32) and the concatenated feature map, and its structure is as follows:

[0164] Transposed convolution layer (filter size 2×2, stride 2, number of output channels 16);

[0165] Batch normalization;

[0166] ReLU activation layer;

[0167] Output feature map size: 2048×2048×16 (This layer is mainly used for feature fusion and may be adjusted according to requirements during actual prediction).

[0168] In this embodiment, each level of the upsampling layer is concatenated with the output of the convolutional block of the encoder at the corresponding level through nested skip connections. For example, the first - level upsampling layer is concatenated with the output of the third - level encoder, the second - level upsampling layer is concatenated with the output of the second - level encoder, and so on.

[0169] The structure of the feature fusion unit is as follows:

[0170] N×M convolutional layer (the number of convolutional kernels is adjusted according to the task requirements, e.g., 1 for the drug-resistant area segmentation task and the number of classes for the drug-resistant type classification task);

[0171] Sigmoid activation function (for the drug-resistant area probability map) or Softmax activation function (for the drug-resistant type probability).

[0172] In this embodiment, the structure of the drug-resistant area segmentation branch is as follows:

[0173] Upsampling layer (adjust the output size to be the same as the original image);

[0174] Sigmoid activation function (output the probability of each pixel belonging to the drug-resistant area);

[0175] Output: Drug-resistant area probability map (the size is the same as the input image, e.g., 1024×1024×1).

[0176] The structure of the volume change rate prediction branch is as follows:

[0177] Global average pooling (convert the feature map into a vector);

[0178] Fully connected layer (the number of neurons is 1, used to regress the volume change rate);

[0179] Linear activation function (directly output the predicted value of the volume change rate);

[0180] Output: Volume change rate (a scalar value representing the change ratio of the tumor volume).

[0181] The structure of the apoptosis resistance intensity prediction branch is as follows:

[0182] Global average pooling (convert the feature map into a vector);

[0183] Fully connected layer (the number of neurons is 1, used to regress the apoptosis resistance intensity);

[0184] Sigmoid activation function (output the probability value of the apoptosis resistance intensity, ranging from 0.0 to 1.0);

[0185] Output: Apoptosis resistance intensity (a scalar value representing the degree of tumor resistance to apoptosis).

[0186] In this embodiment, the structure of the drug-resistant type classification branch is as follows:

[0187] Global average pooling (convert the feature map into a vector);

[0188] Fully connected layer (the number of neurons is adjusted according to the number of drug-resistant type classes);

[0189] Softmax activation function (outputs the probability of each drug resistance type);

[0190] Output: Probability distribution of drug resistance types (in vector form, with length equal to the number of drug resistance type categories).

[0191] In this embodiment, through the above four branches, the output layer of the OncoAGMS-U-Net++ model can completely predict all expected tumor drug resistance related indicators.

[0192] In this embodiment, in addition to the above training set, this application can also be provided with a test set and a validation set. The testing and validation processes are the same as those of existing models and will not be elaborated here.

[0193] In this embodiment, training the OncoAGMS-U-Net++ model through the training set to obtain the trained OncoAGMS-U-Net++ model includes:

[0194] Obtain the OncoAGMS-U-Net++ model;

[0195] Initialize the OncoAGMS-U-Net++ model;

[0196] Define the loss function of the OncoAGMS-U-Net++ model;

[0197] Define the optimizer of the OncoAGMS-U-Net++ model. In this embodiment, the Adam optimizer can be used;

[0198] Perform cyclic training on the OncoAGMS-U-Net++ model until the number of cycles is reached, and then save the information of the model weight parameters with the best performance.

[0199] In this embodiment, the loss function of the OncoAGMS-U-Net++ model includes a drug resistance region segmentation loss function, a volume change rate prediction loss function, an apoptosis resistance intensity prediction loss function, and a drug resistance type classification loss function; among them,

[0200] The drug resistance region segmentation loss function adopts the following formula:

[0201] ;

[0202] Where, is the drug resistance region segmentation loss function, A is the binary image of the drug resistance region predicted by the model, B is the binary image of the true drug resistance region, is the number of pixels in the intersection region of the prediction result and the true label, is the number of pixels in the drug resistance region in the prediction result, is the number of pixels in the drug-resistant region of the true label, is the first smoothing coefficient (usually taking a very small value, such as 1×10 −6 ), which is used to avoid the case where the denominator is zero; is the number of pixels on the boundary of the drug-resistant region, is the boundary pixel index of the drug-resistant region, is the value of the prediction result at the boundary pixel j, is the value of the true label at the boundary pixel j, and λ is the weight of the fixed boundary loss term; in this embodiment, by introducing a penalty term for the predicted boundary, the edge of the drug-resistant region can be captured more precisely.

[0203] The volume change rate prediction loss function adopts the following formula:

[0204] ;

[0205] where is the volume change rate prediction loss function, N is the number of volume change rate samples, is the volume change rate prediction value of the th sample, is the true value of the volume change rate of the th sample, and is the second smoothing coefficient (usually taking a relatively small value, such as 0.1), which is used to avoid the case where the denominator is zero; in this embodiment, a dynamically adjusted factor is introduced to adjust the weight of each sample according to the size of the current error, so that samples with larger errors receive more attention during training.

[0206] The apoptosis resistance intensity prediction loss function adopts the following formula:

[0207] ;

[0208] where is the apoptosis resistance intensity prediction loss function, M is the number of apoptosis resistance intensity samples, is the apoptosis resistance intensity prediction probability of the th sample, is the smoothed value of the true label of the apoptosis resistance intensity of the th sample, and the calculation formula is:

[0209] The loss function for drug resistance type classification adopts the following formula:

[0210] ;

[0211] Among them, is the loss function for drug resistance type classification, N is the number of drug resistance type samples, C is the number of categories of drug resistance types, is the th sample belonging to the predicted probability of the c th drug resistance type, is the th sample belonging to the true label of the c th drug resistance type, T is the temperature scaling factor (usually taking a value greater than 1, such as 2), which is used to adjust the distribution of predicted probabilities; is the predicted probability that the th sample belongs to the kth drug resistance type. In this embodiment, the temperature scaling factor is introduced to scale the predicted probabilities, so that the model pays more attention to samples that are difficult to classify during training.

[0212] In this embodiment, the total loss function is obtained through the following formula:

[0213] ;

[0214] Among them, w D is the reciprocal of the gradient norm of the drug resistance region segmentation loss, w M is the reciprocal of the gradient norm of the volume change rate prediction loss, w B is the reciprocal of the gradient norm of the apoptosis resistance intensity prediction loss, w C is the reciprocal of the gradient norm of the drug resistance type classification loss, L D represents the drug resistance region segmentation loss function, L M is the volume change rate prediction loss function, L B is the apoptosis resistance intensity prediction loss, L C is the drug resistance type classification loss.

[0215] Using the above total loss function, during the training process, we calculate the gradient norm of each loss function and use its reciprocal as the dynamic weight, so that the loss function with a larger gradient accounts for a smaller proportion in the total loss, and vice versa. In this way, the total loss function can automatically balance each task without manual adjustment of weights.

[0216] In this embodiment, obtaining the training set includes:

[0217] Obtaining lung adenocarcinoma organoid sample data, where the lung adenocarcinoma organoid sample data includes sensitive group lung adenocarcinoma organoid sample data and drug-resistant group lung adenocarcinoma organoid sample data;

[0218] Obtaining a sensitive group dual-channel 2D image set and label information according to the sensitive group lung adenocarcinoma organoid sample data;

[0219] Obtaining a drug-resistant group dual-channel 2D image set and label information according to the drug-resistant group lung adenocarcinoma organoid sample data;

[0220] Obtaining gene mutation spatial probability map data and label information.

[0221] The lung adenocarcinoma organoid sample data can be obtained in the following way:

[0222] Obtaining surgical or puncture tissue samples from lung adenocarcinoma patients who have received third-generation EGFR-TKI treatment and are drug-resistant (such as osimertinib resistance), and preferentially selecting samples containing known drug-resistant mutations such as T790M and C797S and unknown subtypes.

[0223] In this embodiment, the culturing of organoids is a prior art and will not be elaborated here.

[0224] In this embodiment, the organoids in the sensitive group can be derived from EGFR-sensitive mutation patients, and the organoids in the drug-resistant group can be derived from third-generation TKI-resistant patients (T790M mutation).

[0225] In this embodiment, drug resistance can be quantified by comprehensively considering the volume change rate (survival rate) and apoptosis resistance intensity. For example, drug resistance RS = 0.6×survival rate + 0.4×apoptosis resistance intensity, where a threshold can be set, and if it is greater than the threshold, it is considered drug-resistant, and if it is less than the threshold, it is considered non-drug-resistant.

[0226] In this embodiment, QuPath or HALO is used for spatial omics analysis. For example, molecular markers are obtained, and the obtained drug-resistant region probability map and molecular markers are superimposed to obtain spatial correlation.

[0227] Alternatively, by comparing the volume change rate before and after treatment, the drug resistance progression speed is evaluated. For example, drug resistance progression speed = (volume change rate after treatment - volume change rate before treatment) / time interval.

[0228] The present application has the following advantages over the prior art:

[0229] Nested dense skip connections:

[0230] The OncoAGMS-U-Net++ model introduces nested dense skip connections, which construct multi-level feature fusion paths between the encoder and the decoder. By fusing the feature representations of the encoder and the decoder at different levels, the model can more effectively capture the details and global information in the image, thereby improving the segmentation accuracy and robustness. These dense skip connections ensure that the feature information from each stage of the encoder can be directly transmitted to the corresponding layers of the decoder, helping the decoder better utilize the low-level and high-level features from the encoder and improving the accuracy of image segmentation.

[0231] Deep supervision mechanism:

[0232] The model sets outputs at decoder layers of different depths and calculates the loss with the corresponding ground truth labels. This deep supervision mechanism enables the model to better learn features at different levels during training, improving the segmentation accuracy. At the same time, deep supervision also helps to accelerate network convergence and improve training efficiency. By performing supervised learning at different levels, the model can learn effective feature representations faster, thus speeding up the training process.

[0233] Flexible architecture and adjustability:

[0234] The architecture of the OncoAGMS-U-Net++ model is more flexible and can be adjusted according to different task requirements and computing resources. By selecting encoders and decoders of different depths, as well as different numbers of nested levels, the model can improve efficiency while ensuring performance. This flexibility allows the OncoAGMS-U-Net++ model to adapt to datasets of different sizes and complexities, meeting different performance and efficiency requirements.

[0235] Multi-task joint optimization:

[0236] The OncoAGMS-U-Net++ model adopts a comprehensive loss function and can simultaneously optimize multiple tasks such as drug-resistant region segmentation, volume change rate prediction, apoptosis resistance intensity prediction, and drug-resistant type classification. This multi-task joint optimization method helps to improve the generalization ability of the model, enabling the model to achieve better performance on multiple related tasks. By sharing feature representations and parameters, multi-task learning can also reduce the risk of overfitting and improve the stability and robustness of the model.

[0237] Gradient normalization technique:

[0238] Introduce the reciprocal of the gradient norm as a weight in the loss function to achieve gradient normalization. This technique can automatically balance the contributions of various tasks, making the loss function with a larger gradient account for a smaller proportion in the total loss, and vice versa. By dynamically adjusting the weights of each task, the gradient normalization technique can prevent a single task from dominating the training process and ensure that each task can be fully optimized. This helps to improve the stability and robustness of the model, enabling the model to still maintain good performance in a complex and changing data environment.

[0239] In summary, through designs such as nested dense skip connections, deep supervision mechanisms, flexible architectures, and comprehensive loss functions, the OncoAGMS-U-Net++ model demonstrates advantages of high accuracy, high efficiency, and high stability in the intelligent recognition of lung adenocarcinoma organoids resistant to third-generation EGFR-TKIs. These advantages make the OncoAGMS-U-Net++ model a powerful tool for solving such problems.

[0240] This application also provides an intelligent recognition device for lung adenocarcinoma organoids resistant to third-generation EGFR-TKIs. The intelligent recognition device for lung adenocarcinoma organoids resistant to third-generation EGFR-TKIs includes a training set acquisition module, a model acquisition module, a training module, a sample image data to be recognized acquisition module, and an output module. Among them,

[0241] The training set acquisition module is used to acquire a training set; among them, the training set includes gene mutation spatial probability map data and label information, a sensitive group dual-channel 2D image set and label information, a drug-resistant group dual-channel 2D image set and label information;

[0242] The model acquisition module is used to acquire the OncoAGMS-U-Net++ model;

[0243] The training module is used to train the OncoAGMS-U-Net++ model with the training set to obtain a trained OncoAGMS-U-Net++ model;

[0244] The sample image data to be recognized acquisition module is used to acquire sample image data to be recognized, and the sample image data to be recognized includes bright-field image data to be recognized, fluorescence image data to be recognized, and gene mutation spatial probability map data to be recognized;

[0245] The output module is used to input the sample image data to be recognized into the OncoAGMS-U-Net++ model to obtain the drug-resistant region probability map, volume change rate, apoptosis resistance intensity, and drug-resistant type probability output by the OncoAGMS-U-Net++ model.

[0246] See Figure 2, Figure 2 This is a performance comparison chart of the OncoAGMS-U-Net++ model of this application compared with the existing U-Net++. It can be seen from the chart that although the inference speed is slightly slower, other indicators are relatively high.

[0247] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of this application, not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An intelligent recognition method for third-generation EGFR-TKIs resistant lung adenocarcinoma organoids, characterized in that, The intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids includes: Obtaining a training set; the training set includes gene mutation spatial probability map data and corresponding label information, a sensitive group dual-channel 2D image set and corresponding label information, and a drug-resistant group dual-channel 2D image set and corresponding label information; Obtaining the OncoAGMS-U-Net++ model; Training the OncoAGMS-U-Net++ model with the training set to obtain the trained OncoAGMS-U-Net++ model; Obtaining the sample image data to be recognized, which includes the bright-field image data to be recognized, the fluorescence image data to be recognized, and the gene mutation spatial probability map data to be recognized; Inputting the sample image data to be recognized into the OncoAGMS-U-Net++ model to obtain the drug-resistant region probability map, volume change rate, apoptosis resistance intensity, and drug-resistant type probability output by the OncoAGMS-U-Net++ model; The sensitive group dual-channel 2D image set includes sensitive group bright-field image data and sensitive group fluorescence image data; The sensitive group dual-channel 2D image set is obtained by the following method: Registering the sensitive group bright-field image data and sensitive group fluorescence image data of the same organoid sample to obtain the registered sensitive group bright-field image data and sensitive group fluorescence image data; Fusing each group of registered sensitive group bright-field image data and sensitive group fluorescence image data to form a sensitive group dual-channel 2D image, and each sensitive group dual-channel 2D image constitutes the sensitive group dual-channel 2D image set; The drug-resistant group dual-channel 2D image set includes drug-resistant group bright-field image data and drug-resistant group fluorescence image data; The drug-resistant group dual-channel 2D image set is obtained by the following method: Registering the drug-resistant group bright-field image data and drug-resistant group fluorescence image data of the same organoid sample to obtain the registered drug-resistant group bright-field image data and drug-resistant group fluorescence image data; Fusing each group of registered drug-resistant group bright-field image data and drug-resistant group fluorescence image data to form a drug-resistant group dual-channel 2D image, and each drug-resistant group dual-channel 2D image constitutes the drug-resistant group dual-channel 2D image set; The OncoAGMS-U-Net++ model includes: An input layer, which includes a dual-channel image input layer and a parallel image input layer; An encoder, which includes four levels of depthwise separable convolution blocks; An attention guidance module, which includes a spatial attention module, a channel attention module, and a dual attention fusion module; A decoder, which includes an upsampling layer, nested skip connections, and a feature fusion unit; An output layer, which includes a drug-resistant region segmentation branch, a volume change rate prediction branch, an apoptosis resistance intensity prediction branch, and a drug-resistant type classification branch; Inputting the sample image data to be recognized into the OncoAGMS-U-Net++ model to obtain a drug resistance region probability map, a volume change rate, an apoptosis resistance intensity, and a drug resistance type probability includes: The dual-channel image input layer is used to receive the registered bright-field image data to be recognized and the fluorescence image data to be recognized and transmit them to the encoder; The parallel image input layer is used to receive the spatial probability map data of the gene mutation to be recognized and transmit it to the encoder; The encoder is used to generate a multi-scale feature map based on the bright-field image data to be recognized, the fluorescence image data to be recognized, and the spatial probability map data of the gene mutation to be recognized; The spatial attention module is used to generate a spatial weight map based on the multi-scale feature map; The channel attention module is used to generate a channel weight map based on the multi-scale feature map; The dual attention fusion module is used to fuse the spatial weight map and the channel weight map into a calibrated feature map; The upsampling layer is used to generate a decoder feature map based on the multi-scale feature map and the calibrated feature map; The feature fusion unit is used to obtain the final fused feature based on each decoder feature map; The drug resistance region segmentation branch is used to obtain a drug resistance region probability map based on the obtained final fused feature; The volume change rate prediction branch is used to obtain a volume change rate based on the obtained final fused feature; The apoptosis resistance intensity prediction branch is used to obtain an apoptosis resistance intensity based on the obtained final fused feature; The drug resistance type classification branch is used to obtain a drug resistance type probability distribution based on the obtained final fused feature.

2. The intelligent recognition method of third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids according to claim 1, wherein Training the OncoAGMS-U-Net++ model with the training set to obtain a trained OncoAGMS-U-Net++ model includes: Obtaining the OncoAGMS-U-Net++ model; Initializing the OncoAGMS-U-Net++ model; Defining the loss function of the OncoAGMS-U-Net++ model; Defining the optimizer of the OncoAGMS-U-Net++ model; Performing cyclic training on the OncoAGMS-U-Net++ model until the number of cycles is reached, and then saving the information of the model weight parameters with the best performance.

3. The intelligent recognition method of third-generation EGFR-TKIs resistant lung adenocarcinoma organoids according to claim 2, wherein The loss function of the OncoAGMS-U-Net++ model includes a drug resistance region segmentation loss function, a volume change rate prediction loss function, an apoptosis resistance intensity prediction loss function, and a drug resistance type classification loss function; among them, The drug resistance region segmentation loss function uses the following formula: ； Among them, is the loss function for drug-resistant region segmentation, A is the binary image of the drug-resistant region predicted by the model, B is the binary image of the true drug-resistant region, is the number of pixels in the intersection region of the prediction result and the true label, is the number of pixels in the drug-resistant region in the prediction result, is the number of pixels in the drug-resistant region in the true label, is the first smoothing coefficient, is the number of pixels on the boundary of the drug-resistant region, is the boundary pixel index of the drug-resistant region, is the value of the prediction result at the boundary pixel j, is the value of the true label at the boundary pixel j, and λ is the weight of the fixed boundary loss term; The volume change rate prediction loss function uses the following formula: ； Among them, is the volume change rate prediction loss function, N is the number of volume change rate samples, is the predicted value of the volume change rate of the th sample, is the true value of the volume change rate of the th sample; is the second smoothing coefficient. The apoptosis resistance intensity prediction loss function uses the following formula: ； Among them, is the prediction loss function of apoptosis resistance intensity, M is the number of apoptosis resistance intensity samples, is the predicted probability of apoptosis resistance intensity of the -th sample, is the smoothed value of the true label of apoptosis resistance intensity of the -th sample; The drug resistance type classification loss function uses the following formula: ； Among them, is the loss function for drug resistance type classification, N is the number of drug resistance type samples, C is the number of categories of drug resistance types, is the th sample belonging to the c th predicted probability of the drug resistance type, is the th sample belonging to the c th true label of the drug resistance type, T is the temperature scaling factor, is the th predicted probability that the sample belongs to the k-th drug resistance type.

4. The intelligent recognition method of third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids according to claim 3, wherein, The total loss function is obtained through the following formula: ； Among them, w D is the reciprocal of the gradient norm of the drug resistance region segmentation loss, w M is the reciprocal of the gradient norm of the volume change rate prediction loss, w B is the reciprocal of the gradient norm of the apoptosis resistance intensity prediction loss, w C is the reciprocal of the gradient norm of the drug resistance type classification loss, L D represents the drug resistance region segmentation loss function, L M is the volume change rate prediction loss function, L B is the apoptosis resistance intensity prediction loss, L C is the drug resistance type classification loss.

5. The intelligent recognition method of third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids according to claim 1, wherein The intelligent recognition method for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids further includes: Obtaining molecular markers; Overlaying the obtained drug resistance region probability map and the molecular markers to obtain spatial correlation; Obtaining the drug resistance progression speed according to the volume change rate.

6. An intelligent recognition device for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids, characterized in that, The intelligent recognition device for third-generation EGFR-TKIs-resistant lung adenocarcinoma organoids includes: A training set acquisition module, which is used to acquire a training set; the training set includes gene mutation spatial probability map data and corresponding label information, a sensitive group dual-channel 2D image set and corresponding label information, and a drug-resistant group dual-channel 2D image set and corresponding label information; A model acquisition module, which is used to acquire the OncoAGMS-U-Net++ model; A training module, which is used to train the OncoAGMS-U-Net++ model with the training set to obtain the trained OncoAGMS-U-Net++ model; A sample image data to be recognized acquisition module, which is used to acquire sample image data to be recognized, and the sample image data to be recognized includes bright-field image data to be recognized, fluorescence image data to be recognized, and gene mutation spatial probability map data to be recognized; An output module, which is used to input the sample image data to be recognized into the OncoAGMS-U-Net++ model to obtain the drug-resistant region probability map, volume change rate, apoptosis resistance intensity, and drug-resistant type probability output by the OncoAGMS-U-Net++ model; The sensitive group dual-channel 2D image set includes sensitive group bright-field image data and sensitive group fluorescence image data; The sensitive group dual-channel 2D image set is obtained by the following method: Register the sensitive group bright-field image data and sensitive group fluorescence image data of the same type of organ sample to obtain the registered sensitive group bright-field image data and sensitive group fluorescence image data; Fuse each group of registered sensitive group bright-field image data and sensitive group fluorescence image data to form a sensitive group dual-channel 2D image, and each sensitive group dual-channel 2D image constitutes the sensitive group dual-channel 2D image set; The drug-resistant group dual-channel 2D image set includes drug-resistant group bright-field image data and drug-resistant group fluorescence image data; The drug-resistant group dual-channel 2D image set is obtained by the following method: Register the drug-resistant group bright-field image data and drug-resistant group fluorescence image data of the same type of organ sample to obtain the registered drug-resistant group bright-field image data and drug-resistant group fluorescence image data; Fuse each group of registered drug-resistant group bright-field image data and drug-resistant group fluorescence image data to form a drug-resistant group dual-channel 2D image, and each drug-resistant group dual-channel 2D image constitutes the drug-resistant group dual-channel 2D image set; The OncoAGMS-U-Net++ model includes: An input layer, which includes a dual-channel image input layer and a parallel image input layer; An encoder, which includes four levels of depthwise separable convolution blocks; An attention guidance module, which includes a spatial attention module, a channel attention module, and a dual attention fusion module; A decoder, which includes an upsampling layer, nested skip connections, and a feature fusion unit; An output layer, which includes a drug-resistant region segmentation branch, a volume change rate prediction branch, an apoptosis resistance intensity prediction branch, and a drug-resistant type classification branch; Inputting the sample image data to be recognized into the OncoAGMS-U-Net++ model to obtain a drug resistance region probability map, a volume change rate, an apoptosis resistance intensity, and a drug resistance type probability includes: The dual-channel image input layer is used to receive the bright-field image data to be recognized and the fluorescence image data to be recognized after registration and transmit them to the encoder; The parallel image input layer is used to receive the spatial probability map data of the gene mutation to be recognized and transmit it to the encoder; The encoder is used to generate multi-scale feature maps according to the bright-field image data to be recognized, the fluorescence image data to be recognized, and the spatial probability map data of the gene mutation to be recognized; The spatial attention module is used to generate a spatial weight map according to the multi-scale feature maps; The channel attention module is used to generate a channel weight map according to the multi-scale feature maps; The dual attention fusion module is used to fuse the spatial weight map and the channel weight map into a calibrated feature map; The upsampling layer is used to generate decoder feature maps according to the multi-scale feature maps and the calibrated feature maps; The feature fusion unit is used to obtain the final fused feature according to each decoder feature map; The drug resistance region segmentation branch is used to obtain a drug resistance region probability map according to the obtained final fused feature; The volume change rate prediction branch is used to obtain a volume change rate according to the obtained final fused feature; The apoptosis resistance intensity prediction branch is used to obtain an apoptosis resistance intensity according to the obtained final fused feature; The drug resistance type classification branch is used to obtain a drug resistance type probability distribution according to the obtained final fused feature.

Images