GIST targeted drug type selection prediction method and system based on artificial intelligence

Abstract

The invention provides a GIST targeted drug type selection prediction method and system based on artificial intelligence, and the method comprises the steps: obtaining a digital pathological image of a tissue pathological section of a target object, and inputting the digital pathological image into a GIST targeted drug type selection prediction model, the generation method of the GIST targeted drug type selection prediction model comprises the steps that the digital pathological image of a tissue pathological section used for model training is obtained, the digital pathological image is cut into a plurality of block images, corresponding position information is obtained, and a target drug type prediction result corresponding to the digital pathological image is obtained. Associating the block images with pseudo labels, inputting the block images into a feature extraction network, learning and training the feature extraction network by using weak supervision features to obtain a feature extractor, and extracting features of all the block images by the feature extractor; and inputting the features and the position information into a neural network model for training to obtain a GIST targeted drug type selection prediction model.

Description

Technical Field

[0001] The present invention specifically relates to the field of artificial intelligence medicine, and more specifically to an artificial intelligence-based GIST targeted drug type selection prediction method and system. Background Art

[0002] Gastrointestinal stromal tumor (GIST) is a relatively rare gastrointestinal tumor, accounting for 0.1%-0.3% of all gastrointestinal malignant tumors. Among the tumors of the gastrointestinal tract originating from mesenchymal tissue, GIST is the most common, accounting for about 80%. GIST is insensitive to traditional radiotherapy and chemotherapy, but the tyrosinase inhibitor (TKI) imatinib can achieve the purpose of treatment. According to the Chinese Consensus on the Diagnosis and Treatment of Gastrointestinal Stromal Tumors (2022 version), when the tumor is larger than 2cm, surgical treatment is the first choice. However, if the patient's preoperative evaluation is difficult to meet the R0 resection standard (no tumor cells at the resection margin), combined organ resection is required, and complete resection is possible but the surgical risk is high, imatinib neoadjuvant therapy should be considered. If the PDGFRA exon 18 mutation is insensitive to imatinib, afatinib is recommended for neoadjuvant chemotherapy. With the in-depth study of the molecular mechanism of tumors and the development and clinical application of targeted drugs, molecular targeted therapy has made important progress. The NCCN and ESMO (European Society for Medical Oncology) GIST diagnosis and treatment guidelines both recommend that when GIST patients have C-KIT and PDGFRA mutations, imatinib can be used as a first-line standard treatment for locally advanced, recurrent and metastatic GIST. If it is a PDGFRA exon 18 mutation or a non-C-KIT mutation, the patient may not benefit from imatinib. Imatinib is a molecular targeted inhibitor of C-KIT and PDGFRA, which has made a breakthrough in the treatment of GIST and can significantly improve the quality of life of patients. Therefore, it is important to clarify whether GIST patients should choose targeted drugs and the type of drugs to choose for clinical treatment options.

[0003] At present, the commonly used detection methods for detecting C-KIT and PDGFRA gene mutations are mainly direct sequencing and the amplification refractory mutation system (ARMS) for detecting known sites. However, these two detection technologies have their own limitations. In actual operation, insufficient sample size often results in the inability to obtain true test results. Therefore, this application is often limited in clinical practice. Nowadays, traditional pathological examinations centered on slides have problems such as a lack of pathologists and long operation time. In addition, there is heterogeneity within the tumor, and the lesions are sometimes atypical. Pathologists have a certain degree of subjectivity in judging whether to choose targeted drugs and the type of medication. Summary of the invention

[0004] A series of simplified concepts are introduced in the Summary of the Invention, which will be further described in detail in the Detailed Description of the Invention. The Summary of the Invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0005] The purpose of the present invention is to provide a method and system for predicting the type of GIST targeted drug selection based on artificial intelligence, so as to solve the problem that the auxiliary prediction system for predicting the type of patient targeted drug selection based on gastrointestinal stromal tumor tissue pathology images proposed in the above background technology has not been studied so far.

[0006] According to one aspect of the present application, a method for predicting the type of targeted drug selection for GIST based on artificial intelligence is provided, the method comprising: obtaining a digital pathological image of a tissue pathological section of a target object, inputting the digital pathological image into a GIST targeted drug type selection prediction model, and obtaining a targeted drug type prediction result corresponding to the digital pathological image, wherein the method for generating the GIST targeted drug type selection prediction model comprises:

[0007] Acquire a digital pathology image of a tissue pathology section for model training, cut the digital pathology image into a plurality of block images and obtain corresponding position information, associate the block images with pseudo labels and input them into a feature extraction network, and train the feature extraction network using weakly supervised feature learning to obtain a feature extractor, which extracts features of all the block images;

[0008] Inputting the features and the location information into a neural network model for training to obtain a GIST targeted drug type selection prediction model;

[0009] The feature and the location information are input into a neural network model for training to obtain a GIST targeted drug type selection prediction model, including:

[0010] Processing the features using a selective state space module to achieve information interaction between features, and aggregating the features using an aggregation module based on an attention module to obtain a first global feature describing the entire digital pathology image;

[0011] In the training stage, the attention scores of all the block image features are calculated by the attention module, and the block images corresponding to the highest multiple attention scores are selected as high-response block images according to the attention scores; the position information of the high-response block images is input into the dynamic local area generator to obtain a feature area of ​​non-fixed shape, and the feature area is processed by the selective state space module to obtain a second global feature of the digital pathology image;

[0012] The consistency of the first global feature and the second global feature is calculated respectively to obtain the divergence loss. The divergence loss and the cross entropy loss function are used together to optimize the entire network through back propagation, and the final GIST targeted drug type selection prediction model is obtained through cyclic iterative training.

[0013] As a further solution of the present invention: the step of cutting the digital pathology image into a plurality of block images and obtaining corresponding position information includes:

[0014] The digital pathology image is cut into non-repetitive, non-fixed-size and background-removed block images by window sliding to obtain a digital pathology image of the whole tissue pathology slice after block division. Where X i is the i-th pathological full-slice image in the dataset D, represents the jth block image in the i-th pathological full slice image, represents the coordinates of the jth block image in the i-th pathological full-slice image, including the row and column information, and n is the number of block images in the i-th pathological full-slice image;

[0015] Acquire a dataset D of labeled histopathological whole-slice images, wherein the dataset D includes digital pathological images of the histopathological whole-slice and targeted drug selection category labels associated with lesion categories of the histopathological slices;

[0016] The data set D={(X1,Y1),(X2,Y2),…,(X i ,Y i ),…,(X M ,Y M )}, where X i represents the digital pathology image of the i-th tissue pathology full slice in the dataset, Y i It represents the targeted drug selection category label of the i-th tissue pathology section in the dataset, and M represents the number of samples of tissue pathology sections in the dataset D.

[0017] As a further solution of the present invention: the block image is associated with a pseudo label and then input into a feature extraction network, and the feature extraction network is trained by using weakly supervised feature learning to obtain a feature extractor, including:

[0018] Associating the block images with pseudo labels, and forming a feature extraction network pre-training data set with the block images with pseudo labels, wherein the pseudo labels are the same as the histopathological section labels of the block images, and the histopathological section labels represent the lesion categories of the histopathological sections, which are associated with the targeted drug selection categories;

[0019] The feature extraction network pre-training data set is input into the feature extraction network for training to generate a feature extractor for the GIST targeted drug type selection prediction method.

[0020] As a further solution of the present invention: the feature extraction network includes a query branch and a key branch;

[0021] The query branch consists of an encoder f q and a projector p q Composition, encoder f q and projector p q By parameter θ q definition;

[0022] The key branch has the same structure as the query branch. The key branch consists of an encoder f k and a projector p k The encoder and projector are composed of parameters θ k Definition, parameter θ of the key branch k The parameter θ of the query branch q Updated based on the exponential moving average mechanism, the exponential moving average mechanism is: θ k ←mθ q +(1-m)θ k ,m∈(0,1).

[0023] As a further solution of the present invention: the feature extraction network pre-training data set is input into the feature extraction network for training to generate a feature extractor for the GIST targeted drug type selection prediction method, including:

[0024] The block image is processed by different data enhancement methods to obtain two enhanced views, a storage queue is constructed for each pathological section lesion category, enhanced views of the same category as the storage queue are stored in the storage queue, and in the storage queue, two enhanced views from the same block image constitute a positive sample pair, and two enhanced views from different block images constitute a negative sample pair;

[0025] Using the feature extraction network to process each enhanced view to obtain a feature vector of the enhanced view;

[0026] The feature extraction network is optimized to increase the similarity between the positive sample pairs and reduce the similarity between the negative sample pairs. After the feature extraction network is optimized, a feature extractor of the GIST targeted drug type selection prediction method is obtained.

[0027] As a further solution of the present invention: two different data enhancement methods are used to process the block image to obtain two different enhanced views v q and v k , will enhance the view v q and v k The query branch and the key branch are sent to the encoder to obtain the encoded features z1 and z2 respectively, and then the enhanced view v is obtained by the projector. q The feature vector q and the enhanced view v k The feature vector k of .

[0028] As a further solution of the present invention: the optimization of the feature extraction network includes:

[0029] Optimizing the feature extraction network through a loss function;

[0030] The loss function is:

[0031]

[0032] Among them, N is the capacity of the storage queue, c is the set of slice categories used as negative sample comparison, and k ji It represents the i-th sample in the storage queue of category j, i and j are positive integers, T is the transposed sign, and τ is the temperature coefficient.

[0033] As a further solution of the present invention: after the feature extraction network is optimized, a feature extractor for the GIST targeted drug type selection prediction method is obtained, comprising:

[0034] After the feature extraction network is optimized, the weight of the encoder of the query branch is taken out as the loading weight of the feature extraction network, and the feature extraction network loaded with the weight is used as a feature extractor of the block image. The feature extractor extracts the features of each block image, and the features can be used for the training of the prediction model for the selection of GIST targeted drug types;

[0035] The feature extractor extracts the features of each block image, including:

[0036] The feature extractor f extracts features from the cut block images, and each block image is extracted as represents a feature vector, d is the dimension of the feature, It represents the jth block image in the i-th pathological full slice image. After feature extraction, the digital pathological image of the tissue pathological slice can be expressed as

[0037] As a further solution of the present invention: the selective state space module is used to process the features to achieve information interaction between features, and the aggregation module based on the attention module is used to aggregate the features to obtain the first global feature describing the entire digital pathology image, including:

[0038] S1. Perform nonlinear mapping on the input feature E to obtain a new feature E. The process is as follows:

[0039] F = W1E;

[0040] Where W1 is a learnable parameter;

[0041] S2, reshape the new feature F into a feature map F' that can reflect the real spatial distribution according to the position information, fill the tissue pathology section area with the corresponding block image feature vector, and fill the background area with a 0 vector;

[0042] By setting the window, the feature map F' is divided into L sub-regions, where L is a positive integer. The size of the window is controlled by the hyperparameter s. Each sub-region contains s 2 features, so that feature F' is expressed as F local =[F'1,F'2,…,F′ L ],

[0043] Add a learnable class label CLS after the feature sequence of each sub-region to obtain a new feature L is a positive integer, S is a positive integer, d is the feature dimension, and d is a positive integer;

[0044] S3, feature F' local Expanding along the second dimension yields a set of feature queues It is input into the selective state space module for information exchange, and the process is as follows:

[0045]

[0046] Where A is the state-state matrix, B is the input-state matrix, C is the state-output matrix, I is the identity matrix, and is the discretization form of A and B, and the discretization process adopts the zero-order hold strategy; h(·) is the state function, x(·) is the input function, and y(·) is the output function; Δ is the step size, which represents the staged hold of the input; the above process is summarized by the Mamba function, and a positive result can be obtained The formula is as follows:

[0047] F f =Mamba(F'local );

[0048] F f Indicates the positive result of the output;

[0049] S4, after reversing the sequence of the new feature F, input it into the selective state space module to obtain the reverse result F b , the positive result F f With the reverse result F b Add together to get the final result of the selective state space module F m , the process is as follows:

[0050] F b =Mamba(reverse(F' local ));

[0051] F m =F f +F b ;

[0052] Where reverse(·) is the flip function;

[0053] S5, the result F m The features at the corresponding class mark position in the extract to get F cls ={CLS1,CLS2,…,CLS L}, CLS L Represents the class label, where L is a positive integer, which is input into the attention module for calculation to calculate the attention score of each sub-region. According to the attention score, all class labels are aggregated to obtain the first global feature h1 that can describe the entire pathological slice. The process is as follows:

[0054]

[0055] In the formula, a l is the attention score corresponding to the l-th class label;

[0056] The first global feature is input into the classifier for classification to obtain the predicted probability

[0057] The loss is calculated using the cross entropy loss function, where the cross entropy loss function L slide for:

[0058]

[0059] In the formula, is the probability of the cth category in the prediction result of the i-th tissue pathology section, Y i is the one-hot encoded label of the i-th full slice.

[0060] According to another aspect of the present application, a GIST targeted drug type selection prediction system based on artificial intelligence is provided, comprising:

[0061] An image acquisition module is used to obtain digital pathology images of tissue pathology sections, cut the digital pathology images into a number of block images and obtain corresponding position information, and divide the block images into a training set and a test set, wherein the training set is used for model training and the test set is used for model testing;

[0062] An image processing module, used for associating the block images with pseudo labels and inputting them into a feature extraction network, and training the feature extraction network using weakly supervised feature learning to obtain a feature extractor, wherein the feature extractor is used to extract features of all block images; and used for inputting the features and position information of the block images into a neural network model for training to obtain a GIST targeted drug type selection prediction model;

[0063] The prediction information export module is used to input the digital pathology image into the GIST targeted drug type selection prediction model to obtain a targeted drug type prediction result corresponding to the digital pathology image.

[0064] The artificial intelligence-based GIST targeted drug type selection prediction method of the present invention combines a feature extractor and a GIST targeted drug type selection prediction model, learns the information of all the block images based on the selective state space module and obtains the first global feature through the aggregation of attention, thereby improving the training and reasoning speed of the model. The local area generated by the dynamic local area generator is input into the selective state space module for learning and obtaining the second global feature, and the first global feature and the second global feature are optimized for consistency target, thereby enhancing the learning ability and robustness of the selective state space module, achieving better classification effect, and improving the generalization ability of the model. The targeted drug type selection prediction model directly obtains the prediction result of the targeted drug use type corresponding to the digital pathological image of the target object, conducts weakly supervised feature learning of a large amount of pseudo-labeled image data for digital pathological images, and assists the model in obtaining pathological image features with more generalization ability. The classified targeted drug selection granularity is finer, which can not only identify whether the targeted drug needs to be used, but also obtain more specific targeted drug type information, and divide it into different types of drug treatment groups according to its different structures, thereby saving medical resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0066] Figure 1 Schematic flow chart of the prediction method for artificial intelligence-based GIST targeted drug type selection;

[0067] Figure 2 Schematic flow chart of the method for generating a prediction model for artificial intelligence-based GIST targeted drug type selection;

[0068] Figure 3 Schematic diagram of the artificial intelligence-based prediction system for selecting targeted drugs for GIST;

[0069] Figure 4 Schematic diagram of the LACL model in the prediction method for selecting the type of GIST targeted drug based on artificial intelligence;

[0070] Figure 5 Schematic diagram of scanning tissue pathology sections for the prediction method of selecting targeted drug types for GIST based on artificial intelligence;

[0071] Figure 6 A schematic diagram of a prediction model for GIST targeted drug type selection in an artificial intelligence-based GIST targeted drug type selection prediction method;

[0072] Figure 7 Comparison chart of the results of the artificial intelligence-based prediction method for GIST targeted drug type selection and existing prediction methods. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0074] First, refer to Figure 1 To describe the artificial intelligence-based GIST targeted drug type selection prediction method used to implement an embodiment of the present invention. Figure 1FIG. 4 is a schematic flow chart of a method for predicting the type of GIST targeted drug selection based on artificial intelligence according to an embodiment of the present application. Figure 1 As shown, the method for predicting the type of GIST targeted drug selection based on artificial intelligence according to the embodiment of the present application may include the following steps:

[0075] In step S10, a digital pathological image of a tissue pathological section of a target object is acquired;

[0076] In step S20, the digital pathological image is input into a GIST targeted drug type selection prediction model to obtain a targeted drug type prediction result corresponding to the digital pathological image.

[0077] Figure 2 FIG. 1 is a schematic flow chart of a method for generating a prediction model for selecting a type of GIST targeted drug based on artificial intelligence according to an embodiment of the present application. Figure 2 As shown, the method for generating a GIST targeted drug type selection prediction model based on artificial intelligence according to an embodiment of the present application may include the following steps:

[0078] In step S100, a digital pathological image of a tissue pathological section for model training is obtained, the digital pathological image is cut into a plurality of block images and corresponding position information is obtained, the block images are associated with pseudo labels and then input into a feature extraction network, and the feature extraction network is trained using weakly supervised feature learning to obtain a feature extractor, which extracts features of all block images;

[0079] In step S200, the features and the location information are input into a neural network model for training to obtain a GIST targeted drug type selection prediction model.

[0080] The artificial intelligence-based GIST targeted drug type selection prediction method of the present invention combines a feature extractor and a GIST targeted drug type selection prediction model, learns the information of all the block images based on a selective state space module, and obtains the first global feature through the aggregation of attention, thereby improving the training and reasoning speed of the model. The local area generated by the dynamic local area generator is input into the selective state space module for learning and obtaining the second global feature, and the first global feature and the second global feature are optimized for consistency target, thereby improving the generalization ability of the model. The targeted drug type prediction result corresponding to the digital pathology image of the target object is directly obtained through the GIST targeted drug type selection prediction model, and a large amount of weakly supervised feature learning with pseudo-labeled image data is carried out for the digital pathology image, and the auxiliary model obtains pathology image features with more generalization ability. The classified targeted drug selection granularity is finer, which can not only identify whether the targeted drug needs to be used, but also obtain more specific targeted drug type information, and divide it into different types of drug treatment groups according to its different structures, thereby saving medical resources.

[0081] In an embodiment of the present application, a digital pathological image of a tissue pathological section of a target object is obtained in step S10. The digital pathological image of the tissue pathological section is obtained, the digital pathological image is cut into a number of fixed-size block images, and the block images are divided into a training set and a test set, wherein the training set is used for model training and the test set is used for model testing.

[0082] See also Figure 5 The method for obtaining a digital pathological image of a tissue pathological section is: obtaining a tissue pathological section and scanning it to obtain a HE-stained whole-section scanned image of the pathological tissue, wherein the HE-stained whole-section scanned image of the pathological tissue is the digital pathological image. Furthermore, the tissue pathological section in this embodiment is scanned using a Motic EasyScan digital section scanner to obtain a corresponding digital pathological image.

[0083] In an embodiment of the present application, in step S100, a digital pathology image of a tissue pathology section for model training is obtained, the digital pathology image is cut into a plurality of block images and corresponding position information is obtained, the block images are associated with pseudo labels and then input into a feature extraction network, and the feature extraction network is trained using weakly supervised feature learning to obtain a feature extractor, which extracts features of all block images.

[0084] In one embodiment, cutting a digital pathology image into a plurality of block images and obtaining corresponding position information includes: cutting the digital pathology image into non-repetitive, non-fixed-size block images with background removed by window sliding, so as to obtain a digital pathology image of the whole tissue pathology slice after the block Where Xi is the i-th pathological full-slice image in the dataset D, represents the jth block image in the i-th pathological full slice image, Represents the coordinates of the jth block image in the i-th pathological full-slice image, including the row and column information, and n is the number of block images in the i-th pathological full-slice image.

[0085] A dataset D of labeled histopathological whole-slice images is obtained, where the dataset D includes digital pathological images of the histopathological whole-slice and targeted drug selection category labels associated with lesion categories of the histopathological slices.

[0086] Dataset D = {(x1,y1),(X2,Y2),…,(X i ,Y i ),…,(X M ,Y M )}, where X i represents the digital pathology image of the i-th tissue pathology full slice in the dataset, Y i It represents the targeted drug selection category label of the i-th tissue pathology section in the dataset, and M represents the number of samples of tissue pathology sections in the dataset D.

[0087] In one embodiment, the block images are associated with pseudo labels and then input into a feature extraction network, and the feature extraction network is trained using weakly supervised feature learning to obtain a feature extractor, including: associating the block images with pseudo labels, and forming a feature extraction network pre-training data set with the block images with pseudo labels, wherein the pseudo labels are the same as the histopathological section labels of the block images, and the histopathological section labels represent the lesion categories of the histopathological sections, which are associated with the targeted drug selection categories. The feature extraction network pre-training data set is input into the feature extraction network for training, and a feature extractor of the GIST targeted drug type selection prediction method is generated, and the features of all the block images are extracted using the feature extractor.

[0088] See also Figure 4 The feature extraction network adopts a lesion-aware pathological image representation learning model (LACL model), which uses a momentum contrastive learning (MoCo) architecture. In one embodiment, the feature extraction network includes a query branch and a key branch; the query branch consists of an encoder f q and a projector p q Composition, encoder f q and projector p q By parameter θ q Definition: The key branch has the same structure as the query branch. The key branch consists of an encoder f k and a projector p kThe encoder and projector are composed of parameters θ k Definition, parameter θ of the key branch k The parameter θ of the query branch q Based on the exponential moving average (EMA) mechanism update, the exponential moving average (EMA) mechanism is: θ k ←mθ q +(1-m)θ k ,m∈(0,1).

[0089] It should be noted that in the embodiment of the present application, the encoder (Backbone) used in the feature extraction network is ResNet50, and the projector is a multi-layer perceptron, which consists of a linear layer, a ReLU layer and another linear layer.

[0090] In one embodiment, a feature extraction network pre-training data set is input into a feature extraction network for training to generate a feature extractor for a GIST targeted drug type selection prediction method, including: using different data enhancement methods to process the block image to obtain two enhanced views, constructing a storage queue for each tissue pathology section lesion category, storing enhanced views of the same category as the storage queue in the storage queue, and in the storage queue, two enhanced views from the same block image constitute a positive sample pair, and two enhanced views from different block images constitute a negative sample pair. Each enhanced view is processed using a feature extraction network to obtain a feature vector of the enhanced view. The feature extraction network is optimized to increase the similarity between positive sample pairs and reduce the similarity between negative sample pairs. After the feature extraction network is optimized, a feature extractor for a GIST targeted drug type selection prediction method is obtained.

[0091] In one embodiment, two different data enhancement methods are used to process the block image to obtain two different enhanced views v q and v k , will enhance the view v q and v k The query branch and key branch are sent to the encoder f respectively. q Processing to obtain the coded features z1 and z2, and then through the projector p k Get enhanced view v q The feature vector q and the enhanced view v k The feature vector k of .

[0092] In one embodiment, optimizing the feature extraction network includes: optimizing the feature extraction network through a loss function. The loss function is:

[0093]

[0094] Among them, N is the capacity of the storage queue, c is the set of slice categories used as negative sample comparison, and k ji It represents the i-th sample in the storage queue of category j, i and j are positive integers, T is the transposed sign, and τ is the temperature coefficient.

[0095] In addition, in order to ensure that the sample representations in the storage queue are representative during the training process, the queue needs to be selectively updated during the training process. The queue purification strategy (QRS) measures the similarity between the input representation and the representation in the queue, and selects a batch of the most representative representations from each minimum batch to update into the lesion storage queue (lesionqueue). Specifically, in the embodiment of the present application, the definition The expected distribution of the input sample representation is: the similarity distribution between the positive sample k0 and the representation in the queue, defined as It represents the expected distribution, and then uses KL divergence to test the consistency of the two distributions and participate in the update of the network parameters of the query branch.

[0096] In one embodiment, after the feature extraction network is optimized, a feature extractor for a GIST targeted drug type selection prediction method is obtained, including: after the feature extraction network is optimized, the weight of the encoder of the query branch is taken out as the loading weight of the feature extraction network, and the feature extraction network with the loaded weight is used as a feature extractor for the block image. The feature extractor extracts features of each block image, and the features can be used for training a GIST targeted drug type selection prediction model.

[0097] The feature extractor extracts the features of each block image, including: the feature extractor f extracts the features of the cut block images, and each block image is extracted as represents a feature vector, d is the dimension of the feature, in the specific implementation, d = 1024, It represents the jth block image in the i-th pathological full slice image. After feature extraction, the digital pathological image of the tissue pathological slice can be expressed as

[0098] In an embodiment of the present application, in step S200, the feature and location information are input into a neural network model for training to obtain a GIST targeted drug type selection prediction model.

[0099] In one embodiment, the neural network model is a weakly supervised pathology full-slice classification network model based on a selective state space module, including two selective state space modules, an attention module and a dynamic local region generator, wherein the selective state space module is used to learn the dependencies and information exchanges of tissue regions of the entire slice, the attention module is used to prompt key regions and aggregate slice-level features, and the dynamic local region generator reshapes the serialized image features into a square-shaped feature map and uses convolution kernels of different sizes to learn the spatial relationship of features of different regions.

[0100] like Figure 6 As shown, in one embodiment, the features are processed using a selective state space module to achieve information interaction between features, and feature aggregation is performed using an aggregation module based on an attention module to obtain a first global feature describing the entire digital pathology image, including: S1, nonlinear mapping of the input feature E to obtain a new feature F, the process is as follows:

[0101] F = W1E;

[0102] Where W1 is a learnable parameter;

[0103] S2, reshape the new feature F into a feature map F' that can reflect the real spatial distribution according to the position information, fill the tissue pathology section area with the corresponding block image feature vector, and fill the background area with a 0 vector;

[0104] By setting the window, the feature map F' is divided into L sub-regions, where L is a positive integer. The size of the window is controlled by the hyperparameter s. Each sub-region contains s 2 features, we can get a fixed length of s 2 The characteristic sequence of F' is represented by F local =[F'1,F'2,…,F' L ],

[0105] In order to aggregate the information of each sub-region, a learnable class label CLS is added after the feature sequence of each sub-region to obtain a new feature L is a positive integer, S is a positive integer, d is the feature dimension, and d is a positive integer.

[0106] S3, feature F' local Expanding along the second dimension yields a set of feature queues It is input into the selective state space module for information exchange, and the process is as follows:

[0107]

[0108]

[0109] Where A is the state-state matrix, B is the input-state matrix, C is the state-output matrix, I is the identity matrix, and is the discretization form of A and B, and the discretization process adopts the zero-order hold strategy; h(·) is the state function, x(·) is the input function, and y(·) is the output function; Δ is the step size, which represents the staged hold of the input; the above process is summarized by the Mamba function, and a positive result can be obtained The formula is as follows:

[0110] F f =Mamba(F' local );

[0111] F f Indicates the positive result of the output;

[0112] S4. In order to enhance the robustness of the model and the completeness of information, a bidirectional selective state space model is used as the selective state space module. The new feature F is sequence-flipped and then input into the selective state space module to obtain the reverse result F. b , the positive result F f With the reverse result F b Add together to get the final result of the selective state space module F m , the process is as follows:

[0113] F b =Mamba(reverse(F' local ));

[0114] F m =F f +F b ;

[0115] Where reverse(·) is the flip function;

[0116] S5, the result F m The features at the corresponding class mark position in the extract to get F cls ={CLS1,CLS2,…,CLS L}, CLS L Represents the class label, where L is a positive integer, which is input into the attention module for calculation to calculate the attention score of each sub-region. According to the attention score, all class labels are aggregated to obtain the first global feature h1 that can describe the entire pathological slice. The process is as follows:

[0117]

[0118] In the formula, a l is the attention score corresponding to the l-th class label;

[0119] Input the first global feature into the classifier for classification and obtain the predicted probability

[0120] The cross entropy loss function is used to calculate the loss, where the cross entropy loss function L slide for:

[0121]

[0122] In the formula, is the probability of the cth category in the prediction result of the i-th tissue pathology section, Y i is the one-hot encoded label of the i-th full slice.

[0123] In an embodiment of the present application, during the training phase, the neural network adds additional constraints, and the attention scores of all the block image features are calculated through the attention module. According to the attention scores, the block images corresponding to the highest multiple attention scores are selected as high-response block images; in this implementation, 10 block images corresponding to the attention scores can be selected as high-response block images. The position information of the high-response block image is input into the dynamic local area generator to obtain a set of feature areas of non-fixed shape, and the number of features in each feature area is the same. This set of feature areas of non-fixed shape is passed through the selective state space module to obtain a set of class labels representing the information of each feature area, and all class labels are averaged and pooled to obtain the second global feature h2 of the digital pathology image;

[0124] In an embodiment of the present application, the consistency of the first global feature h1 and the second global feature h2 is calculated respectively to obtain the KL divergence loss. The KL divergence loss and the cross entropy loss function are used together to optimize the entire network through back propagation, and the final GIST targeted drug type selection prediction model is obtained through cyclic iterative training.

[0125] In an embodiment of the present application, in step S20, the digital pathology image is input into the GIST targeted drug type selection prediction model to obtain a targeted drug type prediction result corresponding to the digital pathology image.

[0126] At present, the way hospitals predict the use of GIST drug types is mostly through molecular testing of GIST specimens, and then medical staff further collect the actual remission rate results of GIST patients using drugs. This application predicts the actual clinical remission of targeted drugs and the actual remission rate of targeted drugs based on sequencing through a deep learning model. The block images cut from digital pathology images are extracted by a feature extractor, and the extracted features are processed by the GIST targeted drug type selection prediction model. Since the second global feature generated by the dynamic local area generator is added to participate in the training and optimization of the model, the robustness of the model is enhanced, thereby enhancing the classification performance of the model and significantly improving the prediction rate.

[0127] In order to facilitate understanding of the superiority of this application, after 3 months of treatment, the targeted drug remission rate was evaluated by computed tomography (CT) or positron emission tomography (PET) imaging. The treatment results were grouped according to the Choi Guidelines (the evaluation criteria currently used by major hospitals). The Choi Guidelines comprehensively evaluate tumors by changes in tumor density and slight changes in tumor size and the presence of new lesions, thereby obtaining Figure 7 , where the AI ​​model is the test method of this application, molecular detection represents the test method currently used in hospitals (i.e., prediction of targeted drugs based on molecular sequencing analysis), response represents complete or partial remission of the tumor after the application of targeted drugs, and no response represents no remission of the tumor after the application of targeted drugs.

[0128] See also Figure 7 As shown, the present application uses a molecular detection method for GIST specimens through actual experiments, such as predicting targeted drugs based on molecular sequencing analysis and comparing them with the prediction method of the present application. Among them, the accuracy of predicting no response to targeted drugs based on molecular sequencing analysis is 0.4286, while the accuracy of predicting no response in the present application is 0.7143; the accuracy of predicting the effectiveness of targeted drugs based on molecular sequencing analysis is 0.8929, and the accuracy of predicting the effectiveness of targeted drugs in the present invention is 0.9286.

[0129] In addition, according to the embodiment of the present application, a GIST targeted drug type selection prediction system based on artificial intelligence is also provided. Figure 3 The following describes an artificial intelligence-based GIST targeted drug type selection prediction system 300 provided according to another aspect of the present application. Figure 3 FIG. 3 is a schematic structural block diagram of a GIST targeted drug type selection prediction system 300 based on artificial intelligence according to an embodiment of the present application. Figure 3 As shown, select the prediction system, including:

[0130] An image acquisition module 310 is used to obtain a digital pathology image of a tissue pathology section, cut the digital pathology image into a plurality of block images and obtain corresponding position information, and divide the block images into a training set and a test set, wherein the training set is used for model training and the test set is used for model testing;

[0131] The image processing module 320 is used to associate the block images with pseudo labels and then input them into the feature extraction network, and use weakly supervised feature learning to train the feature extraction network to obtain a feature extractor, which is used to extract features of all block images; and to input the features and position information of the block images into the neural network model for training to obtain a GIST targeted drug type selection prediction model;

[0132] The prediction information export module 330 is used to input the digital pathology image into the GIST targeted drug type selection prediction model to obtain the targeted drug type prediction result corresponding to the digital pathology image.

[0133] In summary, the present invention uses training set data combined with a neural network model to generate a GIST targeted drug type selection prediction model. Compared with the previous technology, the present application combines a feature extractor and a GIST targeted drug type selection prediction model. The present application learns all the information of the block images based on the selective state space module and obtains the first global feature through the aggregation of attention, thereby improving the training and reasoning speed of the model. In addition, the dynamic local area generator generates a local area input to the selective state space module for learning and obtaining the second global feature, and then optimizes the consistency target of the first global feature and the second global feature, thereby enhancing the learning ability and robustness of the selective state space module, achieving better classification results, and improving the generalization ability of the model. It can better identify the spatial spectrum of digital pathology slices, more accurately identify tumor regions, and more carefully identify different tissue components, cell morphology, and intracellular sub-organ structures in digital pathology slices. The present application can capture almost all target tumor regions, which is more accurate than the existing full-slice analysis technology.

[0134] In addition, the present application directly obtains the prediction result of the targeted drug usage type corresponding to the digital pathology image of the target object through the GIST targeted drug type selection prediction model; it can carry out weakly supervised feature learning of a large amount of pseudo-labeled image data for the digital pathology images of patients with gastrointestinal stromal tumors, and assist the model in obtaining pathology image features with more generalization capabilities. The classified targeted drug selection has a finer granularity, which can not only identify whether the targeted drug needs to be used, but also obtain more specific targeted drug type information, and divide it into different types of drug treatment groups according to its different structures, thereby saving medical resources.

[0135] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0136] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for predicting the type of GIST targeted drug selection based on artificial intelligence, characterized in that: The method comprises: Acquire a digital pathological image of a tissue pathological section of a target object, input the digital pathological image into a GIST targeted drug type selection prediction model, and obtain a targeted drug type prediction result corresponding to the digital pathological image, wherein the method for generating the GIST targeted drug type selection prediction model includes: Acquire a digital pathology image of a tissue pathology section for model training, cut the digital pathology image into a plurality of block images and obtain corresponding position information, associate the block images with pseudo labels and input them into a feature extraction network, and train the feature extraction network using weakly supervised feature learning to obtain a feature extractor, which extracts features of all the block images; Inputting the features and the location information into a neural network model for training to obtain a GIST targeted drug type selection prediction model; The feature and the location information are input into a neural network model for training to obtain a GIST targeted drug type selection prediction model, including: Processing the features using a selective state space module to achieve information interaction between features, and aggregating the features using an aggregation module based on an attention module to obtain a first global feature describing the entire digital pathology image; In the training stage, the attention scores of all the block image features are calculated by the attention module, and the block images corresponding to the highest multiple attention scores are selected as high-response block images according to the attention scores; the position information of the high-response block images is input into the dynamic local area generator to obtain a feature area of ​​non-fixed shape, and the feature area is processed by the selective state space module to obtain a second global feature of the digital pathology image; The consistency of the first global feature and the second global feature is calculated respectively to obtain the divergence loss. The divergence loss and the cross entropy loss function are used together to optimize the entire network through back propagation, and the final GIST targeted drug type selection prediction model is obtained through cyclic iterative training.

2. The selection prediction method according to claim 1, characterized in that: The step of cutting the digital pathology image into a plurality of block images and obtaining corresponding position information includes: The digital pathology image is cut into non-repetitive, non-fixed-size and background-removed block images by window sliding to obtain a digital pathology image of the whole tissue pathology slice after block division. Where X i is the i-th pathological full-slice image in the dataset D, represents the jth block image in the i-th pathological full slice image, represents the coordinates of the jth block image in the i-th pathological full-slice image, including the row and column information, and n is the number of block images in the i-th pathological full-slice image; Acquire a dataset D of labeled histopathological whole-slice images, wherein the dataset D includes digital pathological images of the histopathological whole-slice and targeted drug selection category labels associated with lesion categories of the histopathological slices; The data set D={(X1,Y1),(X2,Y2),…,(X i ,Y i ),…,(X M ,Y M )}, where X i represents the digital pathology image of the i-th tissue pathology full slice in the dataset, Y i It represents the targeted drug selection category label of the i-th tissue pathology section in the dataset, and M represents the number of samples of tissue pathology sections in the dataset D.

3. The selection prediction method according to claim 1, characterized in that: The block images are associated with pseudo labels and then input into a feature extraction network, and the feature extraction network is trained using weakly supervised feature learning to obtain a feature extractor, including: Associating the block images with pseudo labels, and forming a feature extraction network pre-training data set with the block images with pseudo labels, wherein the pseudo labels are the same as the histopathological section labels of the block images, and the histopathological section labels represent the lesion categories of the histopathological sections, which are associated with the targeted drug selection categories; The feature extraction network pre-training data set is input into the feature extraction network for training to generate a feature extractor for the GIST targeted drug type selection prediction method.

4. The selection prediction method according to claim 3, characterized in that: The feature extraction network includes a query branch and a key branch; The query branch consists of an encoder f q and a projector p q Composition, encoder f q and projector p q By parameter θ q definition; The key branch has the same structure as the query branch. The key branch consists of an encoder f k and a projector p k The encoder and projector are composed of parameters θ k Definition, parameter θ of the key branch k The parameter θ of the query branch q Updated based on the exponential moving average mechanism, the exponential moving average mechanism is: θ k ←mθ q +(1-m)θ k ,m∈(0,1).

5. The selection prediction method according to claim 4, characterized in that: The method of inputting the feature extraction network pre-training data set into the feature extraction network for training to generate a feature extractor of the GIST targeted drug type selection prediction method comprises: The block image is processed by different data enhancement methods to obtain two enhanced views, a storage queue is constructed for each pathological section lesion category, enhanced views of the same category as the storage queue are stored in the storage queue, and in the storage queue, two enhanced views from the same block image constitute a positive sample pair, and two enhanced views from different block images constitute a negative sample pair; Using the feature extraction network to process each enhanced view to obtain a feature vector of the enhanced view; The feature extraction network is optimized to increase the similarity between the positive sample pairs and reduce the similarity between the negative sample pairs. After the feature extraction network is optimized, a feature extractor of the GIST targeted drug type selection prediction method is obtained.

6. The selection prediction method according to claim 5, characterized in that: Two different data enhancement methods are used to process the block image to obtain two different enhanced views v q and v k , will enhance the view v q and v k The query branch and the key branch are sent to the encoder to obtain the encoded features z1 and z2 respectively, and then the enhanced view v is obtained by the projector. q The feature vector q and the enhanced view v k The eigenvector k of .

7. The selection prediction method according to claim 5, characterized in that: The optimizing the feature extraction network comprises: Optimizing the feature extraction network through a loss function; The loss function is: Among them, N is the capacity of the storage queue, c is the set of slice categories used as negative sample comparison, and k ji It represents the i-th sample in the storage queue of category j, i and j are positive integers, T is the transposed sign, and τ is the temperature coefficient.

8. The selection prediction method according to claim 1, characterized in that: After the feature extraction network is optimized, a feature extractor of the GIST targeted drug type selection prediction method is obtained, including: After the feature extraction network is optimized, the weight of the encoder of the query branch is taken out as the loading weight of the feature extraction network, and the feature extraction network loaded with the weight is used as a feature extractor of the block image. The feature extractor extracts the features of each block image, and the features can be used for the training of the prediction model for the selection of GIST targeted drug types; The feature extractor extracts the features of each block image, including: The feature extractor f extracts features from the cut block images, and each block image is extracted as represents a feature vector, d is the dimension of the feature, It represents the jth block image in the i-th pathological full slice image. After feature extraction, the digital pathological image of the tissue pathological slice can be expressed as 9. The selection prediction method according to claim 1, characterized in that: The selective state space module is used to process the features to achieve information interaction between the features, and the aggregation module based on the attention module is used to aggregate the features to obtain the first global feature describing the entire digital pathology image, including: S1. Perform nonlinear mapping on the input feature E to obtain a new feature F. The process is as follows: F = W1E; Where W1 is a learnable parameter; S2, reshape the new feature F into a feature map F' that can reflect the real spatial distribution according to the position information, fill the tissue pathology section area with the corresponding block image feature vector, and fill the background area with a 0 vector; By setting the window, the feature map F' is divided into L sub-regions, where L is a positive integer. The size of the window is controlled by the hyperparameter s. Each sub-region contains s 2 features, so that feature F' can be expressed as Add a learnable class label CLS after the feature sequence of each sub-region to obtain a new feature L is a positive integer, S is a positive integer, d is the feature dimension, and d is a positive integer; S3, feature F' local Expanding along the second dimension yields a set of feature queues It is input into the selective state space module for information exchange, and the process is as follows: Where A is the state-state matrix, B is the input-state matrix, C is the state-output matrix, I is the identity matrix, and is the discretization form of A and B, and the discretization process adopts the zero-order hold strategy; h(·) is the state function, x(·) is the input function, and y(·) is the output function; Δ is the step size, which represents the staged hold of the input; the above process is summarized by the Mamba function, and a positive result can be obtained The formula is as follows: F f =Mamba(F local ); F f Indicates the positive result of the output; S4, after reversing the sequence of the new feature F, input it into the selective state space module to obtain the reverse result F b , the positive result F f With the reverse result F b Add together to get the final result of the selective state space module F m , the process is as follows: F b =Mamba(reverse(F′ local )); F m =F f +F b ; Where reverse(.) is the flip function; S5, the result F m The features at the corresponding class mark position in the extract to get F cls = {CLS1, CLS2, ..., CLS L }, CLS L Represents the class label, where L is a positive integer, which is input into the attention module for calculation to calculate the attention score of each sub-region. According to the attention score, all class labels are aggregated to obtain the first global feature h1 that can describe the entire pathological slice. The process is as follows: In the formula, a l is the attention score corresponding to the l-th class label; The first global feature is input into the classifier for classification to obtain the predicted probability The loss is calculated using the cross entropy loss function, where the cross entropy loss function L slide for: In the formula, is the probability of the cth category in the prediction result of the i-th tissue pathology section, Y i is the one-hot encoded label of the i-th full slice.

10. A GIST targeted drug type selection prediction system based on artificial intelligence, characterized in that: include: An image acquisition module is used to obtain digital pathology images of tissue pathology sections, cut the digital pathology images into a number of block images and obtain corresponding position information, and divide the block images into a training set and a test set, wherein the training set is used for model training and the test set is used for model testing; An image processing module, used for associating the block images with pseudo labels and inputting them into a feature extraction network, and training the feature extraction network using weakly supervised feature learning to obtain a feature extractor, wherein the feature extractor is used to extract features of all block images; and used for inputting the features and position information of the block images into a neural network model for training to obtain a GIST targeted drug type selection prediction model; The prediction information export module is used to input the digital pathology image into the GIST targeted drug type selection prediction model to obtain a targeted drug type prediction result corresponding to the digital pathology image.

Images