Method, device and medium for training model

The gene mutation image model trained by deep learning algorithm, using feature extractor and attention network, solves the problems of low efficiency and high cost in FGFR gene mutation detection in traditional methods, realizes rapid and low-cost detection in general hospitals, and improves detection accuracy and efficiency.

CN117152552BActive Publication Date: 2024-11-08SHANGHAI ORIGIMED CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310935033.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-07-27
Publication Date
2024-11-08
Estimated Expiration
2043-07-27

AI Technical Summary

Technical Problem

Existing methods are insufficient to accurately identify pathological images of oncogenic variants in the FGFR gene. Traditional model training methods are inefficient and costly, making them difficult to widely apply in general hospitals.

Method used

We employ a deep learning algorithm based on feature extractors, attention networks, and linear classifiers. By acquiring and processing pathological image samples, we train a model to identify gene variant images. We utilize a pre-trained VisionTransformer model and attention mechanism to reduce noise interference and improve the representativeness of feature extraction.

Benefits of technology

It enables rapid and low-cost gene mutation detection in general hospitals, reduces the computational requirements for high-resolution pathological images, generates highly interpretable visualization heatmaps, helps doctors understand the model's decision-making process, and improves the accuracy and efficiency of detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117152552B_ABST
    Figure CN117152552B_ABST
Patent Text Reader

Abstract

The present invention relates to a method, device and medium for training a model for identifying gene mutation images. The method comprises: obtaining a training image sample set including a gene mutation image labeled, and determining a batch of image samples for a single training model based on the training image sample set; performing feature extraction on the image sample batches in sequence based on a feature extractor, thereby obtaining a feature matrix vector of a predetermined dimension; performing a linear transformation on the obtained feature matrix vector, thereby obtaining a transformed matrix vector; inputting the transformed matrix vector in parallel to a first attention network and a second attention network to obtain a first attention network output and a second attention network output respectively, and performing a linear layer transformation after multiplying the first attention network output and the second attention network output; based on a linear classifier, sequentially outputting the probability that an image in the training image sample set belongs to a gene mutation image, thereby obtaining a trained model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention generally relates to processing of image information, and in particular, to methods, computing devices, and computer storage media for training models for identifying genetic variation images. Background Art

[0002] Since its approval by the U.S. Food and Drug Administration (FDA) in 2019, fibroblast growth factor receptor (FGFR) inhibitors have become the first targeted therapeutic drugs for advanced muscle-invasive bladder urothelial carcinoma. However, the treatment targets of this inhibitor are limited to confirmed FGFR2 protein, FGFR3 protein or FGFR1 protein and FGFR4 protein with lower mutation frequency. Patients who identify gene mutations need molecular testing of tumor tissue samples, but this detection method is not popular, the cost is high and the process cycle is long.

[0003] FGFR (fibroblast growth factor receptor) is a protein that promotes cell growth and classification in normal cells. However, if the FGFR gene mutates, it may cause abnormalities in the encoded protein, leading to the occurrence of tumors. In addition, FGFR gene mutations may also affect the biological and clinical characteristics of urothelial carcinoma, including the invasiveness, efficacy and prognosis of cancer.

[0004] The preparation and observation of hematoxylin & eosin (H&E) stained histopathology slides are part of the routine diagnostic examination of patients with urothelial carcinoma. These sections can be scanned by the corresponding equipment to produce digital WSI pathology images, which are stored in a pyramid structure and occupy a large amount of space. The generated images contain many visual features. Although pathologists usually extract useful pathological information from pathology images, some visual patterns may be too trivial and hidden to capture. In recent years, with the rise of artificial intelligence, a large number of algorithm models have been applied and tested in the field of computer vision, such as convolutional neural networks, VisionTransformer, attention mechanisms, etc. Therefore, computational pathology can now use deep learning algorithms to train models with pathological images of urothelial carcinoma as data samples to predict whether the FGFR gene is carcinogenic. Some literature shows that FGFR gene mutations may cause changes in the morphology of urothelial tumor tissues, such as changes in immune infiltration patterns, papillary morphology differences, epithelial cell polarization, dense but uniform nuclear distribution, and other characteristics. These summarized image patterns and other image features that are difficult to capture by human vision together form the basis for predicting FGFR gene mutations using pathological images.

[0005] In summary, traditional methods for training models to identify gene mutation images are difficult to accurately identify pathological images of oncogenic mutations in FGFR genes. Summary of the invention

[0006] The present invention provides a method, a computing device and a computer storage medium for training a model for identifying gene variation images.

[0007] According to a first aspect of the present invention, a method for training a model for identifying gene mutation images is provided, characterized in that the method comprises: obtaining a training image sample set including gene mutation images annotated therewith, and determining a batch of image samples for a single training model based on the training image sample set; based on a feature extractor, sequentially performing feature extraction on the image sample batches to obtain a feature matrix vector of a predetermined dimension; performing a linear transformation on the acquired feature matrix vector to obtain a transformed matrix vector; inputting the transformed matrix vector into a first attention network and a second attention network in parallel to obtain a first attention network output and a second attention network output, respectively, and performing a linear layer conversion after multiplying the first attention network output and the second attention network output; and based on a linear classifier, sequentially outputting the probability that an image in the training image sample set belongs to a gene mutation image to obtain a model that has completed training.

[0008] According to a second aspect of the present invention, a computing device is also provided, comprising: a memory configured to store one or more computer programs; and a processor coupled to the memory and configured to execute the one or more programs so that the device performs the method of the first aspect of the present invention.

[0009] According to a third aspect of the present invention, a non-transitory computer-readable storage medium is further provided, wherein machine-executable instructions are stored on the non-transitory computer-readable storage medium, and when the machine-executable instructions are executed, the machine executes the method of the first aspect of the present invention.

[0010] In some embodiments, obtaining the transformed matrix vector includes: performing a linear layer, a ReLu activation function layer, and a Dropout layer transformation on the acquired feature matrix vector, thereby obtaining the transformed matrix vector.

[0011] In some embodiments, obtaining a trained model includes: calculating the loss of model training using a loss function based on the probability output by the linear classifier and the gene variation images annotated in the training image sample set; and repeatedly training the model until the calculated model training loss no longer decreases, so as to obtain a trained model.

[0012] In some embodiments, the method also includes: obtaining a training image verification set including images annotated with gene variants; validating the trained model based on the training image verification set to obtain a verification result of the model; and using cross-validation to evaluate the obtained verification result, and adjusting the hyperparameters of the trained model based on the evaluation.

[0013] In some embodiments, using cross-validation to evaluate the obtained validation results and adjusting the hyperparameters of the model based on the evaluation includes: dividing the training image validation set into N equal subsets and selecting one of the subsets as the model validation set, and the remaining N-1 subsets as the model training set; performing N cross-validations, wherein in each cross-validation, the model training set is trained using the trained model and evaluated on the model validation set to determine the performance indicators of the trained model; recording the performance indicators determined for each cross-validation to obtain N independent performance evaluation results; averaging the performance indicators determined by performing N cross-validations to determine an average performance evaluation of the trained model; and selecting model hyperparameters for the trained model based on the determined average performance evaluation.

[0014] In some embodiments, selecting model hyperparameters for a model to complete training includes selecting one or more of the following hyperparameters: learning rate, image sample batch size, regularization parameter, number of hidden layer units, network structure and number of layers, activation function, cross-validation fold number, optimizer, weighted sampling determination parameters, Dropout discard ratio, and weighted attenuation value.

[0015] In some embodiments, performing feature extraction on image samples in batches based on a feature extractor includes: performing feature extraction on image samples in batches based on a VisionTransformer model pre-trained by a pathology dataset.

[0016] In some embodiments, the first attention network includes a linear layer, a Tanh activation function layer, and a Dropout layer and the second attention network includes a linear layer, a Sigmoid activation function layer, and a Dropout layer.

[0017] In some embodiments, the method also includes: acquiring a digitized pathological slice image and determining a tissue region associated with a gene variation at a slice level in the acquired pathological slice image; segmenting the pathological slice image associated with the determined tissue region into images of a predefined size, wherein a height value and a width value of the image of the predefined size are proportional; determining an absorbance value threshold, color saturation, hue, image contour information, and a reference image of the tissue region; performing background filtering on the segmented image of the predefined size based on the determined absorbance value threshold and image contour information, thereby acquiring an image with the tissue region as the main body; and performing a Macenko staining standardization operation on the image with the tissue region as the main body based on the reference image to achieve color normalization processing of the image, thereby generating a color-normalized training image sample set and a training image verification set.

[0018] In some embodiments, the method includes: receiving an input image data set; and

[0019] The received image dataset is input into the model for identifying gene variation images trained by the method described above, so as to determine that variation-based images are included in the image dataset.

[0020] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the invention, nor is it intended to limit the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a system for a method for training a model for identifying gene variation images according to an embodiment of the present invention is shown.

[0022] Figure 2 A flowchart of a method for training a model for identifying gene variation images according to an embodiment of the present invention is shown.

[0023] Figure 3 A method 300 for training a model according to an embodiment of the present invention is shown.

[0024] Figure 4 A flowchart of a method for pre-processing a picture according to an embodiment of the present invention is shown.

[0025] Figure 5 A flowchart of a method for predicting a picture of gene variation using a model for predicting gene variation according to an embodiment of the present invention is shown.

[0026] Figure 6 The block diagram of an electronic device suitable for implementing the embodiments of the present invention is schematically shown.

[0027] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0029] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects.

[0030] In order to at least partially solve the above problems and one or more of other potential problems, the exemplary embodiments of the present invention propose a scheme for training a model for identifying gene mutation images. FGFR (fibroblast growth factor receptor) is a protein that can promote cell growth and classification in normal cells. However, if the FGFR gene mutates, it may cause abnormal encoding protein and lead to the occurrence of tumors. In addition, FGFR gene mutations may also affect the biological and clinical characteristics of urothelial carcinoma, including the invasiveness, efficacy and prognosis of cancer.

[0031] The preparation and observation of hematoxylin & eosin (H&E) stained histopathology slides are part of the routine diagnostic examination of patients with urothelial carcinoma. These sections can be scanned by the corresponding equipment to produce digital WSI pathology images, which are stored in a pyramid structure and occupy a large amount of space. The generated images contain many visual features. Although pathologists usually extract useful pathological information from pathology images, some visual patterns may be too trivial and hidden to capture. In recent years, with the rise of artificial intelligence, a large number of algorithm models have been applied and tested in the field of computer vision, such as convolutional neural networks, VisionTransformer, attention mechanisms, etc. Therefore, computational pathology can now use deep learning algorithms to train models with pathological images of urothelial carcinoma as data samples to predict whether the FGFR gene is carcinogenic. Some literature shows that FGFR gene mutations may cause changes in the morphology of urothelial tumor tissues, such as changes in immune infiltration patterns, papillary morphology differences, epithelial cell polarization, dense but uniform nuclear distribution, and other characteristics. These summarized image patterns and other image features that are difficult to capture by human vision together form the basis for predicting FGFR gene mutations using pathological images.

[0032] NGS testing is currently the most recognized means of detecting FGFR gene mutations, but it is expensive and time-consuming. There are few medical institutions qualified for NGS testing, and they are generally concentrated in first- and second-tier cities. These are all negative factors that affect patient diagnosis, and the present invention can be used as a prerequisite or alternative for NGS testing. At present, most general hospitals have pathology departments, which will perform surgery or puncture pathological sampling of the primary or metastatic lesions of patients diagnosed with solid cancer and make tissue sections. Only the corresponding equipment is needed to read these pathological images, and the present invention can directly make predictions, and the time and financial costs required are much less than NGS testing.

[0033] Figure 1 FIG. 1 is a schematic diagram of a system 100 for training a model for identifying gene mutation images according to an embodiment of the present invention. Figure 1 As shown, the system 100 includes: a computing device 110, an image data management device 130, and a network 140. In some embodiments, the computing device 110 and the image data management device 130 perform data exchange via the network 140.

[0034] Regarding the image data management device 130, it is, for example, a one-stop storage computing structure running on one or more computer nodes for implementing high-concurrency, high-throughput image data management. The image data management server 130 can send, receive or store multiple copies of image data or a collection of image data, the image data including but not limited to pathological reference images including genetic variations, etc.

[0035] Regarding computing device 110, it receives image data or a collection of image data from image data management server 130, for example, via network 140. By performing corresponding processing on the received image data or a collection of image data, computing device 110 can encrypt and decrypt information and generate a key.

[0036] In some embodiments, the computing device 110 may have one or more processing units, including dedicated processing units such as GPU, FPGA and ASIC, and general processing units such as CPU. In addition, one or more virtual machines may also be running on each computing device. The computing device 110 may include, for example:

[0037] The acquisition module 112 is configured to acquire a training image sample set including images annotated with gene mutations, and determine a batch of image samples for a single training model based on the training image sample set.

[0038] The extraction module 114 is configured to perform feature extraction on the image samples in batches in sequence based on the feature extractor, so as to obtain a feature matrix vector of a predetermined dimension.

[0039] The transformation module 116 is configured to perform a linear layer, a ReLu activation function layer, and a Dropout layer transformation on the acquired feature matrix vector, thereby obtaining a transformed matrix vector.

[0040] The attention mechanism module 118 is configured to input the transformed matrix vector into the first attention network and the second attention network in parallel to obtain the first attention network output and the second attention network output respectively and multiply the first attention network output and the second attention network output and then perform a linear layer conversion.

[0041] The output module 120 is configured to output the probabilities of the images in the training image sample set belonging to the gene mutation images in sequence based on the linear classifier, so as to obtain a model that has completed the training.

[0042] The present invention can choose to perform staining normalization processing on the pathological image, thereby reducing the interference of other external factors caused by different equipment and different personnel operations. In addition to the most commonly used ResNet50 or ResNet101 models trained based on the ImageNet dataset, the pre-trained model for extracting image features of the present invention can also use the VisionTransformer model pre-trained by the TCGA pathology dataset, which may improve the representativeness of feature extraction. The present invention does not use the convolutional neural network commonly used in image recognition, but uses the newer gated attention mechanism plus a linear layer to construct an image classification model, which may make the model pay more attention to the tumor part of the pathological image or other important abnormal areas to resist noise interference. The computing requirements can be reduced by processing the features, and there is no need to read, process and predict the entire pathological image of up to 2GB. A highly explanatory visual heat map can be generated to help doctors understand the decision-making process of the model.

[0043] The following will be combined Figure 2 A method for training a model for identifying gene mutation images according to an embodiment of the present invention is described. Figure 2 FIG. 2 is a flowchart of a method for training a model for identifying gene mutation images according to an embodiment of the present invention. It should be understood that the method 200 may be used, for example, in Figure 6 The electronic device 600 described above is executed. Figure 1 The method 200 is executed at the described computing device 110. It should be understood that the method 200 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0044] At step 202 , the computing device 110 obtains a training image sample set including images labeled with gene mutations, and determines a batch of image samples for a single training model based on the training image sample set.

[0045] In one embodiment, the computing device 110 obtains a training image sample set including images labeled with gene mutations from an image database. The batch size of image samples for each training model is determined, for example, set to 32 images. It can be understood by those skilled in the art that the images included in the training image sample set are all images of tumor tissues, which are further divided into images labeled with gene mutations and images not labeled with gene mutations, wherein images labeled with gene mutations refer to images that are labeled or defined as images of tumor tissue gene mutations, and images not labeled with gene mutations refer to images that are labeled or defined as images of tumor tissue without gene mutations.

[0046] The method for obtaining the training image sample set of gene variation images will be described in more detail in the method 400 below.

[0047] At step 204 , the computing device 110 performs feature extraction on the image samples in batches in sequence based on the feature extractor, thereby obtaining a feature matrix vector of a predetermined dimension.

[0048] In one embodiment, the computing device 110 can use a pre-trained deep convolutional neural network (such as ResNet, VGG, etc.) as a feature extractor, or can choose to use a VisionTransformer model pre-trained by a TCGA pathology dataset as a feature extractor. Based on the VisionTransformer model pre-trained by the pathology dataset, feature extraction is performed on image samples in batches. ResNet can, for example, select ResNet (50, 101). The image samples are then input into the network in batches. The backbone network of. The intermediate layer output of the feature extractor is obtained to obtain the corresponding feature matrix vector.

[0049] By performing feature extraction on pathology layers at different magnifications, an N×M feature matrix can be obtained, where N is the number of samples and M is a multidimensional feature vector. Taking M equal to 1024 as an example, 1024 features can be extracted from each image.

[0050] At step 206, the computing device 110 performs a linear transformation on the acquired feature matrix vector to obtain a transformed matrix vector, including performing a linear layer, a ReLu activation function layer, and a Dropout layer transformation on the acquired feature matrix vector to obtain a transformed matrix vector.

[0051] In one embodiment, the computing device 110 performs a linear layer transformation on the feature matrix vector obtained at step 204, and may use a fully connected layer to perform matrix multiplication on the feature vector and add a bias term.

[0052] Then, the ReLU activation function is applied to the output of the linear layer, which maps negative numbers to zero, enhances the nonlinear expression ability of the model, and obtains the activated feature matrix vector.

[0053] Finally, the Dropout layer is applied to the activated feature matrix vector to randomly set some neuron outputs to zero to reduce the risk of overfitting and enhance the generalization ability of the model.

[0054] At step 208, the computing device 110 inputs the transformed matrix vector into the first attention network and the second attention network in parallel to obtain the first attention network output and the second attention network output respectively, and performs a linear layer conversion after multiplying the first attention network output and the second attention network output.

[0055] In one embodiment, the computing device 110 may design two attention networks for the model, respectively process the transformed feature matrix vector, and respectively output attention vectors. The output of the first attention network and the output of the second attention network are element-wise multiplied to obtain a fused attention vector. The fused attention vector is input into the linear layer to adjust the feature weight.

[0056] Specifically, the first attention network adopts the self-attention mechanism to model the association between different positions in the feature matrix vector and output a weight vector.

[0057] The second attention network can use a similar self-attention mechanism or another attention mechanism to capture the global information in the feature matrix vector and output another weight vector. The output of the first attention network and the output of the second attention network are element-wise multiplied to obtain the fused attention vector. The fused attention vector is input into the linear layer to adjust the feature weights and convert it into a more meaningful feature representation.

[0058] In a preferred embodiment, the first attention network includes a linear layer, a Tanh activation function layer and a Dropout layer, and the second attention network includes a linear layer, a Sigmoid activation function layer and a Dropout layer.

[0059] At step 210 , the computing device 110 sequentially outputs the probabilities that the images in the training image sample set belong to gene mutation images based on the linear classifier, thereby obtaining a trained model.

[0060] In one embodiment, the computing device 110 can set a fully connected layer as a linear classifier as an output layer based on the fused feature vector. The number of nodes in the output layer is 2, representing two categories: gene mutation images and non-gene mutation images. After the output layer, the softmax function is applied to normalize the output, and the output value is converted into a probability value, representing the probability that the image belongs to a gene mutation image and a non-gene mutation image. For example, the prediction error of the model can be calculated using a cross entropy loss function, and the weight parameters of the model can be optimized by a back propagation algorithm. Iterate multiple times (can be completed through multiple training iterations) until the model converges and the training process of the model is completed.

[0061] In one embodiment, based on the probability output by the linear classifier and the gene mutation images annotated in the training image sample set, a loss function is used to calculate the loss of the model training. The loss function can be, for example, a SmoothTop1SVM() loss function. The SmoothTop1SVM() loss function is a smoothed version of the Top-1SVM loss function, which itself can be differentiated:

[0062]

[0063] Formula (1) shows the SmoothTop1SVM() loss function used in the present invention.

[0064] The computing device 110 may repeatedly train the model until the calculated loss of the model training no longer decreases, to obtain a trained model.

[0065] The following will be combined Figure 3 A method for training a model according to an embodiment of the present invention is described. Figure 3 FIG. 3 is a flowchart of a method for training a model according to an embodiment of the present invention. It should be understood that the method 300 may be, for example, Figure 6 The electronic device 600 described above is executed. Figure 1 The method 300 is executed at the described computing device 110. It should be understood that the method 300 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0066] Figure 3 A method 300 for training a model according to an embodiment of the present invention is shown. At step 302, the computing device 110 may obtain a training image validation set including images annotated with gene mutations.

[0067] In one embodiment, the computing device 110 divides a portion of the data from the training image sample set into a validation set for verifying the performance of the model. A certain proportion of image samples are randomly selected from the training image sample set, and gene variation category information is annotated for these images to form a training image validation set. Ensure that the image category distribution between the training image validation set and the training image sample set is consistent to reduce the bias problem caused by category imbalance.

[0068] At step 304 , the computing device 110 verifies the trained model based on the training image verification set, thereby obtaining a verification result of the model.

[0069] In one embodiment, the computing device 110 uses the training image verification set to evaluate the trained deep learning model to verify the performance of the model. The image samples in the training image verification set are batch-input into the trained model to obtain the prediction results of the model. By comparing the prediction results of the model with the actual annotation results, the performance indicators of the image verification results, such as accuracy, precision, recall, F1 value, etc., are calculated to measure the performance of the model on the verification set.

[0070] At step 306 , the computing device 110 uses cross-validation to evaluate the obtained validation results and adjusts the hyperparameters of the trained model based on the evaluation.

[0071] In one embodiment, the computing device 110 exits the program after reaching a predetermined epoch or stop condition, and the hyperparameters can be adjusted based on the results of the model on the validation set, and the model training can be restarted. This model uses stratified k-fold cross validation to evaluate model performance and adjust model hyperparameters. In order to improve the credibility and stability of the validation results, the computing device 110 uses cross validation to evaluate the model. The training image sample set is divided into k subsets, and then k-fold cross validation is performed, each subset is used as a validation set in turn, and the remaining k-1 subsets are used as training sets to obtain k independent validation results. The validation results of each cross validation are averaged to obtain the average validation performance index of the model as the performance evaluation of the model on the entire training image validation set. According to the results of cross validation, the hyperparameters of the trained model can be adjusted, such as the learning rate, the number of network layers, the structure of the feature extractor, etc., to optimize the performance of the model.

[0072] Specifically, the computing device 110 divides the training image validation set into N equal subsets and selects one of the sets as the model validation set, and the remaining N-1 subsets as the model training set. Perform N cross validations, wherein in each cross validation, the model training set is trained using the trained model and evaluated on the model validation set, thereby determining the performance index of the trained model. The determined performance index of each cross validation is recorded to obtain N independent performance evaluation results. The performance index of the N cross validations is averaged to determine the average performance evaluation of the trained model. Based on the determined average performance evaluation, a model hyperparameter is selected for the trained model. Selecting a model hyperparameter for the trained model includes selecting one or more of the following hyperparameters: learning rate, image sample batch size, regularization parameter, number of hidden layer units, network structure and number of layers, activation function, cross validation fold, optimizer, weighted sampling determination parameter, Dropout discard ratio, weighted attenuation value.

[0073] The following schematically illustrates a schematic program code of a method for training a model for identifying gene variation images.

[0074]

[0075]

[0076] It should be understood that the predetermined program module may also be a pre-compiled program module of other functions, for example, a code for a method for training a model for identifying gene variation images, etc.

[0077] Based on the above technical means, the training of the model can be completed. In a preferred embodiment, the computing device 110 can also obtain a test set including gene mutation images, and perform a test on the trained model on the test set to obtain a performance evaluation result corresponding to the test set. The performance evaluation result of the trained model on the cross-validation set can be compared with the performance evaluation result performed on the test set. In response to the performance evaluation results of the test set and the validation set being close, it can be determined that the trained model has good stability and persuasiveness.

[0078] The following will be combined Figure 4 A method for preprocessing a picture according to an embodiment of the present invention is described. Figure 4 FIG. 4 is a flowchart of a method for preprocessing a picture according to an embodiment of the present invention. It should be understood that the method 400 may be, for example, Figure 6 The electronic device 600 described above is executed. Figure 1 The method 400 is executed at the described computing device 110. It should be understood that the method 400 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0079] Figure 4 A method 400 for preprocessing an image according to an embodiment of the present invention is shown. At step 402, the computing device 110 acquires a digitized pathological section and determines a tissue region associated with a gene variation at a section level in the acquired pathological section.

[0080] In one embodiment, digitized pathology slides are obtained from a hospital or pathology laboratory, usually stored in a computer in the form of high-resolution images. Computer vision technology and image analysis algorithms are used to process the pathology slides, and pixel-level analysis and feature extraction are used to determine tissue regions associated with genetic variation. Machine learning algorithms, such as convolutional neural networks (CNNs), can be used to classify and segment images and mark tissue regions associated with genetic variation.

[0081] At step 404 , the computing device 110 segments the determined pathological slices of the tissue region into images of a predefined size, wherein the height value and the width value of the image of the predefined size are proportional.

[0082] In one embodiment, the computing device 110 segments the determined pathological slices of the tissue region and cuts them into images of predefined sizes so that the height and width values ​​of the images are proportional. For example, the image resolution is customized, but the height and width values ​​are required to be equal, controlled between 100-800, and the default resolution is 224×224. The segmented images of the predefined size can be more conveniently used for subsequent processing and analysis, while maintaining the information of the tissue structure.

[0083] At step 406 , the computing device 110 determines an absorbance value threshold, color saturation, hue, and image contour information of a tissue region of the pathological section and a reference image.

[0084] In one embodiment, the computing device 110 uses image processing techniques and algorithms to determine the absorbance threshold, color saturation, and hue of the tissue region of the pathological section so as to distinguish the tissue region from the background region. The contour information of the image is extracted for subsequent image segmentation and background filtering operations. A reference image is prepared for subsequent color normalization processing.

[0085] At step 408 , the computing device 110 performs background filtering on the segmented image of the predefined size based on the determined absorbance value threshold and image contour information, thereby obtaining an image with the tissue region as the main body.

[0086] In one embodiment, the computing device 110 uses a predetermined absorbance threshold to perform binarization processing on a picture of a predefined size to extract the tissue region, and further performs background filtering on the image based on the image contour information to remove the portion that does not belong to the tissue region, thereby obtaining a picture with the tissue region as the main body.

[0087] At step 410 , the computing device 110 performs a Macenko staining standardization operation on the image mainly including the tissue region based on the reference image to achieve color normalization processing of the image, thereby generating a training image sample set and a training image verification set that have been color normalized.

[0088] In one embodiment, the computing device 110 uses the reference image to perform a Macenko staining normalization operation, so that all images with tissue regions as the main body after background filtering are normalized in color. The Macenko staining normalization operation can eliminate the color differences between images, making the images more comparable and stable in subsequent processing. A training image sample set and a training image verification set that have been color normalized are generated. After the above processing steps, images with tissue regions as the main body that have been color normalized can be obtained, and these images can constitute a training image sample set and a training image verification set.

[0089] There are many significant or deep image features in pathological images, and some inconspicuous features are difficult for pathologists to identify, summarize and generalize. By using a pre-trained ResNet model or VisionTransformer model to process pathological images, the model can extract those visual features that the model considers to be important, which is likely to make up for the shortcomings of humans. Based on these extracted features, the FGFR gene mutation is predicted. Considering that each slice has many parts that cannot or have not been infiltrated by tumor tissue in addition to the tumor lesion area, the attention mechanism can be used for subsequent result prediction. This is a kind of weak supervision and can effectively avoid the interference of noise (i.e., non-cancerous areas). The images are divided into a training image sample set and a training image verification set according to the required ratio for the training and performance evaluation of the deep learning model.

[0090] The following will be combined Figure 5 A method for predicting a picture of gene variation using a model for predicting gene variation according to an embodiment of the present invention is described. Figure 5 FIG. 5 is a flowchart of a method for predicting a gene variation image using a model for predicting gene variation according to an embodiment of the present invention. It should be understood that method 500 can be used, for example, in Figure 6 The electronic device 600 described above is executed. Figure 1 The method 500 is executed at the described computing device 110. It should be understood that the method 500 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0091] At step 502 , computing device 110 receives an input image data set.

[0092] At step 504 , the computing device 110 inputs the received image dataset into the model for identifying gene variation images trained by the above method, thereby determining that variation-based images are included in the image dataset.

[0093] Based on the above technical means, a combination of multiple algorithms can be used to remove a large amount of blank background in pathological images to the greatest extent, and the ResNet / VisionTransformer network is used to extract relevant features and visual attention mechanisms, which greatly reduces noise interference and the amount of computation required for subsequent model training, accelerates the entire training process, and improves the quality of model training.

[0094] Figure 6 The block diagram of the electronic device 300 suitable for implementing the embodiment of the present invention is schematically shown. The electronic device 600 may be used to implement the execution Figure 2 The methods 200, 300, 400 and 500 shown in FIG. Figure 6As shown, the electronic device 600 includes a central processing unit (i.e., CPU 601), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (i.e., ROM 602) or loaded from a storage unit 608 to a random access memory (i.e., RAM 603). In RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output interface (i.e., I / O interface 605) is also connected to the bus 604.

[0095] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and the CPU 601 performs the various methods and processes described above, such as executing methods 200, 300, 400, and 500. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is stored in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the CPU 601, one or more operations of the method 200 described above may be performed. Alternatively, in other embodiments, the CPU 601 may be configured to perform one or more actions of the method 200 by any other appropriate means (e.g., by means of firmware).

[0096] It should be further noted that the present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0097] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0098] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0099] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0100] Here, various aspects of the present invention are described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to an embodiment of the present invention. It should be understood that each square frame of the flow chart and / or block diagram and the combination of each square frame in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0101] These computer-readable program instructions can be provided to a processor in a voice interaction device, a general-purpose computer, a special-purpose computer, or a processing unit of other programmable data processing devices, thereby producing a machine, so that when these instructions are executed by a processing unit of a computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions enable the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0102] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0103] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of this module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0104] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0105] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for training a model for identifying gene mutation images, characterized in that: The method comprises: Obtaining a training image sample set including images annotated with gene mutations, and determining an image sample batch for a single training model based on the training image sample set; Based on the feature extractor, feature extraction is performed on the image samples in batches in sequence, thereby obtaining a feature matrix vector of a predetermined dimension; Performing a linear transformation on the acquired feature matrix vector to obtain a transformed matrix vector, wherein the linear transformation includes performing matrix multiplication on the feature matrix vector using a fully connected layer and adding a bias term; Inputting the transformed matrix vector into the first attention network and the second attention network in parallel to obtain the first attention network output and the second attention network output respectively, and performing a linear layer conversion after multiplying the first attention network output and the second attention network output, wherein the linear layer conversion includes inputting the fused attention vector obtained by multiplying the first attention network output and the second attention network output vector into the linear layer to adjust the feature weight; and Based on the linear classifier, the probability that the images in the training image sample set belong to the gene mutation image is output in sequence, so as to obtain a model that completes the training.

2. The method according to claim 1, characterized in that Obtaining the transformed matrix vector includes: The obtained feature matrix vector is transformed by performing a linear layer, a ReLu activation function layer, and a Dropout layer to obtain a transformed matrix vector.

3. The method according to claim 1, characterized in that Obtaining the trained model includes: Based on the probability output by the linear classifier and the gene mutation images annotated in the training image sample set, using the loss function to calculate the loss of model training; and The model is repeatedly trained until the calculated model training loss no longer decreases to obtain a model that has completed training.

4. The method according to claim 3, characterized in that The method further comprises: Obtain a validation set of training images including images annotated with gene variants; Based on the training image verification set, verifying the trained model, thereby obtaining a verification result of the model; and Use cross-validation to evaluate the obtained validation results and adjust the hyperparameters of the trained model based on the evaluation.

5. The method according to claim 4, characterized in that Using cross-validation to evaluate the obtained validation results and adjust the model's hyperparameters based on the evaluation includes: Divide the training image validation set into N equal subsets and select one of the subsets as the model validation set, and the remaining N-1 subsets as the model training set; Perform N cross validations, where in each cross validation, the trained model is used to train the model training set and evaluated on the model validation set, thereby determining the performance indicator of the trained model; Record the determined performance indicators of each cross-validation to obtain N independent performance evaluation results; Averaging the performance metrics determined from performing the N cross validations to determine an average performance evaluation of the trained model; and Based on the determined average performance evaluation, model hyperparameters are selected for the trained model.

6. The method according to claim 5, characterized in that Selecting model hyperparameters for a trained model includes selecting one or more of the following hyperparameters: learning rate, image sample batch size, regularization parameter, number of hidden layer units, network structure and number of layers, activation function, number of cross-validation folds, optimizer, weighted sampling determination parameters, Dropout discard ratio, and weighted decay value.

7. The method according to claim 1, characterized in that Performing feature extraction on image samples in batches based on the feature extractor includes: Based on the VisionTransformer model pre-trained by the pathology dataset, feature extraction is performed on image samples in batches.

8. The method according to claim 1, characterized in that The first attention network includes a linear layer, a Tanh activation function layer and a Dropout layer and the second attention network includes a linear layer, a Sigmoid activation function layer and a Dropout layer.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Acquire digitalized pathological section images and determine tissue regions associated with gene mutations at the section level in the acquired pathological section images; Segmenting the pathological slice image associated with the determined tissue region into images of a predefined size, wherein a height value and a width value of the image of the predefined size are proportional; Determine the absorbance threshold, color saturation, hue, image contour information and reference image of the tissue area; Based on the determined absorbance value threshold and image contour information, background filtering is performed on the segmented image of the predefined size, thereby obtaining an image with the tissue region as the main body; and Based on the reference image, a Macenko staining standardization operation is performed on the image with the tissue region as the main body to achieve color normalization processing of the image, thereby generating a training image sample set and a training image verification set that have been color normalized.

10. A method for identifying gene mutation images, characterized in that: The method comprises: receiving an input image dataset; and The received image data set is input into a model for identifying gene variation images trained based on the method of any one of claims 1 to 8, thereby determining that variation-based images are included in the image data set.

11. A computing device, characterized in that: include: at least one processing unit; At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: A computer program is stored on a computer-readable storage medium, and when the computer program is executed by a machine, the method according to any one of claims 1 to 10 is implemented.