Method, system and device for determining prognosis characteristics of nasopharyngeal carcinoma and storage medium

Abstract

The application discloses a nasopharyngeal carcinoma prognosis feature determination method, system and device and a storage medium. The method comprises the following steps: pathological image preprocessing, color normalization based on dye separation is used to standardize the dyeing of the pathological image; a segmentation network is used to automatically segment the lesion area of the preprocessed pathological image to obtain a segmented image; the segmented image is cropped to obtain a target image block; principal component analysis is used to reduce the dimension of the target image block to obtain reduced dimension data; a clustering algorithm is used to perform unsupervised autonomous learning on the reduced dimension data to obtain pathological image features; and finally, the pathological image features are screened through feature inspection to determine a prognosis pathological feature set. The application can obtain and screen key image features of pathological images closely related to local area recurrence and distant metastasis of nasopharyngeal carcinoma from pathological images to assist in prognosis prediction of nasopharyngeal carcinoma, and can be widely applied to the technical field of image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a nasopharyngeal carcinoma prognosis feature determination method, system, device and storage medium. BACKGROUND

[0002] Nasopharyngeal carcinoma (NPC) is a malignant tumor originating from nasopharyngeal epithelial cells. Although the efficacy of nasopharyngeal carcinoma has improved in recent years, the local control rate of nasopharyngeal carcinoma has been significantly improved, but still 10% to 15% of patients have tumor recurrence. In addition, for patients with locally advanced nasopharyngeal carcinoma, 30% of them have distant metastasis after treatment. Local recurrence and distant metastasis are the main reasons for the failure of nasopharyngeal carcinoma treatment and death. How to screen out high-risk groups of nasopharyngeal carcinoma disease progression and thus carry out early intervention and implement individualized treatment is a difficult problem that needs to be solved in clinical practice at present.

[0003] As the "gold standard" of clinical diagnosis, pathology represents the microscopic morphology of tumor cells and their surrounding microenvironment. Pathology plays a very important role in cancer diagnosis, and tumor tissue pathological sections are one of the main methods for physicians to evaluate tumor disease progression, clinical staging, classification and typing. According to the histopathological characteristics, the WHO pathological typing shows that there are three pathological subtypes of nasopharyngeal carcinoma: 1. keratinizing squamous cell carcinoma; 2. non-keratinizing squamous cell carcinoma, divided into differentiated non-keratinizing carcinoma and undifferentiated non-keratinizing carcinoma; 3. basal cell-like squamous cell carcinoma. Among them, keratinizing squamous cell carcinoma is more common in non-endemic areas, accounting for about 20% of cases, while in China's southern region and other global high-incidence regions, more than 95% of nasopharyngeal carcinoma patients are non-keratinizing squamous cell carcinoma. However, studies have shown that non-keratinizing squamous cell carcinoma cannot further stratify the risk of patients in high-incidence regions according to the existing WHO pathological typing. Although pathologists observe that there is obvious morphological heterogeneity in tumor cells of nasopharyngeal carcinoma, the interpretation criteria are difficult to unify and standardize, and the pathological typing proposed so far has not been able to guide the prognosis prediction and individualized treatment of nasopharyngeal carcinoma patients. With the rapid development of digital pathology (DP) of whole-slide imaging (WSI), artificial intelligence has made many progress in computer-aided diagnosis. Utilizing the massive features in pathological images and processing, analyzing and predicting the image feature information through artificial intelligence big data model can help clinicians evaluate tumor diagnosis staging, treatment and prognosis, and become another powerful auxiliary means of modern medicine.

[0004] At present, clinical parameters such as TNM staging of tumor, gender, age, EBV DNA copy number and hematological indicators are used to construct a model for prognosis prediction. However, the prediction model does not include the feature information of the pathological image of nasopharyngeal carcinoma, and it is difficult to comprehensively and objectively reflect the biological characteristics and malignant nature of the evolution stage of nasopharyngeal carcinoma. In the aspect of constructing a prognosis prediction model based on pathological images, only a few studies have constructed pathological prognostic indicators. Liu et al. extracted 429 quantitative pathological features from pathological whole slide images through open software QuPath to obtain pathological labels and predict 5-year progression-free survival. However, due to the lack of simple and specific pathological features, the clinical application is relatively difficult. Shao et al. divided nasopharyngeal carcinoma into epithelioid carcinoma, sarcomatoid carcinoma, sarcoma-epithelioid carcinoma and squamous cell carcinoma according to the histological characteristics of nasopharyngeal carcinoma, and established a prognosis model for each pathological type. However, due to the single pathological type of nasopharyngeal carcinoma, the actual clinical application is very limited. The above results all show that it is urgent to use more abundant information and advanced data analysis means to mine the pathological picture information of nasopharyngeal carcinoma and construct a prognosis prediction model, to provide an auxiliary evaluation means for the diagnosis and treatment of nasopharyngeal carcinoma patients with different risk stratifications, and to help promote the individualized precise diagnosis and treatment of nasopharyngeal carcinoma. SUMMARY

[0005] Therefore, the embodiments of the present application provide a nasopharyngeal carcinoma prognosis feature determination method, system and device and a storage medium, which can screen out key image features of pathological images.

[0006] In a first aspect, the embodiments of the present application provide a nasopharyngeal carcinoma prognosis feature determination method, comprising:

[0007] The pathological image is preprocessed by a color normalization method based on dye separation to perform dye standardization processing on the pathological image;

[0008] The lesion region of the preprocessed pathological image is automatically segmented by a segmentation network to obtain a segmented image; wherein the segmentation network comprises a deep neural network and a neural conditional random field, the deep neural network is used for segmentation of the lesion region, and the neural conditional random field is used for modeling the spatial correlation between adjacent image blocks;

[0009] The segmented image is cropped to obtain a target image block;

[0010] The target image block is subjected to feature dimension reduction by principal component analysis to obtain reduced dimension data;

[0011] According to the reduced dimension data, the pathological image features are obtained by unsupervised autonomous learning through a clustering algorithm;

[0012] The pathological image features are screened by feature inspection to determine a prognosis pathological feature set.

[0013] Optionally, the method further comprises:

[0014] According to the pathological image features, a prognosis prediction is performed through a deep Q network to determine key pathological image features of the patient;

[0015] The deep Q network comprises a behavior network and a target network; the behavior network is used to collect training experience and update network parameters through the training experience; and the target network is used to perform decision operations according to behavior probabilities.

[0016] Optionally, the staining standardization processing of the pathological image through the color normalization method based on staining separation comprises:

[0017] Image channels of each dye are obtained through matrix decomposition;

[0018] The image channels are subjected to mean-variance normalization processing, and a pathological image of an RGB channel is obtained through merging.

[0019] Optionally, the cropping processing of the segmented image to obtain a target image block comprises:

[0020] The segmented image is cropped according to a preset specification to obtain image blocks of tumor tissues and interstitial tissues.

[0021] Optionally, the feature dimension reduction of the target image block through principal component analysis to obtain dimension reduction data comprises:

[0022] A target eigenvector set of a data covariance matrix is obtained;

[0023] A transformation matrix is formed according to the target vector set;

[0024] The target image block is subjected to feature dimension reduction through the transformation matrix to obtain dimension reduction data.

[0025] Optionally, the pathological image features are obtained through unsupervised autonomous learning of the dimension reduction data through a clustering algorithm, comprising:

[0026] All data are clustered into c classes through a K-means clustering algorithm according to a preset hyperparameter; the c is the preset hyperparameter;

[0027] A center point representation of each class cluster is used as a representative of the class;

[0028] The center point representation is used to determine the pathological image features.

[0029] Optionally, an expression of an optimization function minimized by the K-means clustering algorithm is:

[0030]

[0031] wherein J e represents an optimization function, x represents a sample after feature dimension reduction, D i represents a sample set of each cluster, m i represents a center point of the i-th cluster.

[0032] In a second aspect, an embodiment of the present application provides a nasopharyngeal carcinoma prognosis feature determination system, comprising:

[0033] A first module is configured to perform pathological image preprocessing, and perform staining standardization processing on a pathological image based on a color normalization method based on staining separation;

[0034] A second module is configured to perform automatic segmentation on a lesion region of the preprocessed pathological image by using a segmentation network to obtain a segmented image; wherein the segmentation network comprises a deep neural network and a neural conditional random field, the deep neural network is used for lesion region segmentation, and the neural conditional random field is used for modeling spatial correlation between adjacent image blocks.

[0035] A third module is configured to perform cropping processing on the segmented image to obtain a target image block.

[0036] A fourth module is configured to perform feature dimension reduction on the target image block by using principal component analysis to obtain dimension reduction data.

[0037] A fifth module is configured to perform unsupervised autonomous learning on the dimension reduction data by using a clustering algorithm to obtain pathological image features.

[0038] A sixth module is configured to perform screening on the pathological image features by using feature inspection to determine a prognosis pathological feature set.

[0039] In a third aspect, an embodiment of the present application provides a nasopharyngeal carcinoma prognosis feature determination device, comprising a processor and a memory;

[0040] The memory is configured to store a program;

[0041] The processor executes the program to realize the method as described above.

[0042] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the storage medium stores a program, and the program is executed by a processor to realize the method as described above.

[0043] The embodiment of the present application also discloses a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the foregoing method.

[0044] The embodiment of the present application firstly pre-processes a pathological image, performs staining standardization processing on the pathological image based on a color normalization method based on staining separation; performs automatic segmentation on a lesion region of the pre-processed pathological image through a segmentation network to obtain a segmented image; the segmentation network comprises a deep neural network and a neural conditional random field, the deep neural network is used for segmentation of the lesion region, and the neural conditional random field is used for modeling spatial correlation between adjacent image blocks; the segmented image is subjected to clipping processing to obtain a target image block; the target image block is subjected to feature dimension reduction through principal component analysis to obtain reduced dimension data; pathological image features are obtained through unsupervised autonomous learning of a clustering algorithm according to the reduced dimension data; and finally, the pathological image features are screened through feature inspection to determine a prognosis pathological feature set. The present application can obtain and screen key image features of the pathological image closely related to local area recurrence and distant metastasis of nasopharyngeal carcinoma from the pathological image to assist in prognosis prediction of nasopharyngeal carcinoma. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 The overall step flowchart of the nasopharyngeal carcinoma prognosis feature determination method provided by the embodiment of the present application is shown in the figure.

[0047] Figure 2 The structure diagram of the U-Net provided by the embodiment of the present application is shown in the figure.

[0048] Figure 3 The principle flowchart of the pathological omics feature extraction provided by the embodiment of the present application is shown in the figure.

[0049] Figure 4 The principle flowchart of the pathological feature extraction provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0050] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0051] In order to solve the problems in the prior art, in a first aspect, the present application provides a nasopharyngeal carcinoma prognosis feature determination method, as shown in the following formula: Figure 1 The method comprises the following steps:

[0052] The pathological image is preprocessed by a color normalization method based on dye separation to perform dye standardization processing on the pathological image;

[0053] The lesion area of the preprocessed pathological image is automatically segmented by a segmentation network to obtain a segmented image; wherein the segmentation network comprises a deep neural network and a neural conditional random field, the deep neural network is used for segmentation of the lesion area, and the neural conditional random field is used for modeling the spatial correlation between adjacent image blocks;

[0054] The segmented image is cropped to obtain a target image block;

[0055] The target image block is subjected to feature dimension reduction by principal component analysis to obtain reduced dimension data;

[0056] According to the reduced dimension data, pathological image features are obtained by unsupervised autonomous learning through a clustering algorithm;

[0057] The pathological image features are screened by feature inspection to determine a prognosis pathological feature set.

[0058] Optionally, the method further comprises:

[0059] According to the pathological image features, prognosis prediction is performed by a deep Q network to determine key pathological image features of the patient;

[0060] The deep Q network comprises a behavior network and a target network; the behavior network is used to collect training experience, and the network parameters are updated by the training experience; the target network is used to make decision operations according to behavior probability.

[0061] Optionally, the dye standardization processing on the pathological image by the color normalization method based on dye separation comprises:

[0062] The image channels of each dye are obtained by matrix decomposition;

[0063] The image channels are subjected to mean-variance normalization processing to obtain a pathological image of an RGB channel.

[0064] Optionally, the cropping processing on the segmented image is performed to obtain a target image block, and the cropping processing comprises:

[0065] According to a preset specification, the segmented image is cropped to obtain an image block of tumor tissue and interstitial tissue.

[0066] Optionally, the feature dimension reduction on the target image block is performed by principal component analysis to obtain reduced dimension data, and the feature dimension reduction comprises:

[0067] A target eigenvector set of a data covariance matrix is obtained.

[0068] A transformation matrix is formed according to the target eigenvector set.

[0069] The feature dimension reduction on the target image block is performed by the transformation matrix to obtain reduced dimension data.

[0070] Optionally, the pathological image feature is obtained by unsupervised autonomous learning through a clustering algorithm according to the reduced dimension data, and the clustering algorithm comprises:

[0071] According to a preset hyperparameter, all data are clustered into c classes through a K-means clustering algorithm; the c is the preset hyperparameter.

[0072] A center point representation of each class is used as a representative of the class.

[0073] The pathological image feature is determined through the center point representation.

[0074] Optionally, an expression of an optimization function minimized by the K-means clustering algorithm is as follows:

[0075]

[0076] wherein, J e represents the optimization function, x represents a sample after feature dimension reduction, D i represents a sample set of each cluster, m i represents a center point of an i-th cluster.

[0077] In a second aspect, an embodiment of the present application provides a nasopharyngeal carcinoma prognosis feature determination system, comprising:

[0078] A first module is configured to perform pathological image preprocessing, and perform staining standardization processing on a pathological image by using a color normalization method based on staining separation.

[0079] The second module is configured to automatically segment the lesion region of the preprocessed pathological image through a segmentation network to obtain a segmented image; wherein the segmentation network comprises a deep neural network and a neural conditional random field, the deep neural network is configured to segment the lesion region, and the neural conditional random field is configured to model the spatial correlation between adjacent image blocks.

[0080] The third module is configured to perform cropping processing on the segmented image to obtain a target image block.

[0081] The fourth module is configured to perform feature dimension reduction on the target image block through principal component analysis to obtain reduced dimension data.

[0082] The fifth module is configured to perform unsupervised autonomous learning on the reduced dimension data through a clustering algorithm to obtain pathological image features.

[0083] The sixth module is configured to filter the pathological image features through feature inspection to determine a prognosis pathological feature set.

[0084] The content of the method embodiments of the present application is applicable to the system embodiments, the system embodiments specifically realize the same functions as the above-mentioned method embodiments, and achieve the same beneficial effects as the above-mentioned method.

[0085] Another aspect of the embodiment of the present application further provides a nasopharyngeal carcinoma prognosis feature determination device, comprising a processor and a memory.

[0086] The memory is configured to store a program.

[0087] The processor executes the program to realize the method as described above.

[0088] The content of the method embodiments of the present application is applicable to the device embodiments, the device embodiments specifically realize the same functions as the above-mentioned method embodiments, and achieve the same beneficial effects as the above-mentioned method.

[0089] Another aspect of the embodiment of the present application further provides a computer readable storage medium, the storage medium stores a program, and the program is executed by a processor to realize the method as described above.

[0090] The content of the method embodiments of the present application is applicable to the computer readable storage medium embodiments, the computer readable storage medium embodiments specifically realize the same functions as the above-mentioned method embodiments, and achieve the same beneficial effects as the above-mentioned method.

[0091] The embodiment of the application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the foregoing method.

[0092] The implementation principle of the nasopharyngeal carcinoma prognosis feature determination of the application is described in detail as follows:

[0093] First of all, it should be pointed out that pathological omics is one of the cores of the application, a large amount of information related to the disease course and prognosis is contained in the hematoxylin-eosin (HE) staining pathological section image of nasopharyngeal carcinoma, and the clinician usually ignores the omics features closely related to the progression of nasopharyngeal carcinoma contained therein, but a large amount of complementary information is often contained between different modal data, how to extract and screen out the pathological omics features with high stability, small redundancy and closely related to the disease progression is a key scientific problem of the application. Based on the automatically segmented pathological image, the application simultaneously extracts the manually predefined features capable of reflecting the low-level semantic information of nasopharyngeal carcinoma and the deep learning features based on convolutional neural networks (CNN), and adopts a plurality of feature selection algorithms to screen out the pathological omics features with high robustness, high repeatability and closely related to prognosis. How to obtain this part of key omics features is very crucial for establishing a prognosis prediction model of nasopharyngeal carcinoma. The application uses a deep learning method, adopts a mass image feature extraction method, a correlation feature self-learning method and a feature screening method based on the automatically segmented nasopharyngeal carcinoma lesion to realize the acquisition and screening of the key features of the HE pathology and the image closely related to the prognosis of nasopharyngeal carcinoma.

[0094] The application researches and develops a lesion automatic recognition and segmentation method based on a pathological image, and a feature extraction and screening method based on intelligent pathological omics to extract and screen the pathological omics labels related to prognosis. This part of content includes four parts of standardization processing, automatic segmentation, feature extraction and feature screening of the HE section pathological image of nasopharyngeal carcinoma. Specifically, the four parts are as follows:

[0095] (1) Image standardization processing

[0096] The HE staining process and the preservation environment during the pathological section processing also affect the staining of the pathological image, thereby affecting the extraction of tumor features. Therefore, it is necessary to design an algorithm for the specific pathological scene to standardize the lesion image, and eliminate the influence of different HE staining parameters on the image features.

[0097] (2) Automatic segmentation of pathological images

[0098] For the high-resolution pathological large section image of nasopharyngeal carcinoma, the present application uses a deep neural network combined with a neural conditional random field (Neural Conditional Random Field) to simultaneously perform accurate segmentation on multiple adjacent regions in the pathological large section image. Due to the very large size of the pathological large section image, most methods divide a pathological large section image into many small image blocks and perform classification on each image block individually. However, image blocks often share spatial correlations, and ignoring these spatial correlations can result in a loss of classification accuracy. Therefore, while using CNN for lesion segmentation, the spatial correlations between adjacent image blocks are modeled using a neural conditional random field, thereby effectively improving the accuracy of lesion segmentation. In order to support high-precision segmentation, the pathological pictures collected by us need to be annotated by more than two pathologists with more than 10 years of experience, and more than five experienced doctors are organized to check and fine-tune the results of computer automatic segmentation. Using the trained segmentation network, when processing test data, the intelligent system will no longer need manual delineation by imaging experts, but will obtain accurate segmentation results in a fully automatic manner to support subsequent feature extraction of pathological images.

[0099] (3) Feature extraction of pathological images

[0100] After color normalization of the pathological section, the image blocks identified by the segmentation network as tumor tissue and interstitial tissue are flattened, and then feature dimension reduction is achieved using a principal component analysis (PCA) algorithm. Finally, the K-means clustering algorithm is used to cluster all data into c classes, and each pathological section large image is represented by the center point of each corresponding class cluster. PCA is a linear transformation feature reduction method that aims to find the projection that best represents the original data in the sense of minimum mean square error. The resulting space is an optimal reconstruction subspace. The PCA algorithm uses the eigenvectors corresponding to the first k largest eigenvalues of the data covariance matrix to form a transformation matrix, which is then used to reduce the dimension of each sample. The K-means clustering algorithm is an unsupervised self-learning method that divides data objects with high similarity into the same cluster and data objects with high dissimilarity into different clusters based on the similarity principle. It can effectively and automatically learn the distinguishing features of the data. In addition, the present application also uses a deep reinforcement learning strategy to mimic the behavior logic of pathologists during real diagnosis to extract self-supervised learning features. First, a CNN for extracting pathological image features is trained. After inputting the segmented image, the model randomly selects image blocks to train highly sensitive image features related to prognosis. Subsequently, according to the principle of attention mechanism, reinforcement learning is used to search for the most representative image blocks for nasopharyngeal carcinoma pathological evaluation, and the corresponding features are calculated.

[0101] (4) Key image and pathological image feature screening

[0102] After obtaining a large number of pathological image features, the application screens out key image features with robustness and significance through a feature inspection step. First, a stable feature set is screened out using cross-validation, and then a key pathological feature set with strong correlation and small redundancy is screened out using machine learning combined with statistical methods such as correlation inspection, cluster analysis, and autoencoder.

[0103] The implementation principle of the application will be further described in detail in combination with specific embodiments and drawings. It should be understood that the description is an explanation of the application rather than a limitation:

[0104] (I) Pathological image preprocessing:

[0105] An HE-stained pathological slide of a nasopharyngeal biopsy before treatment of a patient is called, and a high-throughput whole-slide digital scanner is used to scan it, with a magnification of 40x objective lens. The pathological whole-slide image is obtained. Since the solubility of the dye, the staining time, and the scanning machine can all affect the staining of the pathological whole slide, the application proposes to use a color normalization method based on staining separation to standardize the pathological image, and to improve the consistency of the color representation of each pathological section. Specifically, the image channels corresponding to each dye are obtained by matrix decomposition, and then mean-variance normalization processing is performed, and finally the pathological image of the RGB channel is merged.

[0106] (II) Automatic segmentation of pathological images

[0107] The application attempts to segment the tumor lesion area of pathological images with fixed magnifications of 40x, 20x, and 10x, and selects the corresponding pathological image with the best effect as the research object. The application uses a U-Net full convolutional neural network algorithm combined with a neural conditional random field to complete the computer automatic segmentation of the tumor lesion. The model structure is shown in Figure 2 It expands the FCN with a U-shaped structure, which allows the combination of features at a shallower layer with features at a deeper layer, and applies a neural conditional random field before the network output to fuse the information between neighbors to improve the stability of the connectivity and accuracy of the segmentation. U-Net mainly includes two parts of encoding and decoding, each part contains several convolutional layers and pooling layers.

[0108] This invention proposes a multi-scale U-Net architecture that pre-uses three different tile sizes (400*400, 200*200, and 100*100), accurately providing contextual information at multiple scales. To prevent these patches from exceeding the boundaries of a given image, a boundary reflection method is used to fill the tiles. Before segmentation, the image is pre-divided into 100×100 sub-tiles, and three tiles of different sizes are extracted around these sub-tiles. Features from the different sized tiles are then concatenated and used as input to the multi-scale U-Net model. The commonly used fully connected layers are replaced with 4×1×1 convolutional layers, the output of which is a segmented image of lesions in a pathological image.

[0109] (III) Extraction of pathomic features

[0110] After obtaining nasopharyngeal carcinoma case data, this invention aims to extract and screen pathomic features closely related to prognosis. (Refer to...) Figure 3 For HE pathological slide images, since a single large pathological slide typically contains approximately one billion pixels, directly using it as a sample for feature extraction is difficult. Furthermore, a single image of a tissue region containing the main lesion cells is insufficient to characterize heterogeneous tumors. Therefore, the large image is cropped into a series of smaller images of fixed size. Then, the PCA algorithm is used to reduce the dimensionality of these smaller images. Finally, K-means clustering is used to cluster the data into c classes, with the center point of each cluster representing that class. Thus, each pathological slide image is represented by the center point of these c classes.

[0111] Pathological image features are obtained by cropping large images of pathological slides, followed by feature reduction, and finally unsupervised self-learning using the K-means clustering algorithm. Specifically, after obtaining standardized large images of pathological slides, color standardization is performed to improve the consistency of color representation among different slides. Then, each slide is cropped into a series of smaller images, which are flattened and compressed into low-dimensional one-dimensional vectors using the PCA algorithm. PCA is a linear transformation feature reduction method that aims to find the projection method that best represents the original data in the sense of minimum mean square error. It uses the eigenvectors corresponding to the k largest eigenvalues ​​of the data covariance matrix to form the transformation matrix W, resulting in a dimensionality-reduced sample for each sample. After feature reduction, the K-means clustering algorithm is used to cluster all data into c classes, using the centroid feature of each class as the representative of that class. Therefore, each pathological slide has a c*d dimensional vector representation. K-means clustering is an unsupervised self-learning method that, based on the principle of similarity, groups data objects with high similarity into the same cluster and data objects with high dissimilarity into different clusters. Its optimization function is minimized as follows:

[0112]

[0113] where J e represents an optimization function, x represents a sample after feature dimension reduction, D i represents a sample set of each cluster, m i represents a center point of the i-th cluster.

[0114] In addition, the application adopts a deep reinforcement learning strategy to imitate the diagnosis behavior mode of radiologists on pathological images, and then extracts self-supervised learning features. Figure 4 The main idea of the model includes two steps. Firstly, a CNN network for extracting pathological image features is trained. In this step, the doctor first outlines the important part of the lesion, and then randomly selects a slice from it and trains a CNN network for extracting deep learning pathological image features. Then, a deep reinforcement learning network for model exploration of key slices is trained. The attention mechanism is used to make the pathological slice slide to the four directions, and the model associates the browsing path of the slice to optimize the search strategy. Finally, the most key slice for nasopharyngeal carcinoma pathological evaluation of the patient is obtained, and the deep learning features of the slice are extracted.

[0115] In the step of constructing the model, the application pre-adopts a deep Q network (DQN) as the basic structure, which has been applied to reproduce the performance of humans in games, sports and other fields. The most famous one is Alpha Go, which first defeated a human professional Go player. We redesigned the behavior strategy and reward mechanism for nasopharyngeal carcinoma prognosis prediction problem. The DQN behavior network (Current Network) and the target network (Target Network) are composed. The input of the network is a one-dimensional vector, mainly representing the position and state of the existing slice. The output is also a one-dimensional vector, representing different behavior sequences. Two hidden layers are composed of a number of neurons. The behavior network is used to collect experiences into the pool during the training process, and its parameters are updated using Q_loss. The formula is as follows:

[0116] Q_loss=E[(Target Q -Current Q ) 2 ]

[0117]

[0118] where E[·] represents an expectation function, Current Q represents the output value of the behavior deep Q network, r represents the reward, γ represents the proportion factor, Q(·) represents the deep Q network, s' represents the state, a' represents the action, and θ represents the parameters of the deep Q network.

[0119] The experience collected by the behavior network contains reward (r), current state (s) and action (a). The reward (r) is defined by its predicted probability Ps,a and the consistency between the predicted label and the true label, the current state (s) represents the prognosis evaluation value output by the model, and the action (a) is a slice behavior action, such as moving left. The basic reward (y s,a ) and the reward rate (a) of the predicted probability (P s,a ) are set to 1 and 0.5. When the experience pool overflows, the return of a single operation in the experience pool is recorded randomly, and the neuron parameters of the current network are assigned to the target network when the cumulative number reaches 100. The training sample includes a patient who provides a pathological section for encoding CNN features, and during the training process, it is ensured that the behavior route is in the lesion area. At the same time, the project realizes data expansion by randomly selecting the starting section, thereby changing the training environment to improve the robustness of the model. In the section search stage, only the target network is used as a decision module to determine the probability of each behavior, and the behavior corresponding to the highest probability will be operated by the network:

[0120] r=y s,a (b+αy s,a P s,a )

[0121] Wherein, b represents a bias value, a represents a reward rate, y s,a represents a basic reward, and P s,a represents a predicted probability.

[0122] To sum up, the present application solves the scientific problem of urgently screening high local area recurrence and metastasis risk of nasopharyngeal carcinoma population in clinical, and can construct a nasopharyngeal carcinoma prognosis prediction model based on pathology by using artificial intelligence technology combined with pathology data, divide different metastasis risk stratification population, realize accurate prediction of nasopharyngeal carcinoma metastasis, guide individualized diagnosis and treatment decision of nasopharyngeal carcinoma patients, and assist in improving the survival period of nasopharyngeal carcinoma patients. The present application innovatively uses artificial intelligence methods throughout the research process, including using FCN for automatic segmentation of lesions, using deep network DenseNet to extract image features, etc. Through the use of various new artificial intelligence methods, it is expected to improve the accuracy of the efficacy prediction model and provide an efficient and intelligent treatment tool for clinical treatment.

[0123] In some alternative embodiments, the function / operations described in the block diagrams can not occur in the order described in the operational illustrations. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality / operations involved. Also, although the embodiments presented in the flow diagrams are shown as a sequence of operations, it is to be understood that the logical flow is merely illustrative of alternative embodiments. The disclosed methods are not limited to the order of operations presented herein. Alternative embodiments can be contemplated where the order of operations is changed, and where sub operations described as part of a larger operation are executed in a different order, or are executed concurrently.

[0124] Further, while the present application has been described in the context of functional modules, it is to be understood that one or more of the functions and / or features described can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the properties, functions and internal relationships of the various functional modules disclosed herein are considered to be of a manner that would be within the purview of an engineer of ordinary skill given the benefit of this disclosure. Accordingly, the present application is not limited to the embodiments illustrated in the figures. Rather, one of ordinary skill in the art having the benefit of this disclosure would be able to implement the application in its broader aspects after attaining an understanding of the pertinent details of which the foregoing summary is intended to provide. It is also to be understood that the particular concepts disclosed are illustrative in nature only and are not intended to limit the scope of the present application as it is defined by the appended claims and equivalents thereof.

[0125] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0126] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be embodied in non-transitory computer-readable media, executed by one or more computing devices, and / or in any other way. The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer. In this context, a "computer-readable medium" can be any means that can store the program for use by or in connection with the instruction execution system, apparatus, or device.

[0127] The foregoing description of various embodiments of the application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed, and various modifications and variations are possible in light of the above teachings or can be acquired from practice of the application. For example, while a particular feature of the application can have been described with respect to only one or more embodiments thereof, the feature is not necessarily limited to that one or more embodiments. Rather, applicants have provided various embodiments of the application and combinations thereof and candidates can combine them in various combinations to produce yet other embodiments of the application. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims.

[0128] It is understood that various portions of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiment, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0129] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner.

[0130] While the embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary and are not to be construed as limiting the scope of the application. The scope of the application is defined by the appended claims and their equivalents.

[0131] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the described embodiment, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for determining a nasopharyngeal carcinoma prognostic signature, characterized in that, The method comprises the following steps: Pathological image preprocessing, the pathological image is subjected to staining standardization treatment through a color normalization method based on staining separation; Automatic segmentation of the lesion area of the preprocessed pathological image is performed through a segmentation network to obtain a segmented image; wherein the segmentation network comprises a deep neural network and a neural conditional random field, the deep neural network is used for segmentation of the lesion area, and the neural conditional random field is used for modeling spatial correlation between adjacent image blocks; The segmented image is subjected to cropping processing to obtain a target image block; Feature dimension reduction is performed on the target image block through principal component analysis to obtain reduced dimension data; Pathological image features are obtained through unsupervised autonomous learning of the reduced dimension data through a clustering algorithm; The pathological image features are screened through feature inspection to determine a prognosis pathological feature set; In the step of constructing the model, a deep Q network is used as a basic structure, the deep Q network comprises a behavior network and a target network, the input of the network is a one-dimensional vector, mainly representing the position and state of the existing slice, and the output is also a one-dimensional vector, representing different behavior sequences, the two hidden layers are respectively composed of a plurality of neurons, the behavior network is used for collecting experience into a pool in the training process, and the parameters thereof are updated using Q_loss, and the formula is as follows: wherein, represents a desired function, represents a behavioral deep Q-network output value, represents a reward, represents a scaling factor, represents a deep Q-network, represents a state, represents an action, represents a parameter of a deep Q-network; The experience collected by the behavioral network includes reward r, current state s, and action a; reward r is defined by its predicted probability Ps,a and the consistency between the predicted label and the true label, current state s represents the prognostic evaluation value output by the model, and action a is the sliced ​​behavior action; the basic reward... and predicted probability reward rate The values ​​are set to 1 and 0.5; when the experience pool overflows, the gains of individual operations in the experience pool are randomly recorded, and when the accumulated number reaches 100, the neuron parameters of the current network are assigned to the target network; training samples include patients who provide pathological slices for encoding CNN features, and during training, it is ensured that the behavioral paths are all within the lesion area; data augmentation is achieved by randomly selecting the starting slice; during the slice search phase, only the target network is used as a decision module to determine the probability of each behavior, and the behavior with the highest probability will be operated on by the network: wherein, wherein, denotes a bias value, denotes a reward rate, denotes a base reward, denotes a prediction probability.

2. The method of prognosis determination of nasopharyngeal carcinoma according to claim 1, wherein, Further comprising: Prognosis prediction is performed through a deep Q network according to the pathological image features to determine key pathological image features of the patient; The deep Q network comprises a behavior network and a target network; the behavior network is used for collecting training experience and updating network parameters through the training experience; and the target network is used for decision operation according to behavior probability.

3. The method of prognosis determination of nasopharyngeal carcinoma according to claim 1, wherein, The staining standardization treatment of the pathological image through the color normalization method based on staining separation comprises the following steps: Image channels of each dye are obtained through matrix decomposition; Mean-variance normalization processing is performed on the image channels to obtain a pathological image of an RGB channel.

4. The method of prognosis determination of nasopharyngeal carcinoma according to claim 1, wherein, The cropping processing of the segmented image to obtain the target image block comprises the following steps: The segmented image is subjected to cropping processing according to a preset specification to obtain image blocks of tumor tissues and interstitial tissues.

5. The method of prognosis determination of nasopharyngeal carcinoma according to claim 1, wherein, The feature dimension reduction of the target image block through principal component analysis to obtain reduced dimension data comprises the following steps: A target eigenvector set of a data covariance matrix is obtained; A transformation matrix is formed according to the target eigenvector set; Feature dimension reduction of the target image block is performed through the transformation matrix to obtain reduced dimension data.

6. The method of prognosis determination for nasopharyngeal carcinoma according to claim 1, wherein, The pathological image features obtained through unsupervised autonomous learning of the reduced dimension data through a clustering algorithm comprise the following steps: All data are clustered into c classes through a K-means clustering algorithm according to a preset hyperparameter; the c is a preset hyperparameter; The center point representation of each cluster is used as a representative of the class; The center point representation is used to determine the pathological image features.

7. The method of claim 6, wherein the method comprises determining the prognosis of nasopharyngeal carcinoma by detecting the expression level of the biomarker in the sample. The expression of the optimization function minimized by the K-means clustering algorithm is as follows: wherein, represents an optimization function, represents the samples after feature dimension reduction, represents the sample set of each cluster, represents the center point of the class cluster.

8. A nasopharyngeal carcinoma prognosis signature determining system, characterized in that, The method comprises the following steps: A first module is used for pathological image preprocessing, and the pathological image is subjected to staining standardization treatment through a color normalization method based on staining separation. The second module is configured to automatically segment the lesion region of the preprocessed pathological image by using a segmentation network to obtain a segmented image; the segmentation network comprises a deep neural network and a neural conditional random field; the deep neural network is used for segmentation of the lesion region; and the neural conditional random field is used for modeling spatial correlation between adjacent image blocks. The third module is configured to perform cropping processing on the segmented image to obtain a target image block. The fourth module is configured to perform feature dimension reduction on the target image block by using principal component analysis to obtain reduced dimension data. The fifth module is configured to perform unsupervised autonomous learning on the reduced dimension data by using a clustering algorithm to obtain pathological image features. The sixth module is configured to perform feature inspection on the pathological image features to determine a prognosis pathological feature set. In the step of constructing the model, a deep Q network is used as a basic structure; the deep Q network comprises a behavior network and a target network; the input of the network is a one-dimensional vector, which mainly represents the position and state of an existing slice; the output of the network is also a one-dimensional vector, which represents different behavior sequences; two hidden layers are respectively composed of a plurality of neurons; the behavior network is used to collect experience into a pool during the training process, and update parameters thereof by using Q_loss; and a formula is as follows: wherein, represents a desired function, represents a behavioral deep Q-network output value, represents a reward, represents a scaling factor, represents a deep Q-network, represents a state, represents an action, represents a parameter of a deep Q-network; The experience collected by the behavioral network includes reward r, current state s, and action a; reward r is defined by its predicted probability Ps,a and the consistency between the predicted label and the true label, current state s represents the prognostic evaluation value output by the model, and action a is the sliced ​​behavior action; the basic reward... and predicted probability reward rate The values ​​are set to 1 and 0.5; when the experience pool overflows, the gains of individual operations in the experience pool are randomly recorded, and when the accumulated number reaches 100, the neuron parameters of the current network are assigned to the target network; training samples include patients who provide pathological slices for encoding CNN features, and during training, it is ensured that the behavioral paths are all within the lesion area; data augmentation is achieved by randomly selecting the starting slice; during the slice search phase, only the target network is used as a decision module to determine the probability of each behavior, and the behavior with the highest probability will be operated on by the network: wherein, wherein, denotes a bias value, denotes a reward rate, denotes a base reward, denotes a prediction probability.

9. A nasopharyngeal carcinoma prognosis feature determination apparatus characterized by comprising: The processor and the memory are included. The memory is configured to store a program. The processor executes the program to implement the method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the method in any one of claims 1 to 7.

Images