Image-based patient-level molecular marker prediction method, system and device

Abstract

The present invention belongs to the field of image analysis, and specifically relates to an image-based patient-level molecular marker prediction method, system and device, which aims to solve the problem that the existing digital pathology image processing methods ignore the additional information provided by key molecular markers related to patient prognosis, resulting in insufficient analysis accuracy. It includes: extracting the image of the region of interest and dividing it into color-standardized rectangular blocks; obtaining deep feature information of the blocks through a deep residual convolutional neural network; merging the deep feature information of the blocks from the same patient; extracting the top features of each molecular marker level through a correlation analysis method, and constructing a marker level prediction model based on machine learning; predicting the patient's marker level through the marker level prediction model based on machine learning to assist in decision-making. The present invention improves the accuracy of automated diagnosis by using the predicted molecular marker level as a prognostic decision factor.

Description

Background Art

[0002] Histopathological sections contain important staging, diagnostic and prognostic information and have been widely used in health analysis. Using artificial intelligence (AI)-based computational pathology to mine high-dimensional information hidden in digital pathology images with hundreds of millions of pixels and then calculate certain properties of digital pathology images is an important application of computer-aided diagnosis. Among them, artificially predefined cell-level feature extraction relies on manual pixel-level annotation or precise segmentation of cells. This method is greatly affected by different sample collection methods and calibration personnel, so its performance is limited and cannot fully reflect disease information. In addition, current digital pathology image processing and modeling methods usually focus on the morphological information of cells to encode high-dimensional features to achieve automated diagnosis, while ignoring the additional information provided by key molecular markers. Different mutation states of key molecular markers will lead to different survival outcomes for patients. In routine clinical practice, diagnosis based on the mutation state of key molecular markers is difficult to be widely used due to the high cost of measurement. However, it is worth noting that these mutations can be expressed at the proteomic level and ultimately hidden in digital pathology images.

[0003] Therefore, we need a more stable and convenient technical means to extract patient-level digital pathology image features and consider the supplementary information brought by molecular markers to digital pathology image analysis when building the model to obtain an accurate auxiliary decision-making model. Summary of the invention

[0004] In order to solve the above-mentioned problems in the prior art, namely, the existing digital pathology image processing and modeling methods usually only focus on the morphological information of cells to encode high-dimensional features to achieve automated diagnosis, ignoring the additional information provided by key molecular markers related to patient prognosis, resulting in insufficient analysis accuracy. The present invention proposes an image-based patient-level molecular marker prediction method, comprising:

[0005] Step S100, acquiring an image to be tested having molecular markers, and then extracting an image of a region of interest;

[0006] Step S200, based on the region of interest image, dividing into color standardized rectangular blocks according to the mode of the acquisition instrument;

[0007] Step S300, based on the color-standardized rectangular blocks, obtaining deep feature information of the blocks one by one through a deep residual convolutional neural network;

[0008] Merge deep feature information of cuts from the same patient to obtain patient-level digital pathology image features;

[0009] Step S400, based on the mutation degree and mutation degree ranking of the molecular marker, the molecular marker is divided into high / low levels by a segmented regression method;

[0010] Step S500, based on the molecular marker high and low levels and patient-level digital pathology image features, extracting the top features of each molecular marker level by a correlation analysis method;

[0011] Step S600, constructing a marker level prediction model based on machine learning based on the top features;

[0012] Step S700, based on the patient-level digital pathology image features, predict the patient's marker level through the marker level prediction model based on machine learning.

[0013] In some preferred embodiments, the deep residual convolutional neural network includes 1 convolutional module, 4 residual modules and 1 fully connected layer connected in sequence;

[0014] The convolution module includes a convolution layer, a batch normalization layer, a nonlinear activation layer and a maximum pooling layer connected in sequence;

[0015] The four residual modules are specifically: the first residual module includes one mapping module connected in sequence and two repeated identity modules, wherein the mapping module includes three convolutional layers, two batch normalization layers, two non-linear activation layers and one residual connection that changes the size of the feature map, and the identity module includes three convolutional layers, two batch normalization layers, two non-linear activation layers and one residual connection that does not change the size of the feature map; the second residual module includes one mapping module connected in sequence and three repeated identity modules; the third residual module includes one mapping module connected in sequence and five repeated identity modules; the fourth residual module includes one mapping module connected in sequence and two repeated identity modules.

[0016] In some preferred implementations, the step S100 specifically includes:

[0017] Step S110, dyeing the target to be tested with a dye and taking a test image;

[0018] Step S120, based on the image to be tested, preliminarily delineating the boundary of the target area at a preset n-magnification viewing distance, and obtaining the image to be tested with the boundary of the target area delineated;

[0019] Step S130, based on the image to be tested that defines the boundary of the target area, the boundary of the target area is reviewed and finely adjusted at a preset m times viewing distance to obtain an image of the region of interest; wherein m is greater than n, and m and n are both constants.

[0020] In some preferred implementations, the step S200 specifically includes:

[0021] Based on the image of the region of interest, the image is divided into non-overlapping rectangular blocks at a preset q-magnification viewing distance, and rectangular blocks whose blank area accounts for a larger proportion than a preset p value are excluded;

[0022] Selecting a rectangular block containing target cells and target cell stroma information from the remaining rectangular blocks as a target block;

[0023] All remaining cuts are decomposed into a color appearance matrix and a sparse non-negative staining density map. The scaled staining density maps of the remaining cuts that are not selected as target cuts are multiplied with the color appearance matrix of the target cuts to generate a normalized source image for structure-preserving color normalization.

[0024] In some preferred implementations, the step S300 specifically includes:

[0025] Step S310, based on the color-standardized rectangular blocks, extracting a feature map of each color-standardized rectangular block one by one through one convolution module and three residual modules in a deep residual convolutional neural network with pre-trained fixed parameters;

[0026] Step S320, performing global average pooling on the feature map of the color-standardized rectangular block to obtain deep feature information of the block;

[0027] Step S330 , merging the deep feature information of the slices from the same patient by average pooling to obtain patient-level digital pathology image features.

[0028] In some preferred implementations, the step S500 specifically includes:

[0029] Through the Pearson correlation analysis method, a preset number of top features with the strongest correlation are extracted from the high-dimensional patient-level digital pathology image features.

[0030] In some preferred implementations, the step S600 specifically includes:

[0031] Taking the high / low level of molecular markers as the target value and the top features as the independent variables, the cross entropy loss function was minimized by the logistic regression method and L2 regularization, and the classification model was fitted to obtain the marker level prediction model based on machine learning.

[0032] In another aspect of the present invention, an image-based patient-level molecular marker prediction system is proposed, comprising a region of interest extraction unit, a standardized slicing unit, a feature extraction unit, a marker grading unit, a top feature acquisition unit, a model building unit, and a grading prediction unit;

[0033] The region of interest extraction unit is configured to obtain the image to be tested having the molecular markers, and then extract the region of interest image;

[0034] The standardized block cutting unit is configured to divide the image of the region of interest into color standardized rectangular blocks according to the mode of the acquisition instrument;

[0035] The feature extraction unit is configured to obtain deep feature information of the blocks one by one through a deep residual convolutional neural network based on the color standardized rectangular blocks;

[0036] Merge deep feature information of cuts from the same patient to obtain patient-level digital pathology image features;

[0037] The marker grade classification unit is configured to classify the molecular marker into high / low grades by a segmented regression method based on the molecular marker mutation degree and mutation degree ranking;

[0038] The top feature acquisition unit is configured to extract the top features of each molecular marker level through a correlation analysis method based on the molecular marker high and low levels and patient-level digital pathology image features;

[0039] The model building unit is configured to build a marker level prediction model based on machine learning based on the top features;

[0040] The grade prediction unit is configured to predict the patient's marker grade based on the patient-level digital pathology image features through the machine learning-based marker grade prediction model.

[0041] The third aspect of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned image-based patient-level molecular marker prediction method.

[0042] In a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned image-based patient-level molecular marker prediction method.

[0043] Beneficial effects of the present invention:

[0044] (1) The present invention uses the predicted molecular marker levels as prognostic decision factors to classify different patients into high and low risk prognoses, and provides patient grouping decisions based on the length of overall survival, thereby improving the accuracy of automated diagnosis;

[0045] (2) The present invention obtains the mutation degree of molecular markers through whole exome sequencing and panel sequencing technology, and then uses pathological characteristics to build a model to predict the gold standard, thereby achieving the gold standard of key molecular marker measurement using only tissue pathological sections and digital pathological images, thereby improving diagnostic accuracy while also improving measurement convenience and reducing measurement costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0047] Figure 1 is a schematic flow chart of the image-based patient-level molecular marker prediction method of the present invention;

[0048] Figure 2 It is a schematic diagram of the product process of the image-based patient-level molecular marker prediction method of the present invention;

[0049] Figure 3 Schematic diagram of a convolutional neural network residual module of the image-based patient-level molecular marker prediction method of the present invention;

[0050] Figure 4 is a schematic diagram of a classification ROC curve of the image-based patient-level molecular marker prediction method of the present invention;

[0051] Figure 5 It is a schematic diagram of the KM curve of the prognostic decision of the image-based patient-level molecular marker prediction method of the present invention on a real experimental data set; DETAILED DESCRIPTION

[0052] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0053] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] The present invention provides an image-based patient-level molecular marker prediction method, the method comprising:

[0055] Step S100, acquiring an image to be tested having molecular markers, and then extracting an image of a region of interest;

[0056] Step S200, based on the region of interest image, dividing into color standardized rectangular blocks according to the mode of the acquisition instrument;

[0057] Step S300, based on the color-standardized rectangular blocks, obtaining deep feature information of the blocks one by one through a deep residual convolutional neural network;

[0058] Merge deep feature information of cuts from the same patient to obtain patient-level digital pathology image features;

[0059] Step S400, based on the mutation degree and mutation degree ranking of the molecular marker, the molecular marker is divided into high / low levels by a segmented regression method;

[0060] Step S500, based on the molecular marker high and low levels and patient-level digital pathology image features, extracting the top features of each molecular marker level by a correlation analysis method;

[0061] Step S600, constructing a marker level prediction model based on machine learning based on the top features;

[0062] Step S700, based on the patient-level digital pathology image features, predict the patient's marker level through the marker level prediction model based on machine learning.

[0063] In order to more clearly illustrate the system of the present invention, Figure 1 and Figure 2 Each step in the embodiment of the present invention is described in detail.

[0064] The steps of the image-based patient-level molecular marker prediction method of the first embodiment of the present invention are described in detail as follows:

[0065] Step S100, acquiring an image to be tested having molecular markers, and then extracting an image of a region of interest;

[0066] In this embodiment, the step S100 specifically includes:

[0067] Step S110, staining the target to be tested with a staining agent, and photographing the image to be tested; in this embodiment, a full-field digital pathology image stained with hematoxylin-eosin is used as the image to be tested.

[0068] Step S120, based on the image to be tested, preliminarily delineate the boundary of the target area at a preset n-magnification viewing distance, and obtain the image to be tested with the boundary of the target area delineated;

[0069] Step S130, based on the image to be tested with the boundary of the target area delineated, the boundary of the target area is reviewed and finely adjusted at a preset m-fold viewing distance to obtain an image of the region of interest; wherein m is greater than n, and both m and n are constants. In this embodiment, n is selected as 5 and m is selected as 10. By performing boundary conformity at a high viewing distance, the interference of information irrelevant to the lesion in the digital pathology image can be minimized. After completing this step, a txt file with complete location information is generated for each pathology image.

[0070] In this embodiment, a tumor is used as the target for illustration, and the region of interest can be determined by automatic, semi-automatic or manual delineation. The automatic method includes using the Otsu threshold segmentation method to determine that the region of interest includes tumor cell clusters and tumor stroma; based on the marked tumor region boundary, the internal region of interest is divided into a set of non-overlapping rectangular blocks that meet the calculation conditions (1024*1024, 512*512 or 256*256), and color standardization is performed on the blocks of different mode images collected by different instruments.

[0071] Step S200, based on the region of interest image, dividing into color standardized rectangular blocks according to the mode of the acquisition instrument;

[0072] In this embodiment, the step S200 specifically includes:

[0073] Based on the image of the area of ​​interest, it is divided into non-overlapping rectangular blocks at a preset q-magnification viewing distance, and the rectangular blocks with a blank area ratio higher than a preset p value are excluded, q>m; in this embodiment, q is selected as 20, the block size is preset to 256*256, and p is 1 / 3.

[0074] A rectangular block containing target cells and target interstitial cell information from the remaining rectangular blocks is selected as a target block; in this embodiment, a block completely containing cancer cells, lymphocytes, and interstitial cell information is selected as a target block;

[0075] All remaining cuts are decomposed into a color appearance matrix and a sparse non-negative staining density map. The scaled staining density maps of the remaining cuts that are not selected as target cuts are multiplied with the color appearance matrix of the target cuts to generate a normalized source image for structure-preserving color normalization.

[0076] Step S300, based on the color-standardized rectangular blocks, deep feature information of the blocks is obtained one by one through a deep residual convolutional neural network; the fixed-size image is analyzed piece by piece through a deep residual convolutional neural network to generate deep feature information that can be used to characterize the original block.

[0077] The deep feature information of the blocks from the same patient is merged to obtain the patient-level digital pathology image features; after analyzing each block, a low-dimensional feature representation is obtained, and after merging all the feature information of each patient through average pooling, a high-dimensional feature representation is obtained.

[0078] In this embodiment, the deep residual convolutional neural network includes 1 convolution module, 4 residual modules and 1 fully connected layer;

[0079] The convolution module includes a convolution layer, a batch normalization layer, a nonlinear activation layer and a maximum pooling layer connected in sequence;

[0080] The four residual modules are specifically as follows: Figure 3 As shown, the first residual module includes one mapping module connected in sequence and two repeated identity modules, wherein the mapping module includes three convolutional layers, two batch normalization layers, two non-linear activation layers and one residual connection that changes the size of the feature map, and the identity module includes three convolutional layers, two batch normalization layers, two non-linear activation layers and one residual connection that does not change the size of the feature map; the second residual module includes one mapping module connected in sequence and three repeated identity modules; the third residual module includes one mapping module connected in sequence and five repeated identity modules; the fourth residual module includes one mapping module connected in sequence and two repeated identity modules.

[0081] In this embodiment, the step S300 specifically includes:

[0082] Step S310, based on the color-standardized rectangular block, extract the feature map of each color-standardized rectangular block one by one through 1 convolution module and 3 residual modules of the deep residual convolutional neural network with pre-trained fixed parameters; the feature map of [14, 14, 1024] obtained in this step, where 14*14 is the size of the feature map and 1024 is the dimension of the feature map; in this embodiment, all parameters of the deep residual convolutional neural network are pre-trained and fixed, and then a residual module is artificially truncated, so that the input image is only encoded by 1 convolution module + 3 residual modules to obtain its relatively low-dimensional features, rather than obtaining high-dimensional features after encoding through all modules, thereby avoiding redundant information to a certain extent;

[0083] Step S320, performing global average pooling on the feature map of the color-standardized rectangular block to obtain deep feature information of the cut block; encoding a total of 1024-dimensional deep feature information of the cut block;

[0084] Step S330, performing average pooling on the deep feature information of the slices from the same patient to obtain patient-level digital pathology image features; the patient-level digital pathology image features are also 1024-dimensional feature representations;

[0085] This embodiment determines the network parameters by pre-training the deep residual convolutional neural network with natural images in ImageNet;

[0086] Step S400, based on the molecular marker mutation degree and mutation degree ranking, the molecular marker is divided into high / low levels by the segmented regression method; in this embodiment, the tumor mutation load is used as the molecular marker to be predicted, and the inflection point is determined by the segmented regression method to determine the high / low tumor mutation load; the mutation and ranking are obtained based on whole exome sequencing, not based on the image. Among them, the mutation degree in the present invention is encoded by a convolutional neural network to encode the image, and further predicted by the gold standard.

[0087] Step S500, based on the molecular marker high and low levels and patient-level digital pathology image features, extracting the top features of each molecular marker level by a correlation analysis method;

[0088] In this embodiment, the step S500 specifically includes:

[0089] By using the Pearson correlation analysis method, a preset number of top features with the strongest correlation are extracted from the high-dimensional patient-level digital pathology image features. The number of top features is determined by {64, 128, 256, 512}, and the number of top features in this example is determined to be 64.

[0090] Step S600, constructing a marker level prediction model based on machine learning based on the top features;

[0091] In this embodiment, the step S600 specifically includes:

[0092] With the high / low level of molecular markers as the target value and the top features as the independent variable, the cross entropy loss function is minimized by the logistic regression method and L2 regularization is applied, and the classification model is fitted to obtain a marker level prediction model based on machine learning. In addition, the prediction model can also be constructed based on molecular markers and top features, according to the support vector machine SVM minimization Hinge Loss and random forest method using integrated ideas. Using the Youden index, the median method defines the optimal threshold for dividing the high and low output of the prediction model, and uses the characteristics of molecular biomarkers related to prognosis to divide patients into high and low risk groups of prognosis based on the high and low output values ​​of the model, and the model results are extended to prognosis prediction. In this embodiment, the regularization parameter is set to 1e-5.

[0093] Step S700, based on the patient-level digital pathology image features, the patient's marker level is predicted by the marker level prediction model based on machine learning. The classification model finally outputs a probability value for predicting high / low tumor mutation load.

[0094] According to the correlation between tumor mutation load and patient prognosis, patients are divided into different prognostic groups by predicting high / low output values ​​of the model, further realizing the auxiliary diagnosis function of prognostic stratification.

[0095] The embodiment uses 566 real data sets with both digital pathology images and tumor mutation load information for testing (302 of which have both digital pathology images, tumor mutation load information and prognosis information), of which 198 are used for training the model, 66 are used to test the classification effect of the present invention, and 302 are used to test the prognostic decision effect of the present invention.

[0096] The method of the present invention is used to obtain the receiver operating characteristic (ROC) curve and the area under the curve (AUC) of the classification method, and the ROC curve and AUC are used as the measurement of the classifier performance. At the same time, the Kaplan-Meier (Kaplan-Meier, KM) curve of the prognostic decision is obtained, and the KM curve is used as the measurement of the prognostic decision performance.

[0097] The classification ROC curve of the classification method on the real experimental data set is Figure 2 The true positive rate refers to the percentage of positive cases that are actually positive and are correctly judged as positive using this method, and the false positive rate refers to the percentage of negative cases that are actually negative and are incorrectly judged as positive using this method. Figure 4 As shown in the figure, the AUC value of this method is 0.813. Generally speaking, the closer the AUC value is to 1, the better the overall performance of the classification method is; when the AUC is greater than 0.8, it proves that the classification method has achieved good classification results. The KM curves of prognostic decision-making on the real experimental data set are Figure 5 The P value was calculated by Log-rank test, and the P value less than 0.05 indicated that there was a significant difference in the prognosis of patients in different groups. Figure 5 As shown, the P value of this method is 0.014, which is much smaller than 0.05, proving that this method has good performance in prognostic decision-making.

[0098] The experimental results show that the patient-level molecular marker prediction system based on digital pathological images described in the present invention can effectively predict the molecular marker level and make prognostic decisions.

[0099] The image-based patient-level molecular marker prediction system of the second embodiment of the present invention includes a region of interest extraction unit, a standardized block cutting unit, a feature extraction unit, a marker grade division unit, a top feature acquisition unit, a type construction unit and a grade prediction unit;

[0100] The region of interest extraction unit is configured to obtain the image to be tested having the molecular markers, and then extract the region of interest image;

[0101] The standardized block cutting unit is configured to divide the image of the region of interest into color standardized rectangular blocks according to the mode of the acquisition instrument;

[0102] The feature extraction unit is configured to obtain deep feature information of the blocks one by one through a deep residual convolutional neural network based on the color standardized rectangular blocks;

[0103] Merge deep feature information of cuts from the same patient to obtain patient-level digital pathology image features;

[0104] The marker grade classification unit is configured to classify the molecular marker into high / low grades by a segmented regression method based on the molecular marker mutation degree and mutation degree ranking;

[0105] The top feature acquisition unit is configured to extract the top features of each molecular marker level through a correlation analysis method based on the molecular marker high and low levels and patient-level digital pathology image features;

[0106] The model building unit is configured to build a marker level prediction model based on machine learning based on the top features;

[0107] The grade prediction unit is configured to predict the patient's marker grade based on the patient-level digital pathology image features through the machine learning-based marker grade prediction model.

[0108] An electronic device according to the third embodiment of the present invention comprises: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned image-based patient-level molecular marker prediction method.

[0109] A fourth embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned image-based patient-level molecular marker prediction method.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the storage device and processing device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0111] The terms "first", "second", etc. are used to distinguish similar objects rather than to describe or indicate a particular order or sequence.

[0112] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus / device.

[0113] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for predicting patient-level molecular markers based on images, characterized in that: The method comprises: Step S100, acquiring an image to be tested having molecular markers, and then extracting an image of a region of interest; Step S200, based on the region of interest image, dividing into color standardized rectangular blocks according to the mode of the acquisition instrument; Step S300, based on the color-standardized rectangular blocks, obtaining deep feature information of the blocks one by one through a deep residual convolutional neural network; Merge deep feature information of cuts from the same patient to obtain patient-level digital pathology image features; Step S400, based on the mutation degree and mutation degree ranking of the molecular marker, the molecular marker is divided into high / low levels by a segmented regression method; Step S500, based on the molecular marker high and low levels and patient-level digital pathology image features, extracting the top features of each molecular marker level by a correlation analysis method; Step S600, constructing a marker level prediction model based on machine learning based on the top features; Step S700, based on the patient-level digital pathology image features, predict the patient's marker level through the marker level prediction model based on machine learning.

2. The image-based patient-level molecular marker prediction method according to claim 1, characterized in that: The deep residual convolutional neural network includes one convolutional module, four residual modules and one fully connected layer connected in sequence; The convolution module includes a convolution layer, a batch normalization layer, a nonlinear activation layer and a maximum pooling layer connected in sequence; The four residual modules are specifically: the first residual module includes one mapping module connected in sequence and two repeated identity modules, wherein the mapping module includes three convolutional layers, two batch normalization layers, two non-linear activation layers and one residual connection that changes the size of the feature map, and the identity module includes three convolutional layers, two batch normalization layers, two non-linear activation layers and one residual connection that does not change the size of the feature map; the second residual module includes one mapping module connected in sequence and three repeated identity modules; the third residual module includes one mapping module connected in sequence and five repeated identity modules; the fourth residual module includes one mapping module connected in sequence and two repeated identity modules.

3. The method for predicting patient-level molecular markers based on images according to claim 1, characterized in that: The step S100 specifically includes: Step S110, dyeing the target to be tested with a dye and taking a test image; Step S120, based on the image to be tested, preliminarily delineate the boundary of the target area at a preset n-magnification viewing distance, and obtain the image to be tested with the boundary of the target area delineated; Step S130, based on the image to be tested that defines the boundary of the target area, the boundary of the target area is reviewed and finely adjusted at a preset m times viewing distance to obtain an image of the region of interest; wherein m is greater than n, and m and n are both constants.

4. The method for predicting patient-level molecular markers based on images according to claim 1, characterized in that: The step S200 specifically includes: Based on the image of the region of interest, the image is divided into non-overlapping rectangular blocks at a preset q-magnification viewing distance, and rectangular blocks whose blank area accounts for a larger proportion than a preset p value are excluded; Selecting a rectangular block containing target cells and target cell stroma information from the remaining rectangular blocks as a target block; All remaining cuts are decomposed into a color appearance matrix and a sparse non-negative staining density map. The scaled staining density maps of the remaining cuts that are not selected as target cuts are multiplied with the color appearance matrix of the target cuts to generate a normalized source image for structure-preserving color normalization.

5. The method for predicting patient-level molecular markers based on images according to claim 2, characterized in that: The step S300 specifically includes: Step S310, based on the color-standardized rectangular blocks, extracting a feature map of each color-standardized rectangular block one by one through one convolution module and three residual modules in a deep residual convolutional neural network with pre-trained fixed parameters; Step S320, performing global average pooling on the feature map of the color-standardized rectangular block to obtain deep feature information of the block; Step S330 , merging the deep feature information of the slices from the same patient by average pooling to obtain patient-level digital pathology image features.

6. The method for predicting patient-level molecular markers based on images according to claim 1, characterized in that: The step S500 specifically includes: Through the Pearson correlation analysis method, a preset number of top features with the strongest correlation are extracted from the high-dimensional patient-level digital pathology image features.

7. The method for predicting patient-level molecular markers based on images according to claim 1, characterized in that: The step S600 specifically includes: Taking the high / low level of molecular markers as the target value and the top features as the independent variables, the cross entropy loss function was minimized by the logistic regression method and L2 regularization, and the classification model was fitted to obtain the marker level prediction model based on machine learning.

8. An image-based patient-level molecular marker prediction system, characterized in that: It includes a region of interest extraction unit, a standardized block cutting unit, a feature extraction unit, a marker grade classification unit, a top feature acquisition unit, a model building unit and a grade prediction unit; The region of interest extraction unit is configured to obtain the image to be tested having the molecular markers, and then extract the region of interest image; The standardized block cutting unit is configured to divide the image of the region of interest into color standardized rectangular blocks according to the mode of the acquisition instrument; The feature extraction unit is configured to obtain deep feature information of the blocks one by one through a deep residual convolutional neural network based on the color standardized rectangular blocks; Merge deep feature information of cuts from the same patient to obtain patient-level digital pathology image features; The marker grade classification unit is configured to classify the molecular marker into high / low grades by a segmented regression method based on the molecular marker mutation degree and mutation degree ranking; The top feature acquisition unit is configured to extract the top features of each molecular marker level through a correlation analysis method based on the molecular marker high and low levels and patient-level digital pathology image features; The model building unit is configured to build a marker level prediction model based on machine learning based on the top features; The grade prediction unit is configured to predict the patient's marker grade based on the patient-level digital pathology image features through the machine learning-based marker grade prediction model.

9. An electronic device, comprising: at least one processor; And a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the image-based patient-level molecular marker prediction method according to any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions, wherein the computer instructions are used to be executed by the computer to implement the image-based patient-level molecular marker prediction method according to any one of claims 1 to 7.

Images