Tumor curative effect prediction device based on image and pathology interaction network

Abstract

The invention discloses a tumor curative effect prediction device based on an image and pathology interaction network in the technical field of medical instruments. The tumor curative effect prediction device comprises a postoperative image acquisition module, a template module, a feature combination module and a prediction module, the postoperative image acquisition module is used for acquiring an image of a patient after tumor excision and generating a postoperative image; the template module is used for correcting the predefined resection area template based on the postoperative image and outputting a personal resection area template of the patient; the feature combination module is used for combining multiple features to form a combined feature vector; and the prediction module is used for establishing a prediction model, receiving the joint feature vector and outputting a curative effect prediction result. According to the invention, the image of the patient after tumor excision is obtained and input into the template model established by the template module to obtain the individualized template of the patient, then the individual sliding template is imported into the prediction module, and the tumor curative effect prediction result is output, so that the prediction precision of the tumor curative effect is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices, and in particular is a tumor efficacy prediction device based on an image and pathology interactive network. Background Art

[0002] Tumors are major diseases that seriously threaten human health. Their accurate diagnosis and efficacy evaluation are the key to improving clinical treatment effects and reducing tumor-related mortality. With the in-depth development of artificial intelligence (AI) in the field of medical imaging, radiomics and pathomic have emerged and have shown great potential in the diagnosis and treatment of tumors. However, both have also shown their limitations: radiomics lacks biological verification, while pathomic lacks the description of the macroscopic properties of tumors. Therefore, combining the two across modalities and scales to complement each other has become a new idea for accurate diagnosis and treatment of tumors. Radiomics is to extract high-throughput quantitative features from image images to quantify tumor characteristics and provide a reference for accurate diagnosis and prognosis prediction of tumors; while pathomic is to apply similar high-throughput image feature extraction technology to convert full-field digital slices into quantitative data sets to objectively reflect the microscopic information of tumor cells and tissue structures. In view of the limitations of radiomics and pathomic, a tumor efficacy prediction device based on the image and pathology interactive network has emerged. The device integrates the interactive information between the modalities of imaging and pathology to construct an efficacy prediction model for tumor immunotherapy, thereby improving the precision and accuracy of efficacy prediction.

[0003] The existing tumor efficacy prediction devices based on image and pathology interaction networks generally have five steps, including data acquisition and preprocessing, feature extraction and matrix construction, image and pathology interaction network construction, multimodal feature combination and efficacy prediction, and model training and optimization. At the same time, when constructing analysis models for efficacy prediction or surgical planning, the postoperative imaging data used are mostly predefined templates. However, this method does have limitations because it cannot fully consider the individual differences in the surgical resection area of ​​each patient.

[0004] In summary, the present invention proposes a tumor efficacy prediction device based on an image and pathology interactive network, which obtains the medical imaging data of the patient before and after treatment and the pathological images of the tumor-adjacent tissue after the patient's surgery, and then uses a pre-established postoperative imaging model to obtain the patient's postoperative individual template. Finally, the template and the above-mentioned image information are passed through the tumor efficacy prediction model, thereby improving the prediction accuracy of the tumor efficacy. Summary of the invention

[0005] In order to solve the above problems, the purpose of the present invention is to provide a tumor efficacy prediction device based on an image and pathology interactive network. By acquiring the medical imaging data of the patient before and after treatment and the pathological images of the tumor-adjacent tissue after the patient's surgery, and then using a pre-established postoperative imaging model, the patient's postoperative individual template is obtained. Finally, the template and the above-mentioned image information are passed through a tumor efficacy prediction model (image and pathology interactive network), thereby improving the prediction accuracy of tumor efficacy.

[0006] In order to achieve the above-mentioned object, the technical scheme of the present invention is as follows: a tumor efficacy prediction device based on an image and pathology interactive network comprises a data acquisition module, a preprocessing module, a feature extraction module, a postoperative image acquisition module, a template module, a feature combination module and a prediction module;

[0007] A data acquisition module, used to acquire the patient's preoperative and postoperative tumor imaging data from medical imaging equipment and to acquire the patient's clinical characteristic information;

[0008] A postoperative image acquisition module is used to obtain pathological images of the patient's postoperative tumor tissue and its surrounding tissues through pathological examination, and is defined as a postoperative image;

[0009] The preprocessing module is used to standardize the image data and pathological images, segment the tumor area of ​​the image data, extract tumor-related image features, and segment and extract tumor cells from the pathological images;

[0010] The feature extraction module is used to extract quantitative features from the preprocessed image data using the radiomics method to form an image feature matrix; and to extract quantitative features from the preprocessed pathological images using the pathological tissue and immunohistochemistry method to form a pathological feature matrix;

[0011] The template module is used to establish a template module, receive postoperative images and postoperative imaging data, extract feature information therein, compare and modify the predefined tumor resection area template, and output the patient's personal resection area template;

[0012] The feature combination module combines the image feature matrix, pathological feature matrix, individual resection area template and clinical features, and uses a deep learning model to fuse multimodal features to form a combined feature vector.

[0013] The prediction module is used to establish a prediction model, receive the joint feature vector, and output the efficacy prediction result.

[0014] Furthermore, it also includes an optimization module, which is used to evaluate the accuracy of the model prediction results through the loss function according to the requirements of the prediction task; and optimize the model parameters through the optimization algorithm.

[0015] Furthermore, the optimization module is used to obtain an independent validation data set based on big data, evaluate the prediction model based on the independent validation data set, and adjust and optimize the prediction model according to the evaluation results.

[0016] Furthermore, the template module is used to construct one or more sets of predefined resection area templates based on preoperative imaging data and clinical experience; after receiving the postoperative images, the postoperative images are preprocessed, and then the key feature information is extracted using image processing technology, and the extracted feature information is matched with the predefined resection area templates. According to the matching results, the predefined resection area templates are modified to output the patient's personal resection area template.

[0017] Furthermore, the preprocessing module is used to identify image data based on the image segmentation algorithm to obtain the tumor area; based on the cell segmentation technology, identify the pathological image to obtain the characteristics of a single cell or a cell group; and use the image processing and analysis algorithm to extract the features related to the tumor or the cell after the image segmentation algorithm or the cell segmentation, including morphological features, statistical features and advanced features.

[0018] Furthermore, the feature extraction module is used to select a feature extraction method according to the research purpose and the characteristics of the data, calculate each eigenvalue from the preprocessed image data, and arrange the value of each image feature in sequence to form a two-dimensional image feature matrix.

[0019] Furthermore, the feature extraction module is used to segment the pathological image, identify and extract the tumor area and non-tumor area, and then extract the features of the cell nucleus and cytoplasm based on the segmented image of the pathological image, arrange the values ​​of each pathological feature, and generate a two-dimensional pathological feature matrix.

[0020] Furthermore, the prediction module is used to concatenate features of different modalities through a deep learning model to generate a joint feature vector, assign different weights to features of different modalities, and perform weighted summation based on the importance of the features; based on machine learning, the weights are adaptively adjusted.

[0021] Furthermore, the postoperative image acquisition module is used to perform image preprocessing on the acquired images of the patient after tumor resection, including at least denoising and contrast enhancement, and regenerate the patient's personal postoperative images.

[0022] Furthermore, a visualization module is included for visualizing the prediction results, joint feature vectors and postoperative images.

[0023] The following beneficial effects were achieved by adopting the above scheme: (1) This scheme, by combining imaging data and pathological images, can extract more comprehensive tumor features, including morphological, statistical and advanced features, thereby improving the accuracy and reliability of efficacy prediction. At the same time, the patient's postoperative images are collected, and the template model established according to the template module is used to generate personal resection area template data, so as to use the deep learning model to combine the four features of imaging features, pathological features, personal resection area template data and clinical features to form a joint feature vector, which further enhances the generalization ability and accuracy of the prediction model, outputs personalized efficacy prediction results for patients, improves the prediction accuracy of tumor efficacy, and provides strong decision support for doctors.

[0024] (2) In this solution, the optimization module evaluates the accuracy of the model prediction results through the loss function and optimizes the model parameters using the optimization algorithm to ensure that the prediction model can continue to learn and improve and improve the prediction performance. At the same time, based on the independent verification data set of big data, the prediction model can be objectively evaluated and adjusted to ensure the stability and reliability of the model.

[0025] (3) In this solution, the preprocessing module and feature extraction module can automatically process image data and pathological images, reduce manual intervention, and further improve the accuracy of tumor prediction.

[0026] (4) In this solution, the visualization module visualizes the prediction results, joint feature vectors, and postoperative images, allowing doctors to intuitively understand the prediction results and tumor conditions, facilitating quick decision making.

[0027] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the process of the tumor efficacy prediction device based on the image and pathology interactive network in this embodiment. DETAILED DESCRIPTION

[0029] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0031] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0032] The following is further described in detail through specific implementation methods:

[0033] Implementation example Figure 1 As shown, a tumor efficacy prediction device based on an image and pathology interactive network includes a data acquisition module, a preprocessing module, a feature extraction module, a postoperative image acquisition module, a template module, a feature combination module and a prediction module;

[0034] The data acquisition module is used to obtain the patient's preoperative and postoperative tumor imaging data and the patient's clinical characteristic information from medical imaging equipment; the medical imaging equipment mainly includes two commonly used medical imaging equipment, CT (computer tomography) and MRI (magnetic resonance imaging), and the patient's tumor imaging data is obtained through these two commonly used medical imaging equipment. At the same time, the quality of the acquired imaging data, including clarity, resolution and contrast, etc., must also be ensured; at the same time, the quality of the pathological image, including staining uniformity, cell structure clarity, etc., must also be ensured.

[0035] The preprocessing module is used to standardize the image data and pathological images, such as image resizing and grayscale correction, segment the image data into tumor regions, extract tumor-related image features, and perform cell segmentation and feature extraction on pathological images, such as cell nucleus size, morphology, staining intensity, etc. The preprocessing module is used to identify image data and obtain tumor regions based on image segmentation algorithms; identify pathological images based on cell segmentation and obtain features of single cells or cell groups; and use image processing and analysis algorithms to extract features related to tumors or cells from images after image segmentation algorithms or cell segmentation, including morphological features, statistical features, and advanced features.

[0036] The feature extraction module is used to extract quantitative features from the preprocessed image data using the imaging omics method. These features can be morphological features (such as the size, shape, edge, etc. of the tumor), texture features (such as the grayscale co-occurrence matrix of the image, local binary pattern, etc.), and more advanced features (such as features obtained by the feature extraction method based on machine learning) to form an image feature matrix. The pathological omics method is used to extract quantitative features from the preprocessed pathological images. These features can be morphological features of the cell nucleus (such as size, shape, number of nucleoli, etc.), staining features of the cytoplasm (such as staining intensity, uniformity, etc.), and spatial relationship features between cells, etc., to form a pathological feature matrix; the feature extraction module is also used to select a feature extraction method according to the research purpose and the characteristics of the data, calculate each feature value from the preprocessed image data, and arrange the values ​​of each image feature in sequence to form a two-dimensional image feature matrix. Among them, the image feature extraction methods are generally divided into three types: manual extraction, semi-automatic extraction, and fully automatic extraction. Here, the fully automatic extraction method is selected, that is, relying on the deep learning algorithm to automatically determine the ROI (region of interest) and calculate the feature value.

[0037] The feature extraction module is used to segment the pathological image, identify and extract the tumor area and non-tumor area, and then extract the features of the cell nucleus and cytoplasm based on the segmented image of the pathological image, arrange the values ​​of each pathological feature, and generate a two-dimensional pathological feature matrix. For pathological features, the image processing algorithm is first used to segment the pathological image into cells to identify single cells or cell groups. This usually involves steps such as threshold segmentation, edge detection, and morphological operations. Various quantitative features are calculated within the identified cell area. These features can be pixel-based statistical features (such as grayscale mean, variance, etc.) or morphological features (such as perimeter, area, etc.).

[0038] The postoperative image acquisition module is used to obtain the pathological images of the patient's postoperative tumor tissue and its surrounding tissue through pathological examination, and is defined as a postoperative image. It mainly uses high-performance image sensors, such as CMOS or CCD sensors, to capture more detailed information, which helps doctors or subsequent prediction modules to more accurately evaluate the surgical effect and patient recovery. These sensors are sensitive to light and can convert the received light into electrical signals. The acquired electrical signals are amplified and converted into digital signals through analog-to-digital conversion (A / D conversion). The postoperative image acquisition module is used to perform image preprocessing on the images obtained after the patient's tumor is removed, including denoising and contrast enhancement, and regenerate the patient's personal postoperative image. Afterwards, the above digital signals are processed by image processing algorithms, such as denoising, contrast enhancement, etc., to improve the quality of the image. The processed image data can be output to a computer or other storage device through various interfaces (such as USB, network interface, etc.). These image data can be used in subsequent tumor efficacy prediction and evaluation and subsequent analysis, diagnosis or teaching by doctors.

[0039] The template module is used to establish a template module, receive postoperative images and postoperative image data, extract feature information therein, compare and modify the predefined tumor resection area template, and output the patient's personal resection area template. The template module is used to build one or more sets of predefined resection area templates based on preoperative image data and clinical experience; when receiving the postoperative image, the postoperative image is preprocessed, and then the key feature information is extracted using image processing technology, and the extracted feature information is matched with the predefined resection area template. According to the matching results, the predefined resection area template is modified, and the patient's personal resection area template is output.

[0040] The feature combination module is used to combine the image feature matrix, pathological feature matrix, personal resection area template and clinical features, and use the deep learning model to fuse multimodal features to form a joint feature vector. Clinical features include the patient's age, gender, tumor stage, previous treatment history, family history, etc. Although these features are relatively simple, they can provide clues about the patient's overall health and disease risk; at the same time, information about the size, shape, location of the tumor and the structure of the surrounding tissue can be extracted from these images through image processing technology. These features are crucial for evaluating the progression of the tumor and predicting the efficacy. At the same time, the deep learning model uses the Transformer network. The core of the Transformer network is the self-attention mechanism, which can model the relationship between any two positions in the input sequence and can automatically learn the mutual dependence between different positions based on the content of the sequence. This mechanism enables the Transformer network to better capture the long-range dependencies in the sequence. Compared with traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), the Transformer network has the following significant advantages: stronger parallel computing capabilities, stronger ability to process long sequences, and better interpretability.

[0041] The prediction module is used to establish a prediction model, receive the joint feature vector, and output the efficacy prediction result. The prediction module is used to concatenate the features of different modalities through a deep learning model to generate a joint feature vector, assign different weights to the features of different modalities, and perform weighted summation according to the importance of the features; based on machine learning, the weights are adaptively adjusted. In the efficacy prediction stage, the joint feature vector is input into the efficacy prediction model. The prediction model can be a classification model or a regression model, and the specific choice depends on the requirements of the prediction task. When the goal is to predict whether a patient will respond to a certain treatment (such as complete remission, partial remission, stability or progression), a classification model can be used. These models take the joint feature vector as input and output one or more class labels to represent the possible efficacy results of the patient; when the goal is to predict the patient's survival time, tumor size change or other continuous variables, a regression model can be used. These models take the joint feature vector as input and output one or more continuous values ​​to represent the possible efficacy indicators of the patient.

[0042] It also includes an optimization module, which is used to evaluate the accuracy of the model's prediction results through a loss function according to the needs of the prediction task; optimize the model parameters through an optimization algorithm; and obtain an independent validation data set based on big data, and evaluate the prediction model based on the independent validation data set, and adjust and optimize the prediction model based on the evaluation results. The loss functions include cross entropy loss and mean square error. Both cross entropy loss and mean square error are commonly used loss functions in machine learning and deep learning. They are used to measure the difference between the model's predicted value and the actual value and guide the optimization of the model. Cross entropy is a concept in information theory that is used to measure the difference between two probability distributions; given two probability distributions P and Q, the cross entropy is defined as H(P,Q) = -∑xP(x)logQ(x), where x represents the sample space, P(x) and Q(x) represent the true distribution and the model's predicted distribution, respectively. The size of the cross entropy reflects the degree of fit of the model to the true distribution. The smaller its value, the closer the model prediction is to the true value. In classification tasks, the cross entropy loss function is usually defined as Where N is the number of samples, yi is the true label of the i-th sample, is the model prediction probability of the ith sample. The mean square error is a commonly used indicator for evaluating the accuracy of regression algorithms. It is defined as: Mean square error (MSE) is a measure of the degree of difference between the estimator and the estimated value; the mean square error is dimensionless and represents the expected value of the sum of squares of the deviations between the model output value and the actual observed value. It is the most general standard for evaluating point estimates, and it is usually hoped that the estimated mean square error is as small as possible. The formula is as follows: MSE = 1 / n∑i = 1n(yi-f(xi))^2, where n is the sample size, yi represents the actual value, and f(xi) represents the predicted value.

[0043] It also includes a visualization module for visualizing the prediction results, joint feature vectors, and postoperative images. For the acquired tumor image data and postoperative images, the visualization module can perform three-dimensional reconstruction to display the spatial morphology of the tumor and the effect after resection, helping doctors to more accurately evaluate the surgical effect. At the same time, the important features in the image feature matrix, pathological feature matrix, and joint feature vector are visualized, such as through color coding, heat map, etc., so that doctors can intuitively see which features have a significant impact on the prediction results, and the efficacy prediction results output by the prediction module are presented to doctors in the form of charts, reports, etc., including key information such as the predicted risk of tumor recurrence and survival time, providing strong support for doctors' subsequent treatment decisions; at the same time, a friendly interactive interface is provided, and doctors can easily view and adjust different visualization results through buttons, sliders and other tools on the interface to achieve a comprehensive understanding and evaluation of the postoperative tumor efficacy.

[0044] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A tumor efficacy prediction device based on an image and pathology interactive network, characterized in that: It includes data acquisition module, preprocessing module, feature extraction module, postoperative image acquisition module, template module, feature combination module and prediction module; A data acquisition module, used to acquire the patient's preoperative and postoperative tumor imaging data from medical imaging equipment and to acquire the patient's clinical characteristic information; A postoperative image acquisition module is used to obtain pathological images of the patient's postoperative tumor tissue and its surrounding tissues through pathological examination, and is defined as a postoperative image; The preprocessing module is used to standardize the image data and pathological images, segment the tumor area of ​​the image data, extract tumor-related image features, and segment and extract tumor cells from the pathological images; The feature extraction module is used to extract quantitative features from the preprocessed image data using the radiomics method to form an image feature matrix; and to extract quantitative features from the preprocessed pathological images using the pathological tissue and immunohistochemistry method to form a pathological feature matrix; The template module is used to establish a template module, receive postoperative images and postoperative imaging data, extract feature information therein, compare and modify the predefined tumor resection area template, and output the patient's personal resection area template; The feature combination module combines the image feature matrix, pathological feature matrix, individual resection area template and clinical features, and uses a deep learning model to fuse multimodal features to form a combined feature vector. The prediction module is used to establish a prediction model, receive the joint feature vector, and output the efficacy prediction result.

2. The tumor efficacy prediction device based on image and pathology interactive network according to claim 1, characterized in that: It also includes an optimization module, which is used to evaluate the accuracy of the model prediction results through the loss function according to the requirements of the prediction task; and optimize the model parameters through the optimization algorithm.

3. The tumor efficacy prediction device based on image and pathology interactive network according to claim 2 is characterized in that: The optimization module is used to obtain an independent validation data set based on big data, evaluate the prediction model based on the independent validation data set, and adjust and optimize the prediction model according to the evaluation results.

4. The tumor efficacy prediction device based on image and pathology interactive network according to claim 3 is characterized in that: The template module is used to construct one or more sets of predefined resection area templates based on preoperative imaging data and clinical experience; after receiving the postoperative images, the postoperative images are preprocessed, and then the key feature information is extracted using image processing technology, and the extracted feature information is matched with the predefined resection area template. According to the matching results, the predefined resection area template is modified to output the patient's personal resection area template.

5. The device for predicting tumor efficacy based on image and pathology interactive network according to claim 4, characterized in that: The preprocessing module is used to identify image data and obtain tumor areas based on image segmentation algorithms; to identify pathological images and obtain the characteristics of single cells or cell groups based on cell segmentation technology; and to extract features related to tumors or cells using image processing and analysis algorithms for images after image segmentation algorithms or cell segmentation, including morphological features, statistical features, and advanced features.

6. The tumor efficacy prediction device based on image and pathology interactive network according to claim 5, characterized in that: The feature extraction module is used to select the feature extraction method according to the research purpose and the characteristics of the data, calculate each eigenvalue from the preprocessed image data, and arrange the value of each image feature in sequence to form a two-dimensional image feature matrix.

7. The device for predicting tumor efficacy based on image and pathology interactive network according to claim 6, characterized in that: The feature extraction module is used to segment the pathological image, identify and extract the tumor area and non-tumor area, and then extract the features of the cell nucleus and cytoplasm based on the segmented image of the pathological image, arrange the values ​​of each pathological feature, and generate a two-dimensional pathological feature matrix.

8. The device for predicting tumor efficacy based on image and pathology interactive network according to claim 7, characterized in that: The prediction module is used to concatenate features of different modalities through a deep learning model to generate a joint feature vector, assign different weights to features of different modalities, and perform weighted summation based on the importance of the features; based on machine learning, the weights are adaptively adjusted.

9. The device for predicting tumor efficacy based on image and pathology interactive network according to claim 8, characterized in that: The postoperative image acquisition module is used to perform image preprocessing on the images acquired after tumor resection of the patient, including at least denoising and contrast enhancement, and regenerate the patient's personal postoperative images.

10. The device for predicting tumor efficacy based on image and pathology interactive network according to claim 9, characterized in that: It also includes a visualization module for visualizing the prediction results, joint feature vectors, and postoperative images.

Images