Visualization method and system of pathological convolutional neural network

Abstract

The invention provides a pathology convolutional neural network visualization method and system, and the method comprises the steps: transmitting various tumor pathology images into a target convolutional layer of a CNN model through forward propagation, recording and extracting the output information of a feature map, and obtaining a multi-channel matrix feature map; the activation value of each channel of the multi-channel matrix feature map is linearly zoomed and visualized into a color image, channels are selected or multiple channels are combined, and a pathological convolutional neural network program penetration visualization model is created through the internal process of a program penetration visualization convolutional neural network; inputting a to-be-predicted gland pathology image data set to the pathology convolutional neural network program to penetrate through the visualization model, and automatically outputting a target convolutional layer feature map; and performing target convolutional layer-by-layer identification on the target convolutional layer feature map by using a multi-class neoplasia intelligent identification model, exploring the internal process of the multi-class neoplasia intelligent identification model through program penetration, and intelligently predicting the neoplasia trend of the gland pathological image.

Description

Technical Field

[0001] The present invention relates to the technical field of image feature extraction visualization prediction, and more specifically, to a visualization method and system for a pathological convolutional neural network. Background Art

[0002] At present, in the study of medical imaging, the interpretability of deep learning models has always been a hot topic in cutting-edge research; existing medical imaging research uses the semantic segmentation model, and its prediction result is the recognition probability of each position in the image belonging to a certain disease, which can be used as a black-box interpretation of the model; however, with the deepening of research, more and more in-depth and large-scale data processing processes require the model to be visualized, and a white-box method is used to understand the deep learning model; through the interpretation of the internal process of the model, people need to understand the machine's recognition process of disease images, and be able to obtain new solutions from the way the machine analyzes problems; at present, although the existing model visualization tools are complex and the degree of visualization is not high, specific problems also include: how to solve the current problems, how to batch output feature maps Feature Map, to show the model's processing process of the original image and how to mine the knowledge inside the model, etc., which have yet to be solved; therefore, it is necessary to propose a visualization method and system for pathological convolutional neural networks to at least partially solve the problems existing in the prior art. Summary of the invention

[0003] A series of simplified concepts are introduced in the content of the invention, which will be further described in detail in the specific implementation method section; the content of the invention of the present invention does not mean to attempt to limit the key features and essential technical features of the technical solution claimed for protection, nor does it mean to attempt to determine the scope of protection of the technical solution claimed for protection.

[0004] To at least partially solve the above problems, the present invention provides a method for visualizing a pathological convolutional neural network, comprising:

[0005] S100, sending multiple types of tumor pathology images into the target convolution layer of the CNN convolutional neural network model through forward propagation, recording and extracting the output information of the feature map, and obtaining a multi-channel matrix feature map;

[0006] S200, linearly scales and visualizes the activation value of each channel of the multi-channel matrix feature map into a color image, selects a channel or combines multiple channels, visualizes the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model;

[0007] S300, inputting the gland pathology image dataset to be predicted into the pathology convolutional neural network program penetration visualization model, and automatically outputting the target convolutional layer feature map;

[0008] S400, based on the target convolution layer feature map, uses a multi-class tumor intelligent recognition model to perform target convolution layer layer-by-layer recognition on the target convolution layer feature map, explores the internal process of the multi-class tumor intelligent recognition model through program penetration, and intelligently predicts the tumor trend of glandular pathology images.

[0009] Preferably, S100 includes:

[0010] S101, select the target convolution layer in the trained CNN convolutional neural network model;

[0011] S102, sending multiple types of tumor pathology images into the target convolution layer of the CNN convolutional neural network model through forward propagation;

[0012] S103, in each target convolution layer, record and extract output information of the feature map to obtain a multi-channel matrix feature map; the target convolution layer includes: a partial selection convolution layer or a full selection convolution layer; the multiple types of tumor pathology images include: intestinal adenoma pathology images, thyroid tumor pathology images or pancreatic tumor pathology images.

[0013] Preferably, S200 includes:

[0014] S201, linearly scaling each channel activation value of the multi-channel matrix feature map to a range of 0 to 255, and converting it into a visual color image;

[0015] S202, selecting multiple channels of a multi-channel matrix feature map, combining the multiple channels to obtain multi-color image features, and creating a pathological convolutional neural network program penetration visualization model by program penetration visualization of the internal process of the convolutional neural network;

[0016] S202, sequentially encode all feature map visualization results and save them as continuously encoded visualization image files; output the continuously encoded visualization image files in a continuous manner to form a visualization image output continuous dynamic stream.

[0017] Preferably, S300 includes:

[0018] S301, acquiring a gland pathology image dataset to be predicted by collecting the gland pathology image to be predicted; the gland pathology image dataset to be predicted includes: normal gland pathology images, pathology images of the junction between normal gland and tumor, and multiple types of tumor gland pathology images;

[0019] S302, inputting the gland pathology image dataset to be predicted into the pathology convolutional neural network program penetration visualization model;

[0020] S303, penetrating the visualization model through the pathological convolutional neural network program, and automatically outputting the target convolutional layer feature map.

[0021] Preferably, S400 includes:

[0022] S401, according to the target convolution layer feature map, using the convolution layer to identify layer by layer, to create a multi-class tumor change intelligent recognition model; the multi-class tumor change intelligent recognition model is a convolution layer layer by layer recognition model;

[0023] S402, using a multi-class tumor intelligent recognition model, performing target convolution layer recognition on the target convolution layer feature map layer by layer to obtain a visualized gland layer by layer recognition result;

[0024] S402, based on the recognition results of the visualized gland shape, the visualized cell nucleus and the visualized cell morphology of the gland, the internal process of the intelligent recognition model of multiple types of neoplasia is explored, and the trend of neoplasia of the gland pathological image is intelligently predicted.

[0025] The present invention provides a visualization system of a pathological convolutional neural network, comprising:

[0026] The multi-channel matrix feature map extraction subsystem sends multiple types of tumor pathology images to the target convolution layer of the CNN convolutional neural network model through forward propagation, records and extracts the output information of the feature map, and obtains the multi-channel matrix feature map;

[0027] The linear scaling program penetration model subsystem visualizes the activation value of each channel of the multi-channel matrix feature map as a color image by linear scaling. It selects channels or combines multiple channels to visualize the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model.

[0028] The visualization model target convolution output subsystem inputs the glandular pathology image dataset to be predicted into the pathology convolution neural network program to penetrate the visualization model and automatically outputs the target convolution layer feature map;

[0029] The gland intelligent recognition and tumor prediction subsystem uses a multi-class tumor intelligent recognition model to perform layer-by-layer recognition of the target convolution layer feature map based on the target convolution layer feature map. It penetrates the program to explore the internal process of the multi-class tumor intelligent recognition model and intelligently predicts the tumor trend of gland pathological images.

[0030] Preferably, the multi-channel matrix feature map extraction subsystem includes:

[0031] The target convolutional layer selection subsystem selects the target convolutional layer in the trained CNN convolutional neural network model;

[0032] The image target convolution layer is passed into the subsystem, and multiple types of tumor pathology images are sent to the target convolution layer of the CNN convolutional neural network model through forward propagation;

[0033] The feature map output extraction subsystem records and extracts the output information of the feature map in each target convolution layer to obtain a multi-channel matrix feature map; the target convolution layer includes: a partial selection convolution layer or a full selection convolution layer; multiple types of tumor pathology images include: intestinal adenoma pathology images, thyroid tumor pathology images or pancreatic tumor pathology images.

[0034] Preferably, the linear scaling program penetration model subsystem includes:

[0035] The activation value linear scaling visualization subsystem linearly scales the activation value of each channel of the multi-channel matrix feature map to the range of 0 to 255 and converts it into a visual color image;

[0036] The channel combination program penetration creates a subsystem, selects multiple channels of the multi-channel matrix feature map, combines multiple channels to obtain multi-color image features, visualizes the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model;

[0037] The feature map storage encoding streaming subsystem sequentially encodes and saves all feature map visualization results as continuously encoded visualization image files; the continuously encoded visualization image files are output continuously to form a continuous dynamic flow of visualization image output.

[0038] Preferably, the visualization model target convolution output subsystem includes:

[0039] The gland pathology image data acquisition subsystem acquires the gland pathology image data set to be predicted by acquiring the gland pathology image to be predicted; the gland pathology image data set to be predicted includes: normal gland pathology images, pathology images of the junction between normal gland and tumor, and multiple types of tumor gland pathology images;

[0040] The visualization model image input subsystem inputs the gland pathology image dataset to be predicted into the pathology convolutional neural network program through the visualization model;

[0041] The target convolution feature map automatic output subsystem penetrates the visualization model through the pathological convolution neural network program and automatically outputs the target convolution layer feature map.

[0042] Preferably, the gland intelligent identification and tumor prediction subsystem includes:

[0043] The convolution layer layer-by-layer recognition model subsystem uses the convolution layer layer-by-layer recognition to create a multi-class tumor change intelligent recognition model based on the target convolution layer feature map; the multi-class tumor change intelligent recognition model is a convolution layer layer-by-layer recognition model;

[0044] The visualized gland layer-by-layer recognition subsystem uses multiple types of tumor intelligent recognition models to perform layer-by-layer recognition of the target convolutional layer feature map to obtain the visualized gland layer-by-layer recognition results;

[0045] The multi-type tumor change intelligent prediction subsystem explores the internal process of the multi-type tumor change intelligent recognition model based on the gland visualization gland shape, gland visualization cell nucleus and gland visualization cell morphology recognition results, and intelligently predicts the tumor change trend of gland pathological images.

[0046] Compared with the prior art, the present invention has at least the following beneficial effects:

[0047] The present invention provides a visualization method and system for a pathological convolutional neural network, including: sending multiple types of tumor pathological images into a target convolution layer of a CNN convolutional neural network model through forward propagation, recording and extracting output information of feature maps, and obtaining a multi-channel matrix feature map; linearly scaling and visualizing each channel activation value of the multi-channel matrix feature map as a color image, selecting a channel or combining multiple channels, and visualizing the internal process of the convolutional neural network through program penetration to create a pathological convolutional neural network program penetration visualization model; inputting a glandular pathological image data set to be predicted into the pathological convolutional neural network program penetration visualization model, and automatically outputting a target convolutional layer feature map; according to the target convolutional layer feature map, using a multi-type tumor intelligent recognition model, performing target convolution layer layer-by-layer recognition on the target convolutional layer feature map, exploring the internal process of the multi-type tumor intelligent recognition model through program penetration, and intelligently predicting the tumor trend of the glandular pathological image; using a given target input image and the location of a model file, through the program The penetration program intelligently searches for the corresponding data entry in the calculation graph, and automatically outputs the feature maps of all convolutional layers to the specified directory; Feature map extraction: Select the target convolutional layer or all convolutional layers in the trained CNN model, and then send the input image to the network through forward propagation; In each target convolutional layer, record and extract the output of the feature map; Feature map visualization: The feature map is usually a multi-channel matrix. By linearly scaling the activation value of each channel to the range of 0 to 255, it is visualized as a color image. You can also select the channel of interest or combine multiple channels to obtain a more comprehensive feature representation; Visualization disk: Save the results of all feature map visualization as an image file; The present invention proposes a TensorFlow model visualization tool program penetration; Program penetration is based on TensorFlow, using powerful computing power and flexibility to provide a simple and efficient model visualization tool; The present invention can batch output feature maps Feature Map can be used to show the model's processing of the original image, making it easy to mine the model's internal knowledge; feature maps can be output in batches, that is, the image features generated by the model in each convolutional layer; these feature maps show the model's processing of the original image, which can provide a deeper understanding of how the model extracts and combines features in the image; program penetration can make it easier to mine the model's internal knowledge; feature maps can be observed and analyzed to understand how the model abstracts and represents images at different levels; this helps to reveal the model's recognition process of disease images and significantly improve the model's operating efficiency and accuracy.

[0048] The visualization method and system of a pathological convolutional neural network described in the present invention, and other advantages, objectives and features of the present invention will be partially reflected through the following description, and will also be partially understood by technicians in this field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 This is a diagram of an embodiment of a visualization method and system for a pathological convolutional neural network described in the present invention.

[0051] Figure 2 This is a diagram of an implementation example of a visualization system framework for a pathological convolutional neural network described in the present invention.

[0052] Figure 3 This is a diagram of an implementation step of a method for visualizing a pathological convolutional neural network according to the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments so that those skilled in the art can implement the invention according to the instructions. Figure 1-Figure 3 As shown, the present invention provides a visualization method of a pathological convolutional neural network, comprising:

[0054] S100, sending multiple types of tumor pathology images into the target convolution layer of the CNN convolutional neural network model through forward propagation, recording and extracting the output information of the feature map, and obtaining a multi-channel matrix feature map;

[0055] S200, linearly scales and visualizes the activation value of each channel of the multi-channel matrix feature map into a color image, selects a channel or combines multiple channels, visualizes the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model;

[0056] S300, inputting the gland pathology image dataset to be predicted into the pathology convolutional neural network program penetration visualization model, and automatically outputting the target convolutional layer feature map;

[0057] S400, based on the target convolution layer feature map, uses a multi-class tumor intelligent recognition model to perform target convolution layer layer-by-layer recognition on the target convolution layer feature map, explores the internal process of the multi-class tumor intelligent recognition model through program penetration, and intelligently predicts the tumor trend of glandular pathology images.

[0058] The principle and effect of the above technical solution are as follows: the present invention provides a visualization method of a pathological convolutional neural network, comprising: sending multiple types of tumor pathological images into the target convolution layer of the CNN convolutional neural network model through forward propagation, recording and extracting the output information of the feature map, and obtaining a multi-channel matrix feature map; linearly scaling and visualizing each channel activation value of the multi-channel matrix feature map as a color image, selecting a channel or combining multiple channels, and visualizing the internal process of the convolutional neural network through program penetration to create a pathological convolutional neural network program penetration visualization model; inputting a dataset of glandular pathological images to be predicted into the pathological convolutional neural network program penetration visualization model, and automatically outputting a target convolutional layer feature map; according to the target convolutional layer feature map, using multiple types of tumor intelligent recognition models, performing target convolutional layer recognition on the target convolutional layer feature map layer by layer, and generating a pathological convolutional neural network program penetration visualization model through program penetration. Dig into the internal process of the multi-class tumor intelligent recognition model and intelligently predict the tumor trend of glandular pathology images; use the given target input image and the location of the model file to intelligently search for the corresponding data entry in the calculation graph through program penetration, and automatically output the feature maps of all convolutional layers to the specified directory; Feature map extraction: select the target convolution layer or all convolution layers in the trained CNN model, and then send the input image into the network through forward propagation; In each target convolution layer, record and extract the output of the feature map; Feature map visualization: The feature map is usually a multi-channel matrix. By linearly scaling the activation value of each channel to the range of 0 to 255, it is visualized as a color image. You can also select the channel of interest or combine multiple channels to obtain a more comprehensive feature representation; Visualization disk: Save the results of all feature map visualizations as image files;

[0059] The present invention sets the model file path, the CNN input image path and the output directory of the feature map; creates a model warehouse; the model warehouse contains: trained model parameters and model structure; automatically searches for data entry, given an input image, the program automatically finds all placeholders in the calculation graph, and searches for the best matching placeholder to input the image into it; when the program fails to run through the model, two autonomous debugging attempts are made: adjusting the size of the input image to match the target data entry, adjusting the target placeholder to be correctly defined in the calculation graph; default placeholder input, the program will automatically obtain all placeholders and fill them with default zero values ​​to make the calculation graph run normally;

[0060] Figure 1 This is a feature visualization result of an intestinal adenoma recognition model according to an embodiment of the present invention; the program penetration model includes:

[0061] Set python conv_vis.py; model #model file path; image_path #CNN input image path; output_dir #feature map output directory;

[0062] The model repository contains: export.ckpt: trained model parameters and export.ckpt.meta: model structure; automatic search for data entry. Given an input image, the program automatically finds all placeholders in the calculation graph and searches for the best matching placeholder to input the image into it;

[0063] When program penetration fails, two debugging attempts are made: resizing the input image to match the target data entry and ensuring that the target placeholder is correctly defined in the computation graph;

[0064] Default placeholder input, the program will automatically get all placeholders and fill them with default zero values ​​to make the calculation graph run normally; the default values ​​are:

[0065] tf.bool: False

[0066] tf.int32:0

[0067] tf.int64:0

[0068] tf.float16:0.0

[0069] tf.float32:0.0

[0070] tf.array(shape): numpy.zeros(shape);

[0071] The present invention proposes a model visualization tool program penetration of TensorFlow; program penetration is based on TensorFlow, and uses powerful computing power and flexibility to provide a concise and efficient model visualization tool; the present invention can batch output feature maps Feature Map to show the process of the model processing the original image, which is easy to mine the knowledge inside the model; feature maps can be batch output, that is, image features generated by the model in each convolutional layer; these feature maps show the model's processing process on the original image, and can more deeply understand how the model extracts and combines features in the image; through program penetration, the knowledge inside the model can be more easily mined; feature maps can be observed and analyzed to understand the model's abstraction and representation of images at different levels; it is helpful to reveal the model's recognition process of disease images and significantly improve the model's operating efficiency and accuracy.

[0072] In one embodiment, S100 includes:

[0073] S101, select the target convolution layer in the trained CNN convolutional neural network model;

[0074] S102, sending multiple types of tumor pathology images into the target convolution layer of the CNN convolutional neural network model through forward propagation;

[0075] S103, in each target convolution layer, record and extract output information of the feature map to obtain a multi-channel matrix feature map; the target convolution layer includes: a partial selection convolution layer or a full selection convolution layer; the multiple types of tumor pathology images include: intestinal adenoma pathology images, thyroid tumor pathology images or pancreatic tumor pathology images.

[0076] The principle and effect of the above technical solution are: selecting a target convolution layer in a trained CNN convolutional neural network model; sending multiple types of tumor pathology images to the target convolution layer of the CNN convolutional neural network model through forward propagation; in each target convolution layer, recording and extracting the output information of the feature map to obtain a multi-channel matrix feature map; the target convolution layer includes: partially selected convolution layers or all selected convolution layers; multiple types of tumor pathology images include: intestinal adenoma pathology images, thyroid tumor pathology images or pancreatic tumor pathology images; the CNN convolutional neural network model and the multi-channel matrix feature map are more accurate.

[0077] In one embodiment, S200 includes:

[0078] S201, linearly scaling each channel activation value of the multi-channel matrix feature map to a range of 0 to 255, and converting it into a visual color image;

[0079] S202, selecting multiple channels of a multi-channel matrix feature map, combining the multiple channels to obtain multi-color image features, and creating a pathological convolutional neural network program penetration visualization model by program penetration visualization of the internal process of the convolutional neural network;

[0080] S202, sequentially encode all feature map visualization results and save them as continuously encoded visualization image files; output the continuously encoded visualization image files in a continuous manner to form a visualization image output continuous dynamic stream.

[0081] The principle and effect of the above technical solution are: linearly scale the activation value of each channel of the multi-channel matrix feature map to the range of 0 to 255, and convert it into a visual color image; select multiple channels of the multi-channel matrix feature map, combine the multiple channels to obtain multi-color image features, and create a pathological convolutional neural network program penetration visualization model through program penetration of the internal process of the convolutional neural network; sequentially encode all feature map visualization results and save them as continuously encoded visualization image files; continuously output the continuously encoded visualization image files to form a continuous dynamic flow of visualization image output; the dynamic visibility of the image output is significantly improved, and the continuity is greatly improved.

[0082] In one embodiment, S300 includes:

[0083] S301, acquiring a gland pathology image dataset to be predicted by collecting the gland pathology image to be predicted; the gland pathology image dataset to be predicted includes: normal gland pathology images, pathology images of the junction between normal gland and tumor, and multiple types of tumor gland pathology images;

[0084] S302, inputting the gland pathology image dataset to be predicted into the pathology convolutional neural network program penetration visualization model;

[0085] S303, penetrating the visualization model through the pathological convolutional neural network program, and automatically outputting the target convolutional layer feature map.

[0086] The principle and effect of the above technical solution are as follows: by collecting the pathological images of the gland to be predicted, a data set of the pathological images of the gland to be predicted is obtained; the data set of the pathological images of the gland to be predicted includes: pathological images of normal glands, pathological images of the junction of normal glands and tumors, and pathological images of multiple types of tumorous glands; the data set of pathological images of the gland to be predicted is input into the pathological convolutional neural network program penetration visualization model; the target convolutional layer feature map is automatically output through the pathological convolutional neural network program penetration visualization model; the target convolutional layer feature map is automatically output through the pathological convolutional neural network program penetration visualization model, including: from the pathological convolutional neural network program penetration visualization The model gradually deepens the number of convolutional network layers from the bottom first convolutional layer to the bottom second convolutional layer and the bottom third convolutional layer until the deepest layer of the target convolutional layer is reached, and the target convolutional layer is output layer by layer; the bottom first convolutional layer extracts the gland visualization edge and visualization color information from the gland pathology image to be predicted; the bottom second convolutional layer outputs the gland visualization edge and visualization color information enhancement data; the bottom third convolutional layer outputs the gland visualization gland morphology of the feature map until the deepest layer of the target convolutional layer outputs the gland visualization gland shape, gland visualization cell nucleus and gland visualization cell morphology; the output accuracy and efficiency of the target convolutional layer feature map are significantly enhanced.

[0087] In one embodiment, S400 includes:

[0088] S401, according to the target convolution layer feature map, using the convolution layer to identify layer by layer, to create a multi-class tumor change intelligent recognition model; the multi-class tumor change intelligent recognition model is a convolution layer layer by layer recognition model;

[0089] S402, using a multi-class tumor intelligent recognition model, performing target convolution layer recognition on the target convolution layer feature map layer by layer to obtain a visualized gland layer by layer recognition result;

[0090] S402, based on the recognition results of the visualized gland shape, the visualized cell nucleus and the visualized cell morphology of the gland, the internal process of the intelligent recognition model of multiple types of neoplasia is explored, and the trend of neoplasia of the gland pathological image is intelligently predicted.

[0091] The principle and effect of the above technical solution are as follows: according to the target convolutional layer feature map, the convolutional layer is used to identify layer by layer to create a multi-class tumor intelligent recognition model; the multi-class tumor intelligent recognition model is a convolutional layer layer by layer recognition model; through the multi-class tumor intelligent recognition model, the target convolutional layer feature map is identified layer by layer to obtain the visualized gland layer by layer recognition result; according to the gland visualized gland shape, gland visualized cell nucleus and gland visualized cell morphology recognition results, the internal process of the multi-class tumor intelligent recognition model is explored, and the tumor trend of the gland pathological image is intelligently predicted; through the multi-class tumor intelligent recognition model, the target convolutional layer feature map is identified layer by layer to obtain The results of visualized gland recognition layer by layer include: identifying the visualized edge and visualized color information extracted from the gland pathology image to be predicted by the first convolutional layer of the bottom layer through a multi-class tumor intelligent recognition model; identifying the visualized edge and visualized color information enhancement data of the second convolutional layer of the bottom layer; identifying the visualized gland morphology of the gradually deepening convolutional network layer feature map of the third convolutional layer of the bottom layer until the output of the deepest convolutional network layer feature map of the target convolutional layer is identified, identifying the visualized gland shape of the gland, the visualized cell nucleus of the gland and the visualized cell morphology of the gland, and obtaining the recognition results of the visualized gland shape, the visualized cell nucleus of the gland and the visualized cell morphology of the gland; the recognition is more accurate and comprehensive.

[0092] The present invention provides a visualization system of a pathological convolutional neural network, comprising:

[0093] The multi-channel matrix feature map extraction subsystem sends multiple types of tumor pathology images to the target convolution layer of the CNN convolutional neural network model through forward propagation, records and extracts the output information of the feature map, and obtains the multi-channel matrix feature map;

[0094] The linear scaling program penetration model subsystem visualizes the activation value of each channel of the multi-channel matrix feature map as a color image by linear scaling. It selects channels or combines multiple channels to visualize the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model.

[0095] The visualization model target convolution output subsystem inputs the glandular pathology image dataset to be predicted into the pathology convolution neural network program to penetrate the visualization model and automatically outputs the target convolution layer feature map;

[0096] The gland intelligent recognition and tumor prediction subsystem uses a multi-class tumor intelligent recognition model to perform layer-by-layer recognition of the target convolution layer feature map based on the target convolution layer feature map. It penetrates the program to explore the internal process of the multi-class tumor intelligent recognition model and intelligently predicts the tumor trend of gland pathological images.

[0097] The principle and effect of the above technical solution are as follows: the present invention provides a visualization system of a pathological convolutional neural network, including: a multi-channel matrix feature map extraction subsystem, which sends multiple types of tumor pathological images into the target convolution layer of the CNN convolutional neural network model through forward propagation, records and extracts the output information of the feature map, and obtains a multi-channel matrix feature map; a linear scaling program penetration model subsystem, which linearly scales and visualizes each channel activation value of the multi-channel matrix feature map into a color image, selects a channel or combines multiple channels, and visualizes the internal process of the convolutional neural network through program penetration to create a pathological convolutional neural network program penetration visualization model; a visualization model target convolution output subsystem, which inputs a glandular pathological image data set to be predicted into the pathological convolutional neural network program penetration visualization model, and automatically outputs a target convolutional layer feature map; a glandular intelligent recognition and tumor prediction subsystem, which uses multiple types of tumor intelligent recognition models according to the target convolutional layer feature map, The target convolutional layer feature map is identified layer by layer, and the internal process of the intelligent recognition model of multiple types of tumor changes is explored through program penetration, so as to intelligently predict the tumor change trend of glandular pathology images; using the location of the given target input image and model file, the program penetration program intelligently searches for the corresponding data entry in the calculation graph, and automatically outputs the feature maps of all convolutional layers to the specified directory; feature map extraction: select the target convolutional layer or all convolutional layers in the trained CNN model, and then send the input image to the network through forward propagation; in each target convolutional layer, record and extract the output of the feature map; feature map visualization: feature maps are usually multi-channel matrices, which are visualized as color images by linearly scaling the activation value of each channel to the range of 0 to 255. You can also select the channel of interest or combine multiple channels to obtain a more comprehensive feature representation; visualization disk: save the results of all feature map visualizations as image files;

[0098] The present invention sets the model file path, the CNN input image path and the output directory of the feature map; creates a model warehouse; the model warehouse contains: trained model parameters and model structure; automatically searches for data entry, given an input image, the program automatically finds all placeholders in the calculation graph, and searches for the best matching placeholder to input the image into it; when the program fails to run through the model, two autonomous debugging attempts are made: adjusting the size of the input image to match the target data entry, adjusting the target placeholder to be correctly defined in the calculation graph; default placeholder input, the program will automatically obtain all placeholders and fill them with default zero values ​​to make the calculation graph run normally;

[0099] Figure 1 This is a feature visualization result of an intestinal adenoma recognition model according to an embodiment of the present invention; the program penetration model includes:

[0100] Set python conv_vis.py; model #model file path; image_path #CNN input image path; output_dir #feature map output directory;

[0101] The model repository contains: export.ckpt: trained model parameters and export.ckpt.meta: model structure; automatic search for data entry. Given an input image, the program automatically finds all placeholders in the calculation graph and searches for the best matching placeholder to input the image into it;

[0102] When program penetration fails, two debugging attempts are made: resizing the input image to match the target data entry and ensuring that the target placeholder is correctly defined in the computation graph;

[0103] Default placeholder input, the program will automatically get all placeholders and fill them with default zero values ​​to make the calculation graph run normally; the default values ​​are:

[0104] tf.bool: False

[0105] tf.int32:0

[0106] tf.int64:0

[0107] tf.float16:0.0

[0108] tf.float32:0.0

[0109] tf.array(shape): numpy.zeros(shape);

[0110] The present invention proposes a model visualization tool program penetration of TensorFlow; program penetration is based on TensorFlow, and uses powerful computing power and flexibility to provide a concise and efficient model visualization tool; the present invention can batch output feature maps Feature Map to show the process of the model processing the original image, which is easy to mine the knowledge inside the model; feature maps can be batch output, that is, image features generated by the model in each convolutional layer; these feature maps show the model's processing process on the original image, and can more deeply understand how the model extracts and combines features in the image; through program penetration, the knowledge inside the model can be more easily mined; feature maps can be observed and analyzed to understand the model's abstraction and representation of images at different levels; it is helpful to reveal the model's recognition process of disease images and significantly improve the model's operating efficiency and accuracy.

[0111] In one embodiment, the multi-channel matrix feature map extraction subsystem includes:

[0112] The target convolutional layer selection subsystem selects the target convolutional layer in the trained CNN convolutional neural network model;

[0113] The image target convolution layer is passed into the subsystem, and multiple types of tumor pathology images are sent to the target convolution layer of the CNN convolutional neural network model through forward propagation;

[0114] The feature map output extraction subsystem records and extracts the output information of the feature map in each target convolution layer to obtain a multi-channel matrix feature map; the target convolution layer includes: a partial selection convolution layer or a full selection convolution layer; multiple types of tumor pathology images include: intestinal adenoma pathology images, thyroid tumor pathology images or pancreatic tumor pathology images.

[0115] The principle and effect of the above technical solution are: using multi-channel matrix feature map extraction, the target convolution layer selection subsystem selects the target convolution layer in the trained CNN convolution neural network model; the image target convolution layer transmission subsystem sends multiple types of tumor pathology images to the target convolution layer of the CNN convolution neural network model through forward propagation; the feature map output extraction subsystem records and extracts the output information of the feature map in each target convolution layer to obtain a multi-channel matrix feature map; the target convolution layer includes: partial selection of convolution layers or all selection of convolution layers; multiple types of tumor pathology images include: intestinal adenoma pathology images, thyroid tumor pathology images or pancreatic tumor pathology images; the CNN convolution neural network model and the multi-channel matrix feature map are more accurate.

[0116] In one embodiment, the linear scaling program penetration model subsystem includes:

[0117] The activation value linear scaling visualization subsystem linearly scales the activation value of each channel of the multi-channel matrix feature map to the range of 0 to 255 and converts it into a visual color image;

[0118] The channel combination program penetration creates a subsystem, selects multiple channels of the multi-channel matrix feature map, combines multiple channels to obtain multi-color image features, visualizes the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model;

[0119] The feature map storage encoding streaming subsystem sequentially encodes and saves all feature map visualization results as continuously encoded visualization image files; the continuously encoded visualization image files are output continuously to form a continuous dynamic flow of visualization image output.

[0120] The principle and effect of the above technical solution are as follows: adopting the linear scaling program penetration model and the activation value linear scaling visualization subsystem, linearly scaling the activation value of each channel of the multi-channel matrix feature map to the range of 0 to 255, and converting it into a visualized color image; the channel combination program penetration creation subsystem selects multiple channels of the multi-channel matrix feature map, combines multiple channels to obtain multi-color image features, and creates a pathological convolutional neural network program penetration visualization model through program penetration visualization of the internal process of the convolutional neural network; the feature map storage encoding streaming subsystem sequentially encodes and saves all feature map visualization results as continuously encoded visualization image files; the continuously encoded visualization image files are output through continuity to form a continuous dynamic flow of visualization image output; the dynamic visibility of the image output is significantly improved, and the continuity is greatly improved.

[0121] In one embodiment, the visualization model target convolution output subsystem includes:

[0122] The gland pathology image data acquisition subsystem acquires the gland pathology image data set to be predicted by acquiring the gland pathology image to be predicted; the gland pathology image data set to be predicted includes: normal gland pathology images, pathology images of the junction between normal gland and tumor, and multiple types of tumor gland pathology images;

[0123] The visualization model image input subsystem inputs the gland pathology image dataset to be predicted into the pathology convolutional neural network program through the visualization model;

[0124] The target convolution feature map automatic output subsystem penetrates the visualization model through the pathological convolution neural network program and automatically outputs the target convolution layer feature map.

[0125] The principle and effect of the above technical solution are as follows: using the visualization model target convolution output, the gland pathology image data acquisition subsystem acquires the gland pathology image data set to be predicted by acquiring the gland pathology image to be predicted; the gland pathology image data set to be predicted includes: normal gland pathology images, pathology images of the junction between normal gland and tumor, and multiple types of tumor gland pathology images; the visualization model image input subsystem inputs the gland pathology image data set to be predicted into the pathology convolution neural network program penetrating the visualization model; the target convolution feature map automatic output subsystem penetrates the visualization model through the pathology convolution neural network program and automatically outputs the target convolution layer feature map; the pathology convolution neural network program penetrates the visualization model and automatically outputs the target convolution layer feature map. The output target convolutional layer feature map includes: from the bottom first convolutional layer of the pathological convolutional neural network program penetrating the visualization model, to the bottom second convolutional layer and the bottom third convolutional layer, the number of convolutional network layers is gradually deepened until the deepest layer of the target convolutional layer, and the target convolutional layer is output layer by layer; the bottom first convolutional layer extracts the gland visualization edge and visualization color information from the gland pathology image to be predicted; the bottom second convolutional layer outputs the gland visualization edge and visualization color information enhancement data; the bottom third convolutional layer outputs the gland visualization gland morphology of the feature map until the deepest layer of the target convolutional layer, outputs the gland visualization gland shape, gland visualization cell nucleus and gland visualization cell morphology; the output accuracy and efficiency of the target convolutional layer feature map are significantly enhanced.

[0126] In one embodiment, the gland intelligent identification and tumor prediction subsystem includes:

[0127] The convolution layer layer-by-layer recognition model subsystem uses the convolution layer layer-by-layer recognition to create a multi-class tumor change intelligent recognition model based on the target convolution layer feature map; the multi-class tumor change intelligent recognition model is a convolution layer layer-by-layer recognition model;

[0128] The visualized gland layer-by-layer recognition subsystem uses multiple types of tumor intelligent recognition models to perform layer-by-layer recognition of the target convolutional layer feature map to obtain the visualized gland layer-by-layer recognition results;

[0129] The multi-type tumor change intelligent prediction subsystem explores the internal process of the multi-type tumor change intelligent recognition model based on the gland visualization gland shape, gland visualization cell nucleus and gland visualization cell morphology recognition results, and intelligently predicts the tumor change trend of gland pathological images.

[0130] The principle and effect of the above technical solution are as follows: using gland intelligent recognition tumor prediction, the convolution layer layer-by-layer recognition model subsystem, according to the target convolution layer feature map, using the convolution layer layer-by-layer recognition, to create a multi-class tumor intelligent recognition model; the multi-class tumor intelligent recognition model is a convolution layer layer-by-layer recognition model; the visualized gland layer-by-layer recognition subsystem, through the multi-class tumor intelligent recognition model, performs the target convolution layer layer-by-layer recognition on the target convolution layer feature map, and obtains the visualized gland layer-by-layer recognition result; the multi-class tumor intelligent prediction subsystem, according to the gland visualized gland shape, gland visualized cell nucleus and gland visualized cell morphology recognition results, explores the internal process of the multi-class tumor intelligent recognition model, and intelligently predicts the tumor trend of the gland pathological image; through the multi-class tumor intelligent recognition Model, target convolution layer feature map is identified layer by layer, and the visualized gland recognition results are obtained layer by layer, including: through multiple types of tumor intelligent recognition models, the visualized edge and visualized color information extracted by the first convolution layer of the bottom layer from the gland pathology image to be predicted are identified; the visualized edge and visualized color information enhancement data of the second convolution layer of the bottom layer are identified; the visualized gland morphology of the gradually deepened convolution network layer feature map of the third convolution layer of the bottom layer is identified until the output of the deepest convolution network layer feature map of the target convolution layer is identified, the visualized gland shape of the gland, the visualized cell nucleus of the gland and the visualized cell morphology of the gland are identified, and the recognition results of the visualized gland shape, the visualized cell nucleus of the gland and the visualized cell morphology of the gland are obtained; the recognition is more accurate and comprehensive.

[0131] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A visualization method for pathological convolutional neural networks, characterized in that: include: S100, sending multiple types of tumor pathology images into the target convolution layer of the CNN convolutional neural network model through forward propagation, recording and extracting the output information of the feature map, and obtaining a multi-channel matrix feature map; S200, linearly scales and visualizes the activation value of each channel of the multi-channel matrix feature map into a color image, selects a channel or combines multiple channels, visualizes the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model; S300, inputting the gland pathology image dataset to be predicted into the pathology convolutional neural network program penetration visualization model, and automatically outputting the target convolutional layer feature map; S400, based on the target convolution layer feature map, uses a multi-class tumor intelligent recognition model to perform target convolution layer layer-by-layer recognition on the target convolution layer feature map, explores the internal process of the multi-class tumor intelligent recognition model through program penetration, and intelligently predicts the tumor trend of glandular pathology images.

2. A visualization method for a pathological convolutional neural network according to claim 1, characterized in that: S100 includes: S101, select the target convolution layer in the trained CNN convolutional neural network model; S102, sending multiple types of tumor pathology images into the target convolution layer of the CNN convolutional neural network model through forward propagation; S103, in each target convolution layer, record and extract output information of the feature map to obtain a multi-channel matrix feature map; the target convolution layer includes: a partial selection convolution layer or a full selection convolution layer; the multiple types of tumor pathology images include: intestinal adenoma pathology images, thyroid tumor pathology images or pancreatic tumor pathology images.

3. The method for visualizing a pathological convolutional neural network according to claim 1, characterized in that: S200 includes: S201, linearly scaling each channel activation value of the multi-channel matrix feature map to a range of 0 to 255, and converting it into a visual color image; S202, selecting multiple channels of a multi-channel matrix feature map, combining the multiple channels to obtain multi-color image features, and creating a pathological convolutional neural network program penetration visualization model by program penetration visualization of the internal process of the convolutional neural network; S202, sequentially encode all feature map visualization results and save them as continuously encoded visualization image files; output the continuously encoded visualization image files in a continuous manner to form a visualization image output continuous dynamic stream.

4. The method for visualizing a pathological convolutional neural network according to claim 1, characterized in that: S300 includes: S301, acquiring a gland pathology image dataset to be predicted by collecting the gland pathology image to be predicted; the gland pathology image dataset to be predicted includes: normal gland pathology images, pathology images of the junction between normal gland and tumor, and multiple types of tumor gland pathology images; S302, inputting the gland pathology image dataset to be predicted into the pathology convolutional neural network program penetration visualization model; S303, penetrating the visualization model through the pathological convolutional neural network program, and automatically outputting the target convolutional layer feature map.

5. The method for visualizing a pathological convolutional neural network according to claim 1, characterized in that: S400 includes: S401, according to the target convolution layer feature map, using the convolution layer to identify layer by layer, to create a multi-class tumor change intelligent recognition model; the multi-class tumor change intelligent recognition model is a convolution layer layer by layer recognition model; S402, using a multi-class tumor intelligent recognition model, performing target convolution layer recognition on the target convolution layer feature map layer by layer to obtain a visualized gland layer by layer recognition result; S402, based on the recognition results of the visualized gland shape, the visualized cell nucleus and the visualized cell morphology of the gland, the internal process of the intelligent recognition model of multiple types of neoplasia is explored, and the trend of neoplasia of the gland pathological image is intelligently predicted.

6. A visualization system for pathological convolutional neural networks, characterized in that: include: The multi-channel matrix feature map extraction subsystem sends multiple types of tumor pathology images to the target convolution layer of the CNN convolutional neural network model through forward propagation, records and extracts the output information of the feature map, and obtains the multi-channel matrix feature map; The linear scaling program penetration model subsystem visualizes the activation value of each channel of the multi-channel matrix feature map as a color image by linear scaling. It selects channels or combines multiple channels to visualize the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model. The visualization model target convolution output subsystem inputs the glandular pathology image dataset to be predicted into the pathology convolution neural network program to penetrate the visualization model and automatically outputs the target convolution layer feature map; The gland intelligent recognition and tumor prediction subsystem uses a multi-class tumor intelligent recognition model to perform layer-by-layer recognition of the target convolution layer feature map based on the target convolution layer feature map. It penetrates the program to explore the internal process of the multi-class tumor intelligent recognition model and intelligently predicts the tumor trend of gland pathological images.

7. A visualization system for pathological convolutional neural networks according to claim 6, characterized in that: The multi-channel matrix feature map extraction subsystem includes: The target convolutional layer selection subsystem selects the target convolutional layer in the trained CNN convolutional neural network model; The image target convolution layer is passed into the subsystem, and multiple types of tumor pathology images are sent to the target convolution layer of the CNN convolutional neural network model through forward propagation; The feature map output extraction subsystem records and extracts the output information of the feature map in each target convolution layer to obtain a multi-channel matrix feature map; the target convolution layer includes: a partial selection convolution layer or a full selection convolution layer; multiple types of tumor pathology images include: intestinal adenoma pathology images, thyroid tumor pathology images or pancreatic tumor pathology images.

8. A visualization system for pathological convolutional neural networks according to claim 6, characterized in that: The linear scaling program penetration model subsystem includes: The activation value linear scaling visualization subsystem linearly scales the activation value of each channel of the multi-channel matrix feature map to the range of 0 to 255 and converts it into a visual color image; The channel combination program penetration creates a subsystem, selects multiple channels of the multi-channel matrix feature map, combines multiple channels to obtain multi-color image features, visualizes the internal process of the convolutional neural network through program penetration, and creates a pathological convolutional neural network program penetration visualization model; The feature map storage encoding streaming subsystem sequentially encodes and saves all feature map visualization results as continuously encoded visualization image files; the continuously encoded visualization image files are output continuously to form a continuous dynamic flow of visualization image output.

9. A visualization system for pathological convolutional neural networks according to claim 6, characterized in that: The visualization model target convolution output subsystem includes: The gland pathology image data acquisition subsystem acquires the gland pathology image data set to be predicted by acquiring the gland pathology image to be predicted; the gland pathology image data set to be predicted includes: normal gland pathology images, pathology images of the junction between normal gland and tumor, and multiple types of tumor gland pathology images; The visualization model image input subsystem inputs the gland pathology image dataset to be predicted into the pathology convolutional neural network program through the visualization model; The target convolution feature map automatic output subsystem penetrates the visualization model through the pathological convolution neural network program and automatically outputs the target convolution layer feature map.

10. A visualization system for pathological convolutional neural networks according to claim 6, characterized in that: The gland intelligent identification and tumor prediction subsystem includes: The convolution layer layer-by-layer recognition model subsystem uses the convolution layer layer-by-layer recognition to create a multi-class tumor change intelligent recognition model based on the target convolution layer feature map; the multi-class tumor change intelligent recognition model is a convolution layer layer-by-layer recognition model; The visualized gland layer-by-layer recognition subsystem uses multiple types of tumor intelligent recognition models to perform layer-by-layer recognition of the target convolutional layer feature map to obtain the visualized gland layer-by-layer recognition results; The multi-type tumor change intelligent prediction subsystem explores the internal process of the multi-type tumor change intelligent recognition model based on the gland visualization gland shape, gland visualization cell nucleus and gland visualization cell morphology recognition results, and intelligently predicts the tumor change trend of gland pathological images.

Images