A hyperspectral image-based benign and malignant tumor identification system

Abstract

The application provides a malignant and benign tumor recognition system based on a hyperspectral image, and relates to the technical field of medical image processing.The system comprises: a feature extraction module, which is used for performing sharpening processing on an acquired hyperspectral image and extracting an edge feature map of the hyperspectral image after the sharpening processing; a feature fusion module, which is used for performing fusion of different frequency band features of the hyperspectral image and the edge feature map by using a Laplacian pyramid image fusion algorithm to obtain a tumor hyperspectral feature enhancement map; and a recognition module, which is used for inputting the tumor hyperspectral feature enhancement map into a pre-trained malignant and benign tumor recognition model, outputting tumor boundary features, and determining a tumor category of the hyperspectral image according to the tumor boundary features.Through the system, more abundant tumor edge information can be extracted, the tumor recognition performance in the case that the malignant tumor is not obvious is strengthened, and thus the accuracy of the malignant and benign tumor recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image processing, and particularly relates to a benign and malignant tumor recognition system based on hyperspectral images. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art that is already known to those skilled in the art.

[0003] The gold standard for benign and malignant detection of tumors is pathological examination, which is performed by observing small tumor tissues removed under a microscope or using immunohistochemical staining. Generally, the edge features and morphological features between malignant tumors, benign tumors and normal cells in the human body are the key to judging the benignity and malignancy of tumors. First, according to the morphology of the tumor, the malignant degree is higher when the tumor has burrs and an uneven surface, and the benign tumor presents a circular or elliptical shape. Second, the relationship with the surrounding tissue, the benign tumor has a capsule distribution, so the boundary with the surrounding tissue is quite clear, and the malignant tumor is different. Because the proliferation rate of cancer cells is very fast, the tumor grows rapidly in the body, and the surface has no capsule distribution, so the morphology of the malignant tumor is irregular and the boundary with the surrounding tissue is not clear. Finally, whether the tumor has metastasis and complications, the benign tumor is not easy to infiltrate and metastasize because of the capsule.

[0004] The pathological biopsy for tumor benignity and malignancy detection usually takes three days to obtain the results, and the time is relatively long. The existing benign and malignant recognition method based on tumor edges is prone to feature loss in the edge extraction process, and thus can only recognize malignant tumors when the performance is obvious. When the survival of the tumor is between benign and malignant, the recognition accuracy is relatively low. SUMMARY

[0005] In order to solve the above problems, the present application provides a benign and malignant tumor recognition system based on hyperspectral images, which extracts more rich tumor edge information by sharpening the high light image and fusing different frequency band features, and enhances the tumor recognition performance in the case of non-obvious malignant tumors.

[0006] In order to achieve the above purpose, the present application mainly includes the following contents:

[0007] The present application provides a benign and malignant tumor recognition system based on hyperspectral images, comprising:

[0008] The feature extraction module is configured to sharpen the acquired hyperspectral image, and extract the edge feature map of the sharpened hyperspectral image;

[0009] a feature fusion module configured to fuse the hyperspectral image and the edge feature map by using a Laplacian pyramid image fusion algorithm to obtain a tumor hyperspectral feature enhanced image;

[0010] a recognition module configured to input the tumor hyperspectral feature enhanced image into a pre-trained benign and malignant tumor recognition model to output tumor boundary features, and determine a tumor category of the hyperspectral image according to the tumor boundary features, wherein the tumor category includes a malignant tumor and a benign tumor.

[0011] In a possible implementation, the feature extraction module is specifically configured to extract high-frequency part features of the hyperspectral image by using a high-pass filter, and generate an edge feature map according to the high-frequency part features, wherein the high-frequency part features include edge features and morphological features of the tumor.

[0012] In a possible implementation, a second-order Butterworth high-pass filter is used to sharpen the hyperspectral image.

[0013] In a possible implementation, the feature fusion module is specifically configured to:

[0014] Laplacian pyramids of the hyperspectral image and the edge feature map are constructed respectively, wherein the Laplacian pyramids have a preset number of layers, and each layer of image is an image obtained by downsampling a previous layer of image by Gaussian blurring;

[0015] For each layer of the Laplacian pyramids, a preset fusion formula is used to fuse the hyperspectral image and the edge feature map;

[0016] The fused Laplacian pyramids are used to reconstruct a tumor hyperspectral feature enhanced image.

[0017] In a possible implementation, the fusion formula includes a maximum operation, a minimum operation, an average value operation, and a region weighting operation.

[0018] In a possible implementation, the fused Laplacian pyramids are used to reconstruct images from top to bottom layer by layer starting from the top layer by using an interpolation method, and a tumor hyperspectral feature enhanced image is output.

[0019] In a possible implementation, the benign and malignant tumor recognition model is constructed based on a residual attention network.

[0020] The residual attention network is stacked by multiple attention modules, and each attention module includes a mask branch and a main branch. The main branch processes features to obtain a feature map. The mask branch uses a combination of bottom-up and top-down attention to learn a mask with the same size as the output of the main branch, which is used to select features of the feature map.

[0021] In a possible implementation, the constructed benign and malignant tumor recognition model is trained by using the tumor hyperspectral feature enhancement pattern sample with a label, and the related parameters of the mask branch and the trunk branch are adjusted to obtain the trained benign and malignant tumor recognition model.

[0022] The above one or more technical solutions have the following beneficial effects:

[0023] (1) The application provides a benign and malignant tumor recognition system based on a hyperspectral image, which comprises a feature extraction module, a feature fusion module and a recognition module, edge feature maps are obtained by sharpening the hyperspectral image, and different band features of the edge feature maps and the hyperspectral image are fused, so that the problem of feature loss in the feature extraction process can be avoided, and more rich tumor edge information can be extracted. Further, the tumor class is recognized according to the fused tumor hyperspectral feature enhancement pattern, which can enhance the tumor recognition performance in the case that the malignant tumor is not obvious, and improve the accuracy of benign and malignant tumor recognition.

[0024] (2) Compared with the traditional tumor benign and malignant identification gold standard pathological detection method, the recognition speed of the system is fast, and no related chemical reactants are needed in the recognition process, which can save costs to a certain extent and improve the recognition efficiency of benign and malignant tumors. BRIEF DESCRIPTION OF DRAWINGS

[0025] The drawings constituting a part of the specification of the application are used to provide further understanding of the application, and the schematic embodiments of the application and the description thereof are used to explain the application, and do not constitute an improper limitation on the application.

[0026] Figure 1 is one of the structure schematic diagrams of the benign and malignant tumor recognition system based on a hyperspectral image provided by the embodiments of the application;

[0027] Figure 2 is a flowchart of constructing a Laplacian pyramid provided by the embodiments of the application;

[0028] Figure 3 is the second structure schematic diagram of the benign and malignant tumor recognition system based on a hyperspectral image provided by the embodiments of the application. DETAILED DESCRIPTION

[0029] The application will be further described below in combination with the drawings and embodiments.

[0030] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0031] It is to be noted that the terms used herein are merely for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise, and it is further understood that the terms "comprising" and / or "including" when used in this specification, specify the presence of stated features, steps, operations, devices, components and / or combinations thereof.

[0032] As shown in Figure 1 The embodiment of the present application provides a malignant and benign tumor recognition system based on hyperspectral images, which comprises:

[0033] A feature extraction module 110 is configured to perform sharpening processing on the acquired hyperspectral image, and extract an edge feature map of the hyperspectral image after the sharpening processing.

[0034] A feature fusion module 120 is configured to perform fusion of different frequency band features on the hyperspectral image and the edge feature map by using a Laplacian pyramid image fusion algorithm, to obtain a tumor hyperspectral feature enhancement map.

[0035] A recognition module 130 is configured to input the tumor hyperspectral feature enhancement map into a pre-trained malignant and benign tumor recognition model, output tumor boundary features, determine a tumor category of the hyperspectral image according to the tumor boundary features, and the tumor category comprises a malignant tumor and a benign tumor.

[0036] In a specific implementation, the purpose of image sharpening processing is to make the edges, contour lines and details of the image clear, and the edge information of the image is mainly concentrated in the high frequency part of the image. Therefore, the tumor hyperspectral image formed by scanning is first subjected to sharpening processing, and an edge feature map of the hyperspectral image after the sharpening processing is extracted.

[0037] As an optional implementation, the feature extraction module is specifically configured to extract high frequency part features of the hyperspectral image by using a high-pass filter, and generate an edge feature map according to the high frequency part features. The high frequency part features include edge features and morphological features of the tumor. In actual application, a second-order Butterworth high-pass filter can be selected to extract features of the hyperspectral image, and the filter has smooth amplitude-frequency characteristics inside and outside the passband, and can generate a feature extraction map mainly including edge features and morphological features. In this way, the tumor boundary and morphological features can be strengthened, so that the malignant and benign tumor recognition model can better learn the tumor features.

[0038] Edge feature maps sharpen images and highlight edge details, but they may lose low-frequency components, potentially resulting in the loss of essential information. Therefore, a Laplacian pyramid image fusion algorithm is used to fuse features from different frequency bands between the hyperspectral image and the edge feature map, supplementing the original image with relevant features. The completeness of edge feature information ensures a high learning rate for the benign / malignant tumor identification model.

[0039] Specifically, the feature fusion module is used to: construct Laplacian pyramids for hyperspectral images and edge feature maps respectively, wherein the Laplacian pyramid has a preset number of layers, and the image of each layer is an image of the previous layer after Gaussian blur downsampling; for each layer of the Laplacian pyramid, the hyperspectral image and the edge feature image are fused using a preset fusion formula; and the tumor hyperspectral feature enhancement map is reconstructed and output using the fused Laplacian pyramid.

[0040] In practical implementation, firstly, for the hyperspectral image and edge feature image to be fused, a Laplacian pyramid is constructed, such as... Figure 2 As shown, G0 is the original image, G... n It is the next level G n-1 A layered image, Gaussian blurred and downsampled. When constructing the Laplacian pyramid, the top layer L... Pn That is, the topmost level of the Gaussian pyramid, G. n Other layer images L Pn-1 It is G n-1 Image minus G n The interpolated image, the size of the interpolated image is related to G. n-1 same.

[0041] Because the edge feature map and the hyperspectral image are the same size, the entire hyperspectral image is selected as the fusion region. For each layer of the Laplacian pyramid, a preset fusion formula is used to fuse the hyperspectral image and the edge feature image; optionally, the fusion formula includes operations such as taking the larger value, taking the smaller value, averaging, and region weighting. For the fused Laplacian pyramid, Gaussian convolution filtering is performed layer by layer from top to bottom, starting from the top layer. The resulting image of each layer is then added to the corresponding residual image stored in the Laplacian pyramid to recover its corresponding Gaussian pyramid, ultimately yielding the original image G0. This is essentially image reconstruction using interpolation starting from the highest layer.

[0042] The embodiment of the present application fuses the edge feature map and the original tumor hyperspectral image through a Laplacian pyramid image fusion algorithm, because the Laplacian pyramid image fusion algorithm firstly decomposes the image into different spatial frequency bands, and then the fusion process is performed on each spatial frequency layer, the features of each dimension are combined, and the feature expression is enhanced, this fusion method better retains the features of each frequency band of the fused image, and finally forms a tumor hyperspectral feature enhanced image.

[0043] As an optional implementation, in the identification module, the benign and malignant tumor identification model is constructed based on a residual attention network; wherein the residual attention network is stacked by multiple attention modules, each attention module includes: a mask branch and a trunk branch; the trunk branch performs feature processing to obtain a feature map; the mask branch uses a bottom-up and top-down attention combination mode to learn a mask with the same size as the output of the trunk, which is used to select the features of the feature map.

[0044] In a specific implementation, each attention module after superposition generates an attention perception function, which adaptively changes, and the bottom-up and top-down feedforward structure is used to unfold the feedforward and feedback the attention process to the single feedforward process, which can be extended by hundreds of layers, so that this network structure can extract deep features.

[0045] Each attention module is divided into two parts: a mask branch and a trunk branch.

[0046] The trunk branch performs feature processing, and in the embodiment of the present application, the basic unit of the trunk branch is a pre-activated residual learning unit, a generative adversarial network (Generative Adversarial Networks) and an inception module.

[0047] The input of the trunk branch is defined as x, and the output is T(x), the mask branch uses a bottom-up and top-down structure to learn a mask with the same size as the output of the trunk, denoted as M(x), which is a soft weight (soft weight) of T(x). The bottom-up and top-down structure simulates the process of fast feedforward and feedback attention. The output mask is used to control the control gate of the trunk branch, which is similar to the Highway Network. The output feature map H of the attention module is:

[0048] H i，c (x)=M i，c (x)*T i，c (x)

[0049] Wherein, i is a spatial position, c is the subscript of the channel, and the whole architecture is trained end-to-end. Finally, the pre-training related results are obtained based on the tumor hyperspectral feature enhancement map. The attention module can be combined with the latest deep network structure in an end-to-end training manner, and the network depth can be easily expanded to hundreds of layers, and the tumor boundary features can be further extracted by using the characteristics thereof.

[0050] As an optional embodiment, the constructed benign and malignant tumor recognition model is trained by using the tumor hyperspectral feature enhancement map sample with labels, the related parameters of the mask branch and the trunk branch are adjusted, and a trained benign and malignant tumor recognition model is obtained.

[0051] In a specific implementation, the training process of the benign and malignant tumor recognition model first is data preparation, sets a reasonable label system, labels comprehensive and balanced data, second is to sort the data set, and divide the samples to generate a training set and a test set, and finally generates a corresponding format and verifies.

[0052] The parameters of the benign and malignant tumor recognition model are adjusted through the pre-training process and the obtained results, reasonable initialization weights w and biases b are set, and the learning rate is gradually optimized, as shown in the formula (1). Figure 3 As shown in the formula (1), the recognition module can be divided into a tumor hyperspectral image benign and malignant identification pre-training module and a tumor hyperspectral image benign and malignant identification parameter adjustment module. The model parameter adjustment module adopts the pre-training model based on the tumor hyperspectral feature enhancement map in the above steps, and adjusts the parameters of the model mask branch, the parameters of the trunk branch and other related parameters through the pre-training model results, so as to improve the model performance, the noise resistance, obtain more accurate prediction results, strengthen the judgment of the non-obvious malignant tumor, and finally determine the tumor category of the hyperspectral image according to the tumor boundary features. The tumor category includes malignant tumor and benign tumor.

[0053] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A benign and malignant tumor identification system based on hyperspectral images, characterized in that, include: The feature extraction module is used to sharpen the acquired hyperspectral image and extract the edge feature map of the sharpened hyperspectral image. The feature fusion module is used to fuse the hyperspectral image and edge feature map with features of different frequency bands using the Laplacian pyramid image fusion algorithm to obtain a tumor hyperspectral feature enhancement map; The feature fusion module is specifically used for: Laplacian pyramids are constructed for hyperspectral images and edge feature maps, respectively. The Laplacian pyramids have a preset number of layers, and the image of each layer is an image of the previous layer after Gaussian blur downsampling. For each layer of the Laplacian pyramid, a preset fusion formula is used to fuse the hyperspectral image and the edge feature image; The fused Laplacian pyramid is used to reconstruct and output an enhanced map of tumor hyperspectral features. The fusion formula includes operations such as taking the largest value, taking the smallest value, calculating the average value, and regional weighting. The identification module is used to input the enhanced hyperspectral feature map of the tumor into a pre-trained benign and malignant tumor identification model, output tumor boundary features, and determine the tumor category of the hyperspectral image based on the tumor boundary features. The tumor category includes malignant tumors and benign tumors.

2. The benign and malignant tumor identification system based on hyperspectral images as described in claim 1, characterized in that, The feature extraction module is specifically used to: extract the high-frequency features of the hyperspectral image using a high-pass filter, and generate an edge feature map based on the high-frequency features; the high-frequency features include the edge features and morphological features of the tumor.

3. The benign and malignant tumor identification system based on hyperspectral images as described in claim 2, characterized in that, The hyperspectral image is sharpened using a second-order Butterworth high-pass filter.

4. The benign and malignant tumor identification system based on hyperspectral images as described in claim 1, characterized in that, For the fused Laplacian pyramid, image reconstruction is performed layer by layer from top to bottom using interpolation, and the output is a tumor hyperspectral feature enhancement map.

5. The benign and malignant tumor identification system based on hyperspectral images as described in claim 1, characterized in that, The benign and malignant tumor identification model is constructed based on a residual attention network. The residual attention network is composed of multiple stacked attention modules. Each attention module includes a mask branch and a trunk branch. The trunk branch performs feature processing to obtain a feature map. The mask branch learns a mask of the same size as the trunk output by using a combination of bottom-up and top-down attention methods, which is used to select features from the feature map.

6. The benign and malignant tumor identification system based on hyperspectral images as described in claim 1, characterized in that, The benign and malignant tumor identification model was trained using labeled hyperspectral feature enhancement map samples. The relevant parameters of the mask branch and the main branch were adjusted to obtain the trained benign and malignant tumor identification model.

Images