A Hyperspectral Microscopic Imaging Optimization Method Applicable to Tumor Diagnosis

Abstract

The present invention discloses a hyperspectral microscopic imaging optimization method applicable to tumor diagnosis, including classifying the features of the input image based on depth pixels and outputting a classification result; preliminarily optimizing the classification result by using a consistency initial registration strategy; constructing an optimization model in combination with a multi-objective optimization strategy, re-optimizing and solving the preliminarily optimized result, and setting the finally optimized value output as the threshold of the output image. Through special feature classification processing means, the present invention obtains key feature descriptors, combines double optimization technologies, ensures the accuracy of key information in image processing, reduces unnecessary complex calculations while retaining key features, improves the imaging speed and accuracy, and provides an ideal help for medical services.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral microscopy imaging and image optimization, and particularly to a hyperspectral microscopy imaging optimization method applicable to tumor diagnosis. Background Art

[0002] Currently, in the diagnosis of tumors, especially liver cancer, a malignant tumor, histological diagnosis of pathological tissues remains the gold standard. Although existing microendoscopes, electronic endoscopes, ultrasonic endoscopes, etc. can provide doctors with clearer images and reduce the pain of patients, it is difficult to detect early lesions only through images, which makes early biopsy random.

[0003] Spectral imaging technology is a combination of imaging technology and spectral technology, which can simultaneously image the same object to be measured in a wide continuous spectral band. While detecting the spatial characteristics of the object, each spatial pixel is dispersed to form dozens to hundreds of bands for imaging, so as to provide spatial domain information and spectral domain information, that is, "the combination of image and spectrum". This cutting-edge technology has been well used in many fields such as military reconnaissance, resource exploration, natural disaster monitoring, and environmental pollution assessment. Although existing spectral imaging technologies have been developed in the field of medical diagnosis and treatment, the effects are not ideal. Their imaging results are relatively blurred, and the operation is complex, making it difficult to provide good services for medical technology. Summary of the Invention

[0004] The purpose of this part is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Simplifications or omissions may be made in this part, as well as in the abstract and title of the specification of this application, to avoid obscuring the purpose of this part, the abstract, and the title. However, such simplifications or omissions shall not be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: including classifying the features of the input image based on depth pixels and outputting a classification result; using a consistency initial registration strategy to preliminarily optimize the classification result; constructing an optimization model by combining a multi-objective optimization strategy, re-optimizing and solving the preliminarily optimized result, and setting the output final optimization value as the threshold of the output image.

[0007] As a preferred solution of the hyperspectral microscopy imaging optimization method applicable to tumor diagnosis according to the present invention, wherein: the classification includes, for a test pixel, classifying the pixel pair composed of the central pixel and each surrounding pixel by a trained CNN, and determining the final label through a voting strategy.

[0008] As a preferred embodiment of the hyperspectral microscopic imaging optimization method for tumor diagnosis according to the present invention, further comprising:

[0009] Image grayscale conversion;

[0010] Using the Gamma correction method to standardize the color space of the input image;

[0011] Calculating the gradient of each pixel in the image;

[0012] Dividing the image into small cells;

[0013] Statistical gradient histograms for each cell, thereby forming descriptors for each cell;

[0014] Combining several cells into a block, and concatenating the feature descriptors of all cells within a block to obtain the HOG feature descriptor of the block;

[0015] Concatenating the HOG feature descriptors of all cells within the image image to obtain the HOG feature descriptor of the target to be detected, that is, the final feature vector available for classification.

[0016] As a preferred embodiment of the hyperspectral microscopic imaging optimization method for tumor diagnosis according to the present invention, wherein: the preliminary optimization includes registration based on feature descriptors and fast point feature histograms.

[0017] As a preferred embodiment of the hyperspectral microscopic imaging optimization method for tumor diagnosis according to the present invention, wherein: the registration includes,

[0018] Finding three feature element values between the query point and each point within its k-neighborhood for the feature descriptor, and summarizing to obtain the SPFH;

[0019] Respectively determining the k-neighborhood for each point in the k-neighborhood, and forming their own SPFH according to the above steps;

[0020] Weighted statistics to generate a final 33-dimensional feature vector, generating a histogram for each feature dimension, and finally connecting them together.

[0021] As a preferred embodiment of the hyperspectral microscopic imaging optimization method for tumor diagnosis according to the present invention, wherein: the preliminary optimization further includes,

[0022] Calculating the feature descriptor and the fast point feature histogram;

[0023] SAC registration, where the sampling record for registration should be greater than a set threshold to ensure that the sampling points have different features, and using Huber as the penalty function;

[0024] Perform fine registration.

[0025] As a preferred embodiment of the hyperspectral microscopic imaging optimization method for tumor diagnosis according to the present invention, wherein: the penalty function includes,

[0026]

[0027]

[0028] wherein, m l is a pre-given value, i.e., a threshold, and l i is the distance difference after transformation of the corresponding points in the i-th group.

[0029] As a preferred embodiment of the hyperspectral microscopic imaging optimization method for tumor diagnosis according to the present invention, wherein: the fine registration includes finding a set of optimal transformations among all transformations such that the value of the error function is minimized, and the transformation at this time is the final registration transformation matrix.

[0030] As a preferred embodiment of the hyperspectral microscopic imaging optimization method for tumor diagnosis according to the present invention, wherein: constructing the optimization model includes,

[0031] Selecting a radial basis function as the objective function of the optimization model, as follows:

[0032]

[0033] wherein, x = {x1; x2; …; x 14}: the registration transformation matrix, y: the amplitude-frequency characteristic vector of the feature descriptor, and σ: the target vector, i.e., the distribution or range characteristic of the feature descriptor.

[0034] As a preferred embodiment of the hyperspectral microscopic imaging optimization method for tumor diagnosis according to the present invention, wherein: the optimization model needs to be trained in advance, including,

[0035] Initializing the penalty parameter and the target vector, and training and testing the optimization model using the feature descriptor;

[0036] If the optimization model does not meet the accuracy threshold requirement, then assign and optimize the penalty parameter and the target vector according to the error;

[0037] Until the accuracy threshold requirement is met, output the optimization model.

[0038] Advantages of the present invention: Through special feature classification processing means, the present invention obtains key feature descriptors, and combines with double optimization technology to ensure the accuracy of key information in image processing. On the basis of retaining key features, unnecessary complex calculations are reduced, the imaging speed and accuracy are improved, which provides an ideal help for medical services, avoids the harm to the human body caused by using fluorescent contrast agents, and has a high diagnostic accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic flowchart of a hyperspectral microscopic imaging optimization method applicable to tumor diagnosis according to an embodiment of the present invention;

[0040] Figure 2 It is a classification schematic diagram based on depth pixel pair features of a hyperspectral microscopic imaging optimization method applicable to tumor diagnosis according to an embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of a test image before registration of a hyperspectral microscopic imaging optimization method applicable to tumor diagnosis according to an embodiment of the present invention;

[0042] Figure 4 It is a schematic diagram of a test image after registration of a hyperspectral microscopic imaging optimization method applicable to tumor diagnosis according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0044] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0045] It should be understood that in various embodiments of the present invention, the magnitude of the serial numbers of the various processes does not mean the order of execution, and the order of execution of the various processes should be determined by their functions and internal logics, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0046] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0047] It should be understood that in the present invention, "a plurality of" means two or more. "And / or" is merely a correlative relationship describing related objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "Including A, B, and C" and "including A, B, C" mean that all of A, B, and C are included. "Including A, B, or C" means including any one of A, B, and C. "Including A, B, and / or C" means including any one, any two, or all three of A, B, and C.

[0048] It should be understood that in the present invention, "B corresponding to A", "B corresponding to A relatively", "A corresponding to B relatively", or "B corresponding to A relatively" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information. The matching of A and B means that the similarity between A and B is greater than or equal to a preset threshold.

[0049] Depending on the context, as used herein, "if" can be interpreted as "when", "while", "in response to determination", or "in response to detection".

[0050] The technical solution of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0051] Embodiment 1

[0052] Referring to Figures 1 to 4 , a specific implementation provided by an embodiment of the present invention specifically includes:

[0053] S1: Classify the input image features based on depth pixels and output a classification result. It should be noted that the classification includes:

[0054] For test pixels, the trained CNN classifies the pixel pairs composed of the central pixel and each surrounding pixel, and determines the final label through a voting strategy.

[0055] Specifically, it further includes:

[0056] Image grayscale conversion (regard the image as a three-dimensional image of x, y, and z (grayscale));

[0057] Use the Gamma correction method to standardize (normalize) the color space of the input image; the purpose is to adjust the contrast of the image, reduce the influence caused by local shadows and lighting changes in the image, and at the same time suppress the interference of noise;

[0058] Calculate the gradient (including magnitude and direction) of each pixel in the image;

[0059] Divide the image into small cells (e.g., 6*6 pixels / square);

[0060] Statistically calculate the gradient histogram (the number of different gradients) of each cell, and thus form the descriptor of each cell;

[0061] Group every few cells into a block (e.g., 3*3 squares / block), and concatenate the feature descriptors of all cells within a block to obtain the HOG feature descriptor of the block;

[0062] Concatenate the HOG feature descriptors of all blocks in the image image to obtain the HOG feature descriptor of the image (the target to be detected), that is, the final feature vector available for classification.

[0063] S2: Use the consistency initial registration strategy to preliminarily optimize the classification result. It should be noted in this step that the preliminary optimization includes registration based on feature descriptors and fast point feature histograms;

[0064] Registration includes:

[0065] Find the three feature element values between the feature descriptor of the query point and each point within its k-neighborhood, and summarize to obtain the SPFH;

[0066] Respectively determine the k-neighborhood for each point in the k-neighborhood, and form their own SPFH according to the above steps;

[0067] Weighted statistics generate the final 33-dimensional feature vector, generate a histogram for each feature dimension, and finally connect them together.

[0068] Furthermore, the preliminary optimization also includes:

[0069] Calculate the feature descriptor and the fast point feature histogram;

[0070] SAC registration, the sampling record of the registration should be greater than the set threshold to ensure that the sampling points have different features, and use Huber as the penalty function;

[0071] Perform fine registration.

[0072] Specifically, the penalty function includes:

[0073]

[0074]

[0075] Among them, m l is a pre-given value, that is, a threshold, and l i is the distance difference after transformation of the corresponding points in the i-th group.

[0076] Fine registration includes finding an optimal set of transformations among all transformations to minimize the value of the error function. The transformation at this time is the final registration transformation matrix.

[0077] S3: Construct an optimization model by combining a multi-objective optimization strategy, re-optimize and solve the preliminary optimization result, and set the output final optimization value as the threshold of the output image. It should also be noted that constructing the optimization model includes:

[0078] Select a radial basis function as the objective function of the optimization model, as follows:

[0079]

[0080] Among them, x = {x1; x2; …; x 14}: Registration transformation matrix, y: Amplitude-frequency characteristic vector of the feature descriptor, σ: Target vector, that is, the distribution or range characteristic of the feature descriptor.

[0081] The optimization model needs to be trained in advance, including:

[0082] Initialize the penalty parameter and the target vector, and use the feature descriptor to train and test the optimization model;

[0083] If the optimization model does not meet the accuracy threshold requirement, assign and optimize the penalty parameter and the target vector according to the error;

[0084] Until the accuracy threshold requirement is met, output the optimization model.

[0085] Referring to Figure 2 , when the number of training samples is large enough, the method of the present invention can provide good hyperspectral image classification performance. Its advantage lies in realizing the augmentation of training samples, ensuring that the advantages of CNN can be truly exerted, and the method of the present invention uses deep CNN to learn pixel pair features, having stronger resolution ability.

[0086] It should also be noted in this embodiment that the feature descriptor - based registration method is less used in medical image software. Instead, more often, some feature points are manually selected first to complete rough registration, and there has never been a registration calculation carried out in the manner provided in this embodiment. Moreover, this embodiment provides a penalty function for fine registration to reduce the registration error, which is a great breakthrough for medical imaging (tumor diagnosis) technology.

[0087] Embodiment 2

[0088] Furthermore, the existing image feature learning methods aim to automatically learn data - adaptive image representations from the original pixel image data. However, the existing technologies are poor at extracting and organizing discriminant information from the data. Most learning frameworks use an unsupervised approach and do not consider the information of class labels. Therefore, in this embodiment, encoding shareable information in the existing category groups is proposed, and the discriminant mode has specific class labels during the image feature learning process (that is, special feature processing is performed on the classification proposed in Embodiment 1). Specifically, it includes:

[0089] Establish a multi - layer feature learning framework: deep discriminant and shared feature learning.

[0090] Purpose: The hierarchical learning transform filter bank transforms the pixel values of local image patches into features.

[0091] For each feature learning layer, the purpose is to learn an over - complete filter bank, which may involve differences in blocks of different classes, while maintaining shared correlation and discriminability for each class among similar classes. The goal can be achieved by randomly searching for training blocks, separately learning the filter banks for each class, and then connecting them together.

[0092] Training process:

[0093] (1) Input the original image or the output of the previous layer features, densely extract image patches or local features;

[0094] (2) Select samples for training;

[0095] (3) Perform the new framework training module and learn the filter bank.

[0096] Testing process:

[0097] (a) Apply the learned filter bank W to the original input image or the previous layer features, and densely extract new framework features for the current layer;

[0098] (b) Perform LLC and SPM, transform the local features into a global image representation, and apply a linear SVM for the final classification.

[0099] Preferably, in the global filter bank W, w1, w2, ..., wD represent filters. During the training process, for each layer, it is forced to activate a small subset of filters. Different classes can share the same filters. The new frame feature of an image patch Xi can be expressed as fi = F(WXi).

[0100] To learn an effective filter bank, each class can only activate a subset of the global filters. In addition to reducing the feature dimension, sharing filters can also make the features more robust. Images belong to different classes but share the same information (for example, in an image, both a computer room and an office contain computers and desks); the amount of information sharing depends on the similarity between different classes. Therefore, allowing filters to be shared means that the same filters can be activated by some classes. Thus, a binary selection vector is introduced to adaptively select which filters to share and in which classes.

[0101] It should be recognized that the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method can be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with the computer program, where the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, for this purpose, the program can run on a dedicated integrated circuit programmed for this purpose.

[0102] In addition, the operations of the processes described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by the context. The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed jointly on one or more processors, by hardware, or a combination thereof. The computer program includes multiple instructions executable by one or more processors.

[0103] Further, the method can be implemented in any type of computing platform operatively connected to a suitable one, including but not limited to personal computers, minicomputers, mainframes, workstations, network or distributed computing environments, separate or integrated computer platforms, or communicating with charged particle tools or other imaging devices, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into the computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it can be read by a programmable computer and, when the storage medium or device is read by the computer, can be used to configure and operate the computer to perform the processes described herein. Additionally, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the inventions described herein include these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself. The computer program is capable of applying to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the transformed data represents physical and tangible objects, including a specific visual depiction of the physical and tangible objects generated on a display.

[0104] Finally, it should be noted that: The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: They can still modify the technical solutions described in the foregoing embodiments, or equivalently replace some or all of the technical features therein; And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral microscopic imaging optimization method applicable to tumor diagnosis, characterized in that: Including, Classify the input image features based on depth pixels and output the classification result; The classification includes, for test pixels, classifying the pixel pairs composed of the central pixel and each surrounding pixel by the trained CNN, and determining the final label through a voting strategy; Grayscale the image; Use the Gamma correction method to standardize the color space of the input image; Calculate the gradient of each pixel in the image; Divide the image into small cells; Statistically calculate the gradient histogram of each cell, and a descriptor for each cell can be formed; Group every several cells into a block, and concatenate the feature descriptors of all cells within a block to obtain the HOG feature descriptor of the block; Concatenate the HOG feature descriptors of all cells in the image image to obtain the HOG feature descriptor of the target to be detected, that is, the final feature vector available for classification; Use the consistency initial registration strategy to preliminarily optimize the classification result; The preliminary optimization includes registering based on the feature descriptor and the fast point feature histogram; The registration includes obtaining three feature element values between the feature descriptor and each point within its k-neighborhood for the query point, and summarizing to obtain the SPFH; respectively determine the k-neighborhood for each point in the k-neighborhood, and form their own SPFH according to the above steps; weighted statistics generate the final 33-dimensional feature vector, with a histogram generated for each feature dimension, and finally connected together; Calculate the feature descriptor and the fast point feature histogram; SAC registration, the sampling record of the registration should be greater than the set threshold to ensure that the sampling points have different features, and use Huber as the penalty function for fine registration; the penalty function includes, where m l is a preset value, i.e., a threshold, and l i is the distance difference after the transformation of the corresponding points in the i-th group; Find a set of optimal transformations among all transformations to minimize the value of the error function, and the transformation at this time is the final registration transformation matrix; Combine the multi-objective optimization strategy to construct an optimization model, perform secondary optimization and solution on the preliminary optimization result, and set the output final optimization value as the threshold of the output image; constructing the optimization model includes, Select the radial basis function as the objective function of the optimization model as follows: where x = {x1; x2; …; x 14}: the registration transformation matrix, y: the amplitude-frequency characteristic vector of the feature descriptor, σ: the target vector, i.e., the distribution or range characteristic of the feature descriptor.

2. The hyperspectral microscopic imaging optimization method applicable to tumor diagnosis according to claim 1, wherein: The optimization model needs to be trained in advance, including, Initialize the penalty parameter and the target vector, and use the feature descriptor to train and test the optimization model; If the optimization model does not meet the accuracy threshold requirement, assign and optimize the penalty parameter and the target vector according to the error; Until the accuracy threshold requirement is met, output the optimization model.

Images