Hyperspectrum-based parathyroid gland identification and positioning method and system

Abstract

The invention discloses a parathyroid gland identification and positioning method and system based on hyperspectrum, and relates to the field of parathyroid gland identification, and the method comprises the steps: employing a hyperspectral imaging device, and carrying out the hyperspectral information collection of a panoramic operation field where a parathyroid gland is exposed in a thyroid operation, so as to obtain a parathyroid gland hyperspectral image; after wave band selection is carried out on the parathyroid gland hyperspectral image, the parathyroid gland hyperspectral image is input into a parathyroid gland recognition and positioning model to be recognized and positioned, and parathyroid gland information is obtained; the parathyroid gland information comprises parathyroid gland pixel sites and parathyroid gland categories corresponding to the pixel sites; the parathyroid gland type is parathyroid gland negative or parathyroid gland positive; the parathyroid gland recognition and positioning model is obtained by training a deep learning classification framework by adopting a parathyroid gland sample set. The parathyroid gland recognition precision can be improved, and real-time high-precision positioning can be achieved.

Description

Technical Field

[0001] The present application relates to the field of parathyroid gland identification, and in particular to a method and system for parathyroid gland identification and positioning based on hyperspectral. Background Art

[0002] Intraoperative identification and preservation of parathyroid glands are crucial in thyroid surgery, and postoperative hypoparathyroidism is an important factor affecting the quality of life of patients. Currently, there are many methods for intraoperative identification of parathyroid glands, including naked eye identification (nanocarbon assisted), autofluorescence imaging technology, etc. The above methods have disadvantages such as low accuracy, leakage of injection fluid and contrast agent, and drug allergy.

[0003] Hyperspectral imaging (HSI) has higher spectral resolution than traditional color digital images, and usually contains dozens or hundreds of bands. Rich spectral information can provide a basis for accurate target identification. However, hyperspectral imaging technology has not yet been applied in parathyroid surgery. In addition, clinical medicine still mostly relies on the subjective judgment of surgeons and their own technical level, and the recognition accuracy fluctuates. Summary of the invention

[0004] The purpose of this application is to provide a method and system for parathyroid gland identification and positioning based on hyperspectral, which can improve the recognition accuracy of parathyroid glands and can locate them in real time with high precision.

[0005] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a method for identifying and locating parathyroid glands based on hyperspectral, comprising: A hyperspectral imaging device is used to collect hyperspectral information of the panoramic surgical field that exposes the parathyroid glands during thyroid surgery to obtain a hyperspectral image of the parathyroid glands. After selecting the band of the parathyroid hyperspectral image, the image is input into a parathyroid identification and positioning model for identification and positioning to obtain parathyroid information; Among them, the parathyroid information includes the pixel sites of the parathyroid glands and the parathyroid gland categories corresponding to the pixel sites; the parathyroid gland category is parathyroid-negative or parathyroid-positive; and the parathyroid gland recognition and positioning model is obtained by training a deep learning classification framework using a parathyroid gland sample set.

[0006] In a second aspect, the present application provides a system for parathyroid gland identification and positioning based on hyperspectral, comprising: The hyperspectral image acquisition module is used to: use a hyperspectral imaging device to collect hyperspectral information of the panoramic surgical field that exposes the parathyroid gland during thyroid surgery to obtain a hyperspectral image of the parathyroid gland; The parathyroid gland identification and positioning module is used to: after selecting the band of the parathyroid gland hyperspectral image, input it into the parathyroid gland identification and positioning model for identification and positioning to obtain the parathyroid gland information; Among them, the parathyroid information includes the pixel sites of the parathyroid glands and the parathyroid gland categories corresponding to the pixel sites; the parathyroid gland category is parathyroid-negative or parathyroid-positive; and the parathyroid gland recognition and positioning model is obtained by training a deep learning classification framework using a parathyroid gland sample set.

[0007] According to the specific embodiments provided by the present application, the present application has the following technical effects: the present application provides a method and system for parathyroid identification and positioning based on hyperspectral, collects parathyroid hyperspectral images, selects bands and inputs them into the parathyroid identification and positioning model for identification and positioning to obtain parathyroid information. Based on the above settings, the hyperspectral information of the parathyroid is combined with the spatial features to avoid reliance on the subjective judgment of the physician; and because the parathyroid identification and positioning model used therein is obtained by training the deep learning classification framework using the parathyroid sample set, the accuracy of parathyroid identification and positioning can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0009] Figure 1 A schematic flow chart of a method for parathyroid gland identification and localization based on hyperspectral according to an embodiment of the present application.

[0010] Figure 2 A schematic diagram of band selection for a parathyroid hyperspectral image provided by an embodiment of the present application.

[0011] Figure 3 A schematic diagram of a deep learning classification framework provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The present application provides a method and system for parathyroid gland identification and positioning based on hyperspectral, which combines the spectral characteristics of parathyroid glands in the surgical field with spatial characteristics, has the advantages of real-time positioning and high-precision positioning, and solves the problem of insufficient accuracy in parathyroid gland identification.

[0014] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0015] In an exemplary embodiment, Figure 1 As shown, a method for identifying and locating parathyroid glands based on hyperspectral is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 101 to 102.

[0016] Step 101, using a hyperspectral imaging device, collects hyperspectral information of a panoramic surgical field that exposes the parathyroid glands during thyroid surgery to obtain a hyperspectral image of the parathyroid glands.

[0017] In an application example, the hyperspectral imaging device includes a mobile support, a hyperspectral camera, a hyperspectral data processing and result feedback device, and a halogen lighting source; wherein the mobile support includes a pulley block base (which can be freely moved in an operating room), an operating platform, and two adjustable support arms; the operating platform and the two adjustable support arms are arranged on the pulley block base; the hyperspectral camera is arranged on one of the adjustable support arms, and the halogen lighting source is arranged on the other adjustable support arm; the hyperspectral data processing and result feedback device is mounted on the operating platform, and the hyperspectral data processing and result feedback device can also display a thyroid hyperspectral image and parathyroid information obtained in the subsequent step 102.

[0018] In another specific application, the halogen illumination source is a halogen lamp cold light source with a diameter of 80 cm and 24V / 150W, and the hyperspectral camera is a hyperspectral camera with a spectral range of 400nm-1000nm and a spectral resolution better than 2.8nm.

[0019] In another specific application, step 101 includes: (11) The hyperspectral camera and the halogen illumination source are placed above the panoramic surgical field for exposing the parathyroid gland during thyroid surgery by means of the two adjustable support arms; specifically, after the thyroid gland is exposed during surgery, a fine dissection of the dorsal side of the thyroid gland is performed to separate and expose the parathyroid gland and surrounding tissues in the surgical field, and the hyperspectral imaging device is placed at a height of 30 cm to 40 cm directly above the surgical field, and the position of the light source is adjusted by means of the adjustable support arms so that the camera can be focused.

[0020] (12) Turn on the hyperspectral camera and the halogen illumination source, use the halogen illumination source for irradiation, use the hyperspectral camera to automatically focus and collect multi-band spectral information to obtain an initial hyperspectral image; specifically, the hyperspectral camera automatically focuses and the halogen illumination source adjusts the brightness until the surgical field is clear, and then collects intraoperative image data and spectral feature information to obtain an initial hyperspectral image. In addition, the collected intraoperative surgical field image data refers to the collected spatial features of the parathyroid glands and surrounding tissues, and a pseudo-color RGB image extracted from the hyperspectral cube is generated at the same time, and the band information corresponding to the image is R: 708nm, G: 539nm, B: 479nm.

[0021] (13) The hyperspectral data processing and result feedback device is used to receive the initial hyperspectral image, and the initial hyperspectral image is subjected to black-and-white calibration and normalization processing to avoid interference from ambient light, thereby obtaining a parathyroid hyperspectral image. When the initial hyperspectral image is subjected to black-and-white calibration and normalization processing, the following formula is used: .

[0022] in, I ref is the reflection spectrum intensity value obtained after black and white calibration normalization processing, I Raw is the reflected spectrum intensity value in the initial hyperspectral image, I dark is the dark current reference intensity, I White is the total reflection reference intensity.

[0023] Step 102, after selecting the band of the parathyroid hyperspectral image, input it into the parathyroid identification and positioning model for identification and positioning to obtain parathyroid information. The parathyroid information includes the pixel sites of the parathyroid glands and the parathyroid gland categories corresponding to the pixel sites; the parathyroid gland categories are parathyroid negative or parathyroid positive; the parathyroid identification and positioning model is obtained by training the deep learning classification framework using the parathyroid gland sample set.

[0024] In a specific application, any parathyroid sample in the parathyroid sample set includes a parathyroid hyperspectral sample image and a corresponding true label; the true label is: the surgeon uses ENVI5.6 to label each pixel site in the parathyroid hyperspectral sample image without knowing the patient's basic information and clinical related data, and the parathyroid category corresponding to each pixel site is obtained; wherein 0 can be used to represent negative parathyroid glands, and 1 can be used to represent positive parathyroid glands. In practical applications, when manually labeling the parathyroid contours using ENVI 5.6, labels can also be created as ROI regions of interest for training or testing. The final results of the position positioning and boundary judgment of the parathyroid glands by an expert group composed of senior surgeons and pathologists are used as the criterion for judging the effectiveness of the validation set.

[0025] In addition, before manual labeling, all parathyroid hyperspectral sample images can be screened for quality, and sample images with poor quality such as blur and severe noise can be removed to ensure the accuracy and reliability of the data set.

[0026] After the labels are manually annotated, they are divided into multiple test sample sets and training sample sets according to different proportions using the cross-validation method. It is necessary to ensure that multiple images of the same case do not appear in both the training sample set and the test sample set. That is, all the images contained in the case are either assigned to the training sample set or to the test sample set to avoid data leakage and bias in the evaluation results.

[0027] In the process of training the deep learning classification framework using the training sample set, it is first necessary to select the bands of the parathyroid hyperspectral images contained in the training sample set, and reduce the influence of factors such as sensor interference and light source instability on the model training accuracy by removing high noise bands. Specifically, Figure 2 As shown, the steps for band selection include: (21) The parathyroid hyperspectral image is sequentially downsampled and upsampled by bicubic interpolation. It should be noted that when bicubic interpolation is used to process an image containing a large number of noise points, the downsampling and upsampling processes will cause noise amplification and affect image reconstruction. Based on this, the present application also sets up the following steps (22) and (23).

[0028] (22) Based on the parathyroid hyperspectral image and the upsampled parathyroid hyperspectral image, a structural similarity calculation is performed to obtain the structural similarity of each band.

[0029] (23) Based on a preset structural similarity threshold and the structural similarity of each band, band screening is performed from the parathyroid hyperspectral image to obtain a parathyroid hyperspectral image after band selection. Specifically, bands with high structural similarity are selected to remove noise.

[0030] Performing band selection on parathyroid hyperspectral images as a step before training the network can improve the performance of downstream image processing tasks, reduce the model's sensitivity to noise, improve classification accuracy, and speed up model training and inference.

[0031] In an application example of the present application, Figure 3 As shown, the deep learning classification framework includes a first convolution layer, a first spatial attention layer, a second convolution layer, a third convolution layer, a second spatial attention layer, a fourth convolution layer, a fifth convolution layer, a third spatial attention layer, a sixth convolution layer, a seventh convolution layer, a first semantic space fusion layer, an eighth convolution layer, a second semantic space fusion layer, a ninth convolution layer and a tenth convolution layer, which are sequentially arranged. The spatial features and spectral features in the parathyroid hyperspectral image after band selection are trained through the above deep learning classification framework.

[0032] Among them, the second convolution layer is jump-connected with the second semantic space fusion layer; the fourth convolution layer is jump-connected with the eighth convolution layer; the first convolution layer, the second convolution layer, the fourth convolution layer, the sixth convolution layer and the ninth convolution layer have the same structure; the third convolution layer and the fifth convolution layer have the same structure; the seventh convolution layer and the eighth convolution layer have the same structure; the third convolution layer and the fifth convolution layer are used to perform a down-sampling operation after convolution; the seventh convolution layer and the eighth convolution layer are used to perform an up-sampling operation after convolution.

[0033] In the deep learning classification framework of the present application, a spatial attention layer is set between convolution layers, which can better focus on key spatial positions. The shallow features in the parathyroid hyperspectral image after band selection are extracted through the first convolution layer, the first spatial attention layer, the second convolution layer, the third convolution layer, the second spatial attention layer, the fourth convolution layer, the fifth convolution layer, the third spatial attention layer, and the sixth convolution layer, and then the hyperspectral image is upsampled to the original size through the seventh convolution layer, the first semantic space fusion layer, the eighth convolution layer, the second semantic space fusion layer, the ninth convolution layer, and the tenth convolution layer, and then the loss is calculated according to the real label (that is, the parathyroid category corresponding to the pixel site of the parathyroid obtained by manual annotation) and the parameters are updated by back propagation, and the model training is completed after convergence. The loss function adopts the classic cross entropy loss.

[0034] In another application example, the process of training a deep learning classification framework using a parathyroid sample set includes: inputting the parathyroid sample set into the deep learning classification framework, obtaining a prediction result through forward propagation calculation, calculating the error between the prediction result and the true label, and then updating the parameters of the deep learning classification framework through back propagation, thereby continuously optimizing the performance of the model, improving the recognition and positioning capabilities of the parathyroid glands, and completing the model training after convergence to obtain a parathyroid gland recognition and positioning model. In addition, the stochastic gradient descent method is used to optimize the model during the training process.

[0035] In another application example, the parathyroid gland recognition and localization model is verified using a validation set. During the verification process, the accuracy rate is used to evaluate the performance of the parathyroid gland recognition and localization model in K-fold cross validation, and the neural network model with the highest accuracy rate is the optimal neural network model.

[0036] In another application example, a confusion matrix is ​​made based on the accuracy of the parathyroid gland identification and positioning model in identifying the location of the parathyroid gland, and a ROC curve is drawn (negative image vs. positive image, accurate positioning vs. wrong positioning), and the true positive rate and false positive rate of the doctors participating in the verification are plotted at the corresponding positions in the ROC curve. If the ROC curve of the optimal neural network model surrounds the doctor's result point, it means that the optimal neural network model can basically reach the level of human eyes. The intersection-over-union ratio is used as an objective indicator to measure the accuracy of the neural network model in identifying the range of parathyroid glands.

[0037] Based on the same inventive concept, the embodiment of the present application also provides a system for parathyroid identification and positioning based on hyperspectral. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more system embodiments provided below can refer to the limitations of the method above, and will not be repeated here. The system for parathyroid identification and positioning based on hyperspectral of the present application includes: The hyperspectral image acquisition module is used to: use a hyperspectral imaging device to collect hyperspectral information of the panoramic surgical field that exposes the parathyroid glands during thyroid surgery to obtain a hyperspectral image of the parathyroid glands.

[0038] The parathyroid identification and positioning module is used to: after performing band selection on the parathyroid hyperspectral image, input it into the parathyroid identification and positioning model for identification and positioning to obtain parathyroid information; wherein the parathyroid information includes the pixel sites of the parathyroid glands and the parathyroid categories corresponding to the pixel sites; the parathyroid category is parathyroid-negative or parathyroid-positive; the parathyroid identification and positioning model is obtained by training a deep learning classification framework using a parathyroid sample set.

[0039] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0040] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for parathyroid gland identification and positioning based on hyperspectral, characterized in that: The method for identifying and locating the parathyroid gland based on hyperspectral includes: A hyperspectral imaging device is used to collect hyperspectral information of the panoramic surgical field that exposes the parathyroid glands during thyroid surgery to obtain a hyperspectral image of the parathyroid glands. After selecting the band of the parathyroid hyperspectral image, the image is input into a parathyroid identification and positioning model for identification and positioning to obtain parathyroid information; Among them, the parathyroid information includes the pixel sites of the parathyroid glands and the parathyroid gland categories corresponding to the pixel sites; the parathyroid gland category is parathyroid-negative or parathyroid-positive; and the parathyroid gland recognition and positioning model is obtained by training a deep learning classification framework using a parathyroid gland sample set.

2. The method for parathyroid gland identification and positioning based on hyperspectral according to claim 1, characterized in that: The hyperspectral imaging device includes a mobile support, a hyperspectral camera, a hyperspectral data processing and result feedback device, and a halogen lighting source; The mobile bracket includes a pulley block base, an operating platform and two adjustable support arms; the operating platform and the two adjustable support arms are arranged on the pulley block base; the hyperspectral camera is arranged on one of the adjustable support arms, and the halogen lighting source is arranged on the other adjustable support arm; The hyperspectral data processing and result feedback device is mounted on the operating platform.

3. The method for parathyroid gland identification and positioning based on hyperspectral according to claim 2, characterized in that: A hyperspectral imaging device is used to collect hyperspectral information of the panoramic surgical field that exposes the parathyroid glands during thyroid surgery to obtain a hyperspectral image of the parathyroid glands, including: The hyperspectral camera and the halogen illumination source are placed above the panoramic surgical field for exposing the parathyroid glands during thyroid surgery through the two adjustable support arms; Turning on the hyperspectral camera and the halogen illumination source, using the halogen illumination source for irradiation, using the hyperspectral camera for automatic focusing and collecting multi-band spectral information to obtain an initial hyperspectral image; The hyperspectral data processing and result feedback device is used to receive the initial hyperspectral image, and performs black-and-white calibration and normalization processing on the initial hyperspectral image to obtain a parathyroid hyperspectral image.

4. The method for parathyroid gland identification and positioning based on hyperspectral according to claim 3, characterized in that: When performing black and white calibration normalization processing on the initial hyperspectral image, the following formula is used: ； in, I ref is the reflection spectrum intensity value obtained after black and white calibration normalization processing, I Raw is the reflected spectrum intensity value in the initial hyperspectral image, I dark is the dark current reference intensity, I White is the total reflection reference intensity.

5. The method for parathyroid gland identification and positioning based on hyperspectral according to claim 1, characterized in that: The parathyroid hyperspectral image is subjected to band selection, including: sequentially performing bicubic interpolation downsampling and upsampling on the parathyroid hyperspectral image; Based on the parathyroid hyperspectral image and the upsampled parathyroid hyperspectral image, a structural similarity calculation is performed to obtain the structural similarity of each band; Based on a preset structural similarity threshold and the structural similarity of each band, band screening is performed from the parathyroid hyperspectral image to obtain a parathyroid hyperspectral image after band selection.

6. The method for parathyroid gland identification and positioning based on hyperspectral according to claim 1, characterized in that: The deep learning classification framework includes a first convolution layer, a first spatial attention layer, a second convolution layer, a third convolution layer, a second spatial attention layer, a fourth convolution layer, a fifth convolution layer, a third spatial attention layer, a sixth convolution layer, a seventh convolution layer, a first semantic space fusion layer, an eighth convolution layer, a second semantic space fusion layer, a ninth convolution layer and a tenth convolution layer, which are arranged in sequence; The second convolution layer is jump-connected with the second semantic space fusion layer; the fourth convolution layer is jump-connected with the eighth convolution layer; The first convolutional layer, the second convolutional layer, the fourth convolutional layer, the sixth convolutional layer and the ninth convolutional layer have the same structure; the third convolutional layer and the fifth convolutional layer have the same structure; the seventh convolutional layer and the eighth convolutional layer have the same structure; The third convolutional layer and the fifth convolutional layer are used to perform a post-convolution downsampling operation; The seventh convolutional layer and the eighth convolutional layer are used to perform a post-convolution upsampling operation.

7. The method for parathyroid gland identification and positioning based on hyperspectral according to claim 1, characterized in that: Any parathyroid sample in the parathyroid sample set includes a parathyroid hyperspectral sample image and a corresponding true label; the true label is: a surgeon labels each pixel site in the parathyroid hyperspectral sample image using ENVI5.6, and obtains the parathyroid category corresponding to each pixel site.

8. The method for parathyroid gland identification and positioning based on hyperspectral according to claim 7, characterized in that: The process of training a deep learning classification framework using a parathyroid sample set includes: inputting the parathyroid sample set into the deep learning classification framework, obtaining a prediction result through forward propagation calculation, calculating the error between the prediction result and the true label, and then updating the parameters of the deep learning classification framework through back propagation until the model training is completed after convergence to obtain a parathyroid recognition and positioning model.

9. A system for parathyroid gland identification and positioning based on hyperspectral, characterized in that: The hyperspectral-based parathyroid gland identification and positioning system comprises: The hyperspectral image acquisition module is used to: use a hyperspectral imaging device to collect hyperspectral information of the panoramic surgical field that exposes the parathyroid gland during thyroid surgery to obtain a hyperspectral image of the parathyroid gland; The parathyroid gland identification and positioning module is used to: after selecting the band of the parathyroid gland hyperspectral image, input it into the parathyroid gland identification and positioning model for identification and positioning to obtain the parathyroid gland information; Among them, the parathyroid information includes the pixel sites of the parathyroid glands and the parathyroid gland categories corresponding to the pixel sites; the parathyroid gland category is parathyroid-negative or parathyroid-positive; and the parathyroid gland recognition and positioning model is obtained by training a deep learning classification framework using a parathyroid gland sample set.

Images