High-throughput endoscopic hyperspectral data acquisition method

Abstract

The invention relates to a high-flux endoscopic hyperspectral data acquisition method, which adopts a broadband optical filter group to replace traditional complex hardware such as a narrow-band optical filter, a beam splitter prism, a grating and the like, and avoids the use of a high-power monochromatic light source, a moving part and a complex optical element in the traditional technology. According to the method, the mapping model of the endoscopic image passing through the broadband optical filter and the corresponding hyperspectral endoscopic image is constructed, and the hyperspectral endoscopic data is acquired by reconstructing the hyperspectral data, so that the complexity of the system is greatly reduced, and the light energy utilization rate and the stability of the system are further improved. The method fully integrates multidisciplinary technologies such as optics, a big data image algorithm and medical diagnosis, has good universality and adaptability, not only can be applied to a hard mirror system, but also can be adapted to a fiberscope, a flexible electronic endoscope system and the like, is expected to be expanded and applied to other types of endoscopic imaging systems, and has wide application prospects. The endoscopic imaging system can be a confocal endoscopic imaging system, an industrial endoscopic imaging system and the like.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and particularly to a method for obtaining high-throughput endoscopic hyperspectral data. Background Art

[0002] With the rapid development of science and technology, various advanced scientific and technological achievements have continuously penetrated into the medical field. The development of technologies such as optical imaging technology, sensing technology, automation technology, and computer science has promoted the progress of medical endoscopes. Various endoscopes have emerged one after another, and their functions are also constantly expanding, gradually developing from a single inspection function into a series of products integrating inspection, diagnosis, treatment, and surgery. As a medical device that can be directly inserted into the internal channels and cavities of the human body for direct observation, diagnosis, and treatment, the endoscope plays an important role in the medical field. Due to the limitations of the actual working scenario requirements, existing endoscopes often have the structural characteristics of small volume, which also leads to relatively single functions and the problem that it is difficult to improve the sampling efficiency. In addition, most of the current endoscopic imaging systems use white light source illumination for imaging, which is prone to missing small lesions with colors and shapes similar to normal tissues. How to distinguish diseased tissues from normal tissues and improve the lesion recognition rate is an important development direction of endoscopy.

[0003] With the development of medical technology, researchers have found that hyperspectral data has unique advantages. It can provide spectral characteristics at the molecular level of substances, and these spectral characteristics contain rich cellular information. Observing the hyperspectral characteristics of these color photos while using an endoscope to observe the internal tissues of the human body helps to more accurately diagnose the patient's physical condition. However, there are still many challenges in the application of endoscopes. On the one hand, endoscopic hyperspectral imaging systems usually require the setting of galvanometers, gratings, and a series of complex lenses, prisms, etc. for scanning imaging, which results in complex equipment, large volume and weight, and at the same time, there is also the problem of difficult and rapid data acquisition. Existing patents (such as CN113016006A, CN115607105A) rely on complex or expensive hardware configurations (such as spectrometers, motion platforms, beam splitting prisms, gratings). Not only are the costs high, but the line scanning imaging technology or the gradient filter used requires precise control of moving parts for multiple imaging, which reduces the time resolution and cannot achieve the effect of real-time hyperspectral imaging. Especially in the case of human body cavity breathing or natural intestinal peristalsis during endoscopic examinations, there are problems of image distortion and motion blur.

[0004] The Chinese invention patent with the publication number CN110115557A discloses a hyperspectral endoscopic imaging device and an imaging method. It adopts a new detection means that combines spectral detection and imaging technology. By using a micro-camera array in the hyperspectral endoscopic imaging system and providing a narrowband light source with adjustable spectrum, the spectral splitting method is a differential combination of light source splitting and filter splitting. This setting improves the spectral resolution while ensuring the utilization rate of the light source, and fully utilizes the advantages of computer technology in the traditional endoscopic optical imaging system, thus avoiding the use of high-power light sources and moving components. However, its disadvantages are that the implementation of this device requires the introduction of an additional narrowband light source device, and at least three narrowband micro-cameras and a black-and-white camera are required for cooperative shooting. At the same time, a filter with a difference from the spectrum of the narrowband light source is set in the narrowband micro-camera, resulting in high optical complexity, high cost, and many unstable factors in the actual operation of the control device. Although the spectral resolution is improved, it is difficult to achieve rapid data acquisition. There is also a problem of introducing image motion blur, which is particularly obvious when generating image data by manually controlling the endoscope.

[0005] The Chinese invention patent with the publication number CN111579498B designs a hyperspectral endoscopic imaging system based on pushbroom imaging. This system combines multiple spectral splitting prisms, detectors, and a moving platform with a white light endoscope. The light emitted by the light source irradiates the biological tissue sample, and the reflected light is divided into two paths by the spectral splitting module. One path enters the first detector for bright-field imaging, and the other path of light enters the next spectral splitting module and is dispersed by a grating and then enters the second detector for hyperspectral imaging. After image fusion of the bright-field image and the hyperspectral image, a high-resolution hyperspectral image is obtained. This method can accurately acquire the spectral information of biological tissues and assist the bright-field channel to accurately acquire the morphological information of biological tissues. However, its disadvantages are that although this device can achieve hyperspectral imaging, it requires the introduction of a moving platform and additional optical components, with a complex structure and many unstable factors in actual operation. The introduction of multiple lenses and spectral splitting prisms results in serious loss of light energy during the light propagation process, and a high-power halogen lamp light source is required, which has problems such as serious heating, high cost, and short service life. Due to the introduction of moving components, the pushbroom process requires the acquisition object to remain stationary. When natural contraction movements such as breathing or intestinal peristalsis occur in the human body cavity, the hyperspectral image will have serious distortion and motion blur phenomena, which are difficult to recover through geometric correction, affecting the final imaging quality and difficult to meet the actual clinical needs. In addition, this invention is based on the optical path design of a rigid endoscope. Its multiple groups of optical lenses require high-precision design and matching, so the manufacturing cost is relatively high, and the reliability decreases after long-term repeated use. The diameter of the rigid endoscope is relatively large, and it is difficult to further reduce it from the principle of the optical system. Therefore, the application scenarios are limited and it cannot be used in endoscopic scenarios with narrow channels, such as in the ear, nose, and throat cavities.

[0006] The US patent with the publication number US20230125377A1 designs a label-free real-time hyperspectral endoscopy system for cancer surgery. This system integrates a high-resolution spectrometer with a white light bronchoscope. The light source is coupled to the illumination channel of the bronchoscope through an integrated light guide to illuminate the imaging site. The light reflected by the target is collected by the imaging lens and transmitted through the image fiber bundle to form an intermediate image. Then, the output image is magnified by a GRIN lens and a 4f imaging system and transmitted to an image mapper, which consists of multiple small planes with specific sizes and angles, thus realizing hyperspectral imaging. However, its disadvantages are that this patent requires the light of the light source to be accurately coupled into the bronchoscope through the integrated light guide, and the coupling efficiency is difficult to control; although this device can achieve real-time hyperspectral imaging, its structure is complex, introducing other additional optical elements such as a hyperspectral imager, a grating, a GRIN lens, and an imaging objective lens, resulting in a high system cost, large volume and weight, and the complex optical structure is prone to deviation or damage during the manufacturing or use process, affecting the final imaging quality. At the same time, the xenon lamp used, although having high brightness and high spectral width, has a high cost and a short lifespan. Summary of the Invention

[0007] Aiming at the problem of how to improve the hyperspectral sampling efficiency of the target area while performing endoscopy, the present invention proposes an imaging system capable of real-time high-throughput acquisition of endoscopy hyperspectral data. Without additionally introducing optical devices such as gratings, monochromators, moving parts, and high-power monochromatic light sources, multiple spectral channel imaging is achieved by adding a broadband filter group. During the imaging process, light passes through each broadband filter, and each broadband filter corresponds to a different spectral curve. Only a single imaging is required to collect different spectral information. The detector receives the optical signal filtered by the filter, converts it into an electrical signal, and then performs endoscopy hyperspectral image reconstruction through a trained deep learning model to obtain a hyperspectral image, thereby increasing the completeness of the lesion information representation, providing spectral features that complement and corroborate the endoscopy image, and improving the diagnostic accuracy and sampling efficiency.

[0008] A high-throughput method for obtaining endoscopy hyperspectral data, characterized by comprising the following steps: 1) Collect endoscopy wide-spectrum images; 2) Input into the hyperspectral network reconstruction model and output the reconstructed endoscopy hyperspectral image, which can be achieved through single-shot wide-spectrum imaging, without precision moving parts, and without registration between the wide-spectrum image and the hyperspectral image.

[0009] Further, the establishment of the hyperspectral network reconstruction model in step 2) includes the following steps: 2.1) First apply a predetermined loss through an end-to-end network; 2.2) Minimize the loss function through the backpropagation algorithm to perform sample training from endoscopic wide-spectrum images to endoscopic hyperspectral images, and the trained network model is used for spectral reconstruction.

[0010] Further, the specific process of step 2.2) is as follows: Extract shallow feature information from the input endoscopic wide-spectrum image through 3×3 convolution operation, and then use stacked residual attention groups to form a deep network for extracting deep spectral features; then connect the feature information of different levels generated by each residual attention group through skip connections to form a connection layer; then use 1×1 convolution operation to reduce the feature dimension; then make full use of the feature information in the original endoscopic wide-spectrum image through the residual structure to improve feature correlation learning, and finally use 3×3 convolution to reconstruct the endoscopic hyperspectral image and save it to update the existing hyperspectral image database.

[0011] Further, the spectral image is jointly encoded by a light source, a filter, a lens, and a camera detector, and the intensity at the spatial position is expressed as: , where is the spectral curve of the object to be measured, is the spectral curve of the light source, is the spectral curve of the nth filter, is the transmittance curve of the objective lens, is the spectral response curve of the camera, and are the spectral ranges of the system, and n is the number of broadband filters.

[0012] Further, the hyperspectral network reconstruction model is obtained by solving the mapping relationship through the data pair of the input endoscopic wide-spectrum image and the corresponding endoscopic hyperspectral image, where the hyperspectral data is expressed as the following formula: , S represents hyperspectral data; I represents wide-spectrum image; F is the function space containing the function that maps data from I to S; represents the set of endoscopic hyperspectral images, is the spectral data; represents the set of endoscopic wide-spectrum images, and n is the number of broadband filters; H is the image height, W is the image width.

[0013] Furthermore, the two cameras are combined and constructed using a precise self-alignment configuration scheme in the hardware design. The optical axes of the two cameras are calibrated to be parallel and in the same direction, and they naturally have overlapping field of view ranges, capable of synchronously capturing image information in the same area. If the initial fields of view are not completely consistent, the fields of view are regularized through translation, scaling, and cropping operations to ensure that the final fields of view targeted by the two cameras are exactly the same, without any post-registration required. The beneficial effects brought by the technical solution of the present invention: The present invention provides a method for obtaining high-throughput endoscopic hyperspectral images. The use of broadband filters greatly improves the light energy utilization rate, and there is no need for any moving parts and high-power light sources, with a small volume and low cost. It can perfectly replace the traditional combination of spectroscopy and push-broom to obtain hyperspectral images, greatly expanding the functions of existing endoscopes. The method of single-shot wide-spectrum imaging and reconstructing endoscopic hyperspectral images based on deep learning improves the time resolution without reducing the spatial and spectral resolutions, and can effectively avoid the influence caused by the breathing of the human body cavity or the natural peristalsis of the intestine during endoscopic examinations, achieving the effect of real-time reconstruction of hyperspectral images. In addition, hyperspectral images can provide additional spectral information to assist endoscopic diagnosis, providing an objective and unified judgment standard for doctors during examinations. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of a method for obtaining high-throughput endoscopic hyperspectral data.

[0015] Figure 2 It is a schematic diagram of the implementation of a broadband filter group through a filter wheel and a sliding filter holder.

[0016] Figure 3 It is a comparison chart of the spectral curves of broadband filters and narrowband filters.

[0017] Figure 4 It is a schematic diagram of a data acquisition method and model training for reconstructing hyperspectral images from wide-spectrum images.

[0018] Figure 5 It is a flowchart of a hyperspectral reconstruction algorithm based on deep learning.

[0019] Figure 6 It is a general flowchart of a high-throughput endoscopic hyperspectral data acquisition system.

[0020] In the figure: Endoscopic imaging system 100, objective lens 101, illumination optical fiber 102, color detector 103, data processing unit 104, image display unit 105, power supply 106, light source 107, halogen lamp or LED lamp 108, broadband filter group 109, focusing lens 110; Training system diagram 200, beam splitter 201, liquid crystal tunable filter 202, monochromatic detector 203, deep learning model 204. Detailed implementation

[0021] The present invention proposes a high-throughput endoscopic imaging system based on hyperspectral reconstruction technology, which can overcome the problems commonly existing in current endoscopic hyperspectral imaging, such as large system volume, low light energy utilization rate, high complexity, and mutual limitation of spatio-temporal resolution. The following introduces the specific technical route of the present invention. Embodiment

[0022] Training and testing stage: As Figure 4 shown, the light emitted by the light source 108 passes through the broadband filter group 109 and is coupled into the illumination optical fiber 102 by the focusing lens 110. The output end of the illumination optical fiber 102 is divided into multiple branched optical fibers, which are evenly distributed around the objective lens 101 and are symmetrically distributed in a ring shape for illumination; a beam splitter 201, an objective lens 101, and a color detector 103 are sequentially arranged from the object plane to the image plane. The beam splitter 201 divides the reflected light from the object to be measured into a reflected optical path and a transmitted optical path in a ratio of 3:7 to 5:5; the two cameras are combined and constructed by adopting a precise self-alignment configuration scheme in hardware design. The optical axes of the two are adjusted to be parallel and in the same direction, and naturally have overlapping field of view ranges, and can synchronously capture image information in the same area. If the initial fields of view are not completely consistent due to slight installation differences or other factors, the fields of view can be regularized through translation, scaling, and cropping operations to ensure that the final fields of view targeted by the two cameras are exactly the same, without any registration algorithm for processing. The beam splitter 201 divides the reflected light from the object to be measured into a reflected optical path and a transmitted optical path in a ratio of 3:7 to 5:5; the transmitted optical path includes the objective lens 101 module including a first lens element, a second lens element, a third lens element, a fourth lens element, a fifth lens element, and a sixth lens element; the first lens element is a negative focal length lens, the second lens element is a negative focal length lens, the third lens element is a positive focal length lens, the fourth lens element is a positive focal length lens, the fifth lens element is a negative focal length lens, and the sixth lens element is a positive focal length lens. The lenses are bonded by an ultraviolet curable adhesive layer; the difference between the reflected optical path and the transmitted optical path is that a liquid crystal tunable filter 202 is added between the objective lens 101 and the monochromatic detector 203 in the reflected optical path. The image processing module contains a deep learning model 204, which receives the hyperspectral image sequence of the monochromatic detector 203 and the broadband image of the color detector 103 for feature fusion training.

[0023] Actual use stage: After the deep learning model 204 is trained, the reflected optical path can be removed, as Figure 1 shown.

[0024] Figure 1Disclosed is an endoscopic imaging system 100 based on hyperspectral reconstruction technology proposed according to the present invention, which includes: an objective lens 101, an illumination optical fiber 102, a color detector 103, a data processing unit 104, an image display unit 105, a power supply 106, and a light source 107. The light source 107 includes a halogen lamp or an LED lamp 108, a broadband filter set 109, and a focusing lens 110. The light emitted by the LED lamp or the halogen lamp 108 passes through the broadband filter set 109 and then is coupled into the illumination optical fiber 102 through the focusing lens 110 to illuminate the object to be measured. After the reflected light of the object to be measured is converged and emitted by the objective lens 101, it is imaged on the target surface of the color detector 103 (the ordinary white light endoscopic image can be realized by removing the broadband filter set 109).

[0025] As Figure 2 shown, the broadband filter set 109 used in the present invention can achieve sequential wide-spectrum imaging through a filter wheel or a sliding filter holder, and the number of broadband filters can be 2, 4, 8, etc., which meet the system requirements. As Figure 3 shown, the broadband filter has a high transmittance and a wide spectral range, and the light energy utilization rate is much higher than that of the narrowband filter sets used in most existing patents. After the wide-spectrum image imaged on the target surface of the color detector 103 is subjected to hyperspectral reconstruction by the image processing unit, a corresponding endoscopic hyperspectral image is generated, and both the endoscopic wide-spectrum image and the endoscopic hyperspectral image can be displayed through the image display unit. The image processing unit includes a trained hyperspectral reconstruction algorithm model. The training of this algorithm requires inputting a data pair of an endoscopic wide-spectrum image and the corresponding endoscopic hyperspectral image to solve the mapping relationship. The method for obtaining the data pair is as Figure 4As shown in the figure, the training system diagram 200 consists of a beam splitter 201, two identical objective lenses 101, a Liquid Crystal Tunable Filter (LCTF) 202, a monochromatic detector 203, a color detector 103, a light source 107, and a deep learning model 204. The two cameras are combined and constructed using a precise self-alignment configuration scheme in the hardware design. The optical axes of the two cameras are calibrated to be parallel and in the same direction, and naturally have overlapping field of view ranges, enabling them to synchronously capture image information in the same area. If the initial fields of view are not completely consistent due to minor installation differences or other factors, the fields of view can be adjusted by translation, zooming, and cropping operations to ensure that the final fields of view targeted by the two cameras are exactly the same, without the need for any registration algorithms. The light emitted by the LED lamp or halogen lamp 108 passes through the broadband filter group 109 and is coupled into the illumination optical fiber 102 through the focusing lens 110 to illuminate the object to be measured. After the reflected light of the object to be measured passes through the beam splitter 201, a part of the light is reflected and enters the Liquid Crystal Tunable Filter (LCTF) 202 through the objective lens 101 for wavelength selection and then is focused and imaged on the target surface of the monochromatic detector 203 in sequence. Another part of the light passes through the objective lens 101 and is imaged on the target surface of the color detector 103. The LCTF is an optical filter designed based on the birefringence effect of liquid crystal molecules and the principle of polarized light interference. By controlling the liquid crystal elements through an electronic unit, it can transmit light of a specific wavelength and exclude light of other wavelengths, achieving a flexible narrowband filtering function for incident light. After this process, the endoscopic hyperspectral image of the object to be measured generated by the monochromatic detector 203 and the endoscopic broadband image of the object to be measured generated by the color detector 103 are formed into a data pair and input into the deep learning model to solve the mapping relationship and train the model. The spectral image is jointly encoded by the light source, filter, lens, and camera detector. The intensity at the spatial position is expressed by Equation (1) as: (1), where is the spectral curve of the object to be measured, is the spectral curve of the light source, is the spectral curve of the nth filter, is the transmittance curve of the objective lens, is the spectral response curve of the camera, and is the spectral range of the system, and n is the number of broadband filters. Reconstructing hyperspectral information from a small number of band images (such as RGB images or multispectral images) is an ill-posed inverse problem that requires hyperspectral reconstruction algorithms to solve. Hyperspectral reconstruction methods are mainly divided into two categories: prior-based methods and data-driven methods. Prior-based methods use the statistical information of hyperspectral images, such as sparsity, spatial structure similarity, and spectral correlation. These methods include least squares regression, Bayesian methods, iterative methods, radial basis function networks, and sparse coding. Data-driven methods rely on encoding image and hyperspectral image datasets to train neural networks for hyperspectral reconstruction. In recent years, with the great success of deep learning in the field of computer vision, researchers have proposed a large number of deep learning-based methods to improve the reconstruction accuracy, and proposed various neural network frameworks, such as deep neural networks (DNNs), convolutional neural networks (CNNs), generative adversarial networks (GANs), and Transformers. It can be obtained from I through Equation (2): (2), where S represents the hyperspectral data, I represents the broadband image, and F is a function space that contains functions that map data from I to S. By solving Equation (2), a nonlinear mapping model between the broadband image and the hyperspectral image can be obtained, and this mapping model is added to the image processing unit for endoscopic hyperspectral image reconstruction. The flowchart of the deep learning-based endoscopic broadband image hyperspectral reconstruction algorithm in the present invention is as Figure 5 shown. The input is the endoscopic broadband image, and the output is the endoscopic hyperspectral image. The main process structure consists of three parts: feature extraction, nonlinear mapping, and spectral reconstruction. The deep learning-based hyperspectral reconstruction method can perform sample training from the broadband image to the hyperspectral image by first applying a predetermined loss through an end-to-end network and then minimizing the loss function through the backpropagation algorithm. Finally, the trained network model is used to directly perform spectral reconstruction to generate a hyperspectral image that can be used for actual analysis and processing.

[0026] An embodiment of the present invention proposes a high-throughput method for obtaining endoscopic hyperspectral data. The overall flowchart is as Figure 6 shown. In the system data acquisition and training stage, the method includes collecting endoscopic broadband images and hyperspectral images, performing preprocessing such as standardization and noise removal on the endoscopic broadband images and endoscopic hyperspectral images to obtain a high-quality endoscopic image pair dataset; solving the mapping relationship and training the hyperspectral reconstruction model for the image pairs based on deep learning methods, and finally obtaining an endoscopic hyperspectral image that meets the system requirements. The acquisition of the image pair of the endoscopic broadband image and the hyperspectral image is through Figure 4The described endoscopic data acquisition system is completed, where the wavelength range is 400nm - 800nm and the number of broadband filters n = 4. The broadband filters used have a wider spectral range and higher light energy utilization rate compared to the narrowband filters used in most existing patents. For example, Figure 3 As shown, taking the narrowband filters with the most commonly used central wavelengths of 415nm, 532nm, 635nm, and 710nm as an example, compared with the broadband filter, the narrowband filter only allows light with a central wavelength of ±5nm to pass through, while the broadband filter can transmit light in the range of 400 - 800nm, and has stable output in each band within this range. The integral area of the normalized spectral curve and the abscissa can be regarded as the light energy utilization rate of the filter. The light energy utilization rate is calculated using the Trapz function in MATLAB software. By calculating, the light energy utilization rate of the broadband filter is about 15 times that of the narrowband filter. Therefore, the exposure time can be greatly shortened, the image acquisition frame rate can be improved, and the influence caused by the respiration of the human body cavity or the natural peristalsis of the intestine can be effectively avoided during endoscopic examination, achieving the effect of real-time reconstruction of hyperspectral images.

[0027] The reconstruction of the hyperspectral image depends on the imaging model, that is, the endoscopic spectral image (I) is essentially the integral of the product of the hyperspectral image (S), light source (G), lens (L), filter transmission function (R), and camera spectral response function (D) within the spectral range (λ). Obtained. Therefore, the conversion from the endoscopic broadband image to the endoscopic hyperspectral image is an inverse mapping problem. A reconstruction algorithm based on deep learning is used to solve this inverse mapping problem, thus changing the previous mode of only obtaining endoscopic hyperspectral images through hardware methods.

[0028] The deep learning method refers to using operations such as convolution and pooling to extract and compress the spatial-spectral information of the endoscopic spectral image pair during the process of solving the inverse mapping, thereby reducing data redundancy and improving the efficiency and accuracy of reconstruction. The deep learning method sequentially includes network model training and the reconstruction of hyperspectral images; the reconstruction of hyperspectral images is to reconstruct the input wide-spectrum image using the trained network model. The network model extracts shallow feature information by performing 3×3 convolution operations on the input endoscopic wide-spectrum image, and then uses stacked residual attention groups to form a deep network, thus facilitating the extraction of deep spectral features; then, different levels of feature information generated by each residual attention group are connected through skip connections to form a connection layer; afterwards, 1×1 convolution operations are used to reduce the feature dimension, which can not only reduce the computational cost and time, but also reduce redundant information; then, the residual structure is used to make full use of the feature information in the original endoscopic wide-spectrum image to improve feature correlation learning, and finally, 3×3 convolution is used to reconstruct the endoscopic hyperspectral image. After the reconstruction of the endoscopic hyperspectral image is completed, it is further saved to update the existing hyperspectral image database. That is to say, the reconstructed hyperspectral image is saved in the database, and by means of cyclic learning and continuous improvement of the mapping relationship from hyperspectral images to wide-spectrum images and adjustment of various network parameters, the accuracy of image reconstruction is ensured. After the reconstruction is completed, the endoscopic hyperspectral image can be further analyzed and processed, such as being superimposed and displayed on the normal endoscopic image through an image display unit to assist doctors in diagnosis based on the lesion spectrum; the analysis and processing of the endoscopic hyperspectral image include interpreting the endoscopic hyperspectral image by using methods such as supervised classification and unsupervised classification.

[0029] The high-throughput endoscopic hyperspectral data acquisition method provided by the present invention can obtain endoscopic hyperspectral information quickly and efficiently, and can greatly enhance the performance and sampling efficiency of the endoscopic imaging system, providing innovative ideas for doctors to carry out relevant diagnosis and analysis. By constructing a mapping model between the endoscopic image passing through the broadband filter and its corresponding hyperspectral endoscopic image, and then realizing the acquisition of hyperspectral endoscopic data by reconstructing the hyperspectral data, the invention does not require any moving parts and monochromatic light sources, which not only greatly reduces the complexity and cost of the system, but also further improves the stability and reliability of the system. The invention can be applied not only to rigid endoscope systems, but also to fiber endoscopes and flexible electronic endoscope systems, etc. At the same time, it fully integrates multidisciplinary technologies such as optics, big data image algorithms, and medical diagnosis, and is an outstanding achievement in the current development of multiple disciplines, showing great advantages. In addition, the present invention is also expected to be extended to other types of endoscopic imaging systems, such as confocal endoscopic imaging systems, industrial endoscopic imaging systems, etc.

[0030] The above examples of the present invention have been described in detail in combination with the embodiments. However, the present invention is not limited to the above examples. Within the scope of knowledge possessed by those of ordinary skill in the art, various changes can be made without departing from the gist of the present invention, and these should also be regarded as the protection scope of the present invention.

Claims

1. A method for obtaining high-throughput endoscopic hyperspectral data, characterized in that, It includes the following steps: 1) Collect endoscopic wide-spectrum images; 2) Input into the hyperspectral network reconstruction model and output the reconstructed endoscopic hyperspectral images. Through single-shot wide-spectrum imaging, without precise moving parts, and without registration between the wide-spectrum images and the hyperspectral images.

2. The high-throughput endoscopic hyperspectral data acquisition method according to claim 1, characterized in that The establishment of the hyperspectral network reconstruction model in step 2) includes the following steps: 2.1) First apply a predetermined loss through an end-to-end network; 2.2) Minimize the loss function through the backpropagation algorithm to perform sample training from endoscopic wide-spectrum images to endoscopic hyperspectral images. The trained network model is used for spectral reconstruction.

3. The high-throughput endoscopic hyperspectral data acquisition method according to claim 3, wherein The specific process of step 2.2) is as follows: The input endoscopic wide-spectrum images are used to extract shallow feature information through 3×3 convolution operations, and then a stacked residual attention group is used to form a deep network for extracting deep spectral features. Then, different levels of feature information generated by each residual attention group are connected through skip connections to form a connection layer. After that, 1×1 convolution operations are used to reduce the feature dimension. Then, the residual structure is used to make full use of the feature information in the original endoscopic wide-spectrum images to improve feature correlation learning. Finally, 3×3 convolution is used to reconstruct the endoscopic hyperspectral images and save them to update the existing hyperspectral image database.

4. The high-throughput endoscopic hyperspectral data acquisition method according to claim 3, wherein The spectral image is jointly encoded by a light source, a filter, a lens, and a camera detector. The intensity at the spatial position is expressed as: ， Among them, is the spectral curve of the object under test, is the spectral curve of the light source, is the spectral curve of the nth filter, is the transmittance curve of the objective lens, is the spectral response curve of the camera, and are the spectral ranges of the system, and n is the number of broadband filters.

5. The high-throughput endoscopic hyperspectral data acquisition method according to any one of claims 1 to 4, characterized in that The hyperspectral network reconstruction model is obtained by solving the mapping relationship through a data pair of the input endoscopic wide-spectrum images and the corresponding endoscopic hyperspectral images, where the hyperspectral data is expressed as the following formula: ， S represents hyperspectral data; I represents broadband image; F is a function space that contains functions for mapping data from I to S; represents the set of endoscopic hyperspectral images, is spectral data; represents the set of endoscopic broadband images, where n is the number of broadband filters; H is the image height, W is the image width.

6. The high-throughput endoscopic hyperspectral data acquisition method according to claim 5, characterized in that The two cameras are combined and constructed using a precise self-alignment configuration scheme in the hardware design. The optical axes of the two are calibrated to be parallel and in the same direction, and naturally have overlapping field of views, and can synchronously capture image information in the same area. If the initial fields of view are not completely consistent, the fields of view are regularized through translation, scaling, and cropping operations to ensure that the final fields of view targeted by the two cameras are exactly the same, without any post-registration.

Images