Endoscopic Detection System and Method Based on Single-Exposure Spectral Imaging of Special Optical Fibers

Abstract

This application proposes an endoscopic detection system and method based on single-exposure spectral imaging of special optical fibers. A special optical fiber bundle with a mask is used to perform spatial distribution modulation imaging on the target object. This optical fiber bundle combines the functions of illumination, encoding, and transmission, and realizes image encoding through the difference in the light transmission characteristics of the sub-fibers. The specific steps include: First, use this optical fiber bundle to obtain an observation image; Second, perform dispersion modulation and compressed acquisition on the observation image to form a compressed observation image; Finally, use image preprocessing algorithms, deep neural reconstruction networks, and image postprocessing algorithms to reconstruct multi-spectral image data of the target endoscopic scene from the compressed observation image. The present invention effectively improves the efficiency and accuracy of endoscopic detection.

Description

Technical Field

[0001] This application relates to the field of endoscopic detection, and particularly to an endoscopic detection system and method based on single-exposure spectral imaging of special optical fibers. Background Art

[0002] As a non-destructive detection method, endoscopic detection demonstrates extensive application value and huge potential demand in fields such as medical diagnosis, industrial inspection, security monitoring, and military reconnaissance. The realization and improvement of its core functions largely depend on the sensor solutions adopted in the endoscopic system, especially key performance indicators such as the adaptability of its application scenarios and the richness of information collected in a single acquisition.

[0003] Current endoscopic sensor solutions on the market mainly include diversified technical routes such as physical parameter electronic sensor solutions, RGB imaging solutions, and multispectral imaging solutions. However, although the above solutions have achieved corresponding application results in their respective fields, there has not yet emerged an endoscopic detection system that can perfectly integrate multiple advantageous features such as multi-dimensional spectral image acquisition, single-shot spectral data capture over a large spatial range, high spatial resolution, high light flux, and miniaturization. This has become a key bottleneck restricting the in-depth development and efficiency improvement of endoscopic detection technology. The main defects are as follows:

[0004] 1. Physical parameter electronic sensor solutions: Such solutions are favored for their advantages such as simple structure and easy implementation of circuit design. However, their main limitation is that most of them can only perform single-point and single-parameter measurements and cannot synchronously collect multiple features in an area at the same time, resulting in severely limited comprehensiveness and depth of information acquisition.

[0005] 2. RGB imaging solutions: This solution can provide clear and intuitive regional images and has significant advantages for observing visual forms. However, it relies on the single-band information of the three primary colors of red, green, and blue, and has weak ability to identify and analyze specific physical and chemical properties (such as pathological features) of the object to be measured, making it difficult to meet the requirements for deep and precise detection.

[0006] 3. Traditional multispectral imaging solutions are mainly divided into two modes: scanning and single-exposure:

[0007] Scanning multispectral imaging: By taking a large number (even thousands) of images to stitch and form complete three-dimensional spectral data, its imaging process takes a long time and cannot achieve real-time or near-real-time video-level imaging. In addition, such solutions usually require large and high-precision rotating mechanisms and have high requirements for the intensity of incident light, which contradicts the characteristics that endoscopic detection systems often need to operate in space-constrained environments and is not conducive to actual deployment and application.

[0008] Single-exposure multispectral imaging: Compared with the scanning scheme, its mechanical structure is more simplified, which better meets the requirements of endoscopic detection systems for compactness and flexibility. However, traditional single-exposure methods can be divided into the filter scheme and the compressive coding scheme, both of which have obvious technical bottlenecks:

[0009] a) Filter scheme: Although the structure is relatively simple, when increasing the number of spectral channels, it will inevitably lead to a decrease in spatial resolution, making it difficult to achieve dual optimization of spectral resolution and spatial resolution.

[0010] b) Compressive coding scheme: Although it overcomes the spatial resolution problem of the filter scheme to a certain extent, it relies on precise high-precision masks. Such masks not only have high manufacturing costs and long processing cycles, but also require an additional optical registration system to ensure that the image of the object to be measured is accurately mapped onto the mask plane for high-precision spatial modulation, increasing the complexity and cost of the system.

[0011] In summary, existing endoscopic detection systems and their sensor schemes generally have problems such as single information acquisition dimension, limited spatial coverage, insufficient spectral resolution ability, slow imaging speed, and unsatisfactory miniaturization degree in practical applications, and cannot simultaneously achieve key performance indicators such as multi-dimensional spectral image acquisition, single-shot large spatial range spectral data acquisition, high spatial resolution, high light flux, and miniaturization of equipment. Therefore, developing a new generation of sensor technology that can overcome the above defects and meet the diversified and refined endoscopic detection requirements has become an urgent task in this field. Summary of the Invention

[0012] The embodiments of the present application provide an endoscopic detection system and method based on single-exposure spectral imaging of special optical fibers, aiming at the problems of single information acquisition dimension, limited spatial coverage, insufficient spectral resolution ability, slow imaging speed, and unsatisfactory miniaturization degree existing in current technologies.

[0013] The core technology of the present invention mainly realizes spatial distribution modulation and coding through a special optical fiber bundle with a mask, combines a dispersion acquisition module for dispersion modulation and compressive observation, uses a deep neural reconstruction network to reconstruct multi-spectral image data, and is supplemented by image preprocessing and post-processing algorithms to achieve video frame rate acquisition of hyperspectral endoscopic images, high spatial resolution, high spectral resolution, and miniaturized endoscopic detection applications.

[0014] In the first aspect, the present application provides an endoscopic detection method based on single-exposure spectral imaging of a special optical fiber bundle, and the method includes the following steps:

[0015] S1. Image the object to be measured and perform spatial distribution modulation through a special optical fiber bundle with a mask to obtain an observation image;

[0016] Among them, the masked special optical fiber bundle is used to simultaneously illuminate, encode, and transmit the target scene image. The light-passing sub-fibers of the masked special optical fiber bundle are used to transmit the image and the illumination beam, and the non-light-passing or partially light-passing sub-fibers are arranged alternately with the light-passing sub-fibers in the transverse space to achieve the encoding of the image;

[0017] S2. Perform dispersion modulation and compressive acquisition on the observed image to obtain a compressed observed image;

[0018] S3. Reconstruct the multi-spectral image data of the target endoscopic scene from the compressed observed image;

[0019] Among them, the reconstruction includes an image preprocessing algorithm, a deep neural reconstruction network, and an image postprocessing algorithm.

[0020] Further, in step S1, the target scene to be measured of the object to be measured is imaged onto the end face of the masked special optical fiber bundle through the first lens group, and the spatial energy is coupled to the optical fiber waveguide structure. The illumination system provides an illumination light source, which enters the masked special optical fiber bundle through the coupler, so that the masked special optical fiber bundle encodes and transmits the target scene image through the alternately arranged light-passing sub-fibers and non-light-passing or partially light-passing sub-fibers.

[0021] Further, in step S2, the dispersion device spectrally disperses and modulates the light beam of the observed image, so that the encoded images of the object to be measured with different wavelengths are imaged on different transverse regions of the detector array.

[0022] Further, in step S3, the image preprocessing algorithm performs noise reduction, gray level equalization, and cropping on the compressed observed image. The deep neural reconstruction network reconstructs the preprocessed compressed observed image into the multi-dimensional spectral image data of the object to be measured through forward propagation. The image postprocessing algorithm performs super-resolution, color image synthesis, contrast enhancement, spectral curve noise reduction, and secondary calibration on the reconstructed spectral image to realize the visualization and calibration of the spectral image.

[0023] Further, the training steps of the deep neural reconstruction network include:

[0024] Prepare a spectral image data set and a system forward model, and obtain the compressed observation of the spectral image data set according to the system forward model;

[0025] Input the compressed observation and the system forward model into the deep neural network for forward propagation to obtain a reconstructed spectral image;

[0026] Input the spectral image data set and the reconstructed spectral image into the loss function to calculate the loss value;

[0027] Perform backpropagation operations on the model based on the loss value, update the parameters of the deep neural network and continuously train until the loss value is less than the target loss value, and stop training to obtain the trained model.

[0028] Further, the inference steps of the deep neural reconstruction network include:

[0029] Input the compressed observation and the system forward model into the trained model to obtain the reconstructed three-dimensional spectral image.

[0030] In a second aspect, the present application provides an endoscopic detection device based on single-exposure spectral imaging with a special fiber bundle, including:

[0031] A special fiber bundle module, including a first lens group, an illumination system, a masked special fiber bundle, and a coupler. The first lens group images the target scene to be measured onto the end face of the masked special fiber bundle and realizes the coupling of spatial energy to the fiber waveguide structure. The illumination system provides an illumination light source that enters the masked special fiber bundle through the coupler. The masked special fiber bundle encodes and transmits the target scene image through alternately arranged light-passing fibers and non-light-passing or partially light-passing fibers;

[0032] A dispersion acquisition module, including a dispersion device, a second lens group, and a detector array. The spatially modulated observation image is sent into the dispersion acquisition module for dispersion modulation to form a compressed observation image. The dispersion device disperses and modulates the outgoing light beam so that the encoded images of the object to be measured with different wavelengths are imaged in different transverse regions of the detector array. The second lens group is used to magnify or reduce the image on the end face of the special fiber bundle to match the size of the detector array. The detector array is used to acquire the compressed observation image;

[0033] A reconstruction module, including an image preprocessing algorithm, a deep neural reconstruction network, and an image postprocessing algorithm. The image preprocessing algorithm performs noise reduction, gray level equalization, and cropping processing on the compressed observation image. The deep neural reconstruction network reconstructs the preprocessed compressed observation image into multi-dimensional spectral image data of the object to be measured through forward propagation. The image postprocessing algorithm performs super-resolution, color image synthesis, contrast enhancement, spectral curve noise reduction, and secondary calibration processing on the reconstructed spectral image to realize the visualization and calibration of the spectral image.

[0034] Further, the masked special fiber bundle replaces some core glasses with materials having different refractive index distributions according to a preset spatial distribution, and ensures that the replaced materials have the same drawing tension as the raw materials.

[0035] In a third aspect, the present application provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the above endoscopic detection method based on single-exposure spectral imaging with a special fiber bundle.

[0036] In a fourth aspect, the present application provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the endoscopic detection method based on single-exposure spectral imaging using a special optical fiber bundle as described above.

[0037] The main contributions and innovations of the present invention are as follows:

[0038] 1. Improvement in the richness of information collected in a single acquisition: Compared with endoscopic RGB imaging and electronic sensor solutions, the present invention can obtain spectral image data with a larger range and higher dimension in a single exposure, effectively solving the problems of single-dimensional information acquisition and limited spatial coverage in the prior art, and significantly improving the comprehensiveness and depth of endoscopic detection.

[0039] 2. High-efficiency and fast imaging: Compared with traditional scanning spectral imaging techniques, the present invention achieves a video frame rate acquisition speed, overcoming the disadvantages of slow imaging and inability to achieve real-time or near-real-time imaging in the scanning solution, and is particularly suitable for endoscopic application scenarios that require fast response and dynamic monitoring.

[0040] 3. Miniaturization and low power consumption: The system of the present invention is small in size and low in power consumption, with a simplified hardware design and economical cost, significantly superior to traditional multi-spectral imaging solutions that require large and high-precision rotating mechanisms, making the deployment of endoscopic detection equipment more convenient in space-constrained scenarios, and having a wider and more flexible scope of application.

[0041] 4. Strong anti-interference and low-light adaptability: Through improved design, the present invention reduces the interference of the endoscopic optical fiber bundle on the spatial distribution of the final hyperspectral image, improving the imaging quality. At the same time, due to a higher light throughput per unit wavelength, the present invention is particularly suitable for endoscopic environments under low-light illumination conditions, enhancing the environmental adaptability and practicality of the system.

[0042] 5. Integration and cost-effectiveness: A special optical fiber bundle is used to construct a mask, and its modulation function is integrated into the existing imaging optical fiber bundle component of the endoscopic system, avoiding the use of an independent and high-precision mask and its complex optical registration system, not only simplifying the system structure, reducing the processing difficulty and cycle, but also significantly saving the system cost and improving the overall cost performance.

[0043] 6. In summary, through the innovative single-exposure spectral imaging technology using a special optical fiber bundle, the present invention has achieved significant improvements in information acquisition efficiency, imaging speed, equipment miniaturization, anti-interference ability, low-light adaptability, and cost-effectiveness in endoscopic detection, effectively solving many pain points in the prior art, and providing an efficient, accurate, and economical solution for endoscopic detection applications in the fields of medicine, industry, security, military, etc.

[0044] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more concise and understandable. Description of the Drawings

[0045] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0046] Figure 1 is the flowchart of the endoscopic detection method based on single-exposure spectral imaging of a special optical fiber bundle according to an embodiment of the present application;

[0047] Figure 2 is the schematic diagram of the system according to an embodiment of the present application;

[0048] Figure 3 is the schematic diagram of the principle according to an embodiment of the present application;

[0049] Figure 4 is the schematic diagram of the end face of the special optical fiber bundle with a mask according to an embodiment of the present application;

[0050] Figure 5 is the schematic diagram of the special optical fiber bundle module modulating the object to be measured according to an embodiment of the present application;

[0051] Figure 6 is the algorithm framework diagram of the deep neural reconstruction network according to an embodiment of the present application;

[0052] Figure 7 is the schematic diagram of the special optical fiber bundle with a mask (right) and without a mask (left) in the simulation according to an embodiment of the present application;

[0053] Figure 8 is the reference RGB image of two test samples and the compressed observation image obtained by the simulation of this system according to an embodiment of the present application;

[0054] Figure 9 is the spectral image of 40 channels reconstructed by two test samples through the simulation algorithm according to an embodiment of the present application;

[0055] Figure 10 is the comparison diagram of the local spectral curves of the original data (left) and the reconstruction result (right) of two test samples according to an embodiment of the present application;

[0056] Figure 11 is the schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application. Detailed Embodiments

[0057] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0058] It should be noted that: In other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0059] Embodiment 1

[0060] This application aims to propose an endoscopic detection method based on single-exposure spectral imaging of a special fiber optic bundle, which can acquire three-dimensional hyperspectral endoscopic images with high spatial resolution and spectral resolution at video frame rate, and can be used for endoscopic detection machine vision applications in narrow spaces. The flow of the endoscopic detection method based on single-exposure spectral imaging of a special fiber optic bundle proposed by the present invention is as Figure 1 shown. Specifically, the embodiments of this application provide an endoscopic detection method based on single-exposure spectral imaging of a special fiber optic bundle. Referring to Figure 1 and Figure 2 , the method includes:

[0061] As shown in step S1, an observation image of the object to be measured (also referred to as the object under test) after spatial modulation by a masked special fiber optic bundle is obtained through a special fiber optic bundle module.

[0062] Among them, the special optical fiber bundle module includes a lens group, an illumination system, a masked special optical fiber bundle, and a coupler. The lens group is located between the masked special optical fiber bundle and the object to be measured, and is connected to one end of the optical fiber bundle. It is used to image the target scene to be measured in a specific field of view onto the end face of the masked special optical fiber bundle, complete the coupling of spatial energy into the optical fiber waveguide structure, and at the same time, it is also used to expand the light source emitted from the optical fiber bundle and couple it into free space; the illumination system is used to provide an illumination light source for the entire system. The illumination light emitted by the illumination system will be coupled into the masked special optical fiber bundle through the coupler, and then emitted through the lens group to illuminate the target scene; the masked special optical fiber bundle is the key device of this module and this invention. It is used to illuminate, encode, and transmit the target scene image at the same time. The light-passing sub-fibers are used to transmit the image and the illumination beam, and the non-light-passing or partially light-passing sub-fibers are arranged alternately with the light-passing sub-fibers in the transverse space to achieve image encoding.

[0063] After that, as shown in step S2, the observed image that has undergone spatial modulation will be sent to the dispersion acquisition module for dispersion modulation to form a compressed observed image, which will be collected and recorded and then transmitted to the reconstruction module.

[0064] Among them, the dispersion acquisition module includes a dispersion device, a lens group, and a detector array. The dispersion device is used to split the broadband light beam emitted by the special optical fiber bundle module and perform dispersion modulation on the light beams of different wavelengths respectively, so that the encoded images of the object to be measured with different wavelengths can finally be imaged in different transverse regions of the detector array to achieve compressed observation; the lens group is used to magnify or reduce the image on the end face of the special optical fiber bundle to match the size of the detector array; the detector array is used to collect the compressed observed image.

[0065] After that, as shown in step S3, the reconstruction module is responsible for reconstructing the multi-spectral image data of the target endoscopic scene from the compressed observed image using algorithms and computing devices to complete endoscopic detection.

[0066] Among them, the reconstruction module includes an image preprocessing algorithm, a deep neural reconstruction network, and an image postprocessing algorithm. The main function of the image preprocessing algorithm is to perform noise reduction, gray level equalization, and cropping on the collected compressed observed image to match the input data requirements of the deep neural reconstruction network; the deep neural reconstruction network mainly performs forward propagation of the preprocessed compressed observed image in the pre-trained deep neural network to reconstruct the compressed observed image into the multi-dimensional spectral image data of the object to be measured; the main function of the image postprocessing algorithm is to perform super-resolution, color image synthesis, contrast enhancement, spectral curve noise reduction, and secondary calibration on the reconstructed spectral image to further visualize and calibrate the spectral image.

[0067] There is no limit on the wavelength range of use for all optoelectronic devices in the above method, and the working wavelength that matches should be selected according to the application scenario.

[0068] In this embodiment, Figure 2 The specific principles and descriptions of the key components in

[0069] The principle is schematically shown as Figure 3 shown. The spectral range of the spectral image of the analyte is [λ min , λ max , and its three-dimensional light field can be expressed as s(x, y, λ), representing the light field intensity at the wavelength λ and the spatial position (x, y). After the light field s(x, y, λ) is modulated and transmitted through a special fiber bundle with a mask and a transfer function h(x, y, λ), the light field S f (x, y, λ) can be expressed as:

[0070] (Equation 1)

[0071] This light field is the encoded spectral image, where h(x, y, λ) can be further expressed as:

[0072] (Equation 2)

[0073] where h f (x, y, λ) is a fiber bundle without a mask, that is, the transfer function of a traditional fiber bundle;

[0074] The light intensity transmittance t corresponding to each spatial position is a random variable M(x, y) that follows a distribution D(t|x, y), which is the random mask modulation. Here, t ∈ [0, 1], and the random mask modulation is used to perform different spatial modulations on different wavelengths to ensure the independence of spectral images of different wavelengths for subsequent reconstruction. Preferably, due to the fact that the special fiber bundle with a mask also needs to have the function of propagating the illumination light source and considering the light flux that can be collected by the entire system, the overall expectation of the light intensity transmittance random variable M(x, y) of the special fiber bundle with a mask should satisfy:

[0075] (Equation 3)

[0076] where γ is a specified constant used to limit the overall light throughput of the special fiber bundle with a mask, and Ω represents the spatial region corresponding to the special fiber bundle. Usually, γ > 0.1. An end face of a special fiber bundle with a mask that conforms to the above description is shown as Figure 4 shown. The black circular area represents that the light intensity transmittance of the corresponding sub-fiber is 0, and the white circular area represents that the light intensity transmittance of the corresponding sub-fiber is 1. The image of the analyte after being modulated by this fiber bundle is shown as Figure 5 shown.

[0077] It should be noted that the principle of the present invention does not limit the outer contour shape and the internal core shape of the fiber bundle used. Further, a method that can be referred to for preparing the described masked special fiber bundle is as follows: in the wire arranging process of the fiber bundle commonly used in the endoscope system, part of the ordinary core glass is replaced with other materials having a refractive index distribution different from that of the ordinary core glass according to the pre-designed spatial distribution M(x,y), and at the same time, it is ensured that the drawing tension of the replaced material is the same as that of the ordinary core glass. Then, the subsequent process flow is the same as that for preparing the fiber bundle of the ordinary endoscope system, eliminating the high-precision mask and its supporting optical registration system in the traditional compressed coding single-exposure system, simplifying the processing flow and reducing the system cost.

[0078] After that, the optical field after being encoded and modulated by the masked special fiber bundle will enter the dispersion element and undergo dispersion modulation, causing the optical fields of different wavelengths to have different offsets in a specific axial direction. Assuming that the axial direction here is the y-axis, then on the detector array plane, the optical field after dispersion offset, that is, the dispersed spectral image S shift (x,y,λ) can be expressed as:

[0079] (Equation Four)

[0080] where d(λ) represents the dispersion offset amount generated by the dispersion element on the detector array plane.

[0081] After that, a pixel spatial coordinate system (u,v) is introduced, which represents the row and column of the pixel respectively. Therefore, on the detector array plane, the photocurrent intensity Y(u,v) that can be collected by each pixel can be expressed as:

[0082] (Equation Five)

[0083] where R(x,y,λ) represents the responsivity of different regions of the detector array to different wavelengths. Further, for quantitative representation of Equation Five, it is defined that the number of pixels in the x and y directions of the spatial dimension is N x , N y , the spectral range [λ min ,λ max contains N λ channels, and the three-dimensional spectrum of the object to be measured is represented as a vector , then Y(u,v) in Equation Five can be represented as a vector, and expanded as:

[0084] (Equation Six)

[0085] where It is the forward model of the system for the special optical fiber bundle module and the dispersion acquisition module, which is obtained through a calibration process in actual use; y can be called the compressive observation; the above formula six can be called the system forward model. In an actual system, due to the existence of system noise, the system forward model will degenerate into:

[0086] y = Hs + g (Formula Seven)

[0087] It should be noted that when the method of the present invention is actually applied, the method of collecting H is different from the traditional single-exposure compressive coding imaging method, because in the present invention, the mask plane is the mask-carrying special optical fiber bundle plane, and the system of the present invention is an active illumination system. Since the two optical end faces of the special optical fiber bundle module are conjugate, the present invention obtains the system forward model H by collecting the image of the special optical fiber bundle module near the coupler end face at the detector array. Preferably, when collecting H, the object-side end face of the special optical fiber bundle module should be as close as possible to a spatially uniform and spectrally flat reflection plate to achieve uniform illumination of the mask-carrying special optical fiber bundle. After the collection of H is completed, the relative spatial relationship between the detector array plane and the special optical fiber bundle module near the coupler end face should remain unchanged to minimize the forward model noise g.

[0088] As described in the above formula, the method of the present invention collects two-dimensional compressive observations in a single-exposure manner without a scanning mechanism, so it can be collected at the video frame rate. Further, to recover the three-dimensional spectral image s of the object to be measured from the compressive observation y, it is necessary to solve the ill-posed inverse problem of Formula Seven. Since the natural spectral image signal has sparse characteristics, according to the compressive sensing theory, when N λ >> 1, the signal can still be recovered from the compressive observation, so the compressive observation will be sent to the reconstruction module for spectral reconstruction.

[0089] The reconstruction module of the present invention mainly uses a deep learning algorithm to obtain a three-dimensional spectral image from the compressive observation, that is, the key component is the deep neural reconstruction network. This deep learning algorithm framework includes two steps: training and inference, as Figure 6 shown. Step one is the deep neural network training stage:

[0090] First, prepare a spectral image data set that matches the system channels , where the subscript n represents the batch of training data;

[0091] Obtain the system forward model H of the acquisition module;

[0092] Obtain the compressive observation y of the data through Formula Six n ;

[0093] Input H and y nInput it into the deep neural network for forward propagation operation to obtain the reconstructed spectral image s n ;

[0094] Input the data and s n into the loss function to calculate the loss value;

[0095] Perform backpropagation operation on the model according to the loss value to update the parameters of the deep neural network;

[0096] After updating the neural network parameters, repeat the operations of inputting the next batch of dataset and compressed observations into the deep neural network and backpropagation until the loss value is less than the target loss value, then stop training, save the current parameters of the deep neural network model, and step one ends.

[0097] Step two is the inference stage of the deep neural network, that is, the actual application stage:

[0098] Input the compressed observation y collected by the acquisition module and the system forward model H into the trained deep neural network to obtain the reconstructed three-dimensional spectral image s. It should be noted that since the spatial distribution of the endoscopic fiber bundle also participates in the training and is learned by the neural network during the training of this method, the interference of the spatial distribution of the fiber bundle on the image will also be reconstructed and restored. The hyperspectral data collected by the traditional hyperspectral camera scheme will be an image modulated by the spatial distribution of the endoscopic fiber bundle, which will deteriorate the image details to a certain extent. The time-consuming of the inference stage in step two is usually in milliseconds, and the time-consuming mainly depends on the parameters of the dispersion acquisition module, the neural network parameters, and the computing resources. After obtaining the trained deep neural network model in the training stage of step one, the reconstruction module only needs to perform image reconstruction through the inference stage of step two and no longer depends on step one. Therefore, the time to obtain a spectral image for the entire system can be in milliseconds.

[0099] Preferably, the deep neural reconstruction network of the present invention can be preferably one of the following types or a combination thereof:

[0100] Convolutional neural network (CNN): Suitable for processing image data, it extracts multi-scale features through operations such as multi-layer convolution, pooling, and non-linear activation, and is especially suitable for restoring multi-spectral images with rich details from compressed observation images. CNN can effectively capture the spatial local correlation of images and is suitable for the super-resolution and color image synthesis tasks of spectral images in the present invention.

[0101] Recurrent Neural Network (RNN) or Long Short-Term Memory Network (LSTM): For spectral data with time series characteristics, such as consecutive frame spectral images in dynamic endoscopic detection scenarios, RNN or LSTM can model time-dependent relationships and effectively utilize historical information to assist in the reconstruction of the current frame. Although the present invention does not explicitly involve time series data, in some application scenarios, such as the processing of real-time endoscopic detection video streams, these network structures may be beneficial choices.

[0102] Generative Adversarial Network (GAN): By simultaneously training a Generator and a Discriminator, GAN can learn to generate realistic spectral images under unsupervised or weakly supervised conditions. This network structure helps to improve the quality and authenticity of the reconstructed images, especially in endoscopic detection applications that require high-fidelity restoration.

[0103] Transformer architecture: Although initially applied to natural language processing, Transformer has been successfully applied to image processing tasks due to its powerful representation ability in processing sequence data with self-attention mechanism. Especially for large-scale, high-resolution spectral images, Transformer can effectively capture long-range dependencies and achieve globally context-aware reconstruction.

[0104] Deep Residual Network (ResNet): By introducing residual blocks, ResNet can alleviate the vanishing gradient problem in deep networks, facilitating the training of deep networks to extract more complex spectral image features. Its good training stability and excellent performance have been widely verified in tasks such as image recognition and super-resolution.

[0105] U-Net architecture: Particularly suitable for image segmentation and reconstruction tasks, its unique encoder-decoder structure, combined with skip connections, can retain low-level detail information while integrating high-level semantic features, making it very suitable for the reconstruction of fine spectral images in endoscopic detection scenarios.

[0106] In this embodiment, the feasibility of the present invention can be verified through simulation experiments. In the simulation experiments, first, a special optical fiber bundle with a mask is generated, which contains approximately 20,000 cores in total, such as Figure 7As shown in the figure, the images of the fiber optic bundle with and without a mask are compared, and the resolution is 1024×1024; the acquisition band of the imaging system has a total of 40 channels, and the central wavelength of the channels is [706.5, 686.1, 667.2, 649.5, 632.9, 617.1, 602.4, 589.5, 578.2, 567.8, 558.1, 549.1, 540.4, 532.0, 523.9, 516.0, 508.4, 501.2, 494.4, 488.1, 482.0, 476.2, 470.5, 465.0, 459.7, 454.8, 450.1, 445.7, 441.7, 438.0, 433.1, 428.9, 424.8, 420.9, 417.1, 413.4, 409.8, 406.4, 403.1, 400.0] nm; the imaging spatial resolution is 1024×1024; the training data set and test samples use the data taken by a 400-700 nm commercial spectral camera. Figure 8 The reference RGB image synthesized from two test samples and the compressed observation image obtained by simulating this system are shown; the deep learning algorithm uses an existing model based on a convolutional neural network and uses Figure 6 the method shown for training and reconstructing test samples. The test results of the model output are visualized, and the reconstructed spectral images of the two test data are as Figure 9 shown. At the same time, the spectra of three local regions of the original data and the reconstructed results of the test samples are simulated and compared, as Figure 10 shown. It can be seen that the reconstructed results have high definition and good spectral accuracy, and at the same time, the reconstructed results remove the sampling aperture effect brought by the fiber optic bundle.

[0107] Embodiment 2

[0108] Based on the same concept, this application also proposes an endoscopic detection device based on single-exposure spectral imaging of a special fiber optic bundle, including:

[0109] A special fiber optic bundle module, including a first lens group, an illumination system, a masked special fiber optic bundle, and a coupler, where the first lens group images the target scene to be measured onto the end face of the masked special fiber optic bundle and realizes the coupling of spatial energy to the fiber waveguide structure, the illumination system provides an illumination light source that enters the masked special fiber optic bundle through the coupler, and the masked special fiber optic bundle encodes and transmits the target scene image through alternately arranged photon-passing fibers and non-photon-passing or partially photon-passing fibers;

[0110] The dispersion acquisition module includes a dispersion device, a second lens group, and a detector array. The spatially modulated observation image is sent into the dispersion acquisition module for dispersion modulation to form a compressed observation image. The dispersion device spectrally disperses and modulates the outgoing light beam, causing the encoded images of the object to be measured at different wavelengths to be imaged on different lateral regions of the detector array. The second lens group is used to magnify or reduce the image of the end face of the special optical fiber bundle to match the size of the detector array. The detector array is used to acquire the compressed observation image;

[0111] The reconstruction module includes an image preprocessing algorithm, a deep neural reconstruction network, and an image postprocessing algorithm. The image preprocessing algorithm performs noise reduction, gray level equalization, and cropping on the compressed observation image. The deep neural reconstruction network reconstructs the preprocessed compressed observation image into multi-dimensional spectral image data of the object to be measured through forward propagation. The image postprocessing algorithm performs super-resolution, color image synthesis, contrast enhancement, spectral curve noise reduction, and secondary calibration on the reconstructed spectral image to achieve visualization and calibration of the spectral image.

[0112] Among them, the special optical fiber bundle with a mask is formed by replacing some of the core glass with a material having a different refractive index distribution according to a preset spatial distribution, and ensuring that the replaced material has the same drawing tension as the original material.

[0113] Embodiment III

[0114] This embodiment also provides an electronic device. Referring to Figure 11 , it includes a memory 404 and a processor 402. A computer program is stored in the memory 404, and the processor 402 is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0115] Specifically, the above-mentioned processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured as one or more integrated circuits for implementing the embodiments of the present application.

[0116] Among them, the memory 404 may include a mass storage 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include removable or non-removable (or fixed) media. In appropriate cases, the memory 404 may be internal or external to the data processing device. In a particular embodiment, the memory 404 is a non-volatile memory. In a particular embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. In appropriate cases, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended date out dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0117] The memory 404 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402.

[0118] By reading and executing the computer program instructions stored in the memory 404, the processor 402 implements any of the endoscopic detection methods based on single-exposure spectral imaging using a special optical fiber bundle in the above embodiments.

[0119] Optionally, the above electronic device may further include a transmission device 406 and an input / output device 408. Among them, the transmission device 406 is connected to the above processor 402, and the input / output device 408 is connected to the above processor 402.

[0120] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above network may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0121] The input / output device 408 is used to input or output information.

[0122] Embodiment 4

[0123] This embodiment also provides a readable storage medium, in which a computer program is stored. The computer program includes program code for controlling a process to execute the process. The process includes the endoscopic detection method based on single-exposure spectral imaging using a special optical fiber bundle according to Embodiment 1.

[0124] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and will not be repeated here.

[0125] Generally, various embodiments can be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the present invention can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention can be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, as a non-limiting example, the blocks, devices, systems, technologies, or methods described herein can be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or a controller or other computing device, or some combination thereof.

[0126] Embodiments of the present invention can be implemented by computer software, which is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. A computer software or program (also referred to as a program product), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product can include one or more computer-executable components configured to perform the embodiments when the program runs. One or more computer-executable components can be at least one software code or a part thereof. Additionally, in this regard, it should be noted that any box in the logical flow in the figure can represent a program step, or interconnected logical circuits, boxes, and functions, or a combination of program steps and logical circuits, boxes, and functions. The software can be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs. The physical media is a non-transitory medium.

[0127] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as falling within the scope described in this specification.

[0128] The above embodiments only represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An endoscope detection method based on single exposure spectrum imaging of a special optical fiber bundle, characterized in that: The following steps are involved: S1, imaging the object to be measured and performing spatial distribution modulation through a masked special optical fiber bundle to obtain an observed image; Among them, the masked special optical fiber bundle is used to illuminate, encode and transmit the target scene image at the same time, and the sub-fibers that can transmit light of the masked special optical fiber bundle are used to transmit the image and the illumination light beam, and the sub-fibers that cannot transmit light or partially transmit light and the sub-fibers that can transmit light are alternately arranged in the horizontal space to realize the encoding of the image; The target scene of the object to be measured is imaged to the end face of the masked special optical fiber bundle through the first lens group and the coupling of spatial energy to the optical fiber waveguide structure is realized. The illumination light source is provided by the illumination system and enters the masked special optical fiber bundle through the coupler, so that the masked special optical fiber bundle encodes and transmits the target scene image through the alternately arranged light-permeable optical fibers and light-impermeable or partially light-permeable optical fibers; S2, performing dispersion modulation and compression acquisition on the observed image to obtain a compressed observed image; S3, reconstructing target endoscopy scene multispectral image data from the compressed observation image; Among them, reconstruction includes image preprocessing algorithm, deep neural reconstruction network and image post-processing algorithm.

2. The endoscopic detection method based on special optical fiber bundle single exposure spectral imaging according to claim 1, characterized in that: In step S2, the light beam of the observed image is split and dispersion-modulated by a dispersion device, so that the encoded images of the object to be measured of different wavelengths are imaged in different lateral areas of the detector array.

3. The endoscopic detection method based on special optical fiber bundle single exposure spectral imaging according to any one of claims 1 to 2, characterized in that: In step S3, the image preprocessing algorithm performs noise reduction, grayscale equalization, and cropping on the compressed observation image. The deep neural reconstruction network reconstructs the preprocessed compressed observation image into multi-dimensional spectral image data of the object to be measured through forward propagation. The image post-processing algorithm performs super-resolution, color image synthesis, contrast enhancement, spectral curve noise reduction, and secondary calibration on the reconstructed spectral image to achieve visualization and calibration of the spectral image.

4. The endoscope detection method based on special optical fiber bundle single exposure spectral imaging according to claim 3, wherein the training step of the deep neural reconstruction network comprises: preparing a spectral image dataset and a system forward model, and obtaining a compressed observation of the spectral image dataset according to the system forward model; Inputting the compressed observation and the system forward model into a deep neural network for forward propagation to obtain a reconstructed spectral image; Inputting the spectral image data set and the reconstructed spectral image into a loss function to calculate a loss value; The model is back-propagated according to the loss value, the deep neural network parameters are updated and trained continuously until the loss value is less than the target loss value, and the training is stopped to obtain the trained model.

5. The endoscopic detection method based on special optical fiber bundle single exposure spectral imaging according to claim 4, characterized in that: The reasoning steps of the deep neural reconstruction network include: The compressed observations and the system forward model are input into the trained model to obtain a reconstructed three-dimensional spectral image.

6. An endoscope detection system based on single exposure spectral imaging of special optical fiber bundles, characterized in that: include: A special optical fiber bundle module comprises a first lens group, an illumination system, a masked special optical fiber bundle and a coupler, wherein the first lens group images a target scene to be measured onto the end face of the masked special optical fiber bundle and realizes coupling of spatial energy to an optical fiber waveguide structure, the illumination system provides an illumination light source to enter the masked special optical fiber bundle through the coupler, and the masked special optical fiber bundle encodes and transmits the target scene image through alternately arranged light-permeable sub-fibers and light-impermeable or partially light-permeable sub-fibers; the masked special optical fiber bundle is used to illuminate, encode and transmit the target scene image at the same time, the light-permeable sub-fibers of the masked special optical fiber bundle are used to transmit images and illumination beams, and the light-impermeable or partially light-permeable sub-fibers and light-permeable sub-fibers are alternately arranged in the lateral space to realize encoding of the image; A dispersion collection module, comprising a dispersion device, a second lens group and a detector array, wherein the spatially modulated observation image is sent to the dispersion collection module for dispersion modulation and forms a compressed observation image, the dispersion device splits and dispersion modulates the outgoing light beam, so that the coded images of the object to be measured of different wavelengths are imaged in different lateral areas of the detector array, the second lens group is used to enlarge or reduce the image of the end face of the special optical fiber bundle to match the size of the detector array, and the detector array is used to collect the compressed observation image; The reconstruction module includes an image preprocessing algorithm, a deep neural reconstruction network and an image post-processing algorithm. The image preprocessing algorithm performs noise reduction, grayscale equalization and cropping on the compressed observation image. The deep neural reconstruction network reconstructs the preprocessed compressed observation image into multi-dimensional spectral image data of the object to be measured through forward propagation. The image post-processing algorithm performs super-resolution, color image synthesis, contrast enhancement, spectral curve noise reduction and secondary calibration on the reconstructed spectral image to achieve visualization and calibration of the spectral image.

7. The endoscope detection system based on special optical fiber bundle single exposure spectrum imaging as claimed in claim 6, characterized in that: The masked special optical fiber bundle replaces part of the core glass with a material with a different refractive index distribution according to a preset spatial distribution, and ensures that the drawing tension of the replaced material is consistent with that of the original material.

8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the endoscopic detection method based on special optical fiber bundle single exposure spectral imaging according to any one of claims 1 to 5.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes the endoscopic detection method based on special optical fiber bundle single exposure spectral imaging according to any one of claims 1 to 5.

Images