Electronic endoscopic imaging system and method

Abstract

This application relates to an electronic endoscopic imaging system and method. The system includes: a light source device for continuously emitting visible light and alternately emitting first near-infrared light and second near-infrared light to an object under observation. Visible light is reflected off the object to form reflected light, and near-infrared light is excited on the object to generate excitation light. An image acquisition device is located in the optical path of the reflected light, the first excitation light, and the second excitation light. An image processing device, electrically connected to the image acquisition device, is used to fuse the visible light image and the first excitation light image formed simultaneously into a first fused image, and to fuse the visible light image and the second excitation light image formed simultaneously into a second fused image, and then generate a target fused image. This allows information about different penetration depths of the object under observation to be presented in a single image, facilitating simultaneous observation of the metabolic and vascular morphological information of the object by the physician, thus aiding in diagnosis and treatment.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to an electronic endoscopic imaging system and method. Background Technology

[0002] With the development of medical technology, in order to clearly observe the lesions in the patient's body and perform surgery accurately, endoscopic technology has emerged. Endoscopes can expand the surgical field of view, obtain images of the lesions, help doctors observe the lesions, and improve doctors' diagnostic and treatment capabilities.

[0003] In traditional techniques, images of the lesion are obtained by emitting visible and near-infrared light toward the lesion.

[0004] However, in practice, in order to observe tissue metabolism during surgery and avoid cutting blood vessels, it is necessary to obtain metabolic information and vascular morphology information of the lesion site at the same time. However, traditional techniques cannot obtain images that simultaneously include metabolic information and vascular morphology information of the lesion site. Summary of the Invention

[0005] Therefore, it is necessary to address the aforementioned technical problems by providing an electronic endoscopic imaging system and method capable of generating images that simultaneously include metabolic information and vascular morphology information of the lesion site.

[0006] An electronic endoscope imaging system includes: a light source device for continuously emitting visible light and alternately emitting first near-infrared light and second near-infrared light to an object to be observed; wherein the wavelength of the first near-infrared light is shorter than the wavelength of the second near-infrared light; the visible light is reflected on the object to be observed to form reflected light, the first near-infrared light excites the object to be observed to generate a first excitation light, and the second near-infrared light excites the object to be observed to generate a second excitation light, wherein the object to be observed contains a developer corresponding to the first near-infrared light; an image acquisition device located in the optical path of the reflected light, the first excitation light, and the second excitation light, for acquiring the reflected light to form a visible light image, acquiring the first excitation light to form a first excitation light image, and acquiring the second excitation light to form a second excitation light image; and an image processing device electrically connected to the image acquisition device, for fusing the visible light image and the first excitation light image formed at the same time into a first fused image, fusing the visible light image and the second excitation light image formed at the same time into a second fused image, and fusing the first fused image and the second fused image generated at adjacent times into a target fused image.

[0007] In one embodiment, the image acquisition device includes: a beam splitter prism located on the optical paths of the reflected light, the first excitation light, and the second excitation light, for changing the optical path of at least one of the reflected light, the first excitation light, and the second excitation light, so that the optical path of the reflected light is not coplanar with the optical paths of the first excitation light and the second excitation light; a first image sensor group located on the optical path of the reflected light passing through the beam splitter prism, for acquiring the reflected light to form a visible light image; and a second image sensor group located on the optical paths of the first excitation light and the second excitation light passing through the beam splitter prism, for acquiring the first excitation light to form a first excitation light image and acquiring the second excitation light to form a second excitation light image.

[0008] In one embodiment, the beam splitter includes a first triangular prism and a second triangular prism glued together with inclined surfaces. The inclined surface of the first triangular prism forms an acute angle of 45° with the optical paths of the reflected light, the first excitation light, and the second excitation light. This is used to change the optical path of at least one of the reflected light, the first excitation light, and the second excitation light, so that the optical path of the reflected light is perpendicular to the optical paths of the first excitation light and the second excitation light. The first triangular prism and the second triangular prism are both right-angled triangular prisms with an angle of 45°.

[0009] In one embodiment, the inclined surface of the first triangular prism is coated with a semi-transparent, semi-reflective multilayer dielectric film for reflecting the reflected light and transmitting the first excitation light and the second excitation light; the inclined surface of the second triangular prism is coated with a high-transmission film for transmitting the first excitation light and the second excitation light.

[0010] In one embodiment, the image acquisition device further includes a light guide lens group located in the optical path of the reflected light, the first excitation light, and the second excitation light, for converging the reflected light, the first excitation light, and the second excitation light into the beam splitter.

[0011] In one embodiment, the image acquisition device further includes a filter located in the optical path of the reflected light, the first excitation light, and the second excitation light, for filtering out the reflected light from the first excitation light and the second excitation light.

[0012] In one embodiment, the light source device includes: a light source emitting module for emitting at least one of the visible light, the first near-infrared light, and the second near-infrared light; and a driving module connected to the light source emitting module for driving the light source emitting module to continuously emit the visible light toward the object to be observed, and to alternately emit the first near-infrared light and the second near-infrared light.

[0013] In one embodiment, the light source emitting module includes: a visible light emitting module for emitting visible light; a first near-infrared light emitting module for emitting the first near-infrared light; and a second near-infrared light emitting module for emitting the second near-infrared light. The driving module includes: a visible light driving module connected to the visible light emitting module for providing driving power to the visible light emitting module; a first near-infrared light driving module connected to the first near-infrared light emitting module for providing driving power to the first near-infrared light emitting module; a second near-infrared light driving module connected to the second near-infrared light emitting module for providing driving power to the second near-infrared light emitting module; and a timing control module connected to the first near-infrared light driving module and the second near-infrared light driving module respectively, for controlling the first near-infrared light driving module and the second near-infrared light driving module to work alternately according to a preset timing signal.

[0014] In one embodiment, the first image sensor group includes two visible light image sensors arranged side by side and coplanarly; the second image sensor group includes two near-infrared light image sensors arranged side by side and coplanarly.

[0015] An electronic endoscope imaging method, the method comprising:

[0016] Visible light is continuously emitted towards the object to be observed, and first near-infrared light and second near-infrared light are emitted alternately; wherein the wavelength of the first near-infrared light is shorter than the wavelength of the second near-infrared light; the visible light is reflected on the object to be observed to form reflected light, the first near-infrared light excites the object to be observed to generate a first excitation light, and the second near-infrared light excites the object to be observed to generate a second excitation light, wherein the object to be observed contains a developer corresponding to the first near-infrared light;

[0017] The reflected light is collected to form a visible light image, the first excitation light is collected to form a first excitation light image, and the second excitation light is collected to form a second excitation light image;

[0018] The visible light image and the first excitation light image formed at the same time are fused into a first fused image, the visible light image and the second excitation light image formed at the same time are fused into a second fused image, and the first fused image and the second fused image generated at adjacent times are fused into a target fused image.

[0019] The aforementioned electronic endoscopic imaging system and method continuously emit visible light and alternately emits first near-infrared light and second near-infrared light towards the object to be observed using a light source device. When the object is irradiated with visible light, it reflects the visible light, forming reflected light. Under the first near-infrared light, the object is excited to emit a first excitation light, and under the second near-infrared light, it is excited to emit a second excitation light. An image acquisition device can acquire the reflected light, as well as the excited first and second excitation lights, and generate a visible light image based on the reflected light. A first excitation light image and a second excitation light image are also generated based on the first and second excitation lights, respectively. Since the wavelength of the first near-infrared light is shorter than that of the second near-infrared light, its penetration depth is less. The first excitation light image obtained using the first near-infrared light and the second excitation light image obtained using the second near-infrared light contain content at different depths of the object to be observed; therefore, images of the object at different penetration depths can be obtained. The system uses an image processing device to fuse a visible light image and a first excitation light image formed at the same time into a first fused image, and to fuse a visible light image and a second excitation light image formed at the same time into a second fused image. Furthermore, it fuses the first fused image and the second fused image generated at adjacent times into a target fused image. Since the first near-infrared light and the second near-infrared light are emitted alternately, the frame rate of both the first excitation light image and the second excitation light image is half that of the visible light image. This ensures that each frame of the first fused image has a corresponding second fused image, resulting in a target fused image containing a one-to-one correspondence between the first fused image and the second fused image. This results in a higher frame rate and clearer, more coherent target fused image, facilitating the simultaneous presentation of information about different depths of the observed object. In summary, the system of this application can present information about different penetration depths of the observed object on a single image, allowing doctors to simultaneously observe the metabolic and vascular morphological information of the observed object, thus facilitating diagnosis and treatment. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of an electronic endoscopic imaging system in one embodiment;

[0022] Figure 2 This is a schematic diagram of the fusion process of a visible light image and a first excitation light image in one embodiment;

[0023] Figure 3 This is a schematic diagram of the fusion process of a visible light image and a second excitation light image in one embodiment;

[0024] Figure 4 This is a schematic diagram of image fusion in one embodiment;

[0025] Figure 5 This is a schematic diagram of the structure of an image acquisition device in one embodiment;

[0026] Figure 6 This is a schematic diagram of the front lens barrel of the image acquisition device in one embodiment;

[0027] Figure 7 This is a schematic diagram of the image 3D fusion process in one embodiment;

[0028] Figure 8 This is a front view of the structure of an image acquisition device in one embodiment;

[0029] Figure 9 This is a structural side view of an image acquisition device in one embodiment;

[0030] Figure 10 This is a graph showing the light intensity detected by the image sensor in one embodiment;

[0031] Figure 11 This is a schematic diagram of the image acquisition device in another embodiment;

[0032] Figure 12 This is a schematic diagram of the light guide lens assembly in one embodiment;

[0033] Figure 13 This is a spectrum of the output beam in one embodiment;

[0034] Figure 14 This is a spectrum of the light beam collected in one embodiment;

[0035] Figure 15 This is a schematic diagram of the signal transmission process in one embodiment;

[0036] Figure 16 This is a schematic diagram of the structure of a light source device in one embodiment;

[0037] Figure 17 This is a detailed structural diagram of the light source device in one embodiment;

[0038] Figure 18 This is a timing diagram of near-infrared light emission in one embodiment;

[0039] Figure 19 This is a schematic diagram of the signal transmission of an electronic endoscope imaging system in one embodiment;

[0040] Figure 20 This is a schematic diagram of the complete workflow of an electronic endoscopic imaging system in one embodiment;

[0041] Figure 21 This is a schematic diagram of the signal transmission output by the light source in one embodiment;

[0042] Figure 22 This is a flowchart of an electronic endoscopic imaging method in one embodiment.

[0043] Explanation of reference numerals in the attached figures: 10-Light source device, 20-Image acquisition device, 30-Image processing device, 11-Light source emission module, 12-Drive module, 21-Beam splitter prism, 22-First image sensor group, 23-Second image sensor group, 211-First triangular prism, 212-Second triangular prism, 100-First plane, 200-Second plane, 221-First visible light sensor, 222-Second visible light sensor, 231-First near-infrared light sensor, 232-Second near-infrared light sensor Sensor, 300 - Light source outlet, 251 - First lens, 252 - Second lens, 24 - Light guide lens group, 240 - Light guide beam, 241 - Lens group, 400 - Visible light, 500 - Excitation light, 110 - Visible light emitting module, 111 - First near-infrared light emitting module, 112 - Second near-infrared light emitting module, 120 - Visible light driving module, 121 - First near-infrared light driving module, 122 - Second near-infrared light driving module, 123 - Timing control module, 40 - Display device. Detailed Implementation

[0044] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0046] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another.

[0047] Spatial relation terms such as “below,” “under,” “below,” “under,” “above,” “above,” etc., are used herein to describe the relationship between one element or feature shown in the figure and other elements or features. It should be understood that, in addition to the orientation shown in the figure, spatial relation terms also include different orientations of the device in use and operation. For example, if the device in the figure is flipped, the element or feature described as “below,” “under,” or “below” will be oriented “above” the other element or feature. Therefore, the exemplary terms “below” and “under” can include both above and below orientations. Furthermore, the device may also include other orientations (e.g., rotated 90 degrees or other orientations), and the spatial descriptive terms used herein will be interpreted accordingly.

[0048] It should be noted that when one element is considered to be "connected" to another element, it can be directly connected to the other element or connected to the other element through an intermediary element. Furthermore, in the following embodiments, "connection" should be understood as "electrical connection," "communication connection," etc., if there is transmission of electrical signals or data between the connected objects.

[0049] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.

[0050] As described in the background section, existing endoscopic imaging systems suffer from the problem of being unable to acquire images that simultaneously include metabolic and vascular morphology information of the lesion site. The inventors have discovered that this problem arises because existing endoscopic imaging systems emit light beams including visible light and near-infrared light of a fixed wavelength. Since the penetration depth of near-infrared light of the same wavelength is fixed, it can only acquire image information of a fixed depth within the tissue of the lesion site. Metabolic and vascular morphology information are located at different tissue depths; therefore, using only near-infrared light of a fixed wavelength allows only one type of information—metabolic or vascular morphology—to be observed.

[0051] For the reasons mentioned above, the present invention provides an electronic endoscopic imaging system and method capable of generating images that simultaneously include metabolic information and vascular morphology information of the lesion site.

[0052] In one embodiment, such as Figure 1As shown, an electronic endoscopic imaging system is provided, the system comprising: a light source device 10, an image acquisition device 20, and an image processing device 30. Wherein:

[0053] The light source device 10 is used to continuously emit visible light to the object to be observed, and to alternately emit first near-infrared light and second near-infrared light.

[0054] Specifically, the wavelength of the first near-infrared light is shorter than that of the second near-infrared light; visible light is reflected on the object to be observed to form reflected light; the first near-infrared light excites the object to be observed to generate the first excitation light; the second near-infrared light excites the object to be observed to generate the second excitation light; and the object to be observed contains a developer corresponding to the first near-infrared light.

[0055] For example, when the contrast agent is indocyanine green, the wavelength of the first near-infrared light is 780nm-811nm. The contrast agent is present in the tissue metabolism of the object being observed. When excited by sufficiently intense near-infrared light, the electrons in the contrast agent enter an excited state and then transition back to the ground state, emitting fluorescence. Generally, about 97% of indocyanine green (contrast agent) is eliminated from the bloodstream 20 minutes after intravenous injection. When liver tumors or cirrhotic nodules are present, the biliary excretion function of hepatocytes in the diseased liver tissue is impaired. Indocyanine green (contrast agent) is targeted and retained in the diseased tissue, exhibiting delayed decay, thus distinguishing it from surrounding normal tissue and allowing the diseased tissue to be visualized. Since the excitation wavelength of indocyanine green (contrast agent) is 808nm, the first near-infrared light can excite the contrast agent to produce fluorescence. The wavelength of the first excitation light generated by exciting indocyanine green (contrast agent) is 830-850nm, preferably 835nm. This allows the metabolic information of the object being observed to be reflected. The developer can also be other chemical substances, such as Rhodamine series dyes, which have an excitation wavelength of 520-600nm. Therefore, when using Rhodamine series dyes, the wavelength of the first near-infrared light is 520-600nm. The excitation wavelength of the Cy (Cyanine) series dyes is 550-780nm, so when using Cy (Cyanine) series dyes, the wavelength of the first near-infrared light is 550-780nm. The excitation wavelength of the Alexa Fluor series dyes is 340-680nm, so when using Alexa Fluor series dyes, the wavelength of the first near-infrared light is 340-680nm.

[0056] For example, the wavelength of the second near-infrared light is 855nm-865nm. Near-infrared light of this wavelength has a penetration depth of approximately 6nm, allowing it to penetrate deeper into the object being observed, thus reflecting the vascular morphology of the object. The wavelength of the second excitation light generated is also 855nm-865nm, thereby reflecting the vascular morphology information of the object being observed.

[0057] Image acquisition device 20 is located in the optical path of reflected light, first excitation light and second excitation light, and is used to acquire reflected light to form a reflected light image, acquire the first excitation light to form a first excitation light image and acquire the second excitation light to form a second excitation light image.

[0058] Specifically, the image acquisition device 20 includes a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, which can convert optical signals into electrical signals.

[0059] Image processing device 30 is electrically connected to image acquisition device 20 and is used to fuse a visible light image and a first excitation light image formed at the same time into a first fused image, fuse a visible light image and a second excitation light image formed at the same time into a second fused image, and fuse the first fused image and the second fused image generated at adjacent times into a target fused image.

[0060] Specifically, such as Figure 2 The diagram illustrates the fusion process of the visible light image and the first excitation light image. Visible light signals are processed using visible light ISP (Image Signal Processing) to obtain the visible light image. The first excitation light signal is then processed using near-infrared ISP to obtain the first excitation light image. Next, the first excitation light image undergoes grayscale processing to enhance brightness and contrast. Thresholding segmentation is then performed to determine different information within the image. Fluorescence coloring is then applied. The first excitation light image is then registered with the visible light image at the same time point. A global motion vector estimation algorithm is used to register the first excitation light image and the visible light image within the same frame. Finally, the first excitation light image and the visible light image are fused using the Laplacian pyramid algorithm. A three-layer pyramid fusion algorithm is employed to balance image quality and fusion speed, resulting in the first fused image, which includes both fluorescence and visible light information.

[0061] Specifically, such as Figure 3The diagram illustrates the fusion process of the visible light image and the second excitation light image. The second excitation light signal is processed using near-infrared ISP to obtain the second excitation light image. Then, the second excitation light image undergoes grayscale processing to enhance brightness and contrast, followed by Gaussian noise reduction to smooth the image. Next, a Hessian matrix is ​​used to segment the second excitation light image, extracting the vascular tissue. The extracted vascular tissue is then electronically stained. The second excitation light image is then registered with the visible light image at the same time point. A global motion vector estimation algorithm is used to register the second excitation light image and the visible light image in the same frame. Finally, the second excitation light image and the visible light image are fused using the Laplacian pyramid algorithm. Due to the complexity of the vascular image, a two-layer pyramid algorithm is used for fusion, balancing image quality and fusion speed, to obtain the second fused image. The second fused image includes both vascular staining information and visible light information.

[0062] For example, such as Figure 4 As shown, the timing of the first excitation light image and the second excitation light image is alternating, while the visible light image is continuous. A frame of the first excitation light image and a frame of the visible light image at the same time are fused to obtain the first fused image. A frame of the second excitation light image and a frame of the visible light image at the same time are fused to obtain the second fused image. Then, the first fused image and the second fused image at adjacent times are fused again to obtain the target fused image.

[0063] In this embodiment, a light source device continuously emits visible light and alternately emits first near-infrared light and second near-infrared light towards the object to be observed. When the object is irradiated by visible light, it reflects the visible light, forming reflected light. Under the irradiation of the first near-infrared light, the object is excited to emit a first excitation light, and under the irradiation of the second near-infrared light, it is excited to emit a second excitation light. An image acquisition device can acquire the reflected light, as well as the excited first and second excitation lights, and generate a visible light image based on the reflected light. A first excitation light image and a second excitation light image are also generated based on the first and second excitation lights, respectively. Since the wavelength of the first near-infrared light is shorter than that of the second near-infrared light, the penetration depth of the first near-infrared light is less than that of the second near-infrared light. The first excitation light image obtained using the first near-infrared light and the second excitation light image obtained using the second near-infrared light contain content at different depths of the object to be observed; therefore, images of the object at different penetration depths can be obtained. The system uses an image processing device to fuse a visible light image and a first excitation light image formed at the same time into a first fused image, and to fuse a visible light image and a second excitation light image formed at the same time into a second fused image. Furthermore, it fuses the first fused image and the second fused image generated at adjacent times into a target fused image. Since the first near-infrared light and the second near-infrared light are emitted alternately, the frame rate of both the first excitation light image and the second excitation light image is half that of the visible light image. This ensures that each frame of the first fused image has a corresponding second fused image, resulting in a target fused image containing a one-to-one correspondence between the first fused image and the second fused image. This results in a higher frame rate and clearer, more coherent target fused image, facilitating the simultaneous presentation of information about different depths of the observed object. In summary, the system of this application can present information about different penetration depths of the observed object on a single image, allowing doctors to simultaneously observe the metabolic and vascular morphological information of the observed object, thus facilitating diagnosis and treatment.

[0064] In one embodiment, such as Figure 5 As shown, the image acquisition device 20 includes: a beam splitter 21, a first image sensor group 22, and a second image sensor group 23. Wherein:

[0065] The beam splitter 21 is located on the optical paths of the reflected light, the first excitation light, and the second excitation light. It is used to change the optical path of at least one of the reflected light, the first excitation light, and the second excitation light so that the optical path of the reflected light is not coplanar with the optical paths of the first excitation light and the second excitation light.

[0066] Specifically, such as Figure 6As shown, the beam splitter 21 includes a first triangular prism 211 and a second triangular prism 212, which are glued together with their inclined surfaces. The acute angle between the inclined surface of the first triangular prism 211 and the optical paths of the reflected light, the first excitation light, and the second excitation light is 45°. This prism is used to change the optical path of at least one of the reflected light, the first excitation light, and the second excitation light, so that the optical path of the reflected light is perpendicular to the optical paths of the first excitation light and the second excitation light. Both the first triangular prism 211 and the second triangular prism 212 are right-angled triangular prisms with an angle of 45°. The reflected light and the first excitation light, along with the second excitation light, which have mutually perpendicular optical paths, respectively illuminate the first plane 100 and the second plane 200.

[0067] Specifically, the inclined surface of the first triangular prism 211 is coated with a semi-transparent and semi-reflective multilayer dielectric film for reflecting reflected light and transmitting the first excitation light and the second excitation light; the inclined surface of the second triangular prism 212 is coated with a high-transmission film for transmitting the first excitation light and the second excitation light, and the inclined surface of the first triangular prism 211 faces the inclined surface of the second triangular prism 212.

[0068] For example, the beam splitter 21 is used to either reflect visible light in the incident light and transmit near-infrared light, or reflect near-infrared light in the incident light and transmit visible light. Depending on whether the beam splitter 21 specifically reflects visible light or near-infrared light, the arrangement of the first and second image sensor groups will be adaptively changed, as long as the first image sensor group is located in the visible light path and the second image sensor group is located in the near-infrared light path.

[0069] The first image sensor group 22 is located on the optical path of the reflected light after passing through the beam splitter 21, and is used to collect the reflected light to form a reflected light image.

[0070] Specifically, the first image sensor group includes two visible light image sensors arranged side-by-side and coplanar. It can simultaneously acquire two sources of visible light to form two reflected light images with horizontal parallax. Side-by-side and coplanar means that the two visible light image sensors are arranged side by side, facing the same direction, and their photosensitive areas are on the same horizontal plane.

[0071] The second image sensor group 23 is located in the optical path of the first excitation light and the second excitation light after passing through the beam splitter 21. It is used to collect the first excitation light to form the first excitation light image and to collect the second excitation light to form the second excitation light image.

[0072] Specifically, the second image sensor group includes two near-infrared image sensors arranged side-by-side and coplanar. It can simultaneously acquire two near-infrared lights to form two excitation light images with horizontal parallax. Side-by-side and coplanar means that the two near-infrared image sensors are arranged side by side, facing the same direction, and the photosensitive areas are on the same horizontal plane.

[0073] Specifically, since the beam splitter 21 separates the reflected light and excitation light into mutually perpendicular reflected and excitation light, the first and second image sensor groups are also arranged perpendicularly to each other, and can be positioned at the first plane 100 and the second plane 200, respectively. This provides a large enough gap in the arrangement space, which is beneficial for sensor heat dissipation and avoids heat accumulation at the front of the sensor lens. Furthermore, it effectively utilizes the space inside the endoscope barrel; one set of sensors is located on the cross-section of the barrel, and the other on the side wall, thus eliminating the need for extra space and not affecting the barrel diameter, maximizing the use of the space inside the barrel. This arrangement also allows both sensors to simultaneously receive visible and near-infrared light, achieving time alignment of the acquired beams and facilitating subsequent image fusion. Sufficient space is also provided for placing the beam splitter 21, whose placement completely covers the photosensitive area of ​​the image sensor, ensuring that the light signal is captured by the image sensor to the maximum extent, maximizing the utilization of the sensor's photosensitive area, and providing a foundation for subsequent high-resolution image processing.

[0074] Specifically, each image sensor group includes two sensors placed side-by-side, thus fulfilling the hardware foundation for 3D (3-dimensional) imaging. It can simultaneously receive two beams of light with horizontal parallax, thereby forming two 2D images with horizontal parallax at the same time. This allows for direct fusion of the two 2D images with horizontal parallax to generate a 3D image, and the resolution and brightness of the resulting 3D image are not attenuated compared to the original 2D image. The images acquired by the two sensors are input separately into the image processing device and processed according to the image fusion process described above. This results in two fused 2D target images with horizontal parallax at the same time. A 3D vision processing algorithm is then used to fuse these two fused 2D target images to obtain the 3D target image. The 3D target image is a stereoscopic image, allowing doctors to more clearly observe metabolic and vascular morphological information of the lesion site during diagnosis.

[0075] For example, such as Figure 7 As shown, the left-path target fusion image and the right-path target fusion image with horizontal parallax can be output directly, or they can be synthesized by 3D vision processing algorithms to obtain a 3D target fusion image.

[0076] For example, such as Figure 8 The image shown is a front view of the image acquisition device. Figure 9The image acquisition device is shown as a side view. Inside the endoscope barrel, two sets of image sensors are arranged perpendicularly to each other on one side of the reflected beam and the other side of the transmitted beam of the beam splitter 21. They can simultaneously acquire visible light and near-infrared light, with two sensors for each. The first image sensor group 22 includes a first visible light sensor 221 and a second visible light sensor 222, and the second image sensor group 23 includes a first near-infrared light sensor 231 and a second near-infrared light sensor 232. The acquired visible light and near-infrared light can be processed for 3D vision to obtain visible light 3D images and near-infrared light 3D images, respectively. Light source exit ports 300 are also provided above and below the endoscope barrel to ensure consistent light intensity within the barrel, resulting in more stable imaging. A first lens 251 and a second lens 252 are also provided in front of the beam splitter 21 to converge the light beam.

[0077] For example, 3D vision processing involves fusing two 2D images with horizontal parallax to obtain a 3D image, which can be achieved using existing image fusion software or algorithms.

[0078] For example, the image sensors are all RGB (red, green, blue) CMOS (Complementary Metal Oxide Semiconductor) sensors, such as... Figure 10 The figure shows a graph of the light intensity detected by the CMOS sensor. It can be seen that the CMOS sensor has high photon conversion efficiency in the 400–700 nm wavelength range, and also relatively high conversion efficiency (approximately 50%) in the near-infrared band (830–865 nm). This results in good imaging quality for both visible and near-infrared images generated by the CMOS sensor. QE (quantum yield): describes the photon conversion efficiency; a higher value indicates better capture of the light signal at that wavelength.

[0079] In this embodiment, a beam splitter is used to separate the reflected light and the excitation light. The reflected light and excitation light are then collected by corresponding image sensor groups, resulting in visible light and excitation light images. The two beams are perpendicular, and the two sets of sensors are also arranged perpendicularly, thus conforming to the shape of the endoscope tube. This allows for the simultaneous generation of two images, facilitating subsequent fusion. Furthermore, since both sets of sensors are two side-by-side coplanar sensors, the resulting images exhibit horizontal parallax, facilitating the generation of 3D images with no loss of resolution or brightness. This generates two high-resolution, high-brightness, and high-definition binocular visible light images and two binocular near-infrared light images, simplifying subsequent image fusion processing.

[0080] In one embodiment, such as Figure 11 As shown, the image acquisition device also includes: a light guide lens group 24.

[0081] The light guide lens group 24 is located in the optical path of the reflected light, the first excitation light and the second excitation light, and is used to converge the reflected light, the first excitation light and the second excitation light into the beam splitter 21.

[0082] For example, such as Figure 6 and Figure 12 As shown, the light guide lens group 24 includes a lens group 241, the specific structure of which is as follows: Figure 6 As shown, it includes a converging objective lens arranged inside the lens barrel, which enables the light beam to be transmitted to the subsequent beam splitter 21 along a preset trajectory.

[0083] In this embodiment, by setting up a light guide lens group, the reflected light and the excitation light can be transmitted along a specific trajectory and converged onto the beam splitter prism, which facilitates subsequent imaging.

[0084] In one embodiment, the image acquisition device 20 further includes a filter. The filter is located in the optical path of the reflected light, the first excitation light, and the second excitation light, and is used to filter out the reflected light from the first excitation light and the second excitation light.

[0085] Specifically, the filter can be an infrared cut-off filter or a lens coating layer covering the infrared cut-off filter, used to filter out infrared light in a fixed wavelength range. For example, such as Figure 13 As shown, the wavelength of visible light is 400nm-700nm, the wavelength of the first near-infrared light is 805nm-810nm, and the wavelength of the second near-infrared light is 855-865nm. Figure 14 As shown, after visible light illuminates the object under observation, the wavelength of the reflected visible light excited is 400-700 nm; after the first near-infrared light illuminates the object under observation, the near-infrared light signal excited by the object is a beam with a wavelength of 830-850 nm; after the second near-infrared light illuminates the object under observation, the near-infrared light signal excited by the object is a beam with a wavelength of 855-865 nm. Therefore, the incident light acquired by the image acquisition device may be reflected visible light with a wavelength of 400 nm-700 nm and near-infrared light with a wavelength of 830-850 nm, or reflected visible light with a wavelength of 400 nm-700 nm and near-infrared light with a wavelength of 855-865 nm. Therefore, by using a filter to filter out beams with wavelengths outside the above ranges, for example, by using an infrared cutoff filter to filter out narrow-band near-infrared light with wavelengths of 798-818 nm, it is ensured that there is no interfering near-infrared light, and all acquired light is the near-infrared light excited by the object under observation. The lens coating layer filters out light beams with wavelengths between visible light and narrow-band near-infrared light.

[0086] For example, such as Figure 15 The diagram shows the signal transmission process of the image acquisition device 20. The incident light passes sequentially through the light guide lens group, the infrared cut-off filter, and the lens coating layer, and then is split into visible light and near-infrared light by the beam splitter, which are then transmitted to the first image sensor group and the second image sensor group, respectively.

[0087] In this embodiment, by setting a filter, the light beams of other wavelengths are filtered out, and the reflected light, the first excitation light, and the second excitation light are selected, thereby ensuring that the collected light beams are all reflected or excited by the object to be observed, and avoiding interference from other ambient light.

[0088] In one embodiment, such as Figure 16 As shown, the light source device 10 includes: a light source emitting module 11 and a driving module 12, wherein:

[0089] The light source emitting module 11 is used to emit at least one of visible light, first near-infrared light, and second near-infrared light.

[0090] Specifically, the wavelength of visible light is 400nm-700nm.

[0091] Specifically, the wavelength of the first near-infrared light is 805nm-810nm, which can excite the fluorescent developer, causing the fluorescent material to emit fluorescence. The fluorescent material requires near-infrared light with a wavelength of 805nm-810nm to be excited.

[0092] Specifically, the wavelength of the second near-infrared light is 855-865nm. Its penetration depth is 6mm, allowing it to penetrate deeper into blood vessels and reveal their morphology.

[0093] Specifically, such as Figure 17 As shown, the light source emitting module 11 includes: a visible light emitting module 110, a first near-infrared light emitting module 111, and a second near-infrared light emitting module 112, wherein:

[0094] Visible light emitting module 110 is used to emit visible light.

[0095] The first near-infrared light emitting module 111 is used to emit first near-infrared light.

[0096] The second near-infrared light emitting module 112 is used to emit second near-infrared light.

[0097] The driving module 12 is connected to the light source emitting module 11 and is used to drive the light source emitting module 11 to continuously emit visible light to the object to be observed, and to alternately emit first near-infrared light and second near-infrared light.

[0098] Specifically, such as Figure 17 As shown, the driving module 12 includes: a visible light driving module 120, a first near-infrared light driving module 121, a second near-infrared light driving module 122, and a timing control module 123, wherein:

[0099] The visible light driving module 120 is connected to the visible light emitting module and is used to provide driving power to the visible light emitting module.

[0100] The first near-infrared light driving module 121 is connected to the first near-infrared light emitting module and is used to provide driving power to the first near-infrared light emitting module.

[0101] The second near-infrared light driving module 122 is connected to the second near-infrared light emitting module and is used to provide driving power to the second near-infrared light emitting module.

[0102] The timing control module 123 is connected to the first near-infrared light driving module 121 and the second near-infrared light driving module 122 respectively, and is used to control the first near-infrared light driving module 121 and the second near-infrared light driving module 122 to work alternately according to the preset timing signal.

[0103] Specifically, after receiving the preset timing signal, the timing control module 123 controls the first near-infrared light driving module 121 and the second near-infrared light driving module 122 to work alternately according to the preset timing signal, so that the first near-infrared light and the second near-infrared light are output alternately according to the preset timing signal.

[0104] For example, the preset timing signal is as follows Figure 18 As shown, for example, PWM (Pulse Width Modulation) 1 and PWM2 correspond to the first near-infrared light and the second near-infrared light, respectively. When PWM is high, near-infrared light is output. Therefore, it can be seen that the first and second near-infrared lights are output alternately. The preset timing signal is consistent with the timing of image processing by the image processing device. When the light source device emits the first near-infrared light signal, the image processing device generates a first fused image based on the visible light image and the first excitation light image; when the light source device emits the second near-infrared light signal, the image processing device generates a second fused image based on the visible light image and the second excitation light image.

[0105] Specifically, the light source device can converge visible light and first near-infrared light, or visible light and second near-infrared light, and output visible light and first near-infrared light, or visible light and second near-infrared light, by emitting a beam of light.

[0106] Specifically, the light guide module is a series of lens groups that can adjust the path of the light beam, thereby converging visible light and first near-infrared light, or visible light and second near-infrared light, into a single output beam through reflection.

[0107] In this embodiment, the light source device is a light source capable of outputting three different wavelengths of light, capable of continuously outputting white light, and capable of alternately outputting the first wavelength according to a timing signal.

[0108] Near-infrared light and second near-infrared light provide the necessary illumination for the system of this application.

[0109] In one embodiment, such as Figure 19 As shown, the electronic endoscope imaging system also includes a display device 40.

[0110] Display device 40 is connected to image processing device 30 and is used to display the images output by image processing device 30 for doctors to observe.

[0111] The image processing device 30 also outputs a preset timing signal to the light source device 10 to control the light source device 10 to alternately output white light and near-infrared light according to the preset timing signal.

[0112] For example, the preset timing signal is as follows Figure 18 As shown, for example, PWM (Pulse Width Modulation) 1 and PWM2 correspond to the first near-infrared light and the second near-infrared light, respectively. When PWM is high, near-infrared light is output. Therefore, it can be seen that the first and second near-infrared lights are output alternately. The preset timing signal is consistent with the timing of image processing by the image processing device. When the light source device emits the first near-infrared light signal, the image processing device generates a first fused image based on the visible light image and the first excitation light image; when the light source device emits the second near-infrared light signal, the image processing device generates a second fused image based on the visible light image and the second excitation light image.

[0113] For example, such as Figure 20 The diagram shows a complete workflow of an electronic endoscopic imaging system.

[0114] The image acquisition device 20 is an endoscope tube structure consisting of an endoscope lens and a detector, which can acquire external light beams.

[0115] The white light and near-infrared light emitted by the light source device 10 are emitted from the center of the image acquisition device 20 through the beam guide 240.

[0116] For example, such as Figure 21 As shown, the light source device 10 focuses the emitted light beam through the light guide beam 240 and then emits it through the image acquisition device 20. The light guide beam 240 can be an optical fiber, which can guide and transmit the light output by the light source device 10 to the image acquisition device 20.

[0117] After receiving visible light and near-infrared light, the image processing device 30 processes the visible light and near-infrared light according to the method of the above embodiment, and finally obtains the target fusion image and the 3D target fusion image after 3D visual processing, and then outputs it to the image display module 40.

[0118] In one embodiment, such as Figure 22 As shown, an electronic endoscopic imaging method is provided, the method comprising:

[0119] Step S2200: Continuously emit visible light and alternately emit first near-infrared light and second near-infrared light towards the object to be observed.

[0120] Specifically, the wavelength of the first near-infrared light is shorter than the wavelength of the second near-infrared light; visible light is reflected on the object to be observed to form reflected light; the first near-infrared light excites the object to be observed to generate the first excitation light; and the second near-infrared light excites the object to be observed to generate the second excitation light.

[0121] Step S2202: Acquire reflected light to form a reflected light image, acquire first excitation light to form a first excitation light image, and acquire second excitation light to form a second excitation light image.

[0122] Step S2204: The visible light image and the first excitation light image formed at the same time are fused into a first fused image, and the visible light image and the second excitation light image formed at the same time are fused into a second fused image.

[0123] Step S2206: The first fusion image and the second fusion image generated at adjacent time points are fused into the target fusion image.

[0124] In this embodiment, visible light is continuously emitted towards the object to be observed, and first near-infrared light and second near-infrared light are emitted alternately. When the object is irradiated by visible light, it reflects the visible light, forming reflected light. Under the irradiation of the first near-infrared light, the object is excited to emit a first excitation light, and under the irradiation of the second near-infrared light, it is excited to emit a second excitation light. The reflected light, as well as the excited first and second excitation lights, are collected. A visible light image is generated based on the reflected light, and a first excitation light image and a second excitation light image are generated based on the first and second excitation lights, respectively. Since the wavelength of the first near-infrared light is shorter than that of the second near-infrared light, the penetration depth of the first near-infrared light is less than that of the second near-infrared light. The first excitation light image obtained using the first near-infrared light and the second excitation light image obtained using the second near-infrared light contain content at different depths of the object to be observed; therefore, images of the object at different penetration depths can be obtained. The method fuses a visible light image and a first excitation light image formed at the same time into a first fused image, and fuses a visible light image and a second excitation light image formed at the same time into a second fused image. Furthermore, it fuses the first fused image and the second fused image generated at adjacent times into a target fused image. Since the first near-infrared light and the second near-infrared light are emitted alternately, the generation frame rate of both the first and second excitation light images is half that of the visible light image. This ensures that each frame of the first fused image has a corresponding second fused image, resulting in a target fused image containing a one-to-one correspondence between the first and second fused images. This results in a higher frame rate and clearer, more coherent target fused image, facilitating the simultaneous presentation of information about different depths of the observed object. In summary, the method of this application can present information about different penetration depths of the observed object on a single image, allowing doctors to simultaneously observe the metabolic and vascular morphological information of the observed object, thus facilitating diagnosis and treatment.

[0125] It should be understood that, although Figure 22 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 22 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0127] In the description of this specification, references to terms such as "some embodiments," "other embodiments," and "ideal embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An electronic endoscopic imaging system, characterized in that, The system includes: A light source device is used to continuously emit visible light and alternately emit first near-infrared light and second near-infrared light towards an object to be observed; wherein the wavelength of the first near-infrared light is shorter than the wavelength of the second near-infrared light; the visible light is reflected on the object to be observed to form reflected light; the first near-infrared light excites the object to be observed to generate a first excitation light; the second near-infrared light excites the object to be observed to generate a second excitation light; wherein the object to be observed contains a developer corresponding to the first near-infrared light; the second excitation light is obtained based on the reflection of the second near-infrared light. An image acquisition device is located in the optical path of the reflected light, the first excitation light, and the second excitation light. It is used to acquire the reflected light to form a visible light image, acquire the first excitation light image formed by the contrast agent excited by the first excitation light (the first excitation light image includes metabolic information of the lesion site at a depth corresponding to the wavelength of the first near-infrared light), and acquire a second excitation light image formed by the second excitation light with a penetration depth greater than the first near-infrared light (the second excitation light image includes vascular morphology information of the object to be observed at a depth corresponding to the wavelength of the second near-infrared light). The device also includes a first image sensor group and a second image sensor group, wherein one group of sensors is disposed on the cross-section of the endoscope barrel, and the other group is disposed on the side wall of the endoscope barrel. An image processing device, electrically connected to the image acquisition device, is used to fuse the visible light image and the first excitation light image formed at the same time into a first fused image, fuse the visible light image and the second excitation light image formed at the same time into a second fused image, and fuse the first fused image and the second fused image generated at adjacent times into a target fused image.

2. The system according to claim 1, characterized in that, The image acquisition device includes: A beam splitter is located on the optical paths of the reflected light, the first excitation light, and the second excitation light, and is used to change the optical path of at least one of the reflected light, the first excitation light, and the second excitation light, so that the optical path of the reflected light is not coplanar with the optical paths of the first excitation light and the second excitation light. The first image sensor group is located on the optical path of the reflected light passing through the beam splitter and is used to collect the reflected light to form a visible light image; The second image sensor group is located on the optical path of the first excitation light and the second excitation light passing through the beam splitter, and is used to collect the first excitation light to form a first excitation light image and collect the second excitation light to form a second excitation light image.

3. The system according to claim 2, characterized in that, The beam splitter includes a first triangular prism and a second triangular prism connected by inclined surfaces. The inclined surface of the first triangular prism forms an acute angle of 45° with the light paths of the reflected light, the first excitation light, and the second excitation light. It is used to change the light path of at least one of the reflected light, the first excitation light, and the second excitation light, so that the light path of the reflected light is perpendicular to the light paths of the first excitation light and the second excitation light. The first triangular prism and the second triangular prism are both right-angled triangular prisms with an angle of 45°.

4. The system according to claim 3, characterized in that, The inclined surface of the first triangular prism is coated with a semi-transparent and semi-reflective multilayer dielectric film, which is used to reflect the reflected light and transmit the first excitation light and the second excitation light. The inclined surface of the second triangular prism is coated with a high-transmission film for transmitting the first excitation light and the second excitation light.

5. The system according to claim 2, characterized in that, The image acquisition device also includes: A light guide lens group is located in the optical path of the reflected light, the first excitation light, and the second excitation light, and is used to converge the reflected light, the first excitation light, and the second excitation light into the beam splitter.

6. The system according to claim 2, characterized in that, The image acquisition device also includes: A filter is located in the optical path of the reflected light, the first excitation light, and the second excitation light, and is used to filter out the reflected light from the first excitation light and the second excitation light.

7. The system according to claim 1, characterized in that, The light source device includes: A light source emitting module is used to emit at least one of the visible light, the first near-infrared light, and the second near-infrared light; A driving module, connected to the light source emitting module, is used to drive the light source emitting module to continuously emit visible light toward the object to be observed, and to alternately emit the first near-infrared light and the second near-infrared light.

8. The system according to claim 7, characterized in that, The light source emitting module includes: Visible light emitting module, used to emit the visible light; A first near-infrared light emitting module is used to emit the first near-infrared light; The second near-infrared light emitting module is used to emit the second near-infrared light; The driving module includes: A visible light driving module, connected to the visible light emitting module, is used to provide driving power to the visible light emitting module; A first near-infrared light driving module is connected to the first near-infrared light emitting module and is used to provide driving power to the first near-infrared light emitting module. The second near-infrared light driving module is connected to the second near-infrared light emitting module and is used to provide driving power to the second near-infrared light emitting module. The timing control module is connected to the first near-infrared light driving module and the second near-infrared light driving module respectively, and is used to control the first near-infrared light driving module and the second near-infrared light driving module to work alternately according to the preset timing signal.

9. The system according to claim 2, characterized in that, The first image sensor group includes two visible light image sensors arranged side by side and coplanarly; The second image sensor group includes two near-infrared light image sensors arranged side by side and coplanarly.

10. The system according to claim 1, characterized in that, The generation frame rate of the first excitation light image and the second excitation light image is half that of the visible light image.

11. An electronic endoscopic imaging method, characterized in that, The method implemented in the electronic endoscopic imaging system as described in any one of claims 1-10 includes: Visible light is continuously emitted towards the object to be observed, and first near-infrared light and second near-infrared light are emitted alternately; wherein the wavelength of the first near-infrared light is shorter than the wavelength of the second near-infrared light; the visible light is reflected on the object to be observed to form reflected light, the first near-infrared light excites the object to be observed to generate a first excitation light, and the second near-infrared light excites the object to be observed to generate a second excitation light, wherein the object to be observed contains a developer corresponding to the first near-infrared light; The reflected light is collected to form a visible light image. The first excitation light image is collected to excite the contrast agent to form a first excitation light image, which includes metabolic information of the lesion site at a depth corresponding to the wavelength of the first near-infrared light. The second excitation light image is collected to form a second excitation light with a penetration depth greater than that of the first near-infrared light. The second excitation light image includes vascular morphology information of the object to be observed at a depth corresponding to the wavelength of the second near-infrared light. The visible light image and the first excitation light image formed at the same time are fused into a first fused image, the visible light image and the second excitation light image formed at the same time are fused into a second fused image, and the first fused image and the second fused image generated at adjacent times are fused into a target fused image.

Images