Ultrahigh-definition multispectral stereo imaging device

Abstract

The invention discloses an ultra-high-definition multispectral stereo imaging device which is applied to the technical field of medical imaging. Comprising a 4K3D fluorescent electron mirror, a control module, a processing module and a centralized management platform which are connected in sequence, and the control module is further connected with a cold light source. The 4K3D fluorescent electron mirror comprises a left light path, a right light path and a flexible connecting cable, wherein the left light path comprises a first multispectral light filter, a first lens module, a first refracting prism and a first multispectral image sensor which are vertically arranged in sequence; and the right light path comprises a second multispectral optical filter, a second lens module, a second refracting prism and a second multispectral image sensor which are vertically arranged in sequence. Compared with the method that only two image sensors are used for completing collection of left and right binocular white light images and fluorescence images, the same sensor is used for color imaging and fluorescence imaging, the picture consistency of the color images and the fluorescence images is guaranteed, pixel-level alignment does not need to be conducted on the structure, and the debugging difficulty is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and more particularly to an ultra-high-definition multi-spectral stereoscopic imaging device. Background Art

[0002] Multispectral stereo imaging is a biomedical imaging technology that can be used in medical endoscopy, surgery, or biological pathology detection. There are two main types of existing stereo multispectral imaging devices: 1. Use four or more image sensors to separately collect left and right color images and black and white fluorescent images, and then fuse the color and fluorescent images; due to the large size of multiple cameras, the imaging light path of the front end of this solution can only be transmitted using an inflexible optical hard tube mirror, and the back end uses a dual-light path lens for imaging. 2. Use two high-definition image sensors to be arranged on the left and right, and by frequently switching the light source, only color images or fluorescent images can be obtained at the same time, and then the two images are synthesized into a stereo multispectral through a software algorithm, and the same sensor obtains color and fluorescent images in a time-sharing manner. The frame rate of the fused image in this solution is relatively low, resulting in a heavy delay in the image. In addition, the illumination light source frequently switches between white light and fluorescence, which easily causes the image to flicker. Existing stereoscopic multispectral imaging devices have the following disadvantages: the structure is relatively large and bulky, and cannot be miniaturized; the overall cost is high, and accessories such as optical hard tube mirrors and binocular lenses greatly increase the cost of the sensor; due to the use of optical hard mirrors, the front end cannot be bent and cannot be used in human cavities such as the gastrointestinal tract. Therefore, how to provide an ultra-high-definition multispectral stereoscopic imaging device is a problem that technicians in this field urgently need to solve. Summary of the invention

[0003] In view of this, the present invention provides an ultra-high-definition multi-spectral stereoscopic imaging device to solve the problems existing in the prior art.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] An ultra-high-definition multi-spectral stereoscopic imaging device comprises a 4K3D fluorescent electron mirror, a control module, a processing module and a centralized management platform which are connected in sequence, wherein the control module is also connected to a cold light source.

[0006] Optionally, the 4K3D fluorescent electron mirror includes a left optical path, a right optical path and a flexible connecting cable, the left optical path includes a first multispectral filter, a first lens module, a first refractive prism and a first multispectral image sensor arranged vertically in sequence, and the first multispectral image sensor is connected to the flexible connecting cable; the right optical path includes a second multispectral filter, a second lens module, a second refractive prism and a second multispectral image sensor arranged vertically in sequence, and the second multispectral image sensor is connected to the flexible connecting cable.

[0007] Optionally, the first lens module includes a first glass lens, a first liquid lens, a second glass lens, a second liquid lens, a third glass lens and a fourth glass lens which are vertically arranged in sequence, and the second lens module includes a fifth glass lens, a third liquid lens, a sixth glass lens, a fourth liquid lens, a seventh glass lens and an eighth glass lens which are vertically arranged in sequence.

[0008] Optionally, the refraction angles of the first refraction prism and the second refraction prism are 45 degrees.

[0009] Optionally, the control module includes a liquid lens control board, a high-speed signal transmission board and a multi-function key board.

[0010] Optionally, the processing module includes a video processing board, a USB flash drive, an SD card, a network port and an LCD screen, and the network port is connected to the centralized management platform via a network cable.

[0011] Optionally, the video processing board uses an AI algorithm and the pixel photosensitivity of the IR channel as a reference to perform pseudo-color intensity conversion on the pixels of the color image at the corresponding position, and then fusions the left and right fluorescent images and splices the left and right multi-spectral images together through a 3D synthesis algorithm; the 3D mode algorithm includes left-right 3D format, top-bottom 3D format, and frame interlaced 3D format.

[0012] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides an ultra-high-definition multi-spectral stereoscopic imaging device, which has the following beneficial effects:

[0013] 1. The present invention reduces the number of sensors and uses only two image sensors to complete the acquisition of left and right binocular white light images and fluorescent images. The image sensor has a color channel and a separate infrared channel, and the color image and the fluorescent image can be acquired at one time;

[0014] 2. The color and fluorescence imaging of the present invention use the same sensor, and the consistency of the color image and the fluorescence image is guaranteed. There is no need for pixel-level alignment in the structure, which greatly reduces the difficulty of debugging;

[0015] 3. While achieving 800W pixel acquisition, the overall size of the product is controlled within 10mm. Two 800W resolution image sensors are designed on a circuit board, back to back, with each sensor facing one side, and vertically installed in a 10mm steel pipe. The circuit board is connected with a flexible cable to ensure that the entire product can be bent to a large extent.

[0016] 4. The optical path of the present invention uses a 45-degree lens for refraction at the front end of the sensor, and through the periscope optical path design, and through the combination of optical glass lenses and multiple groups of liquid lenses, a large-magnification optical magnification is achieved. At the same time, fast automatic focusing is achieved;

[0017] 5. The fluorescence suitable algorithm of the present invention is implemented by AI, which can continuously learn and iterate automatically to ensure that the fluorescence fusion effect in unknown scenes is still guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0019] Figure 1 It is a schematic structural diagram of the ultra-high-definition multi-spectral stereoscopic imaging device of the present invention;

[0020] Figure 2 It is a schematic diagram of the ultra-high-definition multi-spectral stereoscopic imaging device system of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of the 4K3D fluorescent electron mirror of the present invention;

[0022] Figure 4 Schematic diagram of the AI ​​algorithm of the present invention;

[0023] In the figure: 1-4K3D fluorescent electronic mirror, 2-liquid lens control board, 3-high-speed signal transmission board, 4-multi-function key board, 5-video processing board, 6-U disk, 7-SD card, 8-network port, 9-network cable, 10-centralized management platform, 11-cold light source, 12-LCD liquid crystal display, 101-first multi-spectral filter, 102-first refractive prism, 103-first multi-spectral image sensor, 104-second multi-spectral filter, 105-second refractive prism, 1 06-second multispectral image sensor, 107-flexible connecting cable, 108-first glass lens, 109-first liquid lens, 1010-second glass lens, 1011-second liquid lens, 1012-third glass lens, 1013-fourth glass lens, 1014-fifth glass lens, 1015-third liquid lens, 1016-sixth glass lens, 1017-fourth liquid lens, 1018-seventh glass lens, 1019-eighth glass lens. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The embodiment of the present invention discloses an ultra-high-definition multi-spectral stereoscopic imaging device, such as Figure 1 and Figure 2 As shown, it includes a 4K3D fluorescent electron mirror 1, a control module, a processing module and a centralized management platform 10 connected in sequence, wherein the control module is also connected to a cold light source 11.

[0026] Further, such as Figure 3 As shown, the 4K3D fluorescent electronic mirror 1 includes a left optical path, a right optical path and a flexible connecting cable 107. The left optical path includes a first multispectral filter 101, a first lens module, a first refractive prism 102 and a first multispectral image sensor 103 which are vertically arranged in sequence, and the first multispectral image sensor 103 is connected to the flexible connecting cable 107; the right optical path includes a second multispectral filter 104, a second lens module, a second refractive prism 105 and a second multispectral image sensor 106 which are vertically arranged in sequence, and the second multispectral image sensor 106 is connected to the flexible connecting cable 107.

[0027] Furthermore, the first lens module includes a first glass lens 108, a first liquid lens 109, a second glass lens 1010, a second liquid lens 1011, a third glass lens 1012 and a fourth glass lens 1013 which are vertically arranged in sequence, and the second lens module includes a fifth glass lens 1014, a third liquid lens 1015, a sixth glass lens 1016, a fourth liquid lens 1017, a seventh glass lens 1018 and an eighth glass lens 1019 which are vertically arranged in sequence.

[0028] Furthermore, the refraction angles of the first refraction prism 102 and the second refraction prism 105 are 45 degrees.

[0029] Furthermore, the control module includes a liquid lens control board 2 , a high-speed signal transmission board 3 and a multi-function key board 4 .

[0030] Furthermore, the processing module includes a video processing board 5 , a USB flash drive 6 , an SD card 7 , a network port 8 and an LCD screen 12 , and the network port 8 is connected to a centralized management platform 10 via a network cable 9 .

[0031] Furthermore, the video processing board 5 uses the AI ​​algorithm and the pixel sensitivity intensity of the IR channel as a reference to perform pseudo-color intensity conversion on the pixels of the color image at the corresponding position, and then merges the left and right fluorescent images and splices the left and right multi-spectral images together through a 3D synthesis algorithm; the 3D mode algorithm includes left-right 3D format, top-bottom 3D format and frame interlaced 3D format.

[0032] Furthermore, in an embodiment of the present invention, if Figure 4 As shown, the video processing board 5 obtains and trains the matching model of color and black-and-white fluorescent images through the AI ​​algorithm, and determines the part of the image that needs to be superimposed with pseudo-color and the part of the image that does not need to be superimposed with pseudo-color according to the preset spatial parameters; based on the similarity network, the similarity parameters of the color image and the fluorescent image of each picture are calculated in real time, based on the registration network, the initialization parameters are determined from the preset parameters, and according to the preset debugging parameters, the spatial parameters and the training model and the simulation model are initialized to obtain the model debugging image; based on the similarity network, the second similarity parameters of the real-time image and the model debugging image are calculated, and based on the registration network, the debugging parameters are corrected according to the second similarity parameters to obtain the debugging correction parameters, until the second similarity parameters converge, the final debugging correction parameters are determined, and the pseudo-color observation image is generated, and the color plus fluorescent pseudo-color fusion image is superimposed to generate.

[0033] The existing multi-spectral stereo imaging system uses four image sensors and needs to make four image sensor circuit boards. The two cameras on the left collect the color image and fluorescence image on the left to synthesize the left multi-spectral image, and the two cameras on the right collect the color image and fluorescence image on the right to synthesize the right multi-spectral image. Then the left color and fluorescence images are fused, and the image acquisition area needs to be adjusted. The two image sensor circuit boards on the left need to be moved to ensure the pixel-level alignment of the color image and the fluorescence image. The right color and fluorescence images are fused, and the image acquisition area needs to be adjusted. The two image sensor circuit boards on the right need to be moved to ensure the pixel-level alignment of the color image and the fluorescence image. To adjust the 3D effect, it is also necessary to adjust the acquisition areas of the two groups of cameras on the left and right at the same time to ensure that they are on the same horizontal plane. In this way, the image acquisition areas of the four cameras are adjusted by physical methods. Then 3D synthesis is performed. The entire adjustment process is very complicated, requiring a lot of time and very professional equipment. The assembly and debugging costs of the product are extremely high, and the efficiency is relatively low. In addition, the existing solution requires expensive binocular fluorescent hard tube laparoscopes and binocular optical cards to achieve imaging.

[0034] The multi-spectral stereoscopic imaging device disclosed in the embodiment of the present invention does not require an optical hard tube mirror and a binocular optical mount. The image sensor is directly designed at the front end of the endoscope. The image formed on the left side is focused on the first multi-spectral image sensor 103 on the left side through a liquid lens, an optical lens, and a 45-degree first refractive prism 102. The image formed on the right side is focused on the second multi-spectral image sensor 106 on the right side through a liquid lens, an optical lens, and a 45-degree second refractive prism 105. Each image sensor can obtain RGB and a separate IR spectrum, and the simultaneous acquisition of color and fluorescent images can be completed at one time, ensuring the pixel alignment of the image and the synchronization of time. After the color and fluorescent images are collected on both sides at the same time, the pixels of the color image at the corresponding position are converted to pseudo-color intensity through a fusion algorithm, using the pixel sensitivity intensity of the IR channel as a reference, and then the left and right two fluorescent fusion images are spliced ​​together through a 3D synthesis algorithm, and the left and right multi-spectral images are synthesized into a 3D multi-spectral mode.

[0035] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0036] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An ultra-high-definition multi-spectral stereoscopic imaging device, characterized in that: It comprises a 4K3D fluorescent electron mirror (1), a control module, a processing module and a centralized management platform (10) which are connected in sequence, wherein the control module is also connected to a cold light source (11).

2. The ultra-high-definition multi-spectral stereoscopic imaging device according to claim 1, characterized in that: The 4K3D fluorescent electron microscope (1) comprises a left optical path, a right optical path and a flexible connection cable (107), wherein the left optical path comprises a first multispectral filter (101), a first lens module, a first refractive prism (102) and a first multispectral image sensor (103) which are arranged vertically in sequence, and the first multispectral image sensor (103) is connected to the flexible connection cable (107); and the right optical path comprises a second multispectral filter (104), a second lens module, a second refractive prism (105) and a second multispectral image sensor (106) which are arranged vertically in sequence, and the second multispectral image sensor (106) is connected to the flexible connection cable (107).

3. The ultra-high-definition multi-spectral stereoscopic imaging device according to claim 2, characterized in that: The first lens module comprises a first glass lens (108), a first liquid lens (109), a second glass lens (1010), a second liquid lens (1011), a third glass lens (1012) and a fourth glass lens (1013) which are arranged vertically in sequence, and the second lens module comprises a fifth glass lens (1014), a third liquid lens (1015), a sixth glass lens (1016), a fourth liquid lens (1017), a seventh glass lens (1018) and an eighth glass lens (1019) which are arranged vertically in sequence.

4. The ultra-high-definition multi-spectral stereoscopic imaging device according to claim 2, characterized in that: The refraction angles of the first refraction prism (102) and the second refraction prism (105) are 45 degrees.

5. The ultra-high-definition multi-spectral stereoscopic imaging device according to claim 1, characterized in that: The control module comprises a liquid lens control panel (2), a high-speed signal transmission panel (3) and a multifunctional key panel (4).

6. The ultra-high-definition multi-spectral stereoscopic imaging device according to claim 1, characterized in that: The processing module comprises a video processing board (5), a U disk (6), an SD card (7), a network port (8) and an LCD screen (12); the network port (8) is connected to the centralized management platform (10) via a network cable (9).

7. The ultra-high-definition multi-spectral stereoscopic imaging device according to claim 6, characterized in that: The video processing board (5) uses an AI algorithm and the pixel light sensitivity intensity of the IR channel as a reference to perform pseudo-color intensity conversion on the pixels of the color image at the corresponding position, and then merges the left and right fluorescent images and splices the left and right multi-spectral images together through a 3D synthesis algorithm; the 3D mode algorithm includes a left-right 3D format, a top-bottom 3D format, and a frame interlaced 3D format.

Images