A near-infrared fluorescence imaging system based on augmented reality

By introducing augmented reality technology into medical imaging systems, using near-infrared photography and fluorescent laser light sources to collect images, and performing pseudo-color processing through the main control processing device, virtual three-dimensional display on the augmented reality display device is achieved, solving the existing system The inconvenient display of medium fluorescence information and the huge cost of equipment improve the convenience of surgical operation and the portability and economicality of equipment.

CN110226974BActive Publication Date: 2024-12-06UNIV OF SCI & TECH OF CHINA
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN201910611258.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-07-08
Publication Date
2024-12-06
Estimated Expiration
2039-07-08

AI Technical Summary

Technical Problem

The fluorescent information displayed on external monitors in existing medical imaging systems is not convenient for doctors to observe. The equipment is huge and costly, which affects the convenience and economicality of surgical operations.

Method used

Adopt a near-infrared fluorescence imaging system based on augmented reality, fluorescent images are collected through a near-infrared shooting device and a fluorescent laser light source, and pseudo-color processing is performed using the main control processing device, and the image is sent to the augmented reality display device, and the head-mounted device is used to display it. Virtual three-dimensional tissue image enables real-time display of fluorescence information on the surface of patient tissue.

Benefits of technology

It reduces the risk of doctors' distraction during the operation, reduces the size and cost of the equipment, improves the portability and use range of the system, and provides more convenient surgical operation guidance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110226974B_ABST
    Figure CN110226974B_ABST
Patent Text Reader

Abstract

The present invention discloses a near-infrared fluorescence imaging system based on augmented reality, comprising: a near-infrared shooting device and a fluorescent laser light source, which are respectively connected to a main control processing device for communication; the main control processing device is connected to an augmented reality display device for communication, and can collect the fluorescent image of the observed object tissue under the action of the fluorescent laser light source through the near-infrared shooting device, and process the fluorescent image into a pseudo-color fluorescent image and then send it to the augmented reality display device; the augmented reality display device can receive the pseudo-color fluorescent image sent by the main control processing device, and process it into a virtual three-dimensional tissue image and then display it in an augmented reality manner. The imaging system can realize the display of the observed tissue image to the doctor in an augmented reality manner, so that the doctor does not need to watch the display and the observed object tissue at the same time, which can provide great convenience for the doctor's surgical operation on the patient; because the augmented reality display device can adopt a head-mounted device, the system as a whole can be made very small and the cost is low.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical imaging systems, and in particular to a near-infrared fluorescence imaging system based on augmented reality for medical use. Background Art

[0002] Near-infrared fluorescence imaging technology has many applications in the field of clinical medicine, such as lymph node localization in breast cancer and skin cancer. There are also many related products on the market, such as FLARE in the United States and HyperEye in Japan. The principles of these technologies or products are roughly as follows: first, use near-infrared light to excite fluorescence, then use near-infrared imaging equipment to image fluorescent markers, and finally display the imaging information on an external monitor. Such equipment has the following three disadvantages: First, the fluorescence information is displayed on an external monitor, and the doctor needs to observe both the monitor and the patient's tissue at the same time, which can easily distract the doctor's attention; second, the equipment is large and can only be used in large clinical sites with complete equipment, and the portability is poor; third, the equipment cost is high and requires a lot of upfront investment. Summary of the invention

[0003] Based on the problems existing in the prior art, the purpose of the present invention is to provide a near-infrared fluorescence imaging system based on augmented reality, which can solve the problems of the existing medical imaging system in which the fluorescence information is displayed on an external display, it is inconvenient to watch multiple displays, the equipment is bulky and the cost is high.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] An embodiment of the present invention provides a near-infrared fluorescence imaging system based on augmented reality, comprising:

[0006] Near-infrared camera, fluorescent laser light source, main control processing equipment and augmented reality display equipment; among which,

[0007] The near-infrared camera and the fluorescent laser light source are respectively connected to the main control processing device for communication;

[0008] The main control processing device is connected to the augmented reality display device for communication, and can collect a fluorescent image of the observed object tissue under the action of a fluorescent laser light source through the near-infrared shooting device, and process the fluorescent image into a pseudo-color fluorescent image and then send it to the augmented reality display device;

[0009] The augmented reality display device can receive the pseudo-color fluorescence image sent by the main control processing device, and process it into a virtual three-dimensional tissue image and then display it in an augmented reality manner.

[0010] It can be seen from the technical solution provided by the present invention that the near-infrared fluorescence imaging system based on augmented reality provided by the embodiment of the present invention has the following beneficial effects:

[0011] By setting up a main control processing device and connecting the near-infrared shooting device, the fluorescent laser light source and the augmented reality display device, after collecting the fluorescent image of the observed object tissue, it can be processed into a virtual three-dimensional tissue image in cooperation with the augmented reality display device, and then displayed in an augmented reality manner on the augmented reality display device. The imaging system can display the observed tissue image to the doctor in an augmented reality manner, and the doctor does not need to watch the display and the observed object tissue at the same time, which can greatly facilitate the doctor's surgical operation on the patient; because the augmented reality display device can use a head-mounted device, the overall system can be made very small and the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A schematic diagram of a near-infrared fluorescence imaging system based on augmented reality provided by an embodiment of the present invention;

[0014] Figure 2 An imaging process flow chart of an augmented reality-based near-infrared fluorescence imaging system provided in an embodiment of the present invention;

[0015] Figure 3 A diagram of an actual use scenario of the near-infrared fluorescence imaging system based on augmented reality provided in an embodiment of the present invention;

[0016] The marks in the figure are: 1-near-infrared shooting equipment; 11-near-infrared camera; 12-long-pass filter; 2-fluorescent laser light source; 3-main control processing equipment; 31-near-infrared image acquisition module; 32-communication module; 4-augmented reality display equipment; 41-augmented reality imaging module; 42-augmented reality display device; 43-spatial registration submodule; 5-image marker; 6-virtual image marker; 7-virtual three-dimensional tissue model; 8-patient tissue; 9-pseudo-color fluorescence image. DETAILED DESCRIPTION

[0017] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in this field.

[0018] like Figure 1 As shown, an embodiment of the present invention provides a near-infrared fluorescence imaging system based on augmented reality, which can display the fluorescence image captured by a near-infrared camera on a head-mounted display device through enhanced display technology, and conveniently display the position and size of the fluorescent marker in real time, including:

[0019] Near-infrared camera, fluorescent laser light source, main control processing equipment and augmented reality display equipment; among which,

[0020] The near-infrared camera and the fluorescent laser light source are respectively connected to the main control processing device for communication;

[0021] The main control processing device is connected to the augmented reality display device for communication, and can collect a fluorescent image of the observed object tissue under the action of a fluorescent laser light source through the near-infrared shooting device, and process the fluorescent image into a pseudo-color fluorescent image and then send it to the augmented reality display device;

[0022] The augmented reality display device can receive the pseudo-color fluorescence image sent by the main control processing device, and process it into a virtual three-dimensional tissue image and then display it in an augmented reality manner.

[0023] See also Figure 2 In the above imaging system, the main control processing device includes:

[0024] Near infrared image acquisition module and communication module; wherein,

[0025] The near-infrared image acquisition module is communicatively connected to the display end processing device through the communication module, and can acquire a fluorescent image of the observed object tissue under the action of a fluorescent laser light source through the near-infrared shooting device, and process the fluorescent image into a pseudo-color fluorescent image and then send it to the augmented reality display device through the communication module.

[0026] The above-mentioned main control processing device can be a computer, a single-chip microcomputer or a mobile terminal and other computing and processing devices, wherein each module can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the process of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM). The main control processing device can also be a dedicated computing and processing device customized by multiple hardware processor modules, wherein each module can be a processor module integrated with a corresponding program.

[0027] In the above imaging system, the augmented reality display device includes: an augmented reality imaging module and an augmented reality display device; wherein,

[0028] The augmented reality imaging module is communicatively connected with the augmented reality display device, can receive the pseudo-color fluorescence image sent by the main control processing device, and after spatially registering the pseudo-color fluorescence image with a pre-acquired virtual three-dimensional tissue model corresponding to the observed object tissue, fit the pseudo-color fluorescence image to the surface of the virtual three-dimensional tissue model to obtain a virtual three-dimensional tissue image, and send the virtual three-dimensional tissue image to the augmented reality display device;

[0029] The augmented reality display device can display the virtual three-dimensional tissue image in an augmented reality manner.

[0030] Furthermore, the augmented reality imaging module includes a spatial registration submodule for spatially registering the pseudo-color fluorescence image with a pre-acquired virtual three-dimensional tissue model corresponding to the observed object tissue.

[0031] Preferably, the augmented reality display device adopts augmented reality glasses or augmented reality helmet.

[0032] Specifically, the augmented reality imaging module in the augmented reality display device can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the process of the embodiments of the above methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM). The spatial registration module can also use a processor module with an integrated processing program, as long as the corresponding function can be realized.

[0033] In the above imaging system, the near-infrared shooting device includes:

[0034] A near-infrared camera, and a long-pass filter mounted on a lens of the near-infrared camera.

[0035] In the above imaging system, the filtering wavelength of the long-pass filter is 800nm. This long-pass filter can filter out the excitation light source and visible light stray light.

[0036] In the above imaging system, the laser wavelength of the fluorescent laser light source is 785nm.

[0037] In the above imaging system, the augmented reality display device adopts augmented reality glasses or augmented reality helmet.

[0038] The imaging system of the present invention uses augmented reality technology to perform fluorescent imaging on the surface of patient tissue. When using the system, the doctor only needs to observe one direction of the patient tissue, which is more convenient to use. Since the augmented reality display device can use augmented reality glasses or augmented reality helmets, both of which are head-mounted devices, it is free from display devices and solves the problem of bulky equipment on the market. Since it only involves a near-infrared shooting device (a near-infrared imaging industrial camera can be used), a fluorescent laser light source, a main control processing device (a computer that processes data can be used) and an augmented reality display device, the total cost is lower than the near-infrared fluorescent imaging device on the market. When the imaging system of the present invention is used, the doctor can conveniently observe the fluorescent image through the head-mounted display device during the operation to guide the doctor to perform the operation.

[0039] The imaging system of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0040] See also Figure 1 The embodiment of the present invention provides a near-infrared fluorescence imaging system based on augmented reality technology, including: a near-infrared camera, a long-pass filter, a fluorescent laser light source, a main control processing device and augmented reality display glasses; wherein,

[0041] The above-mentioned main control processing device has a built-in Windows operating system and can perform image processing.

[0042] The above-mentioned near-infrared camera uses a CMOS in the near-infrared band and can be connected to the main control processing device through a cable. The optical axis of the near-infrared camera should be perpendicular to the horizontal plane. The near-infrared camera can capture fluorescent images of patient tissues under the action of a fluorescent laser light source.

[0043] The long-pass filter is installed on the lens of the near-infrared camera. The long-pass filter is 800nm ​​and is used to filter out the excitation light source and visible light stray light.

[0044] The laser has a wavelength of 785 nm and is located at the side of the near-infrared camera to excite the fluorescent substance indocyanine green ICG, so that the near-infrared camera can capture fluorescent images of the patient's tissues.

[0045] The augmented reality display glasses are used to display and fit the fluorescent image on the surface of the virtual three-dimensional tissue. After the augmented reality display glasses align the virtual three-dimensional tissue with the real patient tissue, the fluorescent image can fit the actual patient tissue.

[0046] The imaging process of the above imaging system is achieved by injecting indocyanine green ICG into biological tissues. ICG is widely used as a developer in existing navigation instruments. Its main principle is to inject ICG into biological tissues, excite it with external 785nm laser irradiation, and emit fluorescence with a wavelength of 830nm in the blood flow. The near-infrared camera that images in the near-infrared band can capture the ICG image contained in the tissue in real time, which is called a fluorescent image. An 800nm ​​long-pass filter is placed in front of the near-infrared camera lens to filter out the 785nm fluorescent laser light source and visible light stray light, image the fluorescent substance alone, and obtain an accurate fluorescent image. The fluorescent image captured by the near-infrared camera is transmitted to the main control processing device through the network. The fluorescent image is processed by pseudo-color on the main control processing device, so that the fluorescent area appears green, which is used to distinguish it from skin blood tissue; the processed image is transmitted to the augmented reality display glasses with high speed and low latency through the network communication algorithm based on the UDP transmission protocol; at the augmented reality display glasses, the image is mapped to the surface of the virtual three-dimensional object and updated in real time. The direction of the mapping is vertical, which is consistent with the shooting direction, so as to ensure accuracy. Finally, the virtual three-dimensional object and the patient's tissue are aligned in three-dimensional space, and the three-dimensional object appears in the original position of the patient's tissue, while the fluorescent image is attached to the surface of the patient's tissue, so as to achieve the effect of enhanced display, so that doctors can more realistically locate and remove tumors or lesion tissues during the operation.

[0047] Example

[0048] refer to Figure 1 , 2 The embodiment of the present invention is mainly composed of three modules: a near-infrared image acquisition module, a communication module and an augmented reality imaging module; wherein the near-infrared image acquisition module includes an acquisition end control algorithm program in addition to related hardware, and the near-infrared image acquisition module and the communication module are arranged in a main control processing device; the augmented reality imaging module includes an imaging end control algorithm program in addition to related hardware, and the augmented reality imaging module is arranged in an augmented reality display device.

[0049] refer to Figure 2In the main control processing device of the embodiment of the present invention, the near-infrared image acquisition module and the communication module run on the image acquisition end, and the augmented reality imaging module runs on the receiving imaging end. The working process of the two is as follows: first, initialize the near-infrared camera, including setting the frame rate, exposure time, etc.; then initialize the socket to prepare for sending the fluorescent image; after the initialization is completed, take a fluorescent image and perform pseudo-color image processing to become a pseudo-color fluorescent image, and then send it to the receiving imaging end; finally, update the state of the near-infrared camera, if it is still turned on, repeat the previous step; if the near-infrared camera is turned off, stop shooting and close the socket; the processing algorithm of the augmented reality imaging module of the augmented reality display device is as follows Figure 2 As shown in the middle right figure: First, initialize the scene on the augmented reality display device, then import the virtual 3D tissue model corresponding to the patient's tissue; initialize the socket to receive the pseudo-color fluorescence image, and if the connection does not time out, loop the following actions: accept a pseudo-color fluorescence image, set the pseudo-color fluorescence image as the texture of the virtual 3D tissue model corresponding to the patient's tissue; if the connection times out, it means that the image acquisition end stops acquiring images, and then close the socket, ending the entire imaging process. The communication module uses the UDP communication protocol, and uses socket to implement UDP, which can achieve the purpose of real-time transmission.

[0050] refer to Figure 3 In one embodiment, the near-infrared camera 1 is fixed about 50 cm above the patient and connected to the main control processing device 3 through a network cable; the long-pass filter 12 is fixed at the lens of the near-infrared camera 11 to filter out the laser light of the fluorescent laser light source 2 and the visible light stray light, so that the near-infrared camera 11 only images the fluorescent labeled tissue 8; the fluorescent laser light source 2 is a semiconductor laser with a wavelength of 785 nm, the excitation wavelength of indocyanine green ICG is 750 nm to 810 nm, and the fluorescence wavelength in the blood flow is 830 nm. This wavelength can just excite the fluorescent substance and be filtered out by the long-pass filter 12; after receiving the fluorescent image captured by the near-infrared camera 11, the main control processing device 3 performs pseudo-color processing on the fluorescent image to obtain a pseudo-color fluorescent image 9 , the pseudo-color fluorescence image 0 is transmitted to the augmented reality display device 4 with the help of WiFi through the network communication algorithm; the augmented reality display device 4 is worn on the head of the surgeon, and the real image is projected into the surgeon's eyes; the augmented reality display device 4 performs a mapping operation to fit the pseudo-color fluorescence image 9 to the virtual three-dimensional tissue model 7 corresponding to the patient's tissue 4; the image marker 5 is a two-dimensional image, which is used to identify and align with the virtual marker 6, so that the virtual three-dimensional tissue model 7 can be spatially aligned with the patient's tissue 8; after the whole process, the collected pseudo-color fluorescence image 9 is attached to the surface of the virtual three-dimensional tissue model 7 corresponding to the patient's tissue 4 to form a virtual three-dimensional tissue image, thereby achieving the purpose of observing the fluorescence image on the surface of the virtual object.

[0051] In some implementations, Figure 3 The central main control processing device 3 can use an stm32 single-chip microcomputer, and the algorithms of the image acquisition end and the receiving imaging end can be copied into the single-chip microcomputer, which can reduce the size of the device and improve portability.

[0052] In some implementations, Figure 3 The fluorescent laser light source 2 uses a 785nm LED ring array, which can make the brightness of the irradiated area uniform and reduce the harm of the laser to the human body. The fluorescent laser light source 2 and the near-infrared camera 11 can be assembled together.

[0053] The imaging system of the present invention has at least the following beneficial effects, including:

[0054] (1) Due to the use of augmented reality technology, the system can display fluorescent images on the surface of the patient's skin. Surgeons no longer need to look at the display and the patient's skin separately, reducing potential risks caused by distraction.

[0055] (2) Head-mounted augmented reality display devices have no external cables and are relatively light (Hololens, for example, weighs about 600g). Their scope of use is not limited to large clinical environments and they can be easily deployed and used.

[0056] (3) The whole set of equipment includes a near-infrared industrial camera, a fluorescent laser light source, a long-pass filter, a computer and an augmented reality display device. The current price of the whole system is about RMB 50,000. Compared with other near-infrared fluorescence imaging equipment, the cost of this system is relatively low.

[0057] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A near-infrared fluorescence imaging system based on augmented reality, characterized in that: include: Near-infrared camera, fluorescent laser light source, main control processing equipment and augmented reality display equipment; among which, The near-infrared camera and the fluorescent laser light source are respectively connected to the main control processing device for communication; The main control processing device is connected to the augmented reality display device for communication, and can collect a fluorescent image of the observed object tissue under the action of a fluorescent laser light source through the near-infrared shooting device, and process the fluorescent image into a pseudo-color fluorescent image and then send it to the augmented reality display device; The augmented reality display device can receive the pseudo-color fluorescence image sent by the main control processing device, and process it into a virtual three-dimensional tissue image and then display it in an augmented reality manner; The main control processing device includes: a near-infrared image acquisition module and a communication module; wherein, The near-infrared image acquisition module is connected to the display end processing device through the communication module, and can acquire a fluorescent image of the observed object tissue under the action of a fluorescent laser light source through the near-infrared shooting device, and process the fluorescent image into a pseudo-color fluorescent image and then send it to the augmented reality display device through the communication module; The augmented reality display device comprises: an augmented reality imaging module and an augmented reality display device; wherein, The augmented reality imaging module is communicatively connected with the augmented reality display device, can receive the pseudo-color fluorescence image sent by the main control processing device, and after spatially registering the pseudo-color fluorescence image with a pre-acquired virtual three-dimensional tissue model corresponding to the observed object tissue, fit the pseudo-color fluorescence image to the surface of the virtual three-dimensional tissue model to obtain a virtual three-dimensional tissue image, and send the virtual three-dimensional tissue image to the augmented reality display device; The augmented reality display device can display the virtual three-dimensional tissue image in an augmented reality manner; The augmented reality display device adopts augmented reality glasses or augmented reality helmet.

2. The near-infrared fluorescence imaging system based on augmented reality according to claim 1, characterized in that: The near-infrared camera comprises: A near-infrared camera, and a long-pass filter mounted on a lens of the near-infrared camera.

3. The near-infrared fluorescence imaging system based on augmented reality according to claim 2, characterized in that: The filtering wavelength of the long-pass filter is 800 nm.

4. The near-infrared fluorescence imaging system based on augmented reality according to claim 1, characterized in that: The laser wavelength of the fluorescent laser light source is 785 nm.

Citation Information

Patent Citations

  • Head-wearing molecule image navigation system

    CN104305957A

  • Optical navigation system for surgeries

    CN107374730A

  • Optical imaging system and methods thereof

    CN107851176A

  • Handheld fluorescent ultrasonic fusion radiographic navigation system

    CN108186115A

  • Near-infrared fluorescence imaging system based on augmented reality

    CN211325679U