Image processing system, method and device based on medical endoscope

Abstract

The invention relates to an image processing system, method and device based on a medical endoscope. The image processing system based on the medical endoscope comprises a single-path endoscope, and a two-dimensional image is collected through a single imaging channel of the single-path endoscope; the control device is electrically connected with the one-way endoscope, and the control device is set to respond to the two-dimensional image sent by the one-way endoscope and extract an image part of a set area in the two-dimensional image; and performing three-dimensional reconstruction on the image part to obtain a fused image.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of medical equipment, and more specifically, to an image based on a medical endoscope. Background Art

[0002] With the rapid development of medical equipment, doctors can use medical equipment to diagnose or treat patients, which can improve efficiency and accuracy. At present, in order to obtain a three-dimensional image of the patient's body, an electronic mirror or a dual-path optical mirror can be used to obtain two-dimensional images of the patient's body from multiple perspectives, and a three-dimensional image of the patient's body can be constructed based on these two-dimensional images. However, this dual-path imaging method is often limited by process technology and mechanical size, and is difficult to apply to some small endoscopes. Summary of the invention

[0003] One purpose of the embodiments of the present disclosure is to provide a new technical solution for image processing based on medical endoscopes.

[0004] According to a first aspect of the present disclosure, there is provided an image processing system based on a medical endoscope, the system comprising: A single-channel endoscope, wherein a single imaging channel of the single-channel endoscope collects a two-dimensional image; A control device is electrically connected to the single-channel endoscope, and is configured to: extract an image portion of a set area in the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope; and perform three-dimensional reconstruction on the image portion to obtain a fused image.

[0005] Optionally, the diameter of the first channel corresponding to the single imaging channel of the single-channel endoscope is less than or equal to 10 mm.

[0006] Optionally, the first channel diameter is the diameter of the front end of the channel of the single imaging channel.

[0007] Optionally, the system further comprises a display device, wherein the display device is electrically connected to the control device, and the display device is configured to display the fused image in response to the fused image sent by the control device.

[0008] According to a second aspect of the present disclosure, there is also provided an image processing method based on a medical endoscope, the method is applied to the image processing system based on the medical endoscope, the execution subject of the method is a control device in the image processing system based on the medical endoscope, and the method comprises: In response to the two-dimensional image sent by the single-channel endoscope, extracting an image portion of a set area in the two-dimensional image; The image part is reconstructed in three dimensions to obtain a fused image.

[0009] Optionally, in response to the two-dimensional image sent by the single-channel endoscope, extracting an image portion of a set area in the two-dimensional image includes: In response to the two-dimensional image sent by the single-channel endoscope, converting the current image format of the two-dimensional image into an image to be processed in a set image format; The image portion of the set area in the two-dimensional image is extracted through a preset ROI module.

[0010] Optionally, the three-dimensional reconstruction of the image portion to obtain a fused image includes: Performing three-dimensional reconstruction on the image portion through a preset multi-task deep learning model to obtain a reconstructed image; The reconstructed image is registered and fused with the image portion to synthesize a fused image with depth information.

[0011] Optionally, the multi-task deep learning model includes a generation node, a fusion node, and a rendering node connected in sequence; Among them, the generation node is used to generate a first projection image of a predetermined viewpoint based on the image part; the fusion node is used to fuse the first projection image into three-dimensional point cloud data to generate a point cloud image; the rendering node is used to render the point cloud image into a second projection image corresponding to a specific viewpoint to generate a depth image.

[0012] According to a third aspect of the present disclosure, there is also provided an image processing device based on a medical endoscope, the device comprising: A response module, configured to extract an image portion of a set area in the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope; The module is used to perform three-dimensional reconstruction on the image part to obtain a fused image.

[0013] According to a third aspect of the present disclosure, there is also provided an image processing device based on a medical endoscope, comprising a memory and a processor, wherein the memory is used to store a computer program; the processor is used to execute the computer program to implement the method described in the second aspect of the present disclosure.

[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to the second aspect of the present disclosure is implemented.

[0015] According to a fifth aspect of the present disclosure, a computer program product is further provided, comprising a computer program, wherein when the computer program is executed by a processor, the method according to the second aspect of the present disclosure is implemented.

[0016] One beneficial effect of the disclosed embodiment is that the image processing system based on the medical endoscope provided by the present invention can collect two-dimensional images through the single imaging channel of the single-channel endoscope, and extract the image part of the set area in the two-dimensional image sent by the single-channel endoscope through the control device, and perform three-dimensional reconstruction on the image part to obtain a fused image. In other words, by using the control device to perform image processing on a single two-dimensional image to construct a three-dimensional image, it is possible to effectively construct a three-dimensional image of the patient's body in the application scenario of small scopes while ensuring the quality of the three-dimensional image, thereby broadening the application scenario of the image processing system based on the medical endoscope.

[0017] Other features and advantages of the embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0019] Figure 1 is a schematic diagram of the structure of a medical endoscope-based image processing system that can be applied according to an embodiment; Figure 2 is a flowchart of an image processing method based on a medical endoscope according to an embodiment; Figure 3 is a structural topology diagram of a multi-task deep learning model according to an embodiment; Figure 4 is a block diagram of a medical endoscope-based image processing device according to an embodiment; Figure 5 It is a schematic diagram of the hardware structure of an image processing device based on a medical endoscope according to one embodiment. DETAILED DESCRIPTION

[0020] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0021] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0022] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0023] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0024] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0025] <System Embodiment> Figure 1 FIG. 1 is a schematic diagram of the structure of an image processing system based on a medical endoscope that can be applied according to an embodiment. Figure 1 As shown, the medical endoscope-based image processing system may include a single-channel endoscope and a control device, and is applied in medical endoscope scenarios.

[0026] The single imaging channel of this endoscope collects two-dimensional images; The control device is electrically connected to the single-channel endoscope and is configured to: extract an image portion of a set area in the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope; and perform three-dimensional reconstruction on the image portion to obtain a fused image.

[0027] In other words, the image processing system based on the medical endoscope can collect two-dimensional images through the single imaging channel of the single-channel endoscope, and extract the image part of the set area in the two-dimensional image sent by the single-channel endoscope through the control device, and perform three-dimensional reconstruction on the image part to obtain a fused image. In other words, by using the control device to process a single two-dimensional image to construct a three-dimensional image, it is possible to effectively construct a three-dimensional image of the patient's body in the application scenario of small scopes while ensuring the quality of the three-dimensional image, thereby broadening the application scenario of the image processing system based on the medical endoscope.

[0028] In some embodiments, in order to enable image acquisition of a two-dimensional image of a single imaging channel of a single-channel endoscope, the diameter of a first channel corresponding to the single imaging channel of the single-channel endoscope is less than or equal to 10 mm.

[0029] In some examples, the diameter of the first channel corresponding to the single imaging channel of the single-channel endoscope can be directly between 2.7 mm and 10 mm.

[0030] In some embodiments, in order to be applicable to more endoscopes, the first channel diameter is the diameter of the front end of the channel of a single imaging channel.

[0031] In this embodiment, the front end of the single imaging channel may be the end that first extends into the patient's body.

[0032] In some embodiments, in order to facilitate doctors to view the fused image, the system further includes a display device, which is electrically connected to the control device, and the display device is used to display the fused image in response to the fused image sent by the control device.

[0033] As used in the embodiments of the present disclosure, the control device memory is used to store a computer program, which is used to control the control device processor to operate according to the image processing method based on a medical endoscope in any embodiment. A technician can design a computer program according to the scheme of the embodiments of the present disclosure. How the computer program controls the processor to operate is well known in the art, so it will not be described in detail here.

[0034] <Method Example> Figure 2 is a flowchart of an image processing method based on a medical endoscope according to an embodiment. Figure 1 Middle control device.

[0035] like Figure 2 As shown, the image processing method based on medical endoscope in this embodiment may include the following steps S210 and S220: Step S210 , in response to the two-dimensional image sent by the single-channel endoscope, extracting an image portion of a set area in the two-dimensional image.

[0036] In some embodiments, in order to improve the computing efficiency of the control device and save computing resources of the control device, step S210 may include the following steps S2101 and S2102: Step S2101, in response to the two-dimensional image sent by the single-channel endoscope, converting the current image format of the two-dimensional image into an image to be processed in a set image format.

[0037] In this embodiment, after a single-channel endoscope inputs a single-channel two-dimensional image, the control device is able to be compatible with image inputs of various formats and various interfaces, and preprocess the image into a recognizable image format, that is, convert the current image format of the two-dimensional image into an image to be processed in a set image format.

[0038] Step S2102: extracting an image portion of a set area in the two-dimensional image through a preset ROI module.

[0039] In this embodiment, the control device pre-sets a ROI module, which can identify a set area of ​​interest in an existing image. For small endoscopes, the display of a two-dimensional image cannot completely cover the entire screen. For the black edge portion that is not displayed, the ROI module is used to identify the image portion that needs to be used for three-dimensional reconstruction, so as to save the computing power of the control device and provide a better display effect.

[0040] In some examples, in order to further improve the picture quality of the subsequent fused image, after step S2102, the method further includes the following step S2103: Step S2103, performing image enhancement processing on the image portion to obtain an image portion after image enhancement.

[0041] Step S220, performing three-dimensional reconstruction on the image portion to obtain a fused image.

[0042] In some embodiments, step S220 may include the following steps S2201 and S2202: Step S2201, performing three-dimensional reconstruction on the image part through a preset multi-task deep learning model to obtain a reconstructed image.

[0043] In this embodiment, the multi-task deep learning model includes a generation node, a fusion node, and a rendering node that are connected in sequence; wherein the generation node is used to generate a first projection image of a predetermined viewpoint based on an image portion; the fusion node is used to fuse the first projection image into three-dimensional point cloud data to generate a point cloud image; the rendering node is used to render the point cloud image into a second projection image corresponding to a specific viewpoint to generate a depth image.

[0044] In some examples, such as Figure 3 As shown in the figure, in the generation node, 2D structure generation: the generator generates a 2D projection (i.e., the first projection image) of a predetermined viewpoint from a single 2D image (i.e., image part) through a preset image generation algorithm. In the fusion node, point cloud fusion: the 2D projection is fused into the 3D point cloud data to generate a point cloud map (i.e., point cloud image). In the fusion node, pseudo rendering: the fused point cloud map is rendered into different 2D projections through a specific viewpoint to generate a corresponding depth image.

[0045] In some examples, such as Figure 3 As shown, the multi-task deep learning model also includes an optimization node, which can compare the three-dimensional images that can be used as a basis during model training to further optimize the above-mentioned image generation algorithm.

[0046] Step S2202: register and fuse the reconstructed image with the image part to synthesize a fused image with depth information.

[0047] In this embodiment, the fused image may be a three-dimensional RGB image.

[0048] <Equipment Example 1> Figure 4 FIG. 1 is a block diagram of a medical endoscope-based image processing device according to an embodiment. Figure 4 As shown, the image processing device 400 based on medical endoscope may include: A response module 410 is used to extract an image portion of a set area in the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope; The obtaining module 420 is used to perform three-dimensional reconstruction on the image part to obtain a fused image.

[0049] Optionally, the response module 410 is also used to respond to the two-dimensional image sent by the single-channel endoscope, convert the current image format of the two-dimensional image into a to-be-processed image of a set image format; and extract the image portion of the set area in the two-dimensional image through a preset ROI module.

[0050] Optionally, module 420 is obtained, and is also used to perform three-dimensional reconstruction of the image part through a preset multi-task deep learning model to obtain a reconstructed image; and to align and fuse the reconstructed image with the image part to synthesize a fused image with depth information.

[0051] The medical endoscope-based image processing device 400 may be Figure 1 The control device 10 in.

[0052] <Equipment Example 2> Figure 5 It is a schematic diagram of the hardware structure of an image processing device based on a medical endoscope according to another embodiment.

[0053] like Figure 5 As shown, the medical endoscope-based image processing device 500 includes a processor 510 and a memory 520, wherein the memory 520 is used to store an executable computer program, and the processor 510 is used to execute a method such as any of the above method embodiments under the control of the computer program.

[0054] The medical endoscope-based image processing device 500 may be Figure 1 The control device 10 in.

[0055] Each module of the above-mentioned medical endoscope-based image processing device 400 can be implemented by the processor 510 in this embodiment executing a computer program stored in the memory 520, or can be implemented by other structures, which is not limited here.

[0056] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0057] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0058] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0059] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present invention.

[0060] Various aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of the boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0061] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0062] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0063] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0064] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An image processing system based on a medical endoscope, characterized in that: The system comprises: A single-channel endoscope, wherein a single imaging channel of the single-channel endoscope collects a two-dimensional image; A control device is electrically connected to the single-channel endoscope, and is configured to: extract an image portion of a set area in the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope; and perform three-dimensional reconstruction on the image portion to obtain a fused image.

2. The image processing system based on medical endoscope according to claim 1, characterized in that: The diameter of the first channel corresponding to the single imaging channel of the single-channel endoscope is less than or equal to 10 mm.

3. The image processing system based on medical endoscope according to claim 1, characterized in that: The first channel diameter is the diameter of the front end of the channel of the single imaging channel.

4. The image processing system based on medical endoscope according to claim 1, characterized in that: The system further includes a display device, which is electrically connected to the control device, and is used for displaying the fused image in response to the fused image sent by the control device.

5. An image processing method based on medical endoscope, characterized in that: The method is applied to the image processing system based on the medical endoscope, the execution subject of the method is a control device in the image processing system based on the medical endoscope, and the method includes: In response to the two-dimensional image sent by the single-channel endoscope, extracting an image portion of a set area in the two-dimensional image; The image part is reconstructed in three dimensions to obtain a fused image.

6. The image processing method based on medical endoscope according to claim 1, characterized in that: The step of extracting an image portion of a set area in the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope comprises: In response to the two-dimensional image sent by the single-channel endoscope, converting the current image format of the two-dimensional image into an image to be processed in a set image format; The image portion of the set area in the two-dimensional image is extracted through a preset ROI module.

7. The image processing method based on medical endoscope according to claim 1, characterized in that: The three-dimensional reconstruction of the image part to obtain a fused image includes: Performing three-dimensional reconstruction on the image portion through a preset multi-task deep learning model to obtain a reconstructed image; The reconstructed image is registered and fused with the image portion to synthesize a fused image with depth information.

8. The image processing method based on medical endoscope according to claim 7, characterized in that: The multi-task deep learning model includes a generation node, a fusion node, and a rendering node connected in sequence; Among them, the generation node is used to generate a first projection image of a predetermined viewpoint based on the image part; the fusion node is used to fuse the first projection image into three-dimensional point cloud data to generate a point cloud image; the rendering node is used to render the point cloud image into a second projection image corresponding to a specific viewpoint to generate a depth image.

9. An image processing device based on a medical endoscope, characterized in that: The device comprises: A response module, configured to extract an image portion of a set area in the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope; The module is used to perform three-dimensional reconstruction on the image part to obtain a fused image.

10. An image processing device based on a medical endoscope, characterized in that: It comprises a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the image processing method based on medical endoscope according to any one of claims 5 to 8 under the control of the computer program.

Images