Endoscope, endoscope image de-occlusion method, and endoscope system

Abstract

The application discloses an endoscope, an endoscope image unocclusion method and an endoscope system. The endoscope comprises a mirror tube, a light source and an image acquisition device. An instrument channel is formed in the mirror tube. At least part of the mirror tube is adapted to be inserted into a human body to be close to a surgical area. The instrument channel is adapted to extend a surgical instrument into the surgical area. The light source is arranged on the mirror tube and forms a light emitting part at one end of the mirror tube inserted into the human body, so as to provide illumination for the surgical area. The image acquisition device is configured as multiple image acquisition devices. The multiple image acquisition devices are arranged at the one end of the mirror tube inserted into the human body, so as to respectively acquire images of the surgical area in different directions. The application can acquire unoccluded endoscope images, ensure a clear and complete surgical field, and further improve the surgical efficiency and safety.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an endoscope, an endoscope image unblocking method, and an endoscope system. Background Technology

[0002] In spinal endoscopic surgeries such as lumbar discectomy, the endoscope and surgical instruments are accessed via the same side, and the surgeon performs the procedure simultaneously under endoscopic observation. Due to the small surgical area in spinal endoscopic surgery, there are issues with instruments obstructing and limiting the endoscopic field of view. As a result, the surgeon needs to frequently pull the instruments back to observe the surgical tissue, which not only increases the duration of the surgery but also creates a significant surgical burden and increases the risk of complications.

[0003] To address the aforementioned issues, related technologies utilize near-infrared light sources to penetrate and illuminate the surgical site when obstructions exist. Near-infrared images are then obtained using algorithms such as dehazing and noise reduction to achieve sharper edges and improve image clarity. However, the introduction of near-infrared light sources significantly increases the difficulty of system integration under the constraints of narrow access routes in minimally invasive surgery. Furthermore, near-infrared images differ considerably in tone and quality from those perceived by the human eye. Directly displaying these images to the surgeon could lead to misjudgments of the surgical area, failing to completely resolve the issue of obstructions affecting the surgery. Summary of the Invention

[0004] The present invention aims to at least solve one of the technical problems existing in the prior art. To this end, one object of the present invention is to provide an endoscope, an endoscope image unobstructed method, and an endoscope system, which can remove obstructions in the endoscopic image and ensure the integrity of the surgical field of view.

[0005] This application discloses an endoscope, which includes a tube, a light source, and an image acquisition unit. An instrument channel is formed inside the tube, and at least a portion of the tube is adapted to be inserted into the human body to approach the surgical area. The instrument channel is adapted for surgical instruments to be inserted into the surgical area. The light source is disposed in the tube and has a light-emitting part formed at the end of the tube that is inserted into the human body, so as to provide illumination to the surgical area. Multiple image acquisition units are constructed, and multiple image acquisition units are spaced apart at the end of the tube that is inserted into the human body, so as to acquire images of the surgical area from different directions.

[0006] According to the endoscope of this application, since multiple image acquisition devices are set, images of the surgical area can be acquired from multiple directions. The images acquired by the multiple image acquisition devices can fill in the occluded areas in the same coordinate system, thereby obtaining unobstructed endoscopic images, ensuring a clear and complete surgical field of view, and thus improving surgical efficiency and safety.

[0007] According to some embodiments of this application, the axial end face of the end of the endoscope tube that extends into the human body is inclined about the extension direction of the endoscope tube.

[0008] According to some embodiments of this application, the endoscope tube includes a sleeve and a mounting base. An instrument channel is formed inside the sleeve. The mounting base is disposed at one end of the sleeve and has a first through hole that is directly opposite to and communicates with the instrument channel. The mounting base has a first mounting hole and a second mounting hole. The first mounting hole is configured to have multiple holes corresponding one-to-one with an image acquisition component. The image acquisition component is disposed in the first mounting hole, and at least a portion of the light source component is disposed in the second mounting hole. A circuit channel communicating with the first mounting hole and / or the second mounting hole is formed inside the sleeve.

[0009] According to some embodiments of this application, the light source includes an optical fiber and an optical fiber light source. One end of the optical fiber is formed with a light-emitting part, and the other end of the optical fiber is connected to the optical fiber light source. The optical fiber light source is disposed at the end of the lens tube away from the human body.

[0010] According to some embodiments of this application, the endoscope further includes a protective sleeve, which is disposed at the end of the endoscope tube that extends into the human body. The protective sleeve has a second through hole that is directly opposite to and communicates with the instrument channel. A closed cavity is formed between the protective sleeve and the endoscope tube, and the image acquisition element and the light-emitting part are disposed in the closed cavity.

[0011] This application also proposes an endoscopic image de-occlusion method, applied to the aforementioned endoscope. The image acquisition device includes a first image acquisition device and multiple second image acquisition devices. The endoscopic image de-occlusion method includes the following steps:

[0012] Perform intrinsic and extrinsic parameter calibration on each image acquisition component;

[0013] Multiple surgical area images are acquired, among which the surgical area image acquired by the first image acquisition device is the target image;

[0014] Determine if there is occlusion in the target image;

[0015] If there is occlusion in the target image, the target image is segmented into occluded part and preserved part, and the region location and shape mask of the occluded part are extracted;

[0016] Affine transformation is performed on the surgical area images acquired by each second image acquisition device to obtain the transformed images of the surgical area images acquired by each second image acquisition device in the coordinate system of the first image acquisition device.

[0017] The transformed images are cropped based on the location and shape of the occluded area using a mask, and the cropped portion is used to fill in the remaining portion to obtain the de-occluded endoscopic image.

[0018] According to some embodiments of this application, the endoscopic image unobstructed method further includes the following steps:

[0019] Determine whether there are stitching boundaries in the unobstructed endoscopic images;

[0020] If the demasked endoscopic image has a stitching boundary, the stitching boundary is eliminated based on the cropped and retained portions.

[0021] According to some embodiments of this application, the endoscopic image unobstructed method further includes the following steps:

[0022] Determine if there are missing areas in the unobstructed endoscopic image;

[0023] If there are missing areas in the de-masked endoscopic image, the missing areas are filled based on the cropped and retained portions.

[0024] According to some embodiments of this application, the endoscopic image unobstructed method further includes the following steps:

[0025] Determine whether the cropped portions of multiple transformed images overlap;

[0026] If the cropped portions of multiple transformed images overlap, one of the multiple transformed images is selected based on the relative positional relationship between each second image acquisition unit and the first image acquisition unit to fill in the overlapping portion of the retained portion; or the overlapping portions of multiple transformed images are weighted and summed to fill in the retained portion.

[0027] This application also proposes an endoscope system, which includes the aforementioned endoscope, image processing device, and image display device. The endoscope is used to acquire surgical area images from multiple directions; the image processing device is used to generate unobstructed endoscopic images based on the surgical area images from multiple directions; and the image display device is used to display the unobstructed endoscopic images.

[0028] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0030] Figure 1 This is a schematic diagram of the structure of an endoscope according to some embodiments of this application;

[0031] Figure 2 This is a schematic diagram of the structure of an endoscope mounting base according to some embodiments of this application;

[0032] Figure 3 This is a schematic diagram of the structure of the sleeve of an endoscope according to some embodiments of this application;

[0033] Figure 4This is a schematic diagram of the structure of an image acquisition component of an endoscope according to some embodiments of this application;

[0034] Figure 5 This is a schematic diagram of the installation of a protective sleeve for an endoscope according to some embodiments of this application;

[0035] Figure 6 This is a flowchart illustrating an endoscopic image unobstructed method according to some embodiments of this application;

[0036] Figure 7 This is a schematic diagram of the structure of an endoscope system according to some embodiments of this application;

[0037] Figure 8 This is a schematic diagram of the structure of a control system for an endoscope according to some embodiments of this application.

[0038] Figure label:

[0039] Endoscope 100;

[0040] Sleeve 10; Instrument channel 11; Infusion channel 12; Circuit channel 13;

[0041] Image acquisition component 20; camera 21; mounting sleeve 22; first image acquisition component 201; second image acquisition component 202;

[0042] Light source component 30;

[0043] Mounting base 40; First mounting hole 41; Second mounting hole 42; First through hole 43; First filling hole 44;

[0044] Protective sleeve 50; second through hole 51, second injection hole 52. Detailed Implementation

[0045] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0046] The following is for reference. Figures 1-5 An endoscope according to an embodiment of the present invention is described.

[0047] This application discloses an endoscope 100, which includes a tube, a light source 30, and an image acquisition unit 20. An instrument channel 11 is formed inside the tube, and at least a portion of the tube is adapted to be inserted into the human body to approach the surgical area. The instrument channel 11 is adapted for surgical instruments to be inserted into the surgical area. The light source 30 is disposed in the tube and has a light-emitting part formed at the end of the tube that is inserted into the human body, so as to provide illumination to the surgical area. Multiple image acquisition units 20 are constructed, and multiple image acquisition units 20 are spaced apart at the end of the tube that is inserted into the human body, so as to acquire images of the surgical area from different directions respectively.

[0048] According to the endoscope 100 of this application, the light source 30 can provide illumination to the surgical area, facilitating image acquisition and surgery. During surgery, surgical instruments are inserted into the surgical site through the instrument channel 11 to perform the surgical operation. Multiple image acquisition units 20 are provided at the end of the endoscope tube that extends into the human body. Each of the multiple image acquisition units 20 can acquire images of the surgical area. Because the positions of the image acquisition units 20 relative to the surgical site are different, the acquired surgical area images differ. When there is occlusion between the image acquisition unit 20 and the surgical site, the occluded areas differ in the surgical area images acquired by the multiple image acquisition units 20. If one surgical area image is taken as the target image, and the other acquired surgical area images are transformed to the coordinate system of the image acquisition unit 20 corresponding to the target image, the occluded areas of the target image can be selected and cropped. The cropped portion fills in the target image to obtain an unobstructed endoscope image 100.

[0049] According to the endoscope 100 of this application, since multiple image acquisition units 20 are provided, images of the surgical area can be acquired from multiple directions. The images acquired by the multiple image acquisition units 20 can fill in the occluded areas in the same coordinate system, thereby obtaining unobstructed images of the endoscope 100, ensuring a clear and complete surgical field of view, and thus improving surgical efficiency and safety.

[0050] According to some embodiments of this application, the image acquisition element 20 is constructed in 2-6 units, and the light source element 30 is constructed in 1-12 units. The actual number of the image acquisition element 20 and the light source element 30 is selected according to the structural dimensions of the endoscope 100 and the requirements of the endoscope image.

[0051] According to some embodiments of this application, the axial end face of the endoscope tube extending into the human body is inclined about the extension direction of the endoscope tube. In this embodiment, by inclinedly setting the axial end face of the endoscope tube, the insertion of the endoscope 100 into the human body can be facilitated, the friction between the tip of the endoscope 100 and the human tissue can be reduced, unnecessary damage to the human tissue can be avoided, and the surgical operation can be facilitated. In some embodiments, the axial end face of the endoscope tube is constructed as a beveled surface, with an angle of 30° with the extension direction of the endoscope tube.

[0052] According to some embodiments of this application, the endoscope tube includes a sleeve 10 and a mounting base 40. An instrument channel 11 is formed inside the sleeve 10. The mounting base 40 is disposed at one end of the sleeve 10 and has a first through hole 43 that is directly opposite to and communicates with the instrument channel 11. The mounting base 40 has a first mounting hole 41 and a second mounting hole 42. The first mounting hole 41 is configured to have multiple holes corresponding one-to-one with the image acquisition element 20. The image acquisition element 20 is disposed inside the first mounting hole 41, and at least a portion of the light source element 30 is disposed inside the second mounting hole 42. A circuit channel 13 communicating with the first mounting hole 41 and / or the second mounting hole 42 is formed inside the sleeve 10.

[0053] In this embodiment, as Figure 1 As shown, the endoscope tube is constructed as a combination of a sleeve 10 and a mounting base 40, facilitating processing, assembly, and disassembly / replacement. It also facilitates the installation and fixation of the image acquisition component 20 and the light source component 30, making flexible use of limited structural space. Furthermore, the mounting holes enable the installation, fixation, and positioning of the image acquisition component 20 and the light source component 30, improving their stability during endoscope 100 use and enhancing imaging stability. Simultaneously, the mounting base 40 protects the image acquisition component 20 and the light source component 30, preventing bodily fluids from seeping into the mounting holes or even the wiring channel 13, which could lead to short circuits or contamination. Additionally, this installation method reduces the structural impact on the internal environment of the human body.

[0054] In some embodiments, the sleeve 10 and the mounting base 40 are connected by welding, but they can also be connected by bonding, snap-fitting, threaded fixing, or other methods. Furthermore, one axial end face of the mounting base 40 is connected to the sleeve 10, and the other axial end face is inclined about the extension direction of the lens tube.

[0055] In some embodiments, such as Figure 2 , Figure 3As shown, the instrument channel 11 extends in the same direction as the endoscope tube and is located at the center of the sleeve 10. The first through hole 43 is located at the center of the mounting base 40. Further, the first mounting hole 41 and the second mounting hole 42 are both located on the outer periphery of the first through hole 43 and are spaced apart circumferentially to provide illumination and image acquisition around the surgical instrument, avoiding blind spots in illumination and imaging. Simultaneously, the wiring channel 13 is located on the outer periphery of the instrument channel 11 and communicates with the first mounting hole 41 and / or the second mounting hole 42 to facilitate the placement and passage of wiring for the image acquisition element 20 and / or the light source element 30. In some embodiments, to simplify the structure, the wiring channel 13 is directly opposite multiple first mounting holes 41 and / or second mounting holes 42 in the extending direction of the endoscope tube to accommodate wiring for multiple image acquisition elements 20 and / or light source elements 30. Therefore, the wiring channel 13 does not need to be constructed as multiple channels corresponding one-to-one with the first mounting hole 41 and the second mounting hole 42; the number can be selected and set according to the spatial distribution of the sleeve 10 structure.

[0056] In some embodiments, the extension direction of the first mounting hole 41 is consistent with the extension direction of the lens tube, and the first mounting hole 41 is constructed as a stepped hole to limit the image acquisition member 20 in the extension direction of the lens tube.

[0057] According to some embodiments of this application, the light source 30 includes an optical fiber and an optical fiber light source. One end of the optical fiber is disposed in the second mounting hole 42, and the other end of the optical fiber is connected to the optical fiber light source. The optical fiber light source is disposed at the end of the lens tube furthest from the human body. In this embodiment, the light source 30 uses an optical fiber and an optical fiber light source, which can transmit light signals over long distances, save installation space, improve illumination capabilities, help acquire high-definition images, and improve image acquisition efficiency and accuracy. Furthermore, a portion of the optical fiber is disposed in the line channel 13.

[0058] According to some embodiments of this application, such as Figure 4 As shown, the image acquisition component 20 is constructed as a camera assembly, which includes a mounting sleeve 22 and a camera 21. The camera 21 is disposed within the mounting sleeve 22, and the mounting sleeve 22 is disposed in the first mounting hole 41. In this embodiment, the mounting sleeve 22 can protect the camera 21 and can also cooperate with the structure of the first mounting hole 41 to achieve a tight connection, thereby providing a certain sealing effect and preventing surgical fluid from seeping into the endoscope 100.

[0059] According to some embodiments of this application, the endoscope 100 further includes a protective sleeve 50, which is disposed at the end of the endoscope tube that extends into the human body. The protective sleeve 50 has a second through hole 51 that is directly opposite to and communicates with the instrument channel 11. A closed cavity is formed between the protective sleeve 50 and the endoscope tube, and the image acquisition element 20 and the light-emitting part are disposed within the closed cavity. In this embodiment, as... Figure 5As shown, by setting up a protective sleeve 50, the human body environment is isolated from the image acquisition unit 20 and the light source unit 30, which can prevent direct contact between the image acquisition unit 20 and the light source unit 30, avoid contamination of the image acquisition unit 20 by body fluids and avoid affecting imaging, and at the same time avoid structural damage to human tissue. It should be noted that, in order not to affect illumination and image acquisition, the protective sleeve 50 is constructed with a transparent structure, specifically made of transparent optical glass. In addition, the protective sleeve 50 has a second through hole 51 to facilitate the insertion of surgical instruments into the human body for surgical operations. The second through hole 51, the first through hole 43, and the instrument channel 11 are aligned in the extension direction of the endoscope tube. Furthermore, the end face of the protective sleeve 50 near the surgical site is constructed with a beveled surface and is inclined with respect to the extension direction of the endoscope tube to facilitate insertion into the human body.

[0060] In some embodiments, the protective sleeve 50 and the mounting base 40 are connected by adhesive bonding, or by welding, snap-fit, threaded fixing, or other methods.

[0061] According to some embodiments of this application, an infusion channel 12 for delivering media to the surgical area is formed inside the endoscope tube. In this embodiment, the endoscope tube is provided with an infusion channel 12 to introduce surgical media into the surgical environment, thereby enabling the surgery to proceed smoothly. Further, as Figure 3 As shown, the sleeve 10 has the infusion channel 12, the mounting base 40 has a first infusion hole 44 that is opposite to and communicates with the infusion channel 12, and the protective sleeve 50 has a second infusion hole 52 that is opposite to and communicates with the infusion channel 12. The first infusion hole 44, the second infusion hole 52 and the infusion channel 12 are directly opposite to and communicate with the infusion channel 12 in the extension direction of the endoscope tube.

[0062] In some embodiments, the infusion channel 12 is disposed on the outer periphery of the instrument channel 11. Further, multiple infusion channels 12 are configured, and the multiple infusion channels 12 are spaced apart circumferentially. Even further, the infusion channels 12 and the wiring channels 13 are spaced apart circumferentially, and the walls of adjacent infusion channels 12 are connected to the walls of wiring channels 13. The above design allows for a compact structure within the endoscope tube, improving the space utilization of the endoscope tube structure. The number of infusion channels 12 and wiring channels 13 is selected and designed according to the structural dimensions of the endoscope tube and the surgical requirements.

[0063] It should be noted that, according to the endoscope of this application, when acquiring unobstructed endoscopic images, the surgical area image acquired by any one image acquisition device is used as the target image, and surgical area images from other image acquisition devices are used to fill it. Accordingly, the multiple image acquisition devices include one main acquisition device and multiple auxiliary acquisition devices. In some embodiments, since the axial end face of the endoscope tube extending into the human body is inclined about the extension direction of the endoscope tube, the relative distances between the multiple image acquisition devices and the surgical site are different. In practice, the image acquisition device farthest from the surgical site can be constructed as the main acquisition device to capture the surgical area situation to the greatest extent.

[0064] In some embodiments, the image acquisition unit 20 includes a first image acquisition unit 201 and a second image acquisition unit 202, wherein the first image acquisition unit 201 is a primary acquisition unit and the second image acquisition unit 202 is an auxiliary acquisition unit. In some embodiments, the first image acquisition unit 201 is configured as a primary camera and the second image acquisition unit 202 is configured as a secondary camera.

[0065] Furthermore, the first image acquisition unit 201 is configured as one unit; the second image acquisition unit 202 is configured as one to five units.

[0066] In some embodiments, such as Figures 1-5 As shown, the second image acquisition unit 202 is constructed as three units, which are arranged at 60° intervals around the center of the endoscope tube on the side of the endoscope tube axial end closer to the surgical site. The first image acquisition unit 201 is arranged on the side of the endoscope tube axial end away from the surgical site. Further, as... Figure 2 As shown, the light source element 30 is constructed of 6 units, arranged at 60° intervals along the outer periphery of the first through hole 43. Figure 2 As shown, the infusion channel 12 is constructed as two channels, which are set on both sides of the first image acquisition device 201 at a 60° interval from the center of the lens tube.

[0067] This application also proposes an endoscopic image de-occlusion method. Using the aforementioned endoscope, the image acquisition device includes a first image acquisition device and multiple second image acquisition devices. The endoscopic image de-occlusion method includes the following steps:

[0068] S1. Perform intrinsic and extrinsic parameter calibration on each image acquisition component;

[0069] S2. Acquire multiple surgical area images, wherein the surgical area image acquired by the first image acquisition device is the target image;

[0070] S3. Determine if there is occlusion in the target image; if there is occlusion in the target image, segment the target image into occluded part and retained part, and extract the region location and shape mask of the occluded part;

[0071] S4. Perform affine transformation on the surgical area images acquired by each second image acquisition device to obtain the transformed images of the surgical area images acquired by each second image acquisition device in the coordinate system of the first image acquisition device.

[0072] S5. Based on the location and shape of the occluded area, the transformed images are cropped using a mask, and the cropped portion is used to fill in the remaining portion to obtain the unoccluded endoscope image.

[0073] According to the endoscopic image unobstructed method of this application, such as Figure 6 As shown, in step S1, when calibrating the intrinsic parameters of each image acquisition device, Zhang's calibration method can be used to obtain the intrinsic parameter matrix. Intrinsic parameter calibration can correct distortions and errors during the imaging process of the image acquisition devices, improving imaging accuracy and the accuracy of subsequent image processing. When calibrating the extrinsic parameters of each image acquisition device, Zhang's calibration method can also be used. Each second image acquisition device is calibrated relative to the first image acquisition device to obtain the spatial transformation matrix of each second image acquisition device transformed to the pixel space of the first image acquisition device, i.e., the extrinsic parameter matrix. Extrinsic parameter calibration ensures that the image information captured by each image acquisition device remains consistent in space and time, enabling multi-camera collaborative work. The methods for intrinsic and extrinsic parameter calibration are not limited to Zhang's calibration method; other calibration methods can also be used.

[0074] In step S2, each image acquisition device needs to acquire surgical area images in parallel to ensure consistency of imaging time. During the acquisition of surgical area images, the intrinsic parameter matrix of each image acquisition device obtained in step S1 is used to perform distortion correction on each acquired surgical area image.

[0075] Further, step S2 includes: image enhancement of the surgical area image to compensate for the poor image quality of the acquired image. Specifically, a deep learning-based image enhancement model can be used to enhance the acquired image. In addition, in some embodiments, bicubic interpolation, image denoising, and other methods can also be used to enhance the acquired image. In some embodiments, a miniature camera is used as the image acquisition device, which has limited imaging quality; this embodiment can particularly improve its image quality.

[0076] The occlusion mentioned in step S3 mainly refers to the occlusion formed by surgical instruments within the field of view of the image acquisition device, but it is not limited to this, and also includes occlusion formed by some human blood tissue, etc. Surgical instruments include, but are not limited to, common surgical instruments such as clamps, probes, reamers, and electrocautery knives. In step S3, a deep learning-based image segmentation model can be used to extract the position and shape of the instruments in the target image. In some embodiments, graphics-based methods or other common image segmentation methods can also be used to extract the occlusion position and shape in the target image.

[0077] Step S4 specifically includes: using the intrinsic and extrinsic parameter matrices obtained in step S1, transforming the surgical area images of each second image acquisition device from their respective pixel spaces to the pixel space of the first image acquisition device.

[0078] Step S5 specifically includes: using the extracted region location and shape mask of the occluded part to crop the transformed image, and filling the cropped part into the original occluded part of the target image to obtain the de-occluded endoscope image.

[0079] According to the endoscopic image occlusion removal method of this application, under the premise of intelligently identifying the boundaries of instruments and other obstructions in the acquired image, the area obstructed by instruments in the image acquired by the first image acquisition device is filled by the image of the second image acquisition device using calibration and affine transformation methods, thereby removing the obstruction of instruments and other obstructions in the endoscopic image. This can solve the problem of instrument occlusion in endoscopic images, improve surgical convenience and safety, and enhance surgical quality.

[0080] It should be noted that when the transformed images from each of the second image acquisition devices are stitched together to the target image of the first image acquisition device, there may be obvious stitching boundaries, which may affect the surgeon's observation.

[0081] To address the aforementioned issues, according to some embodiments of this application, the endoscopic image de-occlusion method further includes the following steps: determining whether the de-occluded endoscopic image has a stitching boundary; if the de-occluded endoscopic image has a stitching boundary, eliminating the stitching boundary based on the cropped portion and the retained portion.

[0082] Specifically, the infilled target image can be processed based on a deep learning model to eliminate the splicing boundary at the filling point; or the infilled target image can be processed based on a Gaussian blur algorithm to eliminate the splicing boundary at the filling point; or the neighborhood weighting method can be used to process the infilled target image to eliminate the splicing boundary at the filling point.

[0083] In this embodiment, eliminating the stitching boundaries can improve the clarity and accuracy of endoscopic images, facilitating observation and reference during surgery.

[0084] It should also be noted that after the transformed images from each of the second image acquisition devices are stitched together to the target image of the first image acquisition device, there may still be gaps, which may affect the surgeon's observation.

[0085] To address the aforementioned technical problems, according to some embodiments of this application, the endoscopic image de-occlusion method further includes the following steps: determining whether there are missing areas in the de-occluded endoscopic image; if there are missing areas in the de-occluded endoscopic image, filling in the missing areas based on the cropped portion and the retained portion.

[0086] Specifically, depth information of the image can be extracted based on a deep learning model to fill in the missing parts, or neighborhood information can be used to fill in the missing parts. In this embodiment, a complete endoscopic image can be obtained by filling in the missing parts.

[0087] It should also be noted that since the second image acquisition device is constructed in multiple parts, there may be overlapping parts when the transformed images of multiple second image acquisition devices are stitched together. The overlapping parts are not completely identical, which causes the endoscope image to have insufficient clarity, which is not conducive to observation.

[0088] In response, according to some embodiments of this application, the endoscopic image de-occlusion method further includes the following steps: determining whether the cropped portions of multiple transformed images overlap; if the cropped portions of multiple transformed images overlap, selecting one of the multiple transformed images to fill the overlapping portion to the retained portion based on the relative positional relationship between each second image acquisition unit and the first image acquisition unit; or weighted summing of the overlapping portions of multiple transformed images to fill the retained portion. This embodiment can further improve the clarity and reliability of endoscopic images.

[0089] Specifically, when selecting a transformation image, a confidence function can be set according to the relative positional relationship between each of the second image acquisition devices and the first image acquisition device, and the transformation image corresponding to the second image acquisition device with the highest confidence is selected to fill in the overlapping part of the target image.

[0090] This application also proposes an endoscope system, such as Figure 7 As shown, the endoscopic system includes the aforementioned endoscope, as well as an image processing device and an image display device. The endoscope is used to acquire surgical area images from multiple directions; the image processing device is used to process the surgical area images to generate unobstructed endoscopic images based on the surgical area images from multiple directions; and the image display device is used to display the unobstructed endoscopic images. Specifically, the endoscope includes multiple image acquisition components; more specifically, the endoscope can be constructed as a multi-camera endoscope.

[0091] According to some embodiments of this application, such as Figure 7 As shown, the image processing device includes a processor, a memory, a communication module, and an output module. The memory stores an endoscopic image de-occlusion program and is used to store surgical area images acquired by the endoscope and the processor's processing results. The processor executes the endoscopic image de-occlusion program to obtain a de-occluded endoscopic image, and executes the aforementioned endoscopic image de-occlusion method during the execution of the program. The output module outputs the de-occluded endoscopic image to an image display device. The communication module communicates with external devices or servers, and includes a Wi-Fi module, a mobile communication module, and a Bluetooth module, etc.

[0092] The processor's processing results include a calibration matrix, the location and shape mask of the occluded area, the retained portion, various transformed images, and their cropped portions. It is important to note that the surgical area images acquired by the endoscope in the memory, along with the processor's processing results, are updated in real time.

[0093] This application also proposes an endoscope control system to control and implement the above-mentioned method for unobstructed endoscopic images. For example... Figure 8 As shown, the control system includes an image acquisition module, an image processing module, and an image display module. The image acquisition module is used to acquire surgical area images from multiple directions; the image processing module is used to process the surgical area images acquired by the image acquisition module and obtain unobstructed endoscopic images; the image display module is used to output and display the unobstructed endoscopic images obtained by the image processing module. The image acquisition module includes a main acquisition module and an auxiliary acquisition module.

[0094] According to some embodiments of this application, the image processing module includes an instrument segmentation and extraction module, an affine transformation module, and an image filling module; the instrument segmentation and extraction module is used to segment and extract occluded areas in the surgical area image acquired by the main acquisition module; the affine transformation module is used to transform the surgical area image acquired by the auxiliary acquisition module to the pixel space of the main acquisition module using an extrinsic parameter matrix; the image filling module is used to crop the transformed surgical area image of the auxiliary acquisition module and fill in the image of the main acquisition module.

[0095] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0096] In the description of this invention, "first feature" and "second feature" may include one or more of the features.

[0097] In the description of this invention, "a plurality of" means two or more.

[0098] In the description of this invention, the first feature being "above" or "below" the second feature may include the first and second features being in direct contact, or it may include the first and second features not being in direct contact but being in contact through another feature between them.

[0099] In the description of this invention, the terms "above," "over," and "on top" for the first feature and the second feature include the first feature being directly above or diagonally above the second feature, or simply indicating that the first feature is at a higher horizontal level than the second feature.

[0100] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0101] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for unobstructing endoscopic images, characterized in that, An endoscope is used, the endoscope comprising: The endoscope tube has an instrument channel formed inside it. At least a portion of the endoscope tube is adapted to be inserted into the human body to approach the surgical area. The instrument channel is adapted for surgical instruments to be inserted into the surgical area. The axial end face of the endoscope tube that extends into the human body is inclined about the extension direction of the endoscope tube. A light source is disposed on the endoscope tube and has a light-emitting part formed at the end of the endoscope tube that extends into the human body, so as to provide illumination to the surgical area; The image acquisition device is configured in multiple ways, and the multiple image acquisition devices are spaced apart at the end of the endoscope tube that extends into the human body to be suitable for acquiring images of the surgical area from different directions; the image acquisition device includes a first image acquisition device and multiple second image acquisition devices; and the first image acquisition device is configured to be the one farthest from the surgical site; The method for removing obstructions from endoscopic images includes the following steps: Perform intrinsic and extrinsic parameter calibration on each of the image acquisition devices; Multiple surgical area images are acquired, wherein the surgical area image acquired by the first image acquisition device is the target image; Determine whether there is occlusion in the target image; If there is occlusion in the target image, the target image is segmented into occluded part and retained part, and the region location and shape mask of the occluded part are extracted; Affine transformation is performed on the surgical area images acquired by each of the second image acquisition devices to obtain the transformed images of the surgical area images acquired by each of the second image acquisition devices in the coordinate system of the first image acquisition device. The transformed images are cropped according to the region location and shape mask of the occluded part, and the cropped part is used to fill in the retained part to obtain the de-occluded endoscope image.

2. The endoscopic image unobstructed method according to claim 1, characterized in that, The endoscope tube includes: A sleeve, wherein the instrument channel is formed within the sleeve; A mounting base is disposed at one end of the sleeve, and the mounting base has a first through hole that is directly opposite to and communicates with the instrument channel; wherein The mounting base has a first mounting hole and a second mounting hole. The first mounting hole is configured to correspond one-to-one with the image acquisition component. The image acquisition component is disposed in the first mounting hole, and at least a portion of the light source component is disposed in the second mounting hole. A circuit channel communicating with the first mounting hole and / or the second mounting hole is formed inside the sleeve.

3. The endoscopic image unobstructed method according to claim 1, characterized in that, The light source includes: An optical fiber and an optical fiber light source are provided, wherein one end of the optical fiber is formed with the light-emitting part, and the other end of the optical fiber is connected to the optical fiber light source, and the optical fiber light source is located at the end of the lens tube away from the human body.

4. The endoscopic image unobstructed method according to claim 1, characterized in that, The endoscope also includes: A protective sleeve is provided at the end of the endoscope tube that extends into the human body. The protective sleeve has a second through hole that is directly opposite to and communicates with the instrument channel. A closed cavity is formed between the protective sleeve and the endoscope tube. The image acquisition unit and the light-emitting unit are disposed in the closed cavity.

5. The endoscopic image unobstructed method according to claim 1, characterized in that, It also includes the following steps: Determine whether the unobstructed endoscopic image has stitching boundaries; If the de-occluded endoscopic image has a stitching boundary, the stitching boundary is eliminated based on the cropped portion and the retained portion.

6. The endoscopic image unobstructed method according to claim 1, characterized in that, It also includes the following steps: Determine whether there are any missing areas in the unobstructed endoscopic image; If there are missing areas in the de-occluded endoscopic image, the missing areas are filled based on the cropped portion and the retained portion.

7. The endoscopic image unobstructed method according to claim 1, characterized in that, It also includes the following steps: Determine whether the cropped portions of the multiple transformed images overlap; If the cropped portions of multiple transformed images overlap, one of the multiple transformed images is selected to fill in the overlapping portion of the retained portion according to the relative positional relationship between each of the second image acquisition devices and the first image acquisition device; Alternatively, the overlapping portions of multiple transformed images can be weighted and summed to fill in the retained portions.

8. An endoscope system, characterized in that, include: An endoscope, the endoscope being used to acquire surgical area images from multiple directions, and applying the endoscopic image de-obstruction method as described in any one of claims 1-4; An image processing device for generating unobstructed endoscopic images based on surgical area images from the plurality of directions; An image display device for displaying the unobstructed endoscopic image.

Images