Endoscope, endoscope image deblocking method and endoscope system

Abstract

The invention discloses an endoscope, an endoscope image shielding removing method and an endoscope system.The endoscope comprises an endoscope tube, a light source part and an image collecting part, an instrument channel is formed in the endoscope tube, at least part of the endoscope tube is suitable for stretching into a human body to be close to an operation area, and the instrument channel is suitable for a surgical instrument to stretch into the operation area; the light source piece is arranged on the endoscope tube, and a light-emitting part is formed at the end, extending into the human body, of the endoscope tube so as to provide illumination for an operation area. The multiple image acquisition pieces are arranged at the end, extending into the human body, of the endoscope tube at intervals so as to be suitable for acquiring operation area images in different directions. According to the invention, the non-shielding endoscope image can be obtained, the clear and complete operation view is ensured, and the operation efficiency and safety are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to an endoscope, an endoscope image de-occlusion method and an endoscope system. Background Art

[0002] In spinal endoscopic surgeries such as lumbar discectomy, the endoscope and the operating instruments enter the same side, and the doctor performs the operation simultaneously under endoscopic observation. Due to the narrow surgical field of spinal endoscopic surgery, there is a problem of the operating instruments blocking and limiting the endoscope's field of view. In this regard, the doctor needs to frequently pull back the instrument to observe the surgical tissue during the operation, which not only increases the duration of the operation, but also creates a great burden on the operation and increases the risk of the operation.

[0003] In response to the above problems, in the related technology, when there are obstructions in the field of view of the surgical site, a near-infrared light source is used for penetrating illumination to obtain a near-infrared image, and then a near-infrared image with clear edges is obtained using algorithms such as defogging and noise reduction to improve image clarity. However, the introduction of near-infrared light sources has greatly increased the difficulty of system integration under the constraints of the narrow access of minimally invasive surgery. On the other hand, there is a large gap in the color tone and quality of the near-infrared image and the imaging results of the human eye. Directly displaying it to the physician is likely to cause the physician to misjudge the situation in the surgical area, and it is impossible to completely solve the problem of occlusion affecting the surgery. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, one object of the present invention is to provide an endoscope, an endoscope image de-obstruction method and an endoscope system, which can remove obstructions in the endoscope image and ensure the integrity of the surgical field of view.

[0005] The present application proposes an endoscope, which includes a scope tube, a light source and an image acquisition component. An instrument channel is formed inside the scope tube, and at least a portion of the scope tube is suitable for extending into the human body to approach the surgical area. The instrument channel is suitable for surgical instruments to extend into the surgical area; the light source is arranged on the scope tube and a light-emitting portion is formed on the end of the scope tube extending into the human body to provide lighting for the surgical area; the image acquisition component is constructed in multiple components, and the multiple image acquisition components are arranged at intervals on the end of the scope tube extending into the human body to be suitable for respectively acquiring images of the surgical area in different directions.

[0006] According to the endoscope of the present application, since there are multiple image acquisition components, images of the surgical area can be acquired in multiple directions. The images acquired by the multiple image acquisition components can fill each other's blocked areas in the same coordinate system, thereby obtaining unobstructed endoscopic images, ensuring a clear and complete surgical field of view, and thereby improving surgical efficiency and safety.

[0007] According to some embodiments of the present application, the axial end surface of one end of the mirror tube extending into the human body is inclined relative to the extension direction of the mirror tube.

[0008] According to some embodiments of the present application, the mirror tube includes a sleeve and a mounting seat, an instrument channel is formed in the sleeve; the mounting seat is arranged at one end of the sleeve, and the mounting seat is formed with a first through hole that is directly opposite to and connected to the instrument channel; wherein the mounting seat is formed with a first mounting hole and a second mounting hole, and the first mounting hole is configured as a plurality of holes corresponding one to one with the image acquisition component; the image acquisition component is arranged in the first mounting hole, and at least a portion of the light source component is arranged in the second mounting hole; a line channel connected to the first mounting hole and / or the second mounting hole is formed in the sleeve.

[0009] According to some embodiments of the present application, the light source component includes an optical fiber and an optical fiber light source. A light-emitting portion is formed at one end of the optical fiber, and the other end of the optical fiber is connected to the optical fiber light source. The optical fiber light source is arranged at the end of the mirror tube away from the human body.

[0010] According to some embodiments of the present application, the endoscope further includes a protective cover, which is disposed at one end of the scope tube extending into the human body, and the protective cover is formed with a second through hole that is opposite to and connected to the instrument channel; a closed cavity is formed between the protective cover and the scope tube, and the image acquisition component and the light-emitting portion are disposed in the closed cavity.

[0011] The present application also proposes an endoscopic image de-blocking method, which is applied to the above-mentioned endoscope, wherein the image acquisition component includes a first image acquisition component and a plurality of second image acquisition components, and the endoscopic image de-blocking method includes the following steps:

[0012] Perform internal and external calibration on each image acquisition component;

[0013] Acquire multiple surgical area images, wherein the surgical area image acquired by the first image acquisition component is the target image;

[0014] Determine whether there is occlusion in the target image;

[0015] If there is occlusion in the target image, the target image is divided into an occluded part and a retained part, and the area position and shape mask of the occluded part are extracted;

[0016] Performing affine transformation on the surgical area images acquired by each second image acquisition component to obtain a transformed image of the surgical area images acquired by each second image acquisition component in the coordinate system of the first image acquisition component;

[0017] Each transformed image is cropped according to the area position and shape mask of the occluded part, and the cropped part is filled with the retained part to obtain a de-occluded endoscopic image.

[0018] According to some embodiments of the present application, the endoscopic image de-occlusion method further includes the following steps:

[0019] Determine whether the de-occluded endoscopic image has a splicing boundary;

[0020] If there is a stitching boundary in the de-occluded endoscopic image, the stitching boundary is eliminated based on the cropped part and the retained part.

[0021] According to some embodiments of the present application, the endoscopic image de-occlusion method further includes the following steps:

[0022] Determine whether there is a vacant area in the de-occluded endoscopic image;

[0023] If there are missing areas in the de-occluded endoscopic image, the missing areas are filled based on the cropped part and the retained part.

[0024] According to some embodiments of the present application, the endoscopic image de-occlusion method further includes the following steps:

[0025] Determine whether the cropped parts of multiple transformed images overlap;

[0026] If there is overlap in the cropped parts of multiple transformed images, one of the multiple transformed images is selected according to the relative position relationship between each second image acquisition component and the first image acquisition component to fill the retained part with the overlapping part; or the overlapping parts of the multiple transformed images are weightedly summed to fill the retained part.

[0027] The present application also proposes an endoscope system, which includes the above-mentioned endoscope, an image processing device and an image display device. The endoscope is used to collect surgical area images in multiple directions; the image processing device is used to generate unobstructed endoscopic images based on the surgical area images in multiple directions; and the image display device is used to display unobstructed endoscopic images.

[0028] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0030] Figure 1 is a schematic structural diagram of an endoscope according to some embodiments of the present application;

[0031] Figure 2 is a schematic structural diagram of a mounting base of an endoscope according to some embodiments of the present application;

[0032] Figure 3 is a schematic structural diagram of a sleeve of an endoscope according to some embodiments of the present application;

[0033] Figure 4is a schematic diagram of the structure of an image acquisition component of an endoscope according to some embodiments of the present application;

[0034] Figure 5 is a schematic diagram of installing a protective cover of an endoscope according to some embodiments of the present application;

[0035] Figure 6 is a flowchart of an endoscopic image de-occlusion method according to some embodiments of the present application;

[0036] Figure 7 is a schematic structural diagram of an endoscope system according to some embodiments of the present application;

[0037] Figure 8 It is a schematic diagram of the structure of a control system of an endoscope according to some embodiments of the present application.

[0038] Reference numerals:

[0039] Endoscope 100;

[0040] Sleeve 10; instrument channel 11; irrigation channel 12; line channel 13;

[0041] Image acquisition component 20; camera 21; mounting sleeve 22; first image acquisition component 201; second image acquisition component 202;

[0042] Light source 30;

[0043] Mounting seat 40; first mounting hole 41; second mounting hole 42; first through hole 43; first injection hole 44;

[0044] Protective cover 50; second through hole 51, second injection hole 52. DETAILED DESCRIPTION

[0045] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0046] Reference below Figure 1-Figure 5 An endoscope according to an embodiment of the present invention is described.

[0047] The present application proposes an endoscope 100, which includes a mirror tube, a light source component 30 and an image acquisition component 20. An instrument channel 11 is formed inside the mirror tube. At least a portion of the mirror tube is suitable for extending into the human body to approach the surgical area. The instrument channel 11 is suitable for surgical instruments to extend into the surgical area. The light source component 30 is arranged on the mirror tube and a light-emitting portion is formed at one end of the mirror tube extending into the human body to provide lighting for the surgical area. The image acquisition component 20 is constructed in multiple components, and the multiple image acquisition components 20 are spaced apart at one end of the mirror tube extending into the human body to be suitable for respectively acquiring images of the surgical area in different directions.

[0048] According to the endoscope 100 of the present application, the light source component 30 can provide lighting for the surgical area, which is convenient for image acquisition and surgery; during surgery, the surgical instrument is inserted into the surgical site through the instrument channel 11 to perform the surgical operation. A plurality of image acquisition components 20 are provided at one end of the end of the endoscope tube extending into the human body, and the plurality of image acquisition components 20 can all acquire images of the surgical area. Since the positions of the image acquisition components 20 relative to the surgical site are different, the acquired surgical area images are different. When there is an obstruction between the image acquisition component 20 and the surgical site, the obstruction areas in the surgical area images acquired by the plurality of image acquisition components 20 are different. If one of the surgical area images is taken as the target image, the other acquired surgical area images are transformed into the image acquisition component 20 coordinate system corresponding to the target image, and then the target image can be selectively cropped according to the obstruction area of ​​the target image. The cropped part can fill the target image to obtain an unobstructed endoscope 100 image.

[0049] According to the endoscope 100 of the present application, since multiple image acquisition components 20 are provided, images of the surgical area can be acquired in multiple directions, and the images acquired by the multiple image acquisition components 20 can complement each other with respect to the blocked areas in the same coordinate system, thereby obtaining an unobstructed image of the endoscope 100, ensuring a clear and complete surgical field of view, thereby improving surgical efficiency and safety.

[0050] According to some embodiments of the present application, the number of image acquisition components 20 is 2-6, and the number of light source components 30 is 1-12. The actual number of image acquisition components 20 and light source components 30 is selected according to the structural size of the endoscope 100 and the endoscope image requirements.

[0051] According to some embodiments of the present application, the axial end surface of the end of the mirror tube extending into the human body is inclined relative to the extension direction of the mirror tube. In this embodiment, by tilting the axial end surface of the mirror tube, it is possible to facilitate the process of the endoscope 100 extending into the human body, reduce the friction between the front end of the endoscope 100 and the human tissue, avoid unnecessary damage to the human tissue, and facilitate the surgical operation. In some embodiments, the axial end surface of the mirror tube is configured as a beveled surface, and the angle with the extension direction of the mirror tube is 30°.

[0052] According to some embodiments of the present application, the mirror tube includes a sleeve 10 and a mounting seat 40, and an instrument channel 11 is formed in the sleeve 10; the mounting seat 40 is arranged at one end of the sleeve 10, and the mounting seat 40 is formed with a first through hole 43 that is opposite to and connected to the instrument channel 11; the mounting seat 40 is formed with a first mounting hole 41 and a second mounting hole 42, and the first mounting hole 41 is configured to be a plurality of holes corresponding to the image acquisition component 20 one by one; the image acquisition component 20 is arranged in the first mounting hole 41, and at least a portion of the light source component 30 is arranged in the second mounting hole 42; a line channel 13 connected to the first mounting hole 41 and / or the second mounting hole 42 is formed in the sleeve 10.

[0053] In this embodiment, if Figure 1 As shown, the mirror tube is constructed as a combination of a sleeve 10 and a mounting seat 40, which is convenient for processing, assembly, disassembly and replacement; it is also convenient for the installation and fixation of the image acquisition component 20 and the light source component 30, and the limited structural space is flexibly utilized. In addition, the installation, fixation and position limiting of the image acquisition component 20 and the light source component 30 are realized through the mounting hole, which can improve the stability of the image acquisition component 20 and the light source component 30 during the use of the endoscope 100 and improve the stability of imaging; at the same time, the mounting seat 40 can protect the image acquisition component 20 and the light source component 30, and prevent body fluids from penetrating into the mounting hole or even the line channel 13 to cause short circuits, pollution, etc.; in addition, this installation method can also reduce the impact of the structure on the internal environment of the human body.

[0054] In some embodiments, the sleeve 10 and the mounting seat 40 are connected by welding, or by bonding, snap-fitting, threaded fixing, etc. Further, one axial end surface of the mounting seat 40 is connected to the sleeve 10, and the other axial end surface is inclined relative to the extension direction of the mirror tube.

[0055] In some embodiments, Figure 2 , Figure 3As shown, the instrument channel 11 is consistent with the extension direction of the mirror tube, the instrument channel 11 is arranged at the center of the sleeve 10, and the first through hole 43 is arranged at the center of the mounting seat 40. Further, the first mounting hole 41 and the second mounting hole 42 are both arranged on the periphery of the first through hole 43, and are arranged at intervals along the circumferential direction, so as to be suitable for providing lighting and image acquisition on the peripheral side of the surgical instrument, avoiding the existence of lighting blind spots and shooting blind spots. At the same time, the line channel 13 is arranged on the periphery of the instrument channel 11 and is connected with the first mounting hole 41 and / or the second mounting hole 42 to be suitable for the placement and passage of the line of the image acquisition component 20 and / or the light source component 30. In some embodiments, in order to simplify the structure, the line channel 13 is directly opposite to the multiple first mounting holes 41 and / or the second mounting holes 42 in the extension direction of the mirror tube to accommodate the lines of multiple image acquisition components 20 and / or the light source components 30. Therefore, the line channel 13 does not need to be constructed as a plurality of corresponding to the first mounting holes 41 and the second mounting holes 42, and the number can be selected and set according to the spatial distribution of the sleeve 10 structure.

[0056] In some embodiments, the extension direction of the first mounting hole 41 is consistent with the extension direction of the mirror tube, and the first mounting hole 41 is configured as a stepped hole to limit the image acquisition component 20 in the extension direction of the mirror tube.

[0057] According to some embodiments of the present application, the light source component 30 includes an optical fiber and an optical fiber light source, one end of the optical fiber is arranged in the second mounting hole 42, the other end of the optical fiber is connected to the optical fiber light source, and the optical fiber light source is arranged at the end of the mirror tube away from the human body. In this embodiment, the light source component 30 uses an optical fiber and an optical fiber light source, which can transmit optical signals over long distances, can save installation space, improve lighting capabilities, help obtain high-definition images, and improve image acquisition efficiency and accuracy. Furthermore, part of the optical fiber is arranged in the line channel 13.

[0058] According to some embodiments of the present application, Figure 4 As shown, the image acquisition component 20 is constructed as a camera assembly, and the camera assembly includes a mounting sleeve 22 and a camera 21, wherein the camera 21 is disposed in the mounting sleeve 22, and the mounting sleeve 22 is disposed in the first mounting hole 41. In this embodiment, the mounting sleeve 22 can protect the camera 21, and can cooperate with the structure of the first mounting hole 41 to achieve a tight connection, thereby achieving a certain sealing effect and preventing the body fluid in the surgical area from penetrating into the interior of the endoscope 100.

[0059] According to some embodiments of the present application, the endoscope 100 further includes a protective cover 50, which is disposed at the end of the end of the endoscope tube extending into the human body, and the protective cover 50 is formed with a second through hole 51 that is directly opposite to and connected to the instrument channel 11; a closed cavity is formed between the protective cover 50 and the endoscope tube, and the image acquisition component 20 and the light-emitting unit are disposed in the closed cavity. Figure 5As shown, by providing a protective cover 50, the human body environment is isolated from the image acquisition component 20 and the light source component 30, which can avoid direct contact between the image acquisition component 20 and the light source component 30, avoid body fluids from contaminating the image acquisition component 20 and affecting imaging, and avoid the structure from causing damage to human tissue. It should be noted that in order not to affect lighting and image acquisition, the protective cover 50 is constructed as a transparent structure, and can be made of transparent optical glass. In addition, the protective cover 50 is formed with a second through hole 51 to accommodate surgical instruments inserted into the human body for surgical operations, and the second through hole 51, the first through hole 43 and the instrument channel 11 are directly opposite in the extension direction of the mirror tube. Furthermore, the end face of the protective cover 50 at one end close to the surgical site is constructed as a bevel and is inclined with respect to the extension direction of the mirror tube to facilitate insertion into the human body.

[0060] In some embodiments, the protective cover 50 and the mounting base 40 are connected by bonding, and may also be connected by welding, snap-fitting, threaded fixing, etc.

[0061] According to some embodiments of the present application, a perfusion channel 12 is formed in the scope tube for delivering a medium to the surgical area. In this embodiment, the scope tube is provided with a perfusion channel 12 to allow the surgical medium to be introduced into the surgical environment, thereby achieving a smooth operation. Figure 3 As shown, the sleeve 10 is formed with the perfusion channel 12, the mounting seat 40 is formed with a first perfusion hole 44 opposite to and connected to the perfusion channel 12, and the protective cover 50 is formed with a second perfusion hole 52 opposite to and connected to the perfusion channel 12. The first perfusion hole 44 and the second perfusion hole 52 are directly opposite to and connected to the perfusion channel 12 in the extension direction of the mirror tube.

[0062] In some embodiments, the perfusion channel 12 is arranged on the periphery of the instrument channel 11. Further, the perfusion channel 12 is constructed in multiple configurations, and the multiple perfusion channels 12 are arranged at intervals along the circumferential direction. Furthermore, the perfusion channel 12 and the line channel 13 are arranged at intervals along the circumferential direction, and the walls of adjacent perfusion channels 12 are connected to the walls of the line channel 13. The above design can make the structure inside the mirror tube compact and improve the space utilization rate of the mirror tube structure. The number of perfusion channels 12 and line channels 13 is selected and designed according to the structural size of the mirror tube and the surgical needs.

[0063] It should be noted that, according to the endoscope of the present application, when obtaining a de-obstructed endoscopic image, it is necessary to use the surgical area image captured by any image acquisition component as the target image, and fill it with the surgical area images of other image acquisition components. Accordingly, the multiple image acquisition components include a main acquisition component and multiple auxiliary acquisition components. In some embodiments, since the axial end face of the mirror tube extending into the human body is inclined with respect to the extension direction of the mirror tube, the relative distances between the multiple image acquisition components and the surgical site are different. In practice, the image acquisition component farthest from the surgical site can be constructed as the main acquisition component to capture the surgical site to the greatest extent.

[0064] In some embodiments, the image acquisition component 20 includes a first image acquisition component 201 and a second image acquisition component 202. The first image acquisition component 201 is a main acquisition component, and the second image acquisition component 202 is an auxiliary acquisition component. In some embodiments, the first image acquisition component 201 is configured as a main camera, and the second image acquisition component 202 is configured as a slave camera.

[0065] Further, the number of the first image acquisition component 201 is 1; the number of the second image acquisition components 202 is 1-5.

[0066] In some embodiments, Figure 1-Figure 5 As shown, the second image acquisition components 202 are structured as three, and the three second image acquisition components 202 are arranged at a side of the mirror tube shaft end close to the surgical site at an interval of 60° with respect to the center of the mirror tube, and the first image acquisition component 201 is arranged at a side of the mirror tube shaft end away from the surgical site. Figure 2 As shown, there are six light source elements 30, which are arranged at intervals of 60° along the outer circumference of the first through hole 43. Figure 2 As shown, there are two perfusion channels 12 , which are arranged on both sides of the first image acquisition component 201 at an interval of 60° with respect to the center of the mirror tube.

[0067] The present application also proposes an endoscopic image de-blocking method, using the above endoscope, the image acquisition component includes a first image acquisition component and a plurality of second image acquisition components, and the endoscopic image de-blocking method includes the following steps:

[0068] S1. Perform internal and external calibration on each image acquisition component;

[0069] S2, acquiring a plurality of surgical area images, wherein the surgical area image acquired by the first image acquisition component is the target image;

[0070] S3, judging whether there is occlusion in the target image; if there is occlusion in the target image, dividing the target image into an occluded part and a retained part, and extracting the area position and shape mask of the occluded part;

[0071] S4, performing affine transformation on the surgical area images acquired by each second image acquisition component to obtain a transformed image of the surgical area images acquired by each second image acquisition component in the coordinate system of the first image acquisition component;

[0072] S5. Cut each transformed image according to the area position and shape mask of the blocked part, and fill the retained part with the cut part to obtain a de-blocked endoscopic image.

[0073] According to the endoscope image de-occlusion method of the present application, Figure 6 As shown, in step S1, when performing internal parameter calibration on each image acquisition component, Zhang's calibration method can be used to calibrate and obtain the internal parameter matrix. Through the internal parameter calibration, the distortion and error of the image acquisition component in the imaging process can be corrected, and the imaging accuracy and the accuracy of subsequent image processing can be improved; when performing external parameter calibration on each image acquisition component, Zhang's calibration method can be used to calibrate, and each second image acquisition component is calibrated relative to the first image acquisition component to obtain the spatial transformation matrix of each second image acquisition component transformed to the pixel space of the first image acquisition component, that is, the external parameter matrix; through the external parameter calibration, it can be ensured that the image information captured by each image acquisition component is consistent in space and time, and multi-camera collaborative work can be realized. Among them, the methods of internal parameter calibration and external parameter calibration are not limited to Zhang's calibration method, and other calibration methods can also be used.

[0074] In step S2, each image acquisition component needs to acquire the surgical area image in parallel and simultaneously to ensure the consistency of imaging time; in the process of acquiring the surgical area image, the internal parameter matrix of each image acquisition component calibrated in step S1 is used to perform distortion correction on each acquired surgical area image accordingly.

[0075] Further, step S2 includes: performing image enhancement on the surgical area image to compensate for the problem of poor quality of the acquired image. Specifically, an image enhancement model based on deep learning can be used to enhance the acquired image. In addition, in some embodiments, bicubic interpolation, image denoising and other methods can also be used to enhance the acquired image. In some embodiments, the image acquisition component uses a miniature camera with limited imaging quality, and this embodiment can especially improve its image quality.

[0076] The occlusion described in step S3 mainly refers to the occlusion formed by the surgical instrument within the field of view of the image acquisition unit, but is not limited thereto, and also includes the occlusion formed by some human blood tissues, etc. Surgical instruments include but are not limited to common surgical instruments such as clamps, probes, reamers, and electrocautery knives. In step S3, a deep learning-based image segmentation model can be used to extract the position and shape of the instrument in the target image. In some embodiments, a graphics-based method or other common image segmentation methods can also be used to extract the occlusion position and shape in the target image.

[0077] Step S4 specifically includes: using the intrinsic parameter matrix and the extrinsic parameter matrix calibrated in step S1, transforming the surgical area images of each second image acquisition component from their respective pixel spaces to the pixel space of the first image acquisition component.

[0078] Step S5 specifically includes: using the extracted area position and shape mask of the occluded part to crop the transformed image, and filling the cropped part in the original occluded part of the target image to obtain a de-occluded endoscopic image.

[0079] According to the endoscopic image deocclusion method of the present application, under the premise of intelligently identifying the boundaries of occlusions such as instruments in the captured image, calibration and affine transformation and other methods are used to fill in the area occluded by the instrument in the image captured by the first image capture component using the image of the second image capture component, thereby achieving removal of occlusions such as instruments in the endoscopic image. This can solve the problem of instrument occlusion in the endoscopic image, improve the convenience and safety of surgery, and improve the quality of surgery.

[0080] It should be noted that when the transformed images of the second image acquisition components are spliced ​​to the target image of the first image acquisition component, there may be an obvious splicing boundary phenomenon, which will affect the observation of the surgeon.

[0081] In response to the above-mentioned problem, according to some embodiments of the present application, the endoscopic image de-occlusion method further includes the following steps: determining whether the de-occluded endoscopic image has a stitching boundary; if the de-occluded endoscopic image has a stitching boundary, eliminating the stitching boundary based on the cropped part and the retained part.

[0082] Specifically, the padded target image can be processed based on a deep learning model to eliminate the stitching boundaries at the padded location; or the padded target image can be processed based on a Gaussian blur algorithm to eliminate the stitching boundaries at the padded location; or the padded target image can be processed using a neighborhood weighted method to eliminate the stitching boundaries at the padded location.

[0083] In this embodiment, the clarity and accuracy of the endoscopic image can be improved by eliminating the stitching boundary, so as to facilitate observation and reference during surgery.

[0084] It should also be noted that after the transformed images of the second image acquisition components are spliced ​​to the target image of the first image acquisition component, there may still be vacant areas, which will also affect the observation of the surgeon.

[0085] In response to the above-mentioned technical problems, according to some embodiments of the present application, the endoscopic image de-occlusion method also includes the following steps: determining whether there are any missing areas in the de-occluded endoscopic image; if there are any missing areas in the de-occluded endoscopic image, filling the missing areas based on the cropped part and the retained part.

[0086] Specifically, the missing parts can be filled by extracting the depth information of the image based on the deep learning model, or the missing parts can be filled by using the neighborhood information. In this embodiment, a complete endoscopic image can be obtained by filling.

[0087] It should also be noted that since the second image acquisition components are constructed in multiple forms, there may be overlapping parts when the transformed images of the multiple second image acquisition components are spliced. The overlapping parts are not exactly the same, which makes the endoscopic image lack clarity and is not conducive to observation.

[0088] In this regard, according to some embodiments of the present application, the endoscopic image de-occlusion method further includes the following steps: determining whether there is overlap in the cropped parts of multiple transformed images; if there is overlap in the cropped parts of multiple transformed images, selecting one of the multiple transformed images to fill the retained part with the overlapping part according to the relative position relationship between each second image acquisition component and the first image acquisition component; or weighted summing the overlapping parts of the multiple transformed images to fill the retained part. This embodiment can further improve the clarity and reliability of the endoscopic image.

[0089] Specifically, when selecting the transformation image, a confidence function can be set according to the relative position relationship between each of the second image acquisition components and the first image acquisition component, and the transformation image corresponding to the second image acquisition component with the highest confidence is selected to fill the overlapping part of the target image.

[0090] The present application also proposes an endoscope system, such as Figure 7 As shown, the endoscope system includes the above-mentioned endoscope, and also includes an image processing device and an image display device. The endoscope is used to collect images of the surgical area in multiple directions; the image processing device is used to process the surgical area images to generate unobstructed endoscopic images based on the surgical area images in multiple directions; the image display device is used to display the unobstructed endoscopic images. Among them, the endoscope includes multiple image acquisition components, and specifically, the endoscope can be configured as a multi-camera endoscope.

[0091] According to some embodiments of the present application, Figure 7 As shown, the image processing device includes a processor, a memory, a communication module and an output module. The memory stores an endoscopic image de-obstruction program, and the memory is used to store the surgical area image collected by the endoscope and the processing result of the processor; the processor is used to execute the endoscopic image de-obstruction program to obtain a de-obstructed endoscopic image, and the above-mentioned endoscopic image de-obstruction method is executed when the endoscopic image de-obstruction program is executed; the output module is used to output the de-obstructed endoscopic image to the image display device; the communication module is used to communicate with an external device or server, and the communication module includes a Wi-Fi module, a mobile communication module, and a Bluetooth module.

[0092] The processing result of the processor includes the calibration matrix, the area position and shape mask of the blocked part, the retained part, each transformed image and its cropped part, etc. It should be noted that the surgical area image collected by the endoscope in the memory and the processing result of the processor are updated in real time.

[0093] The present application also proposes an endoscope control system, which controls and implements the above-mentioned endoscope image de-obstruction method. Figure 8 As shown, the control system includes an image acquisition module, an image processing module and an image display module. The image acquisition module is used to acquire images of the surgical area in multiple directions; the image processing module is used to process the surgical area images acquired by the image acquisition module and obtain a de-occluded endoscopic image; the image display module is used to output and display the de-occluded endoscopic image obtained by the image processing module. Among them, the image acquisition module includes a main acquisition module and an auxiliary acquisition module.

[0094] According to some embodiments of the present application, the image processing module includes an instrument segmentation and extraction module, an affine transformation module and an image filling module; the instrument segmentation and extraction module is used to segment and extract the occluded area in the surgical area image acquired by the main acquisition module; the affine transformation module is used to transform the surgical area image acquired by the auxiliary acquisition module to the pixel space of the main acquisition module using an external parameter matrix; the image filling module is used to crop the transformed surgical area image of the auxiliary acquisition module and fill the image of the main acquisition module.

[0095] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0096] In the description of the present invention, "first feature" or "second feature" may include one or more of the features.

[0097] In the description of the present invention, "plurality" means two or more.

[0098] In the description of the present invention, a first feature being “on” or “under” a second feature may include that the first and second features are directly in contact with each other, or may include that the first and second features are not in direct contact with each other but are in contact with each other via another feature therebetween.

[0099] In the description of the present invention, “on”, “over” and “above” a first feature from a second feature includes the first feature being directly above and obliquely above the second feature, or simply means that the first feature is horizontally higher than the second feature.

[0100] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0101] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An endoscope, characterized in that: include: A scope tube, wherein an instrument channel is formed inside the scope tube, at least a portion of the scope tube is suitable for extending into the human body to approach the surgical area, and the instrument channel is suitable for surgical instruments to extend into the surgical area; A light source component, which is arranged on the mirror tube and has a light-emitting portion formed at one end of the mirror tube extending into the human body, so as to provide illumination for the surgical area; The image acquisition component is structured in multiple pieces, and the multiple image acquisition components are arranged at intervals at one end of the mirror tube extending into the human body to be suitable for respectively acquiring images of the surgical area in different directions.

2. The endoscope according to claim 1, characterized in that The axial end surface of one end of the mirror tube extending into the human body is arranged obliquely with respect to the extending direction of the mirror tube.

3. The endoscope according to claim 2, characterized in that The mirror tube comprises: a sleeve, wherein the instrument channel is formed in the sleeve; A mounting seat is provided at one end of the sleeve, and the mounting seat is formed with a first through hole which is directly opposite to and communicates with the instrument channel; The mounting seat is formed with a first mounting hole and a second mounting hole, the first mounting hole is configured as a plurality of holes corresponding one to one with the image acquisition component; the image acquisition component is arranged in the first mounting hole, and at least a portion of the light source component is arranged in the second mounting hole; a circuit channel connected to the first mounting hole and / or the second mounting hole is formed in the sleeve.

4. The endoscope according to claim 1, characterized in that The light source comprises: An optical fiber and an optical fiber light source, wherein the light emitting portion is formed at one end of the optical fiber, and the other end of the optical fiber is connected to the optical fiber light source, and the optical fiber light source is arranged at the end of the mirror tube away from the human body.

5. The endoscope according to claim 1, characterized in that Also includes: A protective cover, the protective cover is arranged at the end of the mirror tube extending into the human body, and the protective cover is formed with a second through hole facing and communicating with the instrument channel; A closed cavity is formed between the protective cover and the mirror tube, and the image acquisition component and the light emitting part are arranged in the closed cavity.

6. A method for removing occlusion from an endoscope image, characterized in that: Applied to the endoscope according to any one of claims 1 to 5, the image acquisition component includes a first image acquisition component and a plurality of second image acquisition components, and the endoscope image de-occlusion method includes the following steps: Performing internal and external calibration on each of the image acquisition components; Acquire a plurality of surgical area images, wherein the surgical area image acquired by the first image acquisition component is a target image; Determine whether there is occlusion in the target image; If there is occlusion in the target image, the target image is divided into an occluded part and a retained part, and the regional position and shape mask of the occluded part are extracted; Performing affine transformation on each surgical area image acquired by the second image acquisition component to obtain a transformed image of each surgical area image acquired by the second image acquisition component in the coordinate system of the first image acquisition component; Each of the transformed images is cropped according to the area position and shape mask of the blocked part, and the cropped part is used to fill the retained part to obtain a de-blocked endoscopic image.

7. The method for removing occlusion from an endoscope image according to claim 6, characterized in that: The following steps are also included: Determining whether the de-occluded endoscopic image has a splicing boundary; If a stitching boundary exists in the de-occluded endoscopic image, the stitching boundary is eliminated based on the cropped portion and the retained portion.

8. The method for removing occlusion from an endoscope image according to claim 6, characterized in that: The following steps are also included: Determining whether there is a vacant area in the de-occluded endoscopic image; If there is a missing area in the de-occluded endoscopic image, the missing area is filled based on the cropped part and the retained part.

9. The endoscope image de-occlusion method according to claim 6, characterized in that: The following steps are also included: determining whether there is overlap between the cropped portions of the plurality of transformed images; If the cropped parts of the plurality of transformed images overlap, one of the plurality of transformed images is selected according to the relative positional relationship between each of the second image acquisition components and the first image acquisition component to fill the retained part with the overlapping part; Or, weighted sum is performed on the overlapping parts of the plurality of transformed images to fill the reserved part.

10. An endoscope system, characterized in that: include: An endoscope, the endoscope being used to collect images of the surgical area in multiple directions, the endoscope being constructed as described in any one of claims 1 to 5; An image processing device, the image processing device is used to generate an unobstructed endoscopic image based on the surgical area images in the multiple directions; An image display device is used to display the unobstructed endoscopic image.

Images