Method and device for removing instrument shielding of endoscope image

Abstract

The invention relates to the technical field of medical image processing, in particular to an instrument shielding removing method and device for an endoscope image, and the method comprises the steps: collecting the endoscope image of an endoscope, judging whether a current frame contains a surgical operation instrument or not, and if yes, segmenting an instrument region, and extracting a position and shape mask of the instrument; then, an optical flow field between the current frame and the earlier frame is estimated to fill the motion relationship between the two frames. And finally, on the basis of the extracted instrument position and shape mask and the filled optical flow field, motion transformation is carried out on the early-stage frame, so that the area, which is indeed due to instrument shielding, in the endoscope image of the current frame is filled. Therefore, the problem of serious instrument shielding in the endoscope image in a narrow approach in the prior art is solved.

Description

Technical Field

[0001] This application relates to the technical field of medical image processing, and particularly to a method and device for removing instrument occlusion from endoscopic images. Background Art

[0002] In related technologies, for the method of endoscopic occlusion removal and display, based on detecting whether there is an occluder, a near-infrared light source is used for penetrating irradiation to obtain a near-infrared light image, and algorithms such as defogging and noise reduction are used to obtain a near-infrared light image with clear edges to improve image clarity.

[0003] However, in related technologies, on the one hand, the introduction of the near-infrared light source greatly increases the system integration difficulty under the constraints of the narrow access in minimally invasive surgery. On the other hand, there are significant differences in the hue and quality between the near-infrared light image and the imaging result of the human eye. Directly displaying it to the physician may lead to misjudgment of the surgical area by the physician, and there is an urgent need for improvement. Summary of the Invention

[0004] This application provides a method and device for removing instrument occlusion from endoscopic images to solve the problem of serious instrument occlusion in endoscopic images under narrow access in related technologies.

[0005] In a first aspect embodiment of this application, a method for removing instrument occlusion from endoscopic images is provided, including the following steps: collecting the current endoscopic image of the endoscope; determining whether the current endoscopic image contains surgical operation instruments; if it contains the surgical operation instruments, segmenting the instrument area from the current endoscopic image, and based on the instrument area, extracting the instrument position and shape mask of the surgical operation instrument; based on the instrument position and the shape mask, obtaining the motion relationship between the objects in the current endoscopic image and the objects in the reference endoscopic image according to the optical flow field between the current endoscopic image and the reference endoscopic image; based on the filled optical flow field, performing motion transformation on the reference endoscopic image to fill the instrument occlusion mask area corresponding to the instrument position and the shape mask to obtain an unoccluded endoscopic image.

[0006] Through the above technical solutions, the embodiments of this application can intelligently determine whether there are surgical operation instruments by collecting the current endoscopic image, efficiently segment the instrument area and extract its position and shape mask, and combine the estimation and filling of the optical flow field to ensure the effective filling of the instrument occlusion area in the current frame endoscopic image, thereby significantly improving the visualization effect and safety of endoscopic surgery, reducing the operation risk caused by instrument occlusion, and further improving the surgical quality and efficiency.

[0007] Optionally, in an embodiment of the present application, after filling the instrument occlusion mask area corresponding to the instrument position and the shape mask, it further includes: determining whether the filled endoscopic image completely removes the occlusion; if the filled endoscopic image does not completely remove the occlusion, then filling the remaining occluded part based on the current frame image information of the filled endoscopic image to obtain the unoccluded endoscopic image.

[0008] Through the above technical solution, the embodiment of the present application can ensure that the doctor obtains a clear and accurate view during the operation by determining whether the filled endoscopic image completely removes the occlusion, thereby improving the safety and efficiency of the operation. If it is found that there is still occlusion, the system can further fill the remaining occluded part based on the current frame image information to ensure that the finally presented endoscopic image is complete.

[0009] Optionally, in an embodiment of the present application, filling the remaining occluded part based on the current frame image information of the filled endoscopic image includes: extracting at least one image feature of the current frame image information; filling the remaining occluded part according to the at least one image feature.

[0010] Through the above technical solution, the embodiment of the present application can effectively fill the remaining occluded part by extracting at least one image feature in the current frame image, thereby realizing the integrity and clarity of the endoscopic image. This method not only improves the accuracy of image processing, but also can quickly and intelligently restore the occluded image area without relying on external light sources or complex algorithms, enhancing the safety of minimally invasive surgery and the convenience of the doctor's operation.

[0011] Optionally, in an embodiment of the present application, the reference endoscopic image is the previous frame endoscopic image, wherein obtaining the motion relationship between the objects in the current endoscopic image based on the optical flow field between the current endoscopic image and the reference endoscopic image according to the instrument position and the shape mask includes: performing optical flow estimation on the current endoscopic image and the previous frame endoscopic image to obtain the optical flow field under instrument occlusion; based on the instrument position and the shape mask, perfecting the optical flow field under instrument occlusion to restore the optical flow field information of the area after instrument occlusion; obtaining the motion relationship between the objects in the current endoscopic image based on the optical flow field information of the area after instrument occlusion.

[0012] Through the above technical solution, the embodiment of the present application can effectively restore the image information lost due to instrument occlusion by accurately estimating and perfecting the optical flow field between the current frame and the previous frame endoscopic images, thereby improving the clarity and integrity of the endoscopic image.

[0013] Optionally, in an embodiment of the present application, the motion transformation of the reference endoscopic image based on the filled optical flow field includes: iteratively using the optical flow field information between adjacent frame images to fill multiple previous-frame endoscopic images into the instrument position after motion transformation; replacing the corresponding position of the current endoscopic image with the area of the transformed previous-frame endoscopic image that does not contain the instrument, and for each pixel, selecting a previous frame that meets the preset adjacent condition.

[0014] Through the above technical solution, the embodiment of the present application can perform motion transformation on the previous-frame endoscopic image based on the instrument position and shape mask of the surgical operation instrument and the filled optical flow field, and can effectively iteratively use the optical flow field information between adjacent frame images, so as to accurately fill the instrument occlusion area in the current-frame endoscopic image, thereby improving the clarity and integrity of the endoscopic image.

[0015] An embodiment of the second aspect of the present application provides an endoscopic image de-instrument occlusion device, including: an acquisition module for acquiring the current endoscopic image of the endoscope; a first judgment module for judging whether the current endoscopic image contains a surgical operation instrument; a segmentation module for, if the surgical operation instrument is included, segmenting the instrument area from the current endoscopic image and extracting the instrument position and shape mask of the surgical operation instrument based on the instrument area; an optical flow estimation module for obtaining the motion relationship between the objects in the current endoscopic image and the objects in the reference endoscopic image according to the optical flow field between the current endoscopic image and the reference endoscopic image based on the instrument position and the shape mask; a first filling module for performing motion transformation on the reference endoscopic image based on the filled optical flow field to fill the instrument occlusion mask area corresponding to the instrument position and the shape mask to obtain an unoccluded endoscopic image.

[0016] Through the above technical solution, the embodiment of the present application can collect the current endoscopic image and intelligently judge whether there is a surgical operation instrument, efficiently segment the instrument area and extract its position and shape mask, and combine the estimation and filling of the optical flow field to ensure the effective filling of the instrument occlusion area in the current-frame endoscopic image, thereby significantly improving the visualization effect and safety of the endoscopic surgery, reducing the operation risk caused by instrument occlusion, and further improving the surgical quality and efficiency.

[0017] Optionally, in an embodiment of the present application, it further includes: a second judgment module for judging whether the filled endoscopic image completely removes the occlusion; a second filling module for, if the filled endoscopic image does not completely remove the occlusion, filling the remaining occlusion part based on the current frame image information of the filled endoscopic image to obtain the unoccluded endoscopic image.

[0018] Through the above technical solution, the embodiments of the present application can ensure that doctors obtain a clear and accurate field of view during the operation by determining whether the occlusions in the endoscope image after filling are completely removed, thereby improving the safety and efficiency of the operation. If it is found that there are still occlusions, the system can further fill the remaining occluded parts based on the current frame image information to ensure that the finally presented endoscope image is complete without defects.

[0019] Optionally, in an embodiment of the present application, the second filling module includes: an extraction unit for extracting at least one image feature of the current frame image information; a filling unit for filling the remaining occluded part according to the at least one image feature.

[0020] Through the above technical solution, the embodiments of the present application can effectively fill the remaining occluded parts by extracting at least one image feature in the current frame image, thereby realizing the integrity and clarity of the endoscope image. This method not only improves the accuracy of image processing, but also can quickly and intelligently restore the occluded image area without relying on external light sources or complex algorithms, improving the safety of minimally invasive surgery and the convenience of doctors' operations.

[0021] Optionally, in an embodiment of the present application, the optical flow estimation module includes: an estimation unit for performing optical flow estimation on the current endoscope image and the previous frame endoscope image to obtain an optical flow field under instrument occlusion; a refinement unit for refining the optical flow field under instrument occlusion based on the instrument position and the shape mask to restore the optical flow field information of the area after instrument occlusion; an acquisition unit for obtaining the motion relationship between the objects in the current endoscope image based on the optical flow field information of the area after instrument occlusion.

[0022] Through the above technical solution, the embodiments of the present application can accurately estimate and refine the optical flow field between the current frame and the previous frame endoscope images, effectively restoring the image information lost due to instrument occlusion, thereby improving the clarity and integrity of the endoscope image.

[0023] Optionally, in an embodiment of the present application, the first filling module includes: an iteration unit for iteratively using the optical flow field information between adjacent frame images to fill multiple previous frame endoscope images into the instrument position after motion transformation; a replacement unit for replacing the corresponding position of the current endoscope image with the area of the transformed previous frame endoscope image that does not contain the instrument, and selecting a previous frame that satisfies a preset adjacent condition for each pixel.

[0024] Through the above technical solutions, the embodiments of the present application can perform motion transformation on the previous-frame endoscope image based on the instrument position and shape mask of the surgical operating instrument and the filled optical flow field, effectively iteratively utilize the optical flow field information between adjacent-frame images, so as to accurately fill the instrument occlusion area in the current-frame endoscope image, thereby improving the clarity and integrity of the endoscope image.

[0025] The third aspect of the embodiments of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the method for removing instrument occlusion from an endoscope image as described in the above embodiments.

[0026] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the program is executed by a processor, the method for removing instrument occlusion from an endoscope image as described above is implemented.

[0027] The fifth aspect of the embodiments of the present application provides a computer program product, including a computer program, and when the computer program is executed, it is used to implement the method for removing instrument occlusion from an endoscope image as described above.

[0028] The embodiments of the present application can effectively solve the problem of instrument occlusion by obtaining an endoscope image, segmenting the surgical operating instrument area and extracting its position and shape mask, and combining the estimation and filling of the optical flow field. It not only significantly improves the visualization effect and safety of endoscopic surgery, reduces the operation risk caused by instrument occlusion, but also improves the surgical quality and efficiency. In addition, it can determine in real time whether the occlusion in the filled endoscope image is completely removed, ensuring that doctors obtain a clear and accurate view during the operation. If there is still occlusion, further filling can be performed based on the current-frame image information to ensure that the finally presented endoscope image is complete. By extracting the image features in the current-frame image, the remaining occlusion part can be effectively filled, thereby realizing the integrity and clarity of the endoscope image. This method not only improves the accuracy of image processing, but also can quickly and intelligently restore the occluded image area without relying on external light sources or complex algorithms, improving the safety of minimally invasive surgery and the operation convenience of doctors. At the same time, by accurately estimating and improving the optical flow field between the current-frame and previous-frame endoscope images, the image information lost due to instrument occlusion is effectively restored, further improving the clarity and integrity of the endoscope image and ensuring the smooth progress of the operation.

[0029] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Description of the Drawings

[0030] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:

[0031] Figure 1 FIG. is a flowchart of a method for removing instrument occlusion from an endoscopic image provided according to an embodiment of the present application;

[0032] Figure 2 FIG. is a schematic structural diagram of a device for removing instrument occlusion from an endoscopic image provided according to an embodiment of the present application;

[0033] Figure 3 FIG. is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. Detailed Embodiments

[0034] The method and device for removing instrument occlusion from an endoscopic image according to embodiments of the present application will be described below with reference to the accompanying drawings. In view of the problem of severe instrument occlusion in endoscopic images in a narrow access in the related art mentioned in the above background art, the present application provides a method for removing instrument occlusion from an endoscopic image. In this method, by collecting endoscopic images and intelligently determining whether there are surgical operating instruments, the instrument area can be efficiently segmented and its position and shape mask can be extracted. Combining the estimation and filling of the optical flow field ensures the effective filling of the instrument occlusion area in the current frame of the endoscopic image, thereby significantly improving the visualization effect and safety of endoscopic surgery, reducing the operation risks caused by instrument occlusion, and further improving the surgical quality and efficiency. Thus, the problem of severe instrument occlusion in endoscopic images in a narrow access in the related art is solved.

[0035] It should be noted that the system involved in this embodiment may include a medical endoscope and an image processing system. Among them, the image processing system may be integrated in the medical endoscope or may be a separately provided image processing terminal device or server.

[0036] Specifically, Figure 1 FIG. is a schematic flowchart of a method for removing instrument occlusion from an endoscopic image provided according to an embodiment of the present application.

[0037] As Figure 1 shown, the method for removing instrument occlusion from the endoscopic image includes the following steps:

[0038] In step S101, the current endoscopic image of the endoscope is collected.

[0039] The endoscope camera should have good optical performance and fast image transmission ability to ensure clear capture of image information under different complex conditions. Preferably, while acquiring the image, the system can have the ability to process real-time images, that is, it can perform preliminary processing on the captured images, including but not limited to denoising, enhancing contrast, etc., to improve the visibility and clarity of the images.

[0040] The embodiments of this application can provide accurate and reliable visual information by acquiring endoscope images, providing a data basis for subsequent removal of instrument occlusion.

[0041] In step S102, it is judged whether the current endoscope image contains surgical operation instruments.

[0042] The judgment process can be based on the image processing system integrated in the medical endoscope. The system inputs the acquired endoscope image into the image processing module and analyzes the current frame image using image recognition technology. This analysis process includes but is not limited to extracting the features of each pixel point in the image and identifying the features of possible surgical operation instruments, such as shape, edge, color, and texture, etc.

[0043] The embodiments of this application can effectively analyze and process endoscope images by intelligently identifying the presence of surgical instruments, avoiding image blurring and information loss caused by instrument occlusion.

[0044] In step S103, if it contains surgical operation instruments, the instrument area is segmented from the current endoscope image, and based on the instrument area, the instrument position and shape mask of the surgical operation instrument are extracted.

[0045] Specifically, the method of extracting the instrument position and shape in the endoscope image can adopt a graphics-based method or other common image segmentation methods, usually involving traditional image processing techniques such as edge detection, region growing, and threshold segmentation of images. Preferably, an image segmentation model based on deep learning can be used to extract the instrument position and shape in the endoscope image. The deep learning model can automatically learn the complex features in the image by training a large amount of labeled data. The extracted instrument position and shape are stored in the storage device in the system for subsequent image filling.

[0046] The embodiments of this application can adopt an image segmentation model based on deep learning to automatically learn and identify complex features, improve the accuracy and efficiency of segmentation. The extracted instrument position and shape mask are stored for subsequent image filling, thus providing a reliable data basis for subsequent image processing.

[0047] In step S104, based on the instrument position and shape mask, the motion relationship between the objects in the current endoscopic image and the objects in the reference endoscopic image is obtained according to the optical flow field between the current endoscopic image and the reference endoscopic image.

[0048] Optionally, in an embodiment of the present application, the reference endoscopic image is the previous frame endoscopic image. Among them, based on the instrument position and shape mask, obtaining the motion relationship between the objects in the current endoscopic image according to the optical flow field between the current endoscopic image and the reference endoscopic image includes: performing optical flow estimation on the current endoscopic image and the previous frame endoscopic image to obtain the optical flow field under the instrument occlusion; based on the instrument position and shape mask, improving the optical flow field under the instrument occlusion to restore the optical flow field information of the area after the instrument occlusion; obtaining the motion relationship between the objects in the current endoscopic image based on the optical flow field information of the area after the instrument occlusion.

[0049] Specifically, the optical flow estimation method is used to process the current frame image and the previous frame image to calculate the optical flow field under the instrument occlusion. This process can effectively capture the motion information of the objects in the image, and thus provide basic data for subsequent image filling.

[0050] Preferably, an optical flow estimation model based on deep learning is used to estimate the optical flow field. This method can improve the accuracy and efficiency of the estimation. In some embodiments, the method based on feature point estimation or other common optical flow field estimation methods can also be used to adapt to different application scenarios and requirements.

[0051] Furthermore, based on the instrument position and shape mask segmented in step S103, the estimated optical flow field is improved to restore the optical flow field information in the instrument occlusion area, so as to ensure that the motion state of the object can be more accurately reflected in the subsequent image filling process.

[0052] Finally, the improved optical flow field information is stored in the storage device of the system for subsequent filling use. Optionally, in order to make up for the incomplete optical flow estimation in the instrument occlusion area, a deep learning model can be selected to fill the inter-frame optical flow field.

[0053] The embodiment of the present application can accurately estimate and improve the optical flow field between the current frame endoscopic image and the previous frame image, effectively capturing the motion information of the objects in the image. It not only improves the accuracy and efficiency of image filling, but also ensures the accurate restoration of the optical flow field information in the instrument occlusion area, thereby improving the clarity and usability of the endoscopic image. In addition, the use of an optical flow estimation model based on deep learning makes the estimation result more reliable.

[0054] In step S105, based on the filled optical flow field, perform a motion transformation on the reference endoscopic image to fill the instrument occlusion mask area corresponding to the instrument position and shape mask, so as to obtain an unoccluded endoscopic image.

[0055] In the actual execution process, performing a motion transformation on the reference endoscopic image based on the filled optical flow field includes: iteratively using the optical flow field information between adjacent frame images to fill multiple previous frame endoscopic images into the instrument position after motion transformation; taking the area in the transformed previous frame endoscopic image that does not contain the instrument to replace the corresponding position of the current endoscopic image, and for each pixel, selecting a previous frame that meets the preset adjacent condition.

[0056] Specifically, using the stored instrument position and shape mask and the inter-frame optical flow field, perform iterative processing on multiple previous frame images, perform a motion transformation based on the optical flow field on the previous frames, and obtain the result that the area in the previous frame image that is not occluded by the instrument moves to the area occluded by the instrument in the current frame image, so as to fill the instrument occlusion area in the current frame image. At the same time, for each pixel, select the nearest previous frame as much as possible.

[0057] Preferably, the number of previous frame images used is 20. This number is selected to achieve a balance between real-time performance and authenticity. In other embodiments, this number can be adjusted according to specific circumstances, and the range can be any value between 1 and 100.

[0058] After filling the instrument occlusion mask area corresponding to the instrument position and shape mask, it further includes: determining whether the filled endoscopic image completely removes the occlusion; if the filled endoscopic image does not completely remove the occlusion, then fill the remaining occluded part based on the current frame image information of the filled endoscopic image to obtain an unoccluded endoscopic image.

[0059] Preferably, it is determined whether the instrument is completely removed by determining whether the stored instrument shape and position mask contains the instrument. Or, in some embodiments, different methods such as performing instrument segmentation again can also be used to implement the determination of whether the instrument is completely removed.

[0060] After determining whether the current frame image contains a surgical instrument, if the endoscopic image does not contain instrument occlusion, directly display the endoscopic image on the display device of the system; if the endoscopic image still contains instrument occlusion, estimate the optical flow field between the current frame image and the previous frame image, and save the result in the storage device to provide a basis for filling subsequent instrument-containing frame images.

[0061] Optionally, in an embodiment of the present application, filling the remaining occluded part based on the current frame image information of the filled endoscopic image includes: extracting at least one image feature of the current frame image information; filling the remaining occluded part according to the at least one image feature.

[0062] Preferably, the filling of the remaining occluded part adopts a single-image filling model based on deep learning, and uses the depth information of the image extracted by the model to fill the remaining occluded area. Or, in some embodiments, a method based on neighborhood information or the like can also be used to complete the filling.

[0063] The embodiment of the present application can effectively and accurately fill the occluded area of the surgical instrument in the current frame endoscopic image by iteratively using the optical flow field information of the previous frame endoscopic image, thereby improving the clarity and visibility of the image. This method not only maintains the structural integrity of the surgical instrument, but also achieves a good balance between real-time performance and authenticity, ensuring that doctors obtain an accurate view during minimally invasive surgery, and thus improving the safety and success rate of the surgery. In addition, the deep learning model and image feature extraction technology further enhance the filling effect of the remaining occluded part, significantly improving the integrity and usability of the endoscopic image.

[0064] According to the method for removing instrument occlusion from an endoscopic image proposed by the embodiment of the present application, by collecting the endoscopic image and intelligently judging whether there is a surgical instrument, efficiently segmenting the instrument area and extracting its position and shape mask, and combining the estimation and filling of the optical flow field, it ensures the effective filling of the instrument occlusion area in the current frame endoscopic image, thereby significantly improving the visualization effect and safety of endoscopic surgery, reducing the operation risk caused by instrument occlusion, and thus improving the surgical quality and efficiency.

[0065] Next, a device for removing instrument occlusion from an endoscopic image according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0066] Figure 2 It is a block diagram of a device for removing instrument occlusion from an endoscopic image according to an embodiment of the present application.

[0067] As Figure 2 shown, the device 10 for removing instrument occlusion from an endoscopic image includes: an acquisition module 100, a first judgment module 200, a segmentation module 300, an optical flow estimation module 400, and a first filling module 500.

[0068] Specifically, the acquisition module 100 is configured to acquire the current endoscopic image of the endoscope;

[0069] The first judgment module 200 is configured to judge whether the current endoscopic image contains a surgical instrument.

[0070] The segmentation module 300 is configured to, if a surgical instrument is included, segment the instrument area from the current endoscopic image, and based on the instrument area, extract the instrument position and shape mask of the surgical instrument.

[0071] The optical flow estimation module 400 is configured to obtain the motion relationship between the objects in the current endoscopic image and the objects in the reference endoscopic image according to the optical flow field between the current endoscopic image and the reference endoscopic image based on the instrument position and shape mask.

[0072] The first filling module 500 is configured to perform a motion transformation on the reference endoscopic image based on the filled optical flow field to fill the instrument occlusion mask area corresponding to the instrument position and shape mask, so as to obtain an unoccluded endoscopic image.

[0073] Optionally, in an embodiment of the present application, it further includes: a second judgment module 600 and a second filling module 700.

[0074] Specifically, the second judgment module 600 is configured to judge whether the occlusion in the filled endoscopic image is completely removed.

[0075] The second filling module 700 is configured to, if the occlusion in the filled endoscopic image is not completely removed, fill the remaining occluded part based on the current frame image information of the filled endoscopic image to obtain an unoccluded endoscopic image.

[0076] Optionally, in an embodiment of the present application, the second filling module 700 includes: an extraction unit and a filling unit.

[0077] Wherein, the extraction unit is configured to extract at least one image feature of the current frame image information.

[0078] The filling unit is configured to fill the remaining occluded part according to at least one image feature.

[0079] Optionally, in an embodiment of the present application, the optical flow estimation module 400 includes: an estimation unit, a refinement unit, and an acquisition unit.

[0080] Wherein, the estimation unit is configured to perform optical flow estimation on the current endoscopic image and the previous frame endoscopic image to obtain an optical flow field under instrument occlusion.

[0081] The refinement unit is configured to refine the optical flow field under instrument occlusion based on the instrument position and shape mask to restore the optical flow field information of the area after instrument occlusion.

[0082] The acquisition unit is configured to obtain the motion relationship between the objects in the current endoscopic image based on the optical flow field information of the area after instrument occlusion.

[0083] Optionally, in an embodiment of the present application, the first filling module 500 includes: an iteration unit and a replacement unit.

[0084] Among them, the iteration unit is used to iteratively utilize the optical flow field information between adjacent frame images to fill multiple previous frame endoscopic images into the instrument position after motion transformation.

[0085] The replacement unit is used to take the area of the previous frame endoscopic image that does not contain the instrument after transformation to replace the corresponding position of the current endoscopic image, and select the previous frame that meets the preset adjacent conditions for each pixel.

[0086] It should be noted that the foregoing explanation of the embodiment of the method for removing instrument occlusion from endoscopic images also applies to the device for removing instrument occlusion from endoscopic images in this embodiment, and will not be elaborated here.

[0087] The device for removing instrument occlusion from endoscopic images proposed according to the embodiments of the present application can collect endoscopic images in real time and intelligently judge whether there is a surgical operating instrument, efficiently segment the instrument area and extract its position and shape mask, and combine the estimation and filling of the optical flow field to ensure the effective filling of the instrument occlusion area in the current frame endoscopic image, thereby significantly improving the visualization effect and safety of endoscopic surgery, reducing the operation risk caused by instrument occlusion, and further improving the surgical quality and efficiency.

[0088] Figure 3 The structural schematic diagram of the electronic device provided for the embodiments of the present application. The electronic device may include:

[0089] A memory 301, a processor 302, and a computer program stored on the memory 301 and executable on the processor 302.

[0090] When the processor 302 executes the program, it implements the method for removing instrument occlusion from endoscopic images provided in the above embodiments.

[0091] Further, the electronic device further includes:

[0092] A communication interface 303 for communication between the memory 301 and the processor 302.

[0093] The memory 301 is used to store a computer program executable on the processor 302.

[0094] The memory 301 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0095] If the memory 301, the processor 302, and the communication interface 303 are implemented independently, the communication interface 303, the memory 301, and the processor 302 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 only a thick line is used in Figure 3 , but it does not mean that there is only one bus or one type of bus.

[0096] Optionally, in a specific implementation, if the memory 301, the processor 302, and the communication interface 303 are integrated on a single chip, the memory 301, the processor 302, and the communication interface 303 can communicate with each other through an internal interface.

[0097] The processor 302 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0098] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method for removing instrument occlusion from an endoscopic image as described above is implemented.

[0099] The embodiments of the present application further provide a computer program product, including a computer program, which is used to implement the method for removing instrument occlusion from an endoscopic image as described above when the computer program is executed.

[0100] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0101] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of these features. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0102] Any process or method description depicted in a flowchart or described in other ways herein can be understood to represent a module, segment, or portion of code including one or N executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a manner that is not in the order shown or discussed, including in a substantially simultaneous manner or in the reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of this application pertain.

[0103] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definitional sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or N wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, as the program can be obtained electronically by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0104] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one or a combination of the following techniques known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0105] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0106] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0107] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for removing instrument occlusion from an endoscopic image, characterized in that: The following steps are involved: Acquiring a current endoscopic image of the endoscope; Determining whether the current endoscopic image contains a surgical instrument; If the surgical instrument is included, segmenting the instrument region from the current endoscopic image, and extracting the instrument position and shape mask of the surgical instrument based on the instrument region; Based on the instrument position and the shape mask, obtaining a motion relationship between an object in the current endoscopic image and an object in the reference endoscopic image according to an optical flow field between the current endoscopic image and the reference endoscopic image; Based on the filled optical flow field, the reference endoscopic image is subjected to motion transformation to fill the instrument occlusion mask area corresponding to the instrument position and the shape mask to obtain an unoccluded endoscopic image.

2. The method according to claim 1, characterized in that: After filling the device occlusion mask area corresponding to the device position and the shape mask, the method further includes: Determine whether the filled endoscope image is completely free of occlusion; If the padded endoscopic image is not completely free of occlusion, the remaining occluded portion is padded based on the current frame image information of the padded endoscopic image to obtain the unobstructed endoscopic image.

3. The method according to claim 2, characterized in that The filling of the remaining blocked part based on the current frame image information of the padded endoscopic image includes: Extracting at least one image feature of the current frame image information; The remaining occluded portion is filled according to the at least one image feature.

4. The method according to claim 1, characterized in that The reference endoscopic image is a previous frame endoscopic image, wherein the motion relationship between the object in the current endoscopic image and is obtained based on the instrument position and the shape mask according to the optical flow field between the current endoscopic image and the reference endoscopic image includes: Performing optical flow estimation on the current endoscopic image and the previous frame endoscopic image to obtain an optical flow field under instrument occlusion; Based on the device position and the shape mask, performing optical flow field improvement on the optical flow field under the device occlusion to restore the optical flow field information of the area after the device occlusion; The motion relationship between the object and the current endoscopic image is obtained based on the optical flow field information of the area blocked by the instrument.

5. The method according to claim 1, characterized in that The step of performing motion transformation on the reference endoscopic image based on the padded optical flow field includes: Iteratively utilizing the optical flow field information between adjacent frames of images to fill in the multiple previous frame endoscopic images to the instrument position after motion transformation; The area not containing the instrument in the transformed previous frame endoscopic image is taken to replace the corresponding position of the current endoscopic image, and for each pixel, a previous frame that meets a preset adjacent condition is selected.

6. An endoscope image removal device, characterized in that: include: An acquisition module, used for acquiring the current endoscopic image of the endoscope; A first judgment module, used to judge whether the current endoscopic image contains a surgical instrument; a segmentation module, configured to segment an instrument region from the current endoscopic image if the surgical instrument is included, and extract an instrument position and a shape mask of the surgical instrument based on the instrument region; an optical flow estimation module, configured to obtain a motion relationship between an object in the current endoscopic image and an object in the reference endoscopic image according to an optical flow field between the current endoscopic image and the reference endoscopic image based on the instrument position and the shape mask; The first filling module is used to perform motion transformation on the reference endoscopic image based on the filled optical flow field to fill the instrument occlusion mask area corresponding to the instrument position and the shape mask to obtain an unoccluded endoscopic image.

7. The device according to claim 6, characterized in that Also includes: The second judgment module is used to judge whether the filled endoscope image is completely free of occlusion; The second filling module is used to fill the remaining blocked part based on the current frame image information of the padded endoscopic image to obtain the unblocked endoscopic image if the padded endoscopic image is not completely free of occlusion.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for removing instrument occlusion from endoscopic images as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for removing instrument occlusion from an endoscopic image as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the method for removing instrument occlusion from an endoscopic image as described in any one of claims 1-5.

Images