A method and system for removing instrument occlusion from spinal endoscope images

Abstract

The present application provides a method and system for removing instrument occlusion from spinal endoscope images, which performs contrast enhancement on a local area of ​​a spinal endoscope image to obtain a contrast enhanced image; further performs instrument contour detection on the contrast enhanced image in multi-scale channels to obtain a contour feature map of the instrument in each scale channel, and determines the gradient variance of the pixels in each contour feature map; deconvolution is performed on all contour feature maps based on the edge information loss of the contour in contrast enhancement and each gradient variance to obtain a reconstructed feature map of the instrument contour; an occlusion mask map of the instrument is generated based on the reconstructed feature map and the spinal endoscope image, and then the occluded spinal image in the spinal endoscope image is restored based on the motion vector information of the pixels between adjacent continuous frames and the occlusion mask map. The scheme of the present application can realize multi-scale feature reconstruction of the instrument contour in the spinal endoscope image, thereby improving the robustness of the recovery of the instrument occluded area.

Description

Technical Field

[0001] The present application relates to the technical field of image restoration, and more specifically, to a method and system for removing instrument occlusion from spinal endoscope images. Background Art

[0002] The application of image restoration in removing instrument occlusion from endoscopic images is mainly used to overcome the field of view limitation caused by instrument occlusion. When the endoscopic image is occluded by the instrument, the details of part of the spine or other tissues are lost, thus affecting the doctor's operating precision and diagnostic accuracy. Image restoration technology utilizes the information in adjacent frame images and combines image processing algorithms to restore the details of the occluded area.

[0003] The existing technology usually relies on image restoration methods based on single-scale features in the processing of instrument occlusion removal in spinal endoscope images. However, this method has obvious limitations when dealing with complex occlusion scenes, especially when dealing with interference of highlight spots on instrument contour recognition. It is difficult for a single-scale restoration method to fully capture the multi-level features of the instrument. For example, the edge of the instrument shows different details and forms at different scales. The single-scale restoration method cannot handle these differences at the same time, resulting in the loss of important structural information in the restored image. In addition, when dealing with more serious occlusion, the single-scale restoration method often cannot effectively restore the edge of the occluded area, resulting in blurring, thereby affecting the de-occlusion effect. In contrast, instrument contour detection based on multi-scale features can significantly reduce the error of the single-scale restoration method in the contour feature recognition process, and can effectively improve the accuracy and reliability of instrument occlusion removal in spinal endoscope images. Therefore, how to achieve multi-scale feature reconstruction of instrument contours in spinal endoscope images, thereby improving the robustness of instrument occlusion area recovery has become a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a method and system for removing instrument occlusion from spinal endoscope images, which can realize multi-scale feature reconstruction of instrument contours in spinal endoscope images, thereby improving the robustness of instrument occlusion area recovery.

[0005] In a first aspect, the present application provides a method for removing instrument occlusion from a spinal endoscope image, comprising the following steps:

[0006] Acquiring a spinal endoscopic image blocked by an instrument, performing contrast enhancement on a local area of ​​the spinal endoscopic image to obtain a contrast enhanced image, and then determining the edge information loss of the contour during the contrast enhancement process;

[0007] Based on the distribution information entropy of pixels in the spinal endoscope image, the instrument contour detection of the contrast enhanced image is performed in multi-scale channels to obtain the contour feature map of the instrument in each scale channel, and determine the gradient variance of the pixels in each contour feature map;

[0008] Performing deconvolution reconstruction on all contour feature maps according to the edge information loss and each gradient variance to obtain a reconstructed feature map of the instrument contour;

[0009] The instrument contour is differentially fitted based on the reconstructed feature map and the spinal endoscopic image to generate an occlusion mask map of the instrument. The occluded spinal image in the spinal endoscopic image is restored based on the motion vector information of pixels between adjacent frames of the spinal endoscopic image and the occlusion mask map.

[0010] Preferably, performing contrast enhancement on a local area of ​​the spinal endoscopic image to obtain a contrast enhanced image specifically includes:

[0011] Performing median filtering on the spinal endoscope image to obtain a filtered image;

[0012] Determining a local region of interest in the filtered image where contrast enhancement is required;

[0013] Histogram equalization is performed on the local region of interest to enhance the contrast of the local region of interest, and further obtain a contrast enhanced image.

[0014] Preferably, determining the edge information loss of the contour during the contrast enhancement process specifically includes:

[0015] Acquire a contrast-enhanced image obtained by contrast enhancement;

[0016] Acquire a filtered image obtained by median filtering the spinal endoscope image;

[0017] Determining structural similarity of contour edge information in the contrast enhanced image and the filtered image;

[0018] The edge information loss of the contour during the contrast enhancement process is determined according to the structural similarity.

[0019] Preferably, performing instrument contour detection of multi-scale channels on the contrast-enhanced image based on the distribution information entropy of pixels in the spinal endoscope image to obtain the contour feature map of the instrument in each scale channel specifically includes:

[0020] Determining the distribution information entropy of pixels in the spinal endoscopic image;

[0021] Determine the convolution kernel of each scale channel by using the distribution information entropy;

[0022] The contour of the instrument in the contrast enhanced image is detected according to the convolution kernel of each scale channel to obtain the contour feature map of the instrument in each scale channel.

[0023] Preferably, determining the gradient variance of pixels in each contour feature map specifically includes:

[0024] For each contour feature map, determine the gradient amplitude of each pixel point in the contour feature map in the horizontal direction and the vertical direction, and then determine the gradient value of each pixel point in the contour feature map;

[0025] The gradient variance of the pixels in the contour feature map is determined by the gradient values ​​of all pixels, and then the gradient variance of the pixels in each contour feature map is obtained.

[0026] Preferably, performing differential fitting on the instrument contour based on the reconstructed feature map and the spinal endoscope image to generate an occlusion mask map of the instrument specifically includes:

[0027] Performing a differential operation on the reconstructed feature map and the spinal endoscope image to obtain a differential image;

[0028] Extracting significant edge regions in the differential image based on an edge detection algorithm;

[0029] Least square fitting is performed on edge information in the significant edge region to generate a fitting contour of the instrument, a binary mask image is created in the spinal endoscopy image according to the fitting contour, and the binary mask image is used as an occlusion mask image of the instrument.

[0030] Preferably, the endoscope is a single-channel endoscope system.

[0031] In a second aspect, the present application provides a spinal endoscope image instrument occlusion removal system, comprising:

[0032] An acquisition module, used for acquiring a spinal endoscopic image blocked by an instrument, performing contrast enhancement on a local area of ​​the spinal endoscopic image to obtain a contrast enhanced image, and then determining the edge information loss of the contour during the contrast enhancement process;

[0033] A processing module, configured to perform instrument contour detection of the contrast-enhanced image in multi-scale channels based on the distribution information entropy of pixels in the spinal endoscope image, obtain a contour feature map of the instrument in each scale channel, and determine the gradient variance of the pixels in each contour feature map;

[0034] The processing module is further used to perform deconvolution reconstruction on all contour feature maps according to the edge information loss and each gradient variance to obtain a reconstructed feature map of the instrument contour;

[0035] An execution module is used to perform differential fitting on the instrument contour based on the reconstructed feature map and the spinal endoscopic image, thereby generating an occlusion mask map of the instrument, and restore the occluded spinal image in the spinal endoscopic image based on the motion vector information of pixels between adjacent continuous frames of the spinal endoscopic image and the occlusion mask map.

[0036] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned method for removing instrument occlusion from spinal endoscopic images.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for removing instrument occlusion from spinal endoscopic images is implemented.

[0038] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:

[0039] In an embodiment of the present application, a spinal endoscopic image occluded by an instrument is collected, and the contrast of a local area of ​​the spinal endoscopic image is enhanced to obtain a contrast enhanced image, and then the edge information loss of the contour during the contrast enhancement process is determined; based on the distribution information entropy of pixels in the spinal endoscopic image, the contrast enhanced image is subjected to multi-scale channel instrument contour detection to obtain a contour feature map of the instrument in each scale channel, and the gradient variance of the pixels in each contour feature map is determined; all contour feature maps are deconvoluted and reconstructed according to the edge information loss and each gradient variance to obtain a reconstructed feature map of the instrument contour; based on the reconstructed feature map and the spinal endoscopic image, the instrument contour is differentially fitted to generate an occlusion mask map of the instrument, and the occluded spinal image in the spinal endoscopic image is restored according to the motion vector information of pixels between adjacent continuous frames of the spinal endoscopic image and the occlusion mask map.

[0040] It can be seen that the present application restores the occluded spinal image in the spinal endoscope image through the motion vector information of pixels between adjacent continuous frames of the spinal endoscope image and the occlusion mask map; firstly, the contrast of the local area of ​​the spinal endoscope image is enhanced, and the contrast of the instrument edge in the spinal endoscope image is enhanced so that the instrument contour in the occluded area becomes clearer, thereby improving the recognition accuracy of the instrument contour; secondly, based on the distribution information entropy of pixels in the spinal endoscope image, the instrument contour of the contrast enhanced image is detected in multi-scale channels, and the multi-level features of the instrument contour can be obtained under different scale channels, thereby reducing the image complexity based on a single scale feature. The original method makes errors when dealing with differences in instrument contour details; then, all contour feature maps are reconstructed through the gradient variance of pixels in different contour feature maps, which can significantly retain the detailed features of the instrument contour and effectively make up for the problem of edge blur and information loss caused by the image restoration method based on a single scale feature. At the same time, the edge information loss of the contour during the contrast enhancement process is compensated by deconvolution, making the reconstructed instrument contour clearer and more reliable; finally, the occluded spinal image in the spinal endoscopy image is restored based on the motion vector information of pixels between adjacent continuous frames of the spinal endoscopy image and the occlusion mask map, thereby improving the reliability and robustness of the recovery of the instrument occluded area. In summary, in the embodiment of the present application, multi-scale feature reconstruction of the instrument contour in the spinal endoscopy image can be achieved, thereby improving the robustness of the recovery of the instrument occluded area. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is an exemplary flow chart of a method for removing instrument occlusion from a spinal endoscope image according to some embodiments of the present application;

[0042] Figure 2 is a spinal endoscopy image shown in some embodiments of the present application;

[0043] Figure 3 is a schematic diagram of a process for determining and reconstructing a feature map according to some embodiments of the present application;

[0044] Figure 4 is a schematic diagram of the structure of a spinal endoscope image instrument occlusion removal system according to some embodiments of the present application;

[0045] Figure 5 It is a structural schematic diagram of a computer device for implementing a method for removing instrument occlusion from spinal endoscopic images according to some embodiments of the present application. DETAILED DESCRIPTION

[0046] In order to better understand the technical solution of the present application, the technical solution of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0047] refer to Figure 1 , which is an exemplary flow chart of a method for removing instrument occlusion from a spinal endoscope image according to some embodiments of the present application. The spinal endoscope image removal instrument occlusion method 100 mainly includes the following steps:

[0048] In step 101, a spinal endoscopic image blocked by an instrument is acquired, and contrast enhancement is performed on a local area of ​​the spinal endoscopic image to obtain a contrast enhanced image, and then edge information loss of the contour during the contrast enhancement process is determined.

[0049] It should be noted that during percutaneous endoscopic spinal surgery, the endoscope provides a limited field of view, and surgical instruments such as guide needles and cutters will block the field of view of the endoscope, thereby affecting the doctor's observation and judgment. For example, when the endoscope is a single-channel endoscope system, due to channel limitations, the movement of surgical instruments may limit the field of view of the endoscope, resulting in instrument obstruction. By removing these obstructing instruments, it can ensure that the doctor can clearly see the spinal structure, the lesion site, and the state of the surrounding tissues.

[0050] For specific implementation, refer to Figure 2 As shown, the figure is a spinal endoscopic image in some embodiments of the present application, and the spinal endoscopic image contains instrument obstructions. The collection of spinal endoscopic images obstructed by instruments can be achieved in the following manner, namely: first, use an endoscope to percutaneously detect the spine to obtain an endoscopic video stream, and input the endoscopic video stream into a pre-trained target detection model, and use the target detection model to detect whether the spinal information in each frame of the spinal endoscopic image in the endoscopic video stream is obstructed by the instrument, and use the image detected as obstructed by the instrument as the spinal endoscopic image obstructed by the instrument in the embodiment of the present application; it should be noted that the target detection model in the present application is obtained based on Faster R-CNN training, and the target detection model can be used to detect the target positions of the spine and the instrument, and then determine whether there is occlusion by detecting the relative position of the target.

[0051] In some embodiments, contrast enhancement of a local area of ​​the spinal endoscopic image to obtain a contrast enhanced image may be achieved by the following steps:

[0052] Performing median filtering on the spinal endoscope image to obtain a filtered image;

[0053] Determining a local region of interest in the filtered image where contrast enhancement is required;

[0054] Histogram equalization is performed on the local region of interest to enhance the contrast of the local region of interest, and further obtain a contrast enhanced image.

[0055] In specific implementation, first, a median filter (such as cv2.medianBlur() in OpenCV) can be used to remove noise from the spinal endoscope image to smooth the spinal endoscope image, and then the spinal endoscope image processed by the median filter is used as the filtered image; then, an edge detection algorithm (such as Canny edge detection) or a U-Net segmentation model based on deep learning can be used to analyze and obtain a local region of interest in the filtered image, and the local region of interest is usually a local region in the filtered image where contrast enhancement is required; finally, a fixed-size grid partitioning method can be used to divide the local region of interest into a plurality of blocks of interest, and then for each block of interest, a grayscale histogram within the block of interest is calculated, and the grayscale histogram of each block of interest is processed by equalization to adjust the distribution of pixel values ​​within the block of interest so that the contrast of the block of interest is improved, so that all adjusted blocks of interest can be obtained, and then all blocks of interest are recombined into an image according to the corresponding segmentation positions to obtain a contrast-enhanced local region of interest, and further, the contrast-enhanced local region of interest is used to replace the image of the original local region of interest in the filtered image, and the replaced filtered image is used as the contrast-enhanced image.

[0056] It should be noted that the contrast enhancement of local areas of spinal endoscopic images in the present application can effectively improve the visualization of the details of the spine and instruments in spinal endoscopic images, especially in low-contrast areas, such as the edges of spinal structures, details of the vertebral bodies and the edges of instrument contours, making these key areas clearer and easier to identify. At the same time, through local enhancement rather than full-image enhancement, excessive amplification of noise and artifacts can be avoided, thereby improving the reliability of contrast enhancement of spinal endoscopic images.

[0057] In some embodiments, determining the edge information loss of the contour during the contrast enhancement process may be achieved by using the following steps:

[0058] Acquire a contrast-enhanced image obtained by contrast enhancement;

[0059] Acquire a filtered image obtained by median filtering the spinal endoscope image;

[0060] Determining structural similarity of contour edge information in the contrast enhanced image and the filtered image;

[0061] The edge information loss of the contour during the contrast enhancement process is determined according to the structural similarity.

[0062] In specific implementation, first, a contrast enhanced image and a filtered image are obtained from the above embodiments; then, the Canny edge detection algorithm in the prior art can be used to perform edge detection on the contrast enhanced image and the filtered image respectively, and edge information of the contrast enhanced image and the filtered image can be extracted; then, a similarity value of edge information between the contrast enhanced image and the filtered image is calculated by a structural similarity index (StructuralSimilarity Index, SSIM), and the similarity value is used as the structural similarity of contour edge information in the contrast enhanced image and the filtered image, wherein the structural similarity index can be used to measure the structural similarity between the contrast enhanced image and the filtered image; finally, the inverse of the structural similarity can be used as the edge information loss of the contour during the contrast enhancement process. In other embodiments, other mapping feature values ​​of the structural similarity can also be used as the edge information loss of the contour during the contrast enhancement process, which is not limited here.

[0063] It should be noted that the edge information loss in the present application is an indicator for measuring the degree of edge feature loss during the contrast enhancement process. The greater the edge information loss, the greater the degree of edge feature loss of the spinal endoscopy image during the contrast enhancement process. In addition, the structural similarity in the present application is an indicator for measuring the degree of structural similarity between the contrast enhanced image and the filtered image.

[0064] In step 102, the instrument contour detection of the contrast enhanced image is performed in multi-scale channels based on the distribution information entropy of the pixels in the spinal endoscope image to obtain the contour feature map of the instrument in each scale channel, and determine the gradient variance of the pixels in each contour feature map.

[0065] In some embodiments, based on the distribution information entropy of pixels in the spinal endoscope image, the instrument contour detection of the contrast enhanced image in multi-scale channels is performed to obtain the contour feature map of the instrument in each scale channel, which can be achieved by the following steps:

[0066] Determining the distribution information entropy of pixels in the spinal endoscopic image;

[0067] Determine the convolution kernel of each scale channel by using the distribution information entropy;

[0068] The contour of the instrument in the contrast enhanced image is detected according to the convolution kernel of each scale channel to obtain the contour feature map of the instrument in each scale channel.

[0069] In the specific implementation, first, the spinal endoscope image is gray-scale processed to obtain an endoscope gray-scale image, the pixel difference between each pixel point and other adjacent pixels in the endoscope gray-scale image is obtained, and then the information entropy of all pixel differences is used as the distribution information entropy of the pixels in the spinal endoscope image; then, the convolution kernels of different scale channels are designed according to the size of the distribution information entropy. For example, assuming that there are 3 scale channels, that is, when the distribution information entropy is less than or equal to 3, the convolution kernels of each scale channel are designed to be 7×7, 9×9 and 11×11 respectively, when the distribution information entropy is greater than 3 and less than or equal to 5, the convolution kernels of each scale channel are designed to be 5×5, 7×7 and 9×9 respectively, and when the distribution information entropy is greater than 5, the convolution kernels of each scale channel are designed to be 3×3, 5×5 and 7×7 respectively; finally, the convolution kernel of each scale channel can be used to perform a convolution operation on the contrast-enhanced image to extract the edge features of the contrast-enhanced image at different scale channels, and further the edge features at each scale channel of the pooling layer are pooled to produce the contour feature map of the instrument at each scale channel.

[0070] It should be noted that the distribution information entropy in the present application is an indicator to measure the complexity of the pixel value distribution in the spinal endoscopic image. The higher the distribution information entropy, the richer the details and structure of the spinal endoscopic image, and the greater the local changes. Conversely, the simpler the details and structure of the spinal endoscopic image. In addition, the instrument contour detection through multi-scale channels in the present application can effectively capture the instrument contour information at different scales in the spinal endoscopic image, thereby improving the accuracy and robustness of contour detection. Convolution operations of different scales can process the detail features and macro contours in the image, and avoid detail loss or false detection due to scale mismatch. Subsequently, through the fusion of multi-scale information, it can better adapt to the changes in different image complexities and instrument morphologies, thereby achieving more accurate and stable instrument detection.

[0071] In some embodiments, determining the gradient variance of pixels in each contour feature map may be implemented by the following steps:

[0072] For each contour feature map, determine the gradient amplitude of each pixel point in the contour feature map in the horizontal direction and the vertical direction, and then determine the gradient value of each pixel point in the contour feature map;

[0073] The gradient variance of the pixels in the contour feature map is determined by the gradient values ​​of all pixels, and then the gradient variance of the pixels in each contour feature map is obtained.

[0074] In the specific implementation, for each contour feature map, first, the Sobel operator in the prior art can be used to calculate the gradient amplitude of each pixel in the horizontal and vertical directions, and the average value of the gradient amplitude of each pixel point in the horizontal and vertical directions is used as the gradient value of each pixel point; then, the gradient values ​​of all pixel points can be substituted into the variance calculation formula, and the calculated variance is used as the gradient variance of the pixel in the contour feature map. The gradient variance of the pixel in each contour feature map can be obtained in the above manner.

[0075] It should be noted that the gradient variance in the present application is an indicator for measuring the degree of discreteness of pixel gradient values ​​in the contour feature map, and the gradient variance can reflect the irregularity of edge changes in the contour feature map.

[0076] In step 103, all contour feature maps are deconvolved and reconstructed according to the edge information loss and each gradient variance to obtain a reconstructed feature map of the instrument contour.

[0077] In some embodiments, reference Figure 3 As shown, this figure is a schematic diagram of the process of determining the reconstructed feature map in some embodiments of the present application. In this embodiment, all contour feature maps are deconvoluted and reconstructed according to the edge information loss and each gradient variance, and the reconstructed feature map of the instrument contour can be obtained by the following steps:

[0078] In step 1031, the recognition attention of each scale channel is determined according to the proportion between each gradient variance;

[0079] In step 1032, a convolution reconstruction operation is performed on all contour feature maps through each recognition attention to obtain a contour reconstructed feature map;

[0080] In step 1033, deconvolution compensation is performed on the contour reconstruction feature map using the edge information loss to obtain a reconstructed feature map of the instrument contour.

[0081] In the specific implementation, first, the sum of all gradient variances is recorded as the gradient variance sum, and the ratio of each gradient variance to the gradient variance sum can be used as the recognition attention of the corresponding scale channel. It should be noted that each contour feature map in the present application corresponds to a gradient variance and a scale channel; then, a convolutional network comprising multiple convolutional layers can be designed, wherein each convolutional layer corresponds to a scale channel, and the recognition attention of each scale channel is used as an adjustment factor of the convolution kernel in the corresponding convolutional layer. The adjustment factor can be used to adaptively adjust the size of the convolution kernel in the convolutional layer according to the weight of the recognition attention, thereby adaptively enhancing the learning of the contour feature map, and further all the contour feature maps are input into the convolutional network, and all the contour feature maps are reconstructed by convolution fusion through the convolutional network, and The output result is used as the contour reconstruction feature map; finally, a compensation model is initialized, which is a machine learning model obtained by pre-training based on the convolutional neural network architecture, and the edge information loss is used as the guided compensation parameter of the compensation model. The compensation model can be guided to compensate the contour reconstruction feature map by guiding the compensation parameter, and then the output result of the compensation model is used as the reconstructed feature map of the instrument contour. When the compensation model is guided by the guided compensation parameter to compensate the contour reconstruction feature map, the compensation model can automatically adjust the intensity of the deconvolution operation based on the size of the edge information loss. For example, a larger edge information loss will prompt the compensation process to strengthen the edge recovery of the contour reconstruction feature map, while a smaller loss will maintain the original details of the contour reconstruction feature map, and finally a high-quality instrument contour reconstruction feature map is obtained.

[0082] It should be noted that in the present application, deconvolution compensation is performed on the contour reconstruction feature map through edge information loss, which helps to repair the edge detail information lost in the contrast enhancement process. The deconvolution compensation process can restore the occluded or blurred edge contour by inversely processing the reconstructed feature map, thereby reducing the edge information loss caused by convolution operations or feature map transformations.

[0083] In step 104, differential fitting is performed on the instrument contour based on the reconstructed feature map and the spinal endoscopic image to generate an occlusion mask map of the instrument, and the occluded spinal image in the spinal endoscopic image is restored based on the motion vector information of pixels between adjacent continuous frames of the spinal endoscopic image and the occlusion mask map.

[0084] In some embodiments, differential fitting of the instrument contour based on the reconstructed feature map and the spinal endoscope image to generate an occlusion mask map of the instrument can be implemented by the following steps:

[0085] Performing a differential operation on the reconstructed feature map and the spinal endoscope image to obtain a differential image;

[0086] Extracting significant edge regions in the differential image based on an edge detection algorithm;

[0087] Least square fitting is performed on edge information in the significant edge region to generate a fitting contour of the instrument, a binary mask image is created in the spinal endoscopy image according to the fitting contour, and the binary mask image is used as an occlusion mask image of the instrument.

[0088] In the specific implementation, first, the difference value of each pixel between the reconstructed feature map and the spinal endoscope image is calculated, and the difference values ​​of all pixels are arranged into a two-dimensional image according to the positions of the corresponding pixel points, and then the two-dimensional image is used as a differential image. The differential image can reflect the significant difference between the instrument contour and the background (spinal tissue), thereby highlighting the position and shape of the instrument; then, the gradient amplitude of each difference value in the differential image is calculated, and then all the gradient amplitudes are judged by the Canny edge detection algorithm to extract the significant edge area in the differential image, and the edge area is used as the significant edge area in the differential image. Among them, the significant edge area reflects the significant difference between the instrument contour and other tissue structures, and can provide accurate contour information for subsequent fitting and mask generation; finally, the least squares fitting method can be used to fit the edge information in the significant edge area, and a fitting contour is obtained by minimizing the fitting error, and the fitting contour is used as the fitting contour of the instrument. Further, the fitting contour is used to create a binary mask map in the spinal endoscopy image, in which the contour area is marked as the foreground (usually 1), and other areas (such as spinal tissue) are marked as the background (usually 0), and the obtained binary mask map is used as the occlusion mask map of the instrument.

[0089] It should be noted that the occlusion mask image in the present application is a binary image used to identify the occluded area in the spinal endoscopy image.

[0090] In some embodiments, the following steps may be used to restore the blocked spinal image in the spinal endoscopy image according to the motion vector information of pixels between adjacent frames of the spinal endoscopy image and the occlusion mask map:

[0091] Determining an occluded area in the spinal endoscopy image according to the occlusion mask image;

[0092] The position of each pixel point in the blocked area in the previous frame image is inferred based on the motion vector information of the pixels between adjacent frames of the spinal endoscope image, and then the pixel value of the corresponding pixel point is extracted from the previous frame image;

[0093] The obscured spinal column image in the spinal column endoscope image is restored by using the extracted pixel values.

[0094] In specific implementation, first, the binary mask image (i.e., the occlusion mask image) generated previously can be used to filter out all pixel markers marked as 1 in the binary mask image, and the area composed of the pixel points corresponding to all the pixel markers in the spinal endoscopic image is used as the occluded area in the spinal endoscopic image; then, the motion vector information of the pixel is extracted from the continuous frames adjacent to the spinal endoscopic image, and the motion vector information contains the motion information of the pixel, including direction and motion distance. Based on the motion vector information of the pixel, the position of each pixel point in the occluded area in the previous frame image is inferred by the optical flow method to obtain the position information of the pixel point in the occluded area in the previous frame image, and then the pixel value of the corresponding pixel position is extracted from the previous frame image to obtain the pixel value of all the pixel points in the occluded area corresponding to the previous frame image; finally, the pixel values ​​of all the pixel points in the extracted previous frame image are replaced with the pixel values ​​of the pixel points in the occluded area in the spinal endoscopic image to restore the occluded spinal image.

[0095] On the other hand, in some embodiments, the present application provides a spinal endoscope image instrument occlusion removal system, referring to Figure 4 , which is a schematic diagram of the structure of a spinal endoscope image instrument occlusion removal system according to some embodiments of the present application, the spinal endoscope image instrument occlusion removal system 400 includes: an acquisition module 401, a processing module 402 and an execution module 403, which are respectively described as follows:

[0096] Acquisition module 401, in the present application, acquisition module 401 is mainly used to acquire spinal endoscopic images blocked by instruments, perform contrast enhancement of local areas of the spinal endoscopic images to obtain contrast-enhanced images, and further determine the edge information loss of the contour during the contrast enhancement process;

[0097] Processing module 402, in the present application, is used to perform instrument contour detection of multi-scale channels on the contrast-enhanced image based on the distribution information entropy of pixels in the spinal endoscope image, obtain contour feature maps of the instrument in each scale channel, and determine the gradient variance of pixels in each contour feature map;

[0098] In the present application, the processing module 402 is also used to perform deconvolution reconstruction on all contour feature maps according to the edge information loss and each gradient variance to obtain a reconstructed feature map of the instrument contour;

[0099] Execution module 403, in the present application, the execution module 403 is mainly used to perform differential fitting on the instrument contour based on the reconstructed feature map and the spinal endoscopic image, thereby generating an occlusion mask map of the instrument, and restoring the occluded spinal image in the spinal endoscopic image according to the motion vector information of pixels between adjacent continuous frames of the spinal endoscopic image and the occlusion mask map.

[0100] In addition, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned method for removing instrument occlusion from spinal endoscopic images.

[0101] In some embodiments, reference Figure 5 , which is a schematic diagram of the structure of a computer device for implementing a method for removing instrument occlusion from a spinal endoscope image according to some embodiments of the present application. The method for removing instrument occlusion from a spinal endoscope image in the above embodiment can be performed by Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504.

[0102] The processor 501 may be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0103] The communication bus 502 may be used to transmit information between the above-mentioned components.

[0104] The memory 503 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 503 may exist independently and be connected to the processor 501 via the communication bus 502. The memory 503 may also be integrated with the processor 501.

[0105] The memory 503 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the program code stored in the memory 503. The program code may include one or more software modules. The spinal endoscope image instrument occlusion removal method in the above embodiment can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.

[0106] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0107] In a specific implementation, as an embodiment, a computer device may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0108] The above-mentioned computer device may be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.

[0109] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned method for removing instrument occlusion from spinal endoscopic images is implemented.

[0110] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0111] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for removing instrument occlusion from spinal endoscope images, characterized in that: The steps include: Acquiring a spinal endoscopic image blocked by an instrument, performing contrast enhancement on a local area of ​​the spinal endoscopic image to obtain a contrast enhanced image, and then determining the edge information loss of the contour during the contrast enhancement process; Based on the distribution information entropy of pixels in the spinal endoscope image, the instrument contour detection of the contrast enhanced image is performed in multi-scale channels to obtain the contour feature map of the instrument in each scale channel, and determine the gradient variance of the pixels in each contour feature map; Performing deconvolution reconstruction on all contour feature maps according to the edge information loss and each gradient variance to obtain a reconstructed feature map of the instrument contour; The instrument contour is differentially fitted based on the reconstructed feature map and the spinal endoscopic image to generate an occlusion mask map of the instrument. The occluded spinal image in the spinal endoscopic image is restored based on the motion vector information of pixels between adjacent frames of the spinal endoscopic image and the occlusion mask map.

2. The method according to claim 1, characterized in that Performing contrast enhancement on a local area of ​​the spinal endoscopic image to obtain a contrast enhanced image specifically includes: Performing median filtering on the spinal endoscope image to obtain a filtered image; Determining a local region of interest in the filtered image where contrast enhancement is required; Histogram equalization is performed on the local region of interest to enhance the contrast of the local region of interest, and further obtain a contrast enhanced image.

3. The method according to claim 1, characterized in that Determining the edge information loss of the contour during the contrast enhancement process specifically includes: Acquire a contrast-enhanced image obtained by contrast enhancement; Acquire a filtered image obtained by median filtering the spinal endoscope image; Determining structural similarity of contour edge information in the contrast enhanced image and the filtered image; The edge information loss of the contour during the contrast enhancement process is determined according to the structural similarity.

4. The method according to claim 1, characterized in that Based on the distribution information entropy of pixels in the spinal endoscope image, the instrument contour detection of the contrast enhanced image in multi-scale channels is performed to obtain the contour feature map of the instrument in each scale channel, which specifically includes: Determining the distribution information entropy of pixels in the spinal endoscopic image; Determine the convolution kernel of each scale channel by using the distribution information entropy; The contour of the instrument in the contrast enhanced image is detected according to the convolution kernel of each scale channel to obtain the contour feature map of the instrument in each scale channel.

5. The method according to claim 1, characterized in that Determining the gradient variance of pixels in each contour feature map specifically includes: For each contour feature map, determine the gradient amplitude of each pixel point in the contour feature map in the horizontal direction and the vertical direction, and then determine the gradient value of each pixel point in the contour feature map; The gradient variance of the pixels in the contour feature map is determined by the gradient values ​​of all pixels, and then the gradient variance of the pixels in each contour feature map is obtained.

6. The method according to claim 1, characterized in that Performing differential fitting on the instrument contour based on the reconstructed feature map and the spinal endoscope image to generate an occlusion mask map of the instrument specifically includes: Performing a differential operation on the reconstructed feature map and the spinal endoscope image to obtain a differential image; Extracting significant edge regions in the differential image based on an edge detection algorithm; Least square fitting is performed on edge information in the significant edge region to generate a fitting contour of the instrument, a binary mask image is created in the spinal endoscopy image according to the fitting contour, and the binary mask image is used as an occlusion mask image of the instrument.

7. The method according to claim 1, characterized in that The endoscope is a single-channel endoscope system.

8. A spinal endoscope image instrument occlusion removal system, characterized in that: include: An acquisition module, used for acquiring a spinal endoscopic image blocked by an instrument, performing contrast enhancement on a local area of ​​the spinal endoscopic image to obtain a contrast enhanced image, and then determining the edge information loss of the contour during the contrast enhancement process; A processing module, configured to perform instrument contour detection of the contrast-enhanced image in multi-scale channels based on the distribution information entropy of pixels in the spinal endoscope image, obtain a contour feature map of the instrument in each scale channel, and determine the gradient variance of the pixels in each contour feature map; The processing module is further used to perform deconvolution reconstruction on all contour feature maps according to the edge information loss and each gradient variance to obtain a reconstructed feature map of the instrument contour; An execution module is used to perform differential fitting on the instrument contour based on the reconstructed feature map and the spinal endoscopic image, thereby generating an occlusion mask map of the instrument, and restore the occluded spinal image in the spinal endoscopic image based on the motion vector information of pixels between adjacent continuous frames of the spinal endoscopic image and the occlusion mask map.

9. A computer device, comprising a memory and a processor, wherein the memory stores a code, characterized in that: The processor is configured to obtain the code and execute the method for removing instrument occlusion from spinal endoscopy images as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for removing instrument occlusion from spinal endoscopy images as described in any one of claims 1 to 7 is implemented.

Images