A method for automatically cutting out the effective visible area of ​​a digestive tract endoscope

Abstract

The present invention provides a method for automatically cropping an effective visible area of ​​a digestive tract endoscope, comprising: using a fuzzy image and a binary image of a digestive tract endoscope image for joint calculation to obtain an enhanced fuzzy image of the digestive tract endoscope image; binarizing the enhanced fuzzy image to obtain an enhanced mask image; using contour detection to cyclically screen the image information of the enhanced mask image to determine an image area that meets the requirements; cropping the digestive tract endoscope image according to the image area to obtain an effective visible area. The present invention can intelligently identify images, automatically obtain effective image areas, and achieve the effect of automatically cropping images. Effective areas can be extracted in various scenes, various models, and images of different surgical types. Its success rate is very high, at more than 99.9%, mainly because the effect is very stable and the cropped areas are basically correct.

Description

Technical Field

[0001] The invention belongs to the field of intelligent recognition, and in particular relates to a method for automatically cutting an effective visible area of ​​a digestive tract endoscope. Background Art

[0002] Problems with current technology: For various models of digestive endoscopes and different endoscope display settings, the effective display area in the digestive endoscope surgery video stream may appear at any position. If the preset parameters are used to crop the effective area, each endoscope host needs to be debugged separately, which is troublesome and may cause errors. If it is not cropped, it is not conducive to the training of deep learning models. Summary of the invention

[0003] The invention provides a method for automatically cutting the effective visible area of ​​a digestive tract endoscope, which can automatically cut different endoscope displays.

[0004] The present invention provides a method for automatically cutting an effective visible area of ​​a digestive tract endoscope, comprising:

[0005] The fuzzy image and the binary image of the digestive tract endoscopy image are used for joint calculation to obtain the enhanced fuzzy image of the digestive tract endoscopy image;

[0006] Binarize the enhanced blur image to obtain an enhanced mask image;

[0007] Use contour detection to cyclically screen the image information of the enhanced mask map to determine the image area that meets the requirements;

[0008] The digestive tract endoscopy images were cropped according to the image area to obtain the effective visible area.

[0009] Furthermore, the method of performing joint calculation using the fuzzy image and the binary image of the digestive tract endoscopy image to obtain the enhanced fuzzy image of the digestive tract endoscopy image includes:

[0010] S11. Let the blurred image of the digestive tract endoscopy image be t1, let the binary image of the digestive tract endoscopy image be t2, then the enhanced blurred image t3 = t1 × (t2 + n), where n is the superposition value;

[0011] Furthermore, the binarization of the enhanced blur image to obtain the enhanced mask image includes:

[0012] S12. adjusting the threshold value of the binarization according to the properties of the enhanced fuzzy map;

[0013] S13. Use the threshold to binarize the enhanced blur image to obtain an enhanced mask image:

[0014] Furthermore, the contour detection is used to perform cyclic screening on the image information of the enhanced mask image to determine that the image area that meets the requirements includes:

[0015] S14. Perform contour detection in the enhanced mask image to determine whether the minimum bounding rectangle of the contour meets the screening conditions. If so, the contour is used as the image area that meets the requirements and the contour detection is exited. If the minimum bounding rectangles of all contours do not meet the conditions, it is considered that the cropping is unsuccessful, and the original image is returned and input.

[0016] Furthermore, the method for generating the fuzzy image of the digestive tract endoscopy image is:

[0017] The three-channel input image is converted into a grayscale image, and the blur kernel is calculated according to the image size to obtain a blur image.

[0018] Furthermore, the binary image generation method is:

[0019] S21 performs binarization processing on the three channels of the original input image by subtracting them two by two;

[0020] S22. Calculate the dilation and erosion kernel according to the image size, perform dilation first, and then perform erosion to obtain a binary image processed by mask morphology.

[0021] Furthermore, the original input graph is obtained by the following process:

[0022] Collect pictures and analyze the cropping status of the pictures according to preset parameters. If it is a cropped image, it will not be cropped and the subsequent process will be stopped; if it is a non-cropped image, it will be used as the original input image.

[0023] Furthermore, the original input graph is obtained by the following process:

[0024] S31. Collect pictures and start padding parameters, the padding parameters include at least one of dynamic random padding, up, down, left, and right four directions of percentage parameters;

[0025] S32. Determine whether the image has been cropped based on the padding parameter.

[0026] Furthermore, the use of contour detection to cyclically screen the image information of the enhanced mask image to determine the image area that meets the requirements also includes:

[0027] After determining the image area that meets the requirements, create a candidate box for the image area based on the padding parameters;

[0028] The digestive tract endoscopy image is cropped according to the image area to obtain an effective visible area, including:

[0029] Export images based on image area and candidate box.

[0030] Furthermore, the padding parameters include dynamic random padding and percentage parameters in four directions: up, down, left, and right.

[0031] Compared with the prior art, the present invention can intelligently identify images, automatically obtain effective image areas, and achieve the effect of automatically cropping images. It can extract effective areas in various scenes, various models, and different types of surgery. Its success rate is very high, at more than 99.9%, mainly because the effect is very stable and the cropped areas are basically correct. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of an embodiment of the present invention;

[0033] Figure 2 is a grayscale image of an embodiment of the present invention;

[0034] Figure 3 A fuzzy graph according to an embodiment of the present invention;

[0035] Figure 4 A mask diagram according to an embodiment of the present invention;

[0036] Figure 5 This is a mask morphology processing diagram of an embodiment of the present invention;

[0037] Figure 6 Enhance the blur map for the embodiment of the present invention;

[0038] Figure 7 Strengthening the mask map for the embodiment of the present invention;

[0039] Figure 8 This is a cropping result image with padding parameters of all 0s in an embodiment of the present invention;

[0040] Fig. 9 This is the result graph of the embodiment of the present invention where the padding parameters are all 0.2 and dynamic random padding is turned off;

[0041] Fig.10 In this embodiment of the present invention, the padding parameter is all 0.2, and the result graph of dynamic random padding is turned on. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only embodiments of a part of the present invention, rather than all embodiments.

[0043] The embodiment of the present invention provides a method for automatically cutting the effective visible area of ​​a digestive tract endoscope. Figure 1 As shown:

[0044] S101. Transmit images and set padding parameters;

[0045] Among them, this step is mainly to input the picture. There are two padding parameters here. One is whether to turn on dynamic random padding, and the second is the percentage of the four directions of up, down, left, and right, for example [0.2, 0.2, 0.2, 0.2]. The padding parameter can be used in other steps later;

[0046] S102. Determine whether the image has been cropped based on the image size and padding parameters;

[0047] If the aspect ratio of the image tends to be square, this embodiment considers it to be a cropped image and no further cropping is performed. If 0.8 < aspect ratio < 1.2, it is considered to have been cropped and no subsequent steps are performed. The padding parameter needs to be judged for validity, and the value in each direction should be in the range of 0-1.

[0048] S103. Convert the color space to obtain a grayscale image;

[0049] Convert the three-channel input image into a grayscale image to facilitate numerical statistics and morphological operations between pixel values, as shown below: Figure 2 Grayscale image shown.

[0050] S104. Figure 3 As shown, the blur kernel is calculated according to the image size, and blur processing is performed to obtain a blur image;

[0051] This step can effectively remove noise. Set the size of the blur kernel to "the minimum value between the length and width divided by 100". If the result is an even number, add 1 to ensure that the kernel is an odd number. Use median filtering or Gaussian filtering to blur the image. The resulting blurred image is as follows: Figure 3 shown.

[0052] S105 Figure 4 As shown, the mask map is calculated based on the channel difference;

[0053] By using the three channels of the original input image in S101 and subtracting them two by two, we can get the difference maps of the two channels, a total of 3 images; Set the difference threshold to 5. In these 3 difference images, set the position with a value less than the difference threshold to 0, and set the position with a value greater than the threshold to 1. Finally, perform an OR operation on the 3 binary difference images to get the mask image as follows. The purpose of this step is mainly to filter most of the background area and text. In the example, the position of the white text has almost the same value in the three channels, so after subtraction, the value less than the threshold is also filtered out.

[0054] S106 calculates the expansion and corrosion kernel according to the image size, and then performs expansion first and then corrosion to obtain a mask morphological processing image, such as Figure 5 As shown;

[0055] Among them, the mask image obtained in S105 shows that although most of the effective area has been identified, there are still some holes and rough edges. The result can be made more accurate by using the operation of dilation first and then erosion. Dilation is used to eliminate holes, and erosion is used to restore to the original size. Such a pair of combinations is also called the morphological closing operation of the image. The size of the dilation structure element and the erosion structure element is "the minimum value between the length and the width divided by 200". If the result is an even number, +1 is required to ensure that the kernel is an odd number.

[0056] S107 jointly calculates the enhanced fuzzy image Figure 6 As shown:

[0057] Here, the result of S104 is t1, the result of step 106 is t2, and the result of this step is t3; t3 = t1x (t2 + 0.5).

[0058] It should be noted that t2 is a binary image, with white areas as 1 and black areas as 0. Then t3 is processed with upper and lower thresholds, with the upper threshold as 255 and the lower threshold as 20. If the pixel value in t3 is greater than the upper threshold, it is set to 255, and if it is less than the lower threshold, it is set to 0. The result is as follows Figure 6 As shown, the difference in pixel values ​​between the effective area and the background area can be further enlarged, and the text area is also reduced.

[0059] S108 adjusts the threshold value of binarization according to the properties of the enhanced fuzzy image;

[0060] The initial threshold t is 30; the enhanced fuzzy image of S107 is eb, and eb is needed to readjust the threshold. There are four rules in the following form:

[0061] ifeb.min() > t / 3:

[0062] t = tx (eb.min() / 10)

[0063] ifeb.mean() > tx 3:

[0064] t = tx (eb.mean() x 0.75)

[0065] ifeb.max() < tx 3:

[0066] t = tx (eb.max() / 255)

[0067] ifeb.mean() < t:

[0068] t = eb.mean() x 0.5

[0069] By executing the above four rules sequentially, the accuracy of subsequent binarization can be improved.

[0070] S109 uses the threshold to binarize the enhanced blur image to obtain an enhanced mask image, such as Figure 7 As shown;

[0071] Among them, in the valid area, the enhanced mask image has a smoother boundary than the mask image.

[0072] S110 performs contour detection in the enhanced mask image;

[0073] Among them, from the binary image S109, the contour of each white area is found, and even the smallest white area has its contour. Therefore, dozens of contours are usually obtained, and each contour is a collection of a bunch of coordinates. The embodiment of the present invention directly uses a function called findContours in opencv.

[0074] S111 determines whether the bounding rectangle of the detected contour meets the conditions from large to small;

[0075] Among them, these contours are sorted from large to small according to the number of coordinates. At this time, the contour ranked first has the most coordinate points, and the minimum enclosing rectangles of these contours are judged in turn. The judgment conditions are as follows:

[0076] Condition 1: The aspect ratio of the bounding box is greater than 0.5 and less than 2;

[0077] Condition 2: The minimum value of the length and width of the bounding box is greater than half of the minimum value of the length and width of the uncropped image, that is, the original image.

[0078] Once one of the boxes is satisfied, the loop is exited and the image cutting is considered successful.

[0079] S112 returns the incoming image

[0080] If in step 111, all the contour bounding boxes do not meet the conditions, it is regarded as an unsuccessful cropping, and the original input image is returned.

[0081] S113 Modify the candidate box according to the padding parameter:

[0082] The purpose of the padding parameter is to increase the black border area and improve the generalization performance of deep learning training. If in S111, a candidate box that meets the conditions is found, then this box is adjusted according to the padding parameter. In step S101 of this embodiment, there are two padding parameters. One is whether to turn on dynamic random padding, and the second is the percentage in the four directions of up, down, left, and right. If the padding parameter in a certain direction is 0.2, if dynamic random padding is turned on, a floating point number will be randomly selected within the range of (0, 0.2); conversely, if dynamic random padding is turned off, the bounding box will be padded according to a ratio of 0.2. The padding in the horizontal direction is based on the length of the bounding box, and the padding in the vertical direction is based on the height of the bounding box.

[0083] S114 Crop the image according to the candidate box and return the cropped image:

[0084] If the padding parameter is all 0, the cropping result is as Figure 8 shown;

[0085] If the padding parameter is all 0.2 and dynamic random padding is turned off, the result is as Fig. 9 shown;

[0086] Fig. 9 In [figure], the black borders on the left and right are of equal length. Since the original image only has this height in the up and down directions, all of them are retained.

[0087] If the padding parameter is all 0.2 and dynamic random padding is turned on, the result is as Fig.10 shown.

[0088] The embodiment of the present invention can perform intelligent recognition on images, automatically obtain effective image regions, and achieve the effect of automatically cropping images. It can extract effective regions for images in various scenarios, various models, and different surgical types. Its success rate is very high, above 99.9%. The main reason is that the effect is very stable, and the cropped regions are basically correct.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the specification of this application, the technicians can still modify or replace the specific implementation mode of the present invention with equivalents, but these modifications or changes do not deviate from the scope of protection of the pending claims of the present application.

Claims

1. A method for automatically cropping the effective visual area of ​​a digestive tract endoscope. It is characterized in that include: The fuzzy image and the binary image of the digestive tract endoscopy image are used for joint calculation to obtain the enhanced fuzzy image of the digestive tract endoscopy image; Binarize the enhanced blur image to obtain an enhanced mask image; Use contour detection to cyclically screen the image information of the enhanced mask map to determine the image area that meets the requirements; The digestive tract endoscopy image is cropped according to the image area to obtain an effective visible area; The method of performing joint calculation using the fuzzy image and the binary image of the digestive tract endoscopy image to obtain the enhanced fuzzy image of the digestive tract endoscopy image includes: S11. Let the blurred image of the digestive tract endoscopy image be t1, let the binary image of the digestive tract endoscopy image be t2, then the enhanced blurred image t3 = t1 × (t2 + n), where n is the superposition value.

2. According to claim 1, a method for automatically cutting the effective visible area of ​​a digestive tract endoscope, Features The step of binarizing the enhanced blur image to obtain the enhanced mask image comprises: S12. adjusting the threshold value of the binarization according to the properties of the enhanced fuzzy map; S13. Use the threshold to binarize the enhanced blurred image to obtain an enhanced mask image.

3. According to claim 1, a method for automatically cutting the effective visible area of ​​a digestive tract endoscope, It is characterized in that The contour detection is used to perform cyclic screening on the image information of the enhanced mask image to determine the image areas that meet the requirements, including: S14. Perform contour detection in the enhanced mask image to determine whether the minimum bounding rectangle of the contour meets the screening conditions. If so, the contour is used as the image area that meets the requirements and the contour detection is exited. If the minimum bounding rectangles of all contours do not meet the conditions, it is considered that the cropping is unsuccessful, and the original image is returned and input.

4. According to claim 1, a method for automatically cutting the effective visible area of ​​a digestive tract endoscope, It is characterized in that The method for generating a fuzzy image of a digestive tract endoscopy image is as follows: the original input image of the three channels is converted into a grayscale image, and a fuzzy kernel is calculated according to the image size to obtain a fuzzy image.

5. According to claim 1, a method for automatically cutting the effective visible area of ​​a digestive tract endoscope, It is characterized in that The binary image generation method is: S21 performs binarization processing on the three channels of the original input image by subtracting them two by two, and performs OR operation on the three binary difference images; S22. Calculate the dilation and erosion kernel according to the image size, perform dilation first, and then perform erosion to obtain a binary image processed by mask morphology.

6. A method for automatically cutting the effective visible area of ​​a digestive tract endoscope according to claim 4, It is characterized in that The original input image is obtained by the following process: collecting an image and analyzing the cropping of the image according to preset parameters. If it is a cropped image, it will not be cropped and the subsequent process will be stopped; if it is a non-cropped image, it will be used as the original input image.

7. A method for automatically cutting the effective visible area of ​​a digestive tract endoscope according to claim 6, It is characterized in that The original input graph is obtained specifically by the following process: S31. Collect pictures and start padding parameters, the padding parameters include at least one of dynamic random padding and percentage parameters in four directions: up, down, left, and right; S32. Determine whether the image has been cropped according to the padding parameter.

8. According to claim 7, a method for automatically cutting the effective visible area of ​​a digestive tract endoscope, It is characterized in that The method of using contour detection to cyclically screen the image information of the enhanced mask image to determine the image area that meets the requirements also includes: After determining the image area that meets the requirements, a candidate frame is established for the image area according to the padding parameter; cropping the digestive tract endoscopy image according to the image area to obtain a valid visible area includes: exporting the image according to the image area and the candidate frame.

9. According to claim 8, a method for automatically cutting the effective visible area of ​​a digestive tract endoscope, It is characterized in that The padding parameters include dynamic random padding and percentage parameters in four directions: up, down, left, and right.

Images