A method for realizing autonomous navigation of endoscope and endoscope device

Abstract

The present invention discloses a method for realizing autonomous navigation of an endoscope, the method comprising: extracting the image contour by taking a picture of the channel where the endoscope is located, performing image preprocessing and contour extraction operations, and when a clear contour is extracted, adopting different algorithms for analysis and processing according to the different types and quantities of the contours, obtaining and storing steering data; when a clear contour cannot be extracted, reading the previously stored steering data, designing a triple control strategy to control the endoscope to capture a clear contour, and transmitting the control data to a single-chip microcomputer through a serial port after obtaining the steering parameters, and the single-chip microcomputer controls the steering gear to rotate to realize the control of the endoscope. The present invention can realize rapid autonomous navigation of an endoscope. The present invention also provides an endoscope device using the method.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscopes. Background Art

[0002] An endoscope is a commonly used detection instrument with components such as an image sensor, an optical lens, a light source, a mechanical device, etc. Since the environment in which the endoscope is used is an irregular and complex channel, such as the intestine, etc., the endoscope in the prior art has the ability to autonomously navigate in the irregular channel.

[0003] In the endoscope autonomous navigation algorithm, the existing endoscope autonomous navigation strategies are mostly aimed at detecting the deepest / darkest area in the image, that is, the direction of the deepest / darkest area in the image is the direction the endoscope needs to turn, but this method does not take into account the impact of irregular structures in complex channels on the camera's acquisition of images. For example, the folded structure in the intestine will affect the movement of the endoscope. These folded structures are closer to the lens. If this structure is not taken into account, it may cause serious consequences such as perforation. Summary of the invention

[0004] In order to solve the above problems, the present application provides a method for implementing endoscope autonomous navigation, which can improve the efficiency of endoscope autonomous navigation in complex environments.

[0005] The present invention also provides an endoscope device using the method for implementing the endoscope autonomous navigation.

[0006] To achieve the above objectives, the method for implementing endoscope autonomous navigation provided by the present invention can adopt the following technical solutions:

[0007] A method for realizing autonomous navigation of an endoscope, characterized in that:

[0008] The intestinal images are acquired through the camera at the end of the endoscope and transmitted to the computer for image preprocessing;

[0009] Perform contour extraction on the preprocessed image to obtain the image contour;

[0010] If a clear contour can be obtained, first set the contour area threshold to determine the closed contour and the open contour. If a closed contour exists, the endoscope will go straight without turning;

[0011] When only open contours exist, the contours are classified, and the contours of each frame image are divided into left contours and right contours.

[0012] When the number of contours N on one side is greater than the number of contours P on the other side, this includes the cases where N=3 or 4, N=2 or N=1;

[0013] When N=3 or 4, the AHP hierarchical analysis method is used to assign corresponding weights to each contour based on the distance from the contour to the center of the image, and the turning determination coefficient E is obtained through matrix operation.

[0014] When N=2, the steering determination coefficient E is obtained through matrix operation.

[0015] When N=1, the steering determination coefficient E is determined by detecting the contour opening direction;

[0016] When N=P, the steering determination coefficient E is determined by detecting the number of leftward or rightward contours;

[0017] The steering determination coefficient E is used to determine the steering angle of the endoscope.

[0018] Furthermore, the image preprocessing method includes:

[0019] Convert each frame into a grayscale image;

[0020] Use contrast-constrained adaptive histogram equalization method to enhance the contrast of grayscale images;

[0021] A 9×9 Gaussian kernel is used to perform Gaussian filtering on the image.

[0022] Furthermore, after the image preprocessing is completed, the method for extracting the image contour includes:

[0023] The adaptive threshold of the mean method is used for the Gaussian filtered image to remove the influence of the background color;

[0024] After the background color is initially removed, a 3×3 convolution kernel is used to perform a morphological closing operation on the image to supplement the missing contour information;

[0025] Use the built-in findContours function of opencv to extract the contours of the completed image, calculate the area of ​​each contour, and fill the smaller contours by setting the threshold;

[0026] After padding is completed, a 3×3 convolution kernel is used again for morphological closing operation;

[0027] After the morphological operation is completed, the findContours contour extraction operation is performed.

[0028] Finally, the minimum circumscribed circle is extracted for each contour to obtain the minimum circumscribed circle radius and center.

[0029] Furthermore, the difference in the contour information lies in the position of the center of the minimum circumscribed circle of each contour relative to the center point of the image.

[0030] Furthermore, if a clear outline cannot be obtained, a triple control strategy is designed based on the existing parameters to control the steering of the endoscope; the triple strategy includes:

[0031] Read the previously stored steering control parameters and profile state parameters, take the sum of the five adjacent parameters, if the sum of the profile state parameters is greater than or equal to the threshold, set the steering amplitude according to the sum of the steering control parameters, and the positive and negative numbers indicate the direction. If the sum of the profile state parameters is less than the threshold, the triple control strategy is triggered. The first step is to control the size of the steering amplitude according to the size of the sum of the steering control parameters, and the positive and negative numbers indicate the direction.

[0032] The second step is to take a turn that is different from the first step and larger than the first step if a clear outline cannot be captured after the first step is completed.

[0033] The third step is to take a turn in the same direction as the first step and with a larger amplitude than the second step if a clear outline still cannot be captured after executing the second step.

[0034] Furthermore, when N = 3 or 4, the corresponding weights are calculated

[0035]

[0036] ω 0 i is a transition parameter used to calculate the contour weight; N = 3 or 4; a ij Represents the parameter of row i and column j in the judgment matrix, 1≤i≤N, 1≤j≤N;

[0037]

[0038] ω i It refers to the weight of a certain contour calculated, 1≤i≤N;

[0039] in

[0040] After obtaining the weights corresponding to each contour, the steering determination coefficient E is calculated by the formula:

[0041]

[0042] Where left / right is the normalized left / right lateral error, RL / RR is the left / right minimum circumscribed radius after mapping, Δ represents the suppression parameter, and N / P is the number of open contours on the left / right side;

[0043] When N=2, calculate the steering determination coefficient E:

[0044]

[0045] When N=1, set E to medium-amplitude steering;

[0046] The steering angle ω is:

[0047]

[0048] When N=P, if the difference between the number of contours on one side and the number on the other side is greater than 2, it is set as a medium-amplitude turn, otherwise it is set as a small-amplitude turn.

[0049] Furthermore, the steering control parameter is defined as: after processing each frame of the image, a steering control result is obtained based on the image information or stored data, where positive and negative indicate the direction.

[0050] Furthermore, the contour state parameter is defined as follows: when a clear contour can be extracted, the contour state parameter corresponding to the frame is 1, and if not, it is 0.

[0051] Furthermore, the steering amplitude refers to the angle of rotation of the endoscope, which is determined according to a steering determination coefficient during the analysis and judgment process. The steering determination coefficient is calculated by analyzing the profile information or the steering control parameters.

[0052] Beneficial effects:

[0053] Compared with the prior art, the image processing and contour extraction method of the present invention is simple and fast, and the analysis and processing method designed for different contours can greatly reduce the time required for contour calculation. The processing speed of the present invention for each frame of image is 20-25ms, which fully meets the requirements of real-time processing, can overcome the impact of image errors on endoscope autonomous navigation in complex environments, and improve the efficiency and accuracy of endoscope autonomous navigation.

[0054] The present invention also provides an endoscope device using the above-mentioned implementation method, wherein the device includes an endoscope, a straight-moving drive module, a steering control module, and a user interface; the straight-moving drive module drives the straight-moving movement of the endoscope through the friction of two servos; the steering control module is connected to the endoscope steering knob through the servo to control the steering of the endoscope. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a method for implementing endoscope autonomous navigation;

[0056] Figure 2 This is a schematic diagram of the processing after image preprocessing and contour extraction;

[0057] Figure 3 It is a schematic diagram of the analysis of clear outlines;

[0058] Figure 4 It is a schematic diagram of the processing flow for clear contours;

[0059] Figure 5 It is a schematic diagram of the processing flow for unclear contours;

[0060] Figure 6 It is a contour extraction and analysis diagram of the intestinal image;

[0061] Figure 7 It is a schematic diagram of image processing verification using the method for implementing endoscope autonomous navigation of the present invention. DETAILED DESCRIPTION

[0062] A specific embodiment of the present invention is described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] Figure 1 The figure shows a specific flow chart of the implementation of the entire control method. As shown in the figure, the camera at the end of the endoscope takes images of the intestines and transmits the images (640×480) to the image processing module (computer) through the USB interface. After the images are analyzed and processed, control judgment is performed, and then the control parameters of the endoscope are transmitted to the single-chip microcomputer. The single-chip microcomputer controls the rotation of the servo according to the data, thereby realizing the control of the endoscope. The specific implementation steps are as follows:

[0064] First, the image is preprocessed, including grayscale, contrast-limited adaptive histogram equalization, and Gaussian filtering. The clipLimit of contrast-limited adaptive histogram equalization is set to 2, tileGridSize is (12, 12), ksize of Gaussian filtering is (9, 9), and sigmaX is set to 1.

[0065] After the image preprocessing is completed, the image is subjected to the adaptive threshold of the mean method to remove the influence of the background color, and then a 3×3 convolution kernel is used to perform a morphological closing operation on the image to supplement some missing contour information. The contours that have been supplemented are subjected to the opencv built-in findContours contour extraction operation, and the area of ​​each contour is calculated. The contours whose area is smaller than the threshold are filled with the background color. After the filling is completed, the morphological closing operation is performed again, and finally the contour extraction is completed through the findContours contour extraction operation.

[0066] The present invention processes some fine contours from the level of the entire image, and uses the findContours operation built into OpenCV combined with the contour area threshold to quickly remove fine noise points in the image, and can better preserve large contour information with a faster processing speed.

[0067] If clear contour information can be obtained, first perform the minimum circumscribed circle extraction operation on each contour to obtain the center coordinates of the minimum circumscribed circle radius. According to the position of the center coordinates relative to the image center (320, 240), the contours are classified into left and right. Taking the left and right as an example (the same applies to the vertical direction), the distance from the center of the minimum circumscribed circle of each contour to the image center is calculated, and the data is normalized. The normalization formula is:

[0068]

[0069] Combined with the real intestinal images, taking the left side as an example, the contour information is divided into four categories, specifically, the number of contours on the left side of the image center (N) is greater than or equal to the right side (P), N = 3 / 4, N = 2, N = 1, N = P.

[0070] When N=3 or 4, combined with the AHP hierarchical analysis method, different weights are assigned according to the distance from the center of the minimum circumscribed circle of each contour to the center of the image. Because the contour closer to the center of the image is closer to the camera, the following judgment matrix is ​​manually given. When N=3, each contour is divided into three categories of Close, Mid, and Far according to the distance from the center of the image, giving a 3×3 numerical matrix. When N=4, each contour is divided into four categories of Close, Mid, Far, and Farthest according to the distance from the center of the image, giving a 4×4 numerical matrix. The corresponding weights are calculated using the following formulas (2) and (3). The data in the i-th row and j-th column of the matrix corresponds to a in the formula ij , the values ​​in the matrix are given manually to calculate the weight corresponding to each contour. From the column point of view, the size of the value represents the importance:

[0071]

[0072]

[0073]

[0074] ω 0 i is a transition parameter used to calculate the contour weight; N = 3 or 4; a ij Represents the parameter of row i and column j in the judgment matrix, 1≤i≤N, 1≤j≤N. The judgment matrix is ​​a 3×3 or 4×4 numerical matrix given based on experience.

[0075]

[0076] ω i It refers to the calculated weight of a certain contour, 1≤i≤N.

[0077] After obtaining the weights corresponding to each profile, the steering determination coefficient E is calculated using formula (4):

[0078]

[0079] where ω represents the weight calculated by (2) and (3), left / right is the normalized left / right lateral error, RL / RR is the left / right minimum circumscribed radius after mapping, Δ represents the suppression parameter (when p>0, 1-Δ=10Δ), and N / P is the number of open contours on the left / right side.

[0080] When N=2, the steering determination coefficient E is calculated by formula (5):

[0081]

[0082] When N=1, only an open contour information can be extracted. At this time, the turning direction is determined by calculating the opening direction of the contour, that is, the position of the center of the contour's minimum circumscribed circle relative to the contour concave point (the tangent point of the contour and its minimum circumscribed circle), and E is set to a medium-amplitude turning.

[0083] When N=P, the steering coefficient E is determined by detecting the number of left or right contours. If the difference between the number of contours on one side and the number on the other side is greater than 2, it is set as a medium-amplitude steering, otherwise it is set as a small-amplitude steering.

[0084] In the entire steering control process, 0 is used to represent a straight line, ω min Indicates extremely small amplitude (1°), ω small Indicates small amplitude (2°), ω mid Indicates medium amplitude (3°), ω big Indicates large amplitude (5°), ω max Indicates the maximum amplitude (10°) and ω extreme Indicates the maximum amplitude (20°). The positive and negative signs indicate the left and right directions. The control amplitude in the clear contour stage is 0, ω min ,ω small and ω mid .

[0085] After the steering determination coefficient E is determined, the steering angle is determined according to (6). Figure 6 It is the result of extracting and judging the intestinal contour.

[0086]

[0087] If clear contour information cannot be extracted, then according to the existing steering control parameters and contour state parameters,

[0088] The first step is to read the steering control parameters and contour state parameters before this frame image, and take the sum of the five adjacent parameters. If the sum of the contour state parameters is greater than or equal to the threshold, the steering is slightly turned by the sum of the steering control parameters, and the positive and negative indicate the direction. If the sum of the contour state parameters is less than the threshold, the triple control strategy is triggered. The first step is to control the magnitude of the steering amplitude according to the sum of the steering control parameters (ω big ), positive and negative indicate direction.

[0089] The second step is to take a large turn (ω) in a different direction from the first step if a clear outline cannot be captured after the first step is completed. max ).

[0090] The third step is to take a larger turn in the same direction as the first step if a clear outline cannot be captured after the second step. extreme ). In every five consecutive operations, only one step using this triple control strategy will occur.

[0091] In order to verify the processing efficiency of the endoscope autonomous navigation implementation method provided in the above embodiment for real-time image acquisition, the above implementation method is specifically applied to endoscope acquisition images in the human intestinal environment and the images are processed according to the above navigation implementation method. Figure 7 As shown in the figure, in the image collection and verification, 96 images were collected to test the verification speed of each image, 90.62% of which were within 25ms, and the average time taken for all images was 22.072ms.

[0092] Therefore, the image processing speed of the above-mentioned implementation method can generally be controlled within 20-25ms, which fully meets the requirements of real-time processing, can overcome the impact of image errors on endoscope autonomous navigation in complex environments, and improve the efficiency and accuracy of endoscope autonomous navigation.

Claims

1. A method for implementing endoscope autonomous navigation, characterized in that: The intestinal images are acquired through the camera at the end of the endoscope and transmitted to the computer for image preprocessing; Perform contour extraction on the preprocessed image to obtain the image contour; If a clear contour can be obtained, first set the contour area threshold to determine the closed contour and the open contour. If a closed contour exists, the endoscope will go straight without turning; If a clear outline cannot be obtained, a triple control strategy is designed to control the steering of the endoscope based on the existing parameters; The triple control strategy includes: Read the previously stored steering control parameters and profile state parameters, take the sum of the five adjacent parameters, if the sum of the profile state parameters is greater than or equal to the threshold, set the steering amplitude according to the sum of the steering control parameters, and the positive and negative numbers indicate the direction. If the sum of the profile state parameters is less than the threshold, the triple control strategy is triggered. The first step is to control the size of the steering amplitude according to the size of the sum of the steering control parameters, and the positive and negative numbers indicate the direction. The second step is to take a turn that is different from the first step and larger than the first step if a clear outline cannot be captured after the first step is completed. The third step is to take a turn in the same direction as the first step and with a larger amplitude than the second step if a clear outline cannot be captured after the second step is executed; When only open contours exist, the contours are classified, and the contours of each frame image are divided into left contours and right contours. When the number of contours N on one side is greater than the number of contours P on the other side, this includes the cases where N=3 or 4, N=2 or N=1; When N=3 or 4, the AHP hierarchical analysis method is used to assign corresponding weights to each contour based on the distance from the contour to the center of the image, and the turning determination coefficient E is obtained through matrix operation; when N=3 or 4, the corresponding weights are calculated ω 0 i is a transition parameter used to calculate the contour weight; N = 3 or 4; a ij Represents the parameter of row i and column j in the judgment matrix, 1≤i≤N, 1≤j≤N; ω i It refers to the weight of a certain contour calculated, 1≤i≤N; in After obtaining the weights corresponding to each contour, the steering determination coefficient E is calculated by the formula: Where left / right is the normalized left / right lateral error, RL / RR is the left / right minimum circumscribed radius after mapping, Δ represents the suppression parameter, and N / P is the number of open contours on the left / right side; When N=2, calculate the steering determination coefficient E: When N=1, set E to medium-amplitude steering; The steering angle ω is: When N=P, if the difference between the number of contours on one side and the number on the other side is greater than 2, it is set as a medium-amplitude turn, otherwise it is set as a small-amplitude turn; When N=2, the steering determination coefficient E is obtained through matrix operation; When N=1, the steering determination coefficient E is determined by detecting the contour opening direction; When N=P, the steering determination coefficient E is determined by detecting the number of leftward or rightward contours; The steering determination coefficient E is used to determine the steering angle of the endoscope.

2. The method for realizing endoscope autonomous navigation according to claim 1, characterized in that: The image preprocessing method comprises: Convert each frame into a grayscale image; Use contrast-constrained adaptive histogram equalization method to enhance the contrast of grayscale images; A 9×9 Gaussian kernel is used to perform Gaussian filtering on the image.

3. The method for realizing endoscope autonomous navigation according to claim 1 or 2, characterized in that: After the image preprocessing is completed, the method for extracting the image contour includes: The adaptive threshold of the mean method is used for the Gaussian filtered image to remove the influence of the background color; After the background color is initially removed, a 3×3 convolution kernel is used to perform a morphological closing operation on the image to supplement the missing contour information; Use the built-in findContours function of opencv to extract the contours of the completed image, calculate the area of ​​each contour, and fill the smaller contours by setting the threshold; After padding is completed, a 3×3 convolution kernel is used again for morphological closing operation; After the morphological operation is completed, the findContours contour extraction operation is performed; Finally, the minimum circumscribed circle is extracted for each contour to obtain the minimum circumscribed circle radius and center.

4. The method for realizing endoscope autonomous navigation according to claim 3, characterized in that: The difference in the contour information lies in the position of the center point of the minimum circumscribed circle of each contour relative to the center point of the image.

5. The method for realizing endoscope autonomous navigation according to claim 1, characterized in that: The steering control parameter is defined as: after processing each frame of the image, the steering control result is obtained according to the image information or the stored data, and the positive and negative values ​​represent the direction.

6. The method for realizing endoscope autonomous navigation according to claim 1, characterized in that: The definition of the contour state parameter is: when a clear contour can be extracted, the contour state parameter corresponding to the frame is 1, otherwise it is 0.

7. The method for realizing endoscope autonomous navigation according to claim 1, characterized in that: The steering amplitude refers to the angle of rotation of the endoscope, and is determined according to a steering determination coefficient during the analysis and judgment process. The steering determination coefficient is calculated by analyzing the profile information or the steering control parameters.

8. An endoscope device using the implementation method as claimed in any one of claims 1 to 7, characterized in that: The device includes an endoscope, a straight driving module, a steering control module, and a user interface; the straight driving module drives the endoscope to move straight through the friction of two servos; and the steering control module is connected to an endoscope steering knob through the servo to control the steering of the endoscope.

Images