Endoscope abnormal bending detection method and system, terminal and storage medium

Abstract

The invention belongs to the technical field of medical robots, and discloses an endoscope abnormal bending detection method and system, a terminal and a storage medium, and the method comprises the steps: obtaining relative motion data between the far end of an endoscope and an instrument box; obtaining an endoscope image sequence, and analyzing according to the endoscope image sequence to obtain an endoscope motion state; propulsive force change data detected by a force sensor are obtained; and performing fusion analysis according to the relative motion data, the endoscope motion state and the propulsive force change data to obtain an abnormal bending detection result of the endoscope. According to the method, the abnormal bending condition of the endoscope is accurately recognized through real-time data monitoring and analysis in combination with multi-modal information of displacement, vision and propulsive force sensors.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical robots, and in particular to an endoscope abnormal bending detection method, system, terminal and storage medium. BACKGROUND

[0002] In a bronchoscope surgery robot system, the movement system of an endoscope usually involves a feed slide, a drive box and an endoscope, etc. Among them, the endoscope is composed of an instrument box, an insertion tube and a snake bone, etc. The drive box controls the advance and retreat of the snake bone through a pushing force to realize the precise navigation of the endoscope in a complex path in the body.

[0003] When the endoscope moves in a winding bronchus, the applied pushing force may not be enough to overcome the friction between the snake bone and the bronchus wall, causing the distal end of the endoscope to stop moving. When the endoscope appears abnormal bending, the insertion tube can cause excessive contact force on the tracheal wall, thereby causing bronchial tissue damage or even rupture. In the current surgical robot system, the pushing force and obstacle feedback of the endoscope cannot be perceived, and there is a lack of real-time abnormal monitoring means.

[0004] The existing endoscope abnormal bending detection method mainly compares the displacement difference between the distal end and the proximal end of the endoscope. If the displacement difference is greater than a threshold value, it is considered that the insertion tube has abnormal bending. However, the detection result of this method is not accurate enough. On the one hand, the movement of the distal end of the endoscope is a combination of feed and snake bone bending, and the displacement change of the distal end of the endoscope caused by the bending of the snake bone will affect the judgment. On the other hand, it is a big problem to choose the appropriate threshold value in different scenarios. Although the use of a shape sensor can simplify this problem, the shape sensor is generally high in price, which limits the application scenarios.

[0005] In summary, the existing endoscope abnormal bending detection method is relatively simple, which usually compares the displacement difference between the instrument box and the distal end of the endoscope to determine whether the insertion tube has abnormal bending. This method has insufficient accuracy and faces problems such as difficulty in threshold selection and insufficient sensitivity.

[0006] Therefore, the prior art still needs to be improved. SUMMARY

[0007] The technical problem to be solved by the present application is that, in view of the defects of the prior art, the present application provides an endoscope abnormal bending detection method, system, terminal and storage medium to solve the problem of poor detection accuracy of the existing endoscope abnormal bending detection method.

[0008] The technical solution adopted by the present application to solve the technical problem is as follows: In a first aspect, the present application provides an endoscope abnormal bending detection method, comprising: obtaining relative movement data between the distal end of the endoscope and the instrument box; acquire an endoscope image sequence, and analyze an endoscope motion state according to the endoscope image sequence; acquire push force change data detected by a force sensor; perform fusion analysis according to the relative motion data, the endoscope motion state, and the push force change data to obtain an endoscope abnormal bending detection result.

[0009] In an implementation manner, the acquiring of the relative motion data between the endoscope distal end and the instrument box comprises: acquire and record position data of the endoscope distal end; perform dynamic processing and analysis on the acquired position data to obtain the relative motion data between the endoscope distal end and the instrument box.

[0010] In an implementation manner, the performing of the dynamic processing and analysis on the acquired position data to obtain the relative motion data between the endoscope distal end and the instrument box comprises: perform filtering processing on the continuously acquired position data to obtain a continuous displacement trajectory curve after noise reduction; calculate a displacement difference between the endoscope distal end and the instrument box and a displacement difference change rate between the endoscope distal end and the instrument box according to the continuous displacement trajectory curve to obtain the relative motion data between the endoscope distal end and the instrument box.

[0011] In an implementation manner, the acquiring of the endoscope image sequence and the analyzing of the endoscope motion state according to the endoscope image sequence comprise: acquire the endoscope image sequence, and calculate optical flow vectors of each pixel point in each frame of image in the endoscope image sequence by using an optical flow method; perform statistical analysis on the optical flow vectors of all pixels in each frame of image to obtain an average optical flow amplitude of each frame of image; determine an optical flow vector change trend between two or more continuous frames of endoscope image according to the average optical flow amplitudes of the images; determine the endoscope motion state according to the optical flow vector change trend; wherein the endoscope motion state comprises a normal pushing state and a motion stagnation state.

[0012] In an implementation manner, the acquiring of the push force change data detected by the force sensor comprises: acquire force sensor data connected to the bottom of the driving box to obtain push force data; perform filtering processing on the push force of the continuous sampling points in the push force data, and calculate an average force value at each time according to the filtered push force data; Calculate a difference value of the average force values of adjacent time points, and calculate a change speed of the propulsion force according to the difference value and a sampling interval to obtain the propulsion force change data.

[0013] In an implementation manner, the fusion analysis according to the relative motion data, the endoscope motion state and the propulsion force change data to obtain the endoscope abnormal bending detection result comprises: Abnormal bending detection is performed according to displacement difference and displacement difference change rate of consecutive N sampling points in the relative motion data to obtain a position-velocity difference detection result; wherein N is an integer greater than 1; Propulsion resistance detection is performed according to the propulsion force change data to obtain a propulsion resistance detection result; The position-velocity difference detection result, the endoscope motion state and the propulsion resistance detection result are fused by using a preset module weight coefficient to obtain the endoscope abnormal bending detection result.

[0014] In an implementation manner, the abnormal bending detection according to the displacement difference and the displacement difference change rate of the consecutive N sampling points in the relative motion data to obtain the position-velocity difference detection result comprises: The probability of abnormal bending occurring at each bronchial branch junction is obtained based on prior knowledge, and a dynamic detection threshold corresponding to the motion process of the endoscope distal end is determined according to the obtained probability; wherein the dynamic detection threshold comprises a displacement difference threshold and a displacement difference change rate threshold; It is judged whether the displacement difference and the displacement difference change rate of the consecutive N sampling points are higher than the corresponding thresholds; When the displacement difference and the displacement difference change rate of the consecutive N sampling points are lower than the corresponding thresholds, it is determined that the endoscope has abnormal bending, and the position-velocity difference detection result is obtained.

[0015] In an implementation manner, the probability of abnormal bending occurring at each bronchial branch junction is obtained based on prior knowledge, and a dynamic detection threshold corresponding to the motion process of the endoscope distal end is determined according to the obtained probability, comprising: The probability of abnormal bending occurring at each bronchial branch junction in the path L is obtained, and the overall abnormal bending probability when passing through the path L is calculated according to the obtained probability accumulation; The displacement difference threshold and the displacement difference change rate threshold corresponding to the motion process of the endoscope distal end are obtained by dynamically adjusting the displacement difference threshold and the displacement difference change rate threshold according to the overall abnormal bending probability.

[0016] In an implementation manner, the propulsion resistance detection according to the propulsion force change data to obtain the propulsion resistance detection result comprises: Obtaining mechanical statistical data and prior knowledge under normal propulsion conditions, determining double threshold conditions of the propulsion force and the propulsion force change speed according to the mechanical statistical data and the prior knowledge; According to the double threshold conditions, the propulsion resistance detection is performed by using a threshold judgment mechanism of continuous sampling points to obtain the propulsion resistance detection result.

[0017] In an implementation manner, the propulsion resistance detection is performed by using the threshold judgment mechanism of the continuous sampling points according to the double threshold conditions to obtain the propulsion resistance detection result, including: According to the threshold judgment mechanism of the continuous sampling points, it is judged whether the propulsion force and the propulsion force change speed of the continuous N sampling points in the propulsion force change data satisfy the double threshold conditions; When the propulsion force and the propulsion force change speed of the continuous N sampling points satisfy the double threshold conditions, it is determined that the endoscope is abnormally bent, and the propulsion resistance detection result is obtained.

[0018] In a second aspect, the present application provides an endoscope abnormal bending detection system, including: A distal-proximal motion difference detection module is configured to obtain relative motion data between a distal end of an endoscope and an instrument box. An image-based distal end motion detection module is configured to obtain an endoscope image sequence and analyze an endoscope motion state according to the endoscope image sequence. A propulsion resistance detection module is configured to obtain propulsion force change data detected by a force sensor. A fusion analysis module is configured to perform fusion analysis according to the relative motion data, the endoscope motion state and the propulsion force change data to obtain an endoscope abnormal bending detection result.

[0019] In a third aspect, the present application provides a terminal including a processor and a memory, wherein the memory stores an endoscope abnormal bending detection program, and the endoscope abnormal bending detection program is configured to perform operations of the endoscope abnormal bending detection method according to the first aspect when executed by the processor.

[0020] In a fourth aspect, the present application further provides a computer readable storage medium storing an endoscope abnormal bending detection program, and the endoscope abnormal bending detection program is configured to perform operations of the endoscope abnormal bending detection method according to the first aspect when executed by a processor.

[0021] The present application has the following effects by using the above technical solutions: The application monitors the relative motion data between the distal end of the endoscope and the instrument box through a sensor, processes the endoscope image sequence by using an optical flow analysis method, analyzes the endoscope motion state in real time according to the endoscope image sequence, detects the change data of the pushing force through a force sensor, and intelligently evaluates whether the endoscope is abnormally bent in a multi-modal information fusion analysis manner. The application combines the multi-modal information of displacement, vision and pushing force sensors, realizes real-time data monitoring and analysis, and effectively improves the sensitivity and accuracy of endoscope detection. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained according to the structures shown in the drawings without creative labor.

[0023] Figure 1 is a flow chart of the endoscope abnormal bending detection method in the present application.

[0024] Figure 2 is a bronchoscope motion schematic diagram of the bronchoscope surgery robot in the present application.

[0025] Figure 3 is a motion state schematic diagram of the endoscope in the bronchus in the present application.

[0026] Figure 4 is a displacement schematic diagram of the driving box in the present application.

[0027] Figure 5 is a displacement schematic diagram of the distal end of the endoscope in the present application.

[0028] Figure 6 is a threshold value schematic diagram on different paths in the present application.

[0029] Figure 7 is an endoscope pushing resistance detection schematic diagram in the present application.

[0030] Figure 8 is a functional principle diagram of a terminal in one implementation manner of the present application.

[0031] In the figure: 1, feed slide; 2, driving box; 3, endoscope; 4, instrument box (proximal end of endoscope); 5, insertion tube; 6, snake bone; 7, distal end of endoscope; 8, bronchus; 10, force sensor.

[0032] The purpose implementation, functional characteristics and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0033] In order to make the objects, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not intended to limit the present application.

[0034] Exemplary method The existing endoscope abnormal bending detection method mainly compares the displacement difference between the distal end and the proximal end of the endoscope. If the displacement difference is greater than a threshold value, it is considered that the insertion tube is abnormally bent. However, the detection result of this method is not accurate enough. On the one hand, the movement of the distal end of the endoscope is a composite movement of feeding and snake bending, and the displacement change of the distal end of the endoscope caused by snake bending will affect the judgment. On the other hand, how to choose a suitable threshold value in different scenes is a big problem. Although the use of a shape sensor can simplify this problem, the shape sensor is generally more expensive, which limits the application scenarios.

[0035] In summary, the existing endoscope abnormal bending detection method is relatively simple, which usually compares the displacement difference between the instrument box and the distal end of the endoscope to determine whether the insertion tube is abnormally bent. This method has insufficient accuracy and faces problems such as difficulty in threshold selection and insufficient sensitivity.

[0036] In view of the above technical problems, the present application provides an endoscope abnormal bending detection method, which comprises the following steps:

[0037] As shown in Figure 1 The present application provides an endoscope abnormal bending detection method, which comprises the following steps: Step S100, acquiring relative motion data between the distal end of the endoscope and the instrument box.

[0038] In this embodiment, the endoscope abnormal bending detection method is applied to the endoscope abnormal bending detection scene of a bronchoscope surgery robot system; or applied to the abnormal bending detection scene of other detection systems, such as the bending detection scene of an industrial endoscope, for abnormal bending detection in complex environments such as deep cavities and narrow welds.

[0039] As an example, as shown in Figure 2As shown, in the bronchoscope surgery robot system, the structure responsible for the movement of the endoscope (endoscope) generally includes a feeding slide 1, a drive box 2, and an endoscope 3. Among them, the endoscope 3 is composed of components such as instrument box 4, insertion tube 5, and snake bone 6. The active movement of the endoscope 3 includes two parts, one is the feeding movement provided by the feeding slide 1, and the other is the bending movement provided by the drive box 2 to the snake bone 6. The drive box 2 can realize the feeding movement along the feeding slide 1.

[0040] The working principle of the above bronchoscope surgery robot system is: in the use process, the drive box 2 applies a pushing force or a pulling force to the instrument box 4 to control the advance and retreat of the snake bone 6. However, when the endoscope 3 penetrates into the lesion along the tortuous path of the bronchus, the force transmission path of the endoscope 3 is a complex curve. When the friction between the snake bone 6 and the bronchial wall is greater than the pushing force transmitted to the snake bone 6, the endoscope distal end 7 stops moving, and the insertion tube 5 appears an unexpected abnormal bending. The normal bending of the insertion tube 5 is shown in Figure 3 (a) as a comparison, the abnormal bending of the insertion tube 5 is shown in Figure 3 (b); the abnormal bending of the insertion tube 5 generates a larger contact force on the tracheal wall, which may cause bronchial 8 tissue damage or even rupture. If the endoscope is still pushed forward after the insertion tube 5 appears abnormal bending, it may cause danger.

[0041] In this embodiment, the endoscope abnormal bending detection method is an endoscope abnormal bending detection scheme based on multi-information fusion, which mainly judges whether the endoscope abnormal bending occurs through the following ways: 1) Use the displacement difference and displacement difference change rate between the endoscope distal end and the instrument box to judge whether the endoscope abnormal bending occurs; 2) Use the visual information of the endoscope distal end to judge whether the endoscope abnormal bending occurs; if the adjacent frame changes little, it means that the lens is no longer moving, so as to judge whether the endoscope abnormal bending occurs; 3) Use the force sensor to monitor the pushing force to judge whether the endoscope abnormal bending occurs; if the force size and force change rate reach the threshold, it is determined that the endoscope abnormal bending occurs.

[0042] As the first way of the endoscope abnormal bending detection scheme based on multi-information fusion, this embodiment is called distal-end-proximal-end motion difference detection method. This detection method mainly obtains the relative motion data between the endoscope distal end and the instrument box, including the displacement difference and the displacement difference change rate, to further judge whether the endoscope abnormal bending occurs. Among them, the relative motion data between the endoscope distal end and the instrument box refers to the displacement difference between the endoscope distal end and the instrument box caused by motion, and the displacement difference change rate between the endoscope distal end and the instrument box.

[0043] Specifically, in one implementation manner of the embodiment, the step S100 includes the following steps: Step S101, collect and record the position data of the distal end of the endoscope; Step S102, dynamically process and analyze the collected position data to obtain the relative motion data between the distal end of the endoscope and the instrument box.

[0044] In this embodiment, the distal end of the endoscope is equipped with a high-precision position sensor, which can collect and record the three-dimensional spatial position of the distal end of the endoscope in real time. To achieve timely and accurate detection of the abnormal bending state of the endoscope, continuous and dynamic monitoring and analysis of the collected position data are needed to obtain the relative motion data between the distal end of the endoscope and the instrument box.

[0045] Since the collected raw position data is limited by sensor measurement error, environmental interference and other factors, it usually contains certain high-frequency noise and occasional abnormal fluctuations. If the untreated data is directly used for judgment, it is easy to cause false positives and false negatives. Therefore, further preprocessing of the collected position data is needed.

[0046] In one implementation of the embodiment, step S102 includes the following steps: Step S1021, filtering the continuously collected position data to obtain a denoised continuous displacement trajectory curve; Step S1022, calculating the displacement difference between the distal end of the endoscope and the instrument box according to the continuous displacement trajectory curve, and calculating the displacement difference change rate between the distal end of the endoscope and the instrument box to obtain the relative motion data between the distal end of the endoscope and the instrument box.

[0047] Specifically, when preprocessing the collected position data, a set window M (i.e., window size M) is used to smooth the continuously collected position data to obtain a denoised continuous displacement trajectory curve. The filtering method can use any one or combination of sliding average filtering, median filtering and adaptive mean filtering techniques.

[0048] As an example, in this embodiment, adaptive mean filtering is used. For the continuously collected position data, the change amplitude (i.e., local standard deviation) in the neighborhood of each data point is calculated, and then the window M size is adjusted according to the change amplitude. For example, when the change is small, the window M is reduced to preserve the details of the trajectory; when the change is large, the window M is increased to enhance the smoothing effect and noise suppression ability.

[0049] In this embodiment, after obtaining the denoised continuous displacement trajectory curve, the displacement difference between the distal end of the endoscope and the instrument box is calculated according to the continuous displacement trajectory curve, and the displacement difference change rate between the distal end of the endoscope and the instrument box is calculated. Since the instrument box is directly driven by the driving box, in this embodiment, the displacement difference between the distal end of the endoscope and the instrument box can be determined by calculating the displacement of the driving box and the displacement of the distal end of the endoscope. The displacement of the driving box and the displacement of the distal end of the endoscope are shown in Figure 4 and Figure 5 respectively.

[0050] In this embodiment, the difference between the displacement of the driving box and the displacement of the distal end of the endoscope in the same time period is taken as The difference is the displacement difference between the distal end of the endoscope and the instrument box, and the difference is calculated in the Euclidean distance; and the change speed (i.e., the change rate) of the displacement difference between the distal end of the endoscope and the instrument box between consecutive time points is taken as In this way, the relative motion data between the distal end of the endoscope and the instrument box is obtained, so that subsequent abnormal bending detection can be performed according to the relative motion data, and a position-velocity difference detection result is obtained.

[0051] As shown in Figure 1 , the present embodiment provides an endoscope abnormal bending detection method, which comprises the following steps: Step S200, acquiring an endoscope image sequence, and analyzing the endoscope motion state according to the endoscope image sequence.

[0052] In this embodiment, as a second way of the endoscope abnormal bending detection scheme based on multi-information fusion, this embodiment is called endoscope image detection method. This method performs image detection through an image-based distal end motion detection module, and uses an optical flow method to detect whether the endoscope is in a normal advancing state or a motion stagnation state. The optical flow method is an important technology in the field of computer vision, and is mainly used to estimate the motion state of objects in an image sequence. The basic principle is that the image brightness is assumed to remain constant in a short time, and the brightness change of adjacent pixels in consecutive two frames of images is compared to calculate the corresponding motion vector, so as to determine whether the object is in a motion state. The optical flow method can effectively capture subtle motion information in the image, and is suitable for dynamic scene analysis, target tracking and motion detection and other applications.

[0053] Specifically, in one implementation manner of this embodiment, step S200 comprises the following steps: Step S201, acquiring the endoscope image sequence, and calculating the optical flow vector of each pixel point in each frame of image in the endoscope image sequence by the optical flow method. Step S202, statistical analysis is performed on the optical flow vectors of all pixels in each frame of image to obtain the average optical flow amplitude of each frame of image; Step S203, the optical flow vector variation trend between two or more consecutive endoscope images is determined according to the average optical flow amplitudes of the images; Step S204, the endoscope motion state is determined according to the optical flow vector variation trend; wherein the endoscope motion state includes: normal advancing state and motion stagnation state.

[0054] In this embodiment, after obtaining the endoscope image sequence, the horizontal component and the vertical component of each pixel point, i.e. the optical flow vector, are calculated by combining the brightness change of adjacent frames with the optical flow constraint equation. Specifically, the optical flow vector of each pixel point in each frame of image in the endoscope image sequence is calculated by the optical flow method; then, statistical analysis is performed on the optical flow vectors of all pixels in each frame of image, i.e. all pixels in the whole image are counted to calculate the average optical flow size to reflect the overall motion intensity and obtain the motion mode of the whole image. This is particularly important for endoscope image analysis because the motion of the distal end of the endoscope directly affects the change characteristics of the image sequence.

[0055] As an example, in actual application, the motion state of the distal end image of the endoscope can be judged by using the optical flow method to detect the motion state of the distal end image of the endoscope to determine whether there is motion stagnation or abnormality. This is because there will be obvious optical flow change between two or more consecutive frames in the normal advancing process; on the contrary, if the optical flow vector change is very small or tends to zero, it usually means that the distal end position of the endoscope does not move effectively, and there may be abnormal bending or blockage phenomenon. By setting a reasonable optical flow threshold, the abnormal state can be discovered in time. Specifically, the determination of the optical flow threshold can be achieved by collecting a large amount of optical flow data of the endoscope under normal advancing and abnormal bending state, and counting the optical flow vector variation range under normal state to obtain the optical flow threshold.

[0056] Based on the result analysis of the optical flow detection, the system can dynamically judge the actual motion of the endoscope and take corresponding response measures accordingly.

[0057] As shown in Figure 1 The embodiment of the present application provides an endoscope abnormal bending detection method, which includes the following steps: Step S300, acquiring the advancing force change data detected by the force sensor.

[0058] In this embodiment, as the third way of the endoscope abnormal bending detection scheme based on multi-information fusion, the method is called advancing resistance detection method in this embodiment, which first needs to collect real-time advancing force change data.

[0059] Specifically, in one implementation manner of this embodiment, step S300 includes the following steps: Step S301, acquiring force sensor data connected to the bottom of the driving box to obtain propulsion force data; Step S302, filtering the propulsion force of the continuous sampling points in the propulsion force data, and calculating the average force value at each time according to the filtered propulsion force data; Step S303, calculating the difference value of the average force values of adjacent time points, and calculating the change speed of the propulsion force according to the difference value and the sampling interval to obtain the propulsion force change data.

[0060] In this embodiment, as shown in Figure 7 The force sensor 10 is connected to the bottom of the driving box 2, and the force sensor 10 can monitor the propulsion force during the endoscope propulsion. In order to ensure the accuracy and stability of the propulsion force signal, real-time propulsion force data needs to be collected first.

[0061] Because the original data of the force sensor has certain noise and mutations, directly using the original data can easily lead to misjudgment, so it is necessary to filter the collected propulsion force data. Common filtering methods include sliding average filtering and low-pass filtering, which can effectively suppress high-frequency noise and short-time abnormal fluctuations, thereby extracting a relatively smooth and representative force value curve.

[0062] Specifically, a set window N (the window size is N, the number of sampling points in which is N, and N is an integer greater than 1) is used to perform a sliding average operation on the propulsion force of the continuous sampling points, and the average force value at each time is calculated to reduce the influence of random noise. Subsequently, by calculating the difference value of the sliding average values of adjacent time points, the calculated difference value is divided by the sampling interval , the change speed of the propulsion force (that is, the propulsion force change data) can be obtained, which provides accurate and stable parameter basis for subsequent threshold judgment.

[0063] As shown in Figure 1 , the embodiment of the present application provides an endoscope abnormal bending detection method, which comprises the following steps: Step S400, performing fusion analysis according to the relative motion data, the endoscope motion state, and the propulsion force change data to obtain an endoscope abnormal bending detection result.

[0064] In this embodiment, the relative motion data, the endoscope motion state, and the propulsion force change data are combined, and fusion analysis is performed using a preset module weight coefficient, so that the endoscope abnormal bending detection result can be quickly obtained; wherein the preset module weight coefficient is a weight obtained according to corresponding experimental data and experience (that is, prior knowledge).

[0065] Specifically, in one implementation manner of the embodiment, step S400 comprises the following steps: Step S401, according to the displacement difference and the displacement difference change rate of the continuous N sampling points in the relative motion data, the abnormal bending detection is performed to obtain a position-velocity difference detection result.

[0066] In an implementation manner of the embodiment, step S401 includes the following steps. Step S4011, a probability of abnormal bending occurring at each bronchial branch intersection is obtained based on prior knowledge, and a corresponding dynamic detection threshold of the endoscope distal end in the movement process is determined according to the obtained probability; wherein the dynamic detection threshold includes a displacement difference threshold and a displacement difference change rate threshold.

[0067] In an implementation manner of the embodiment, the probability of abnormal bending occurring at each bronchial branch intersection is obtained, and the corresponding dynamic detection threshold of the endoscope distal end in the movement process is determined according to the obtained probability, including: obtaining the probability of abnormal bending occurring at each bronchial branch intersection in the path L, and calculating the overall abnormal bending probability when passing through the path L according to the obtained probability accumulation; the displacement difference threshold and the displacement difference change rate threshold are dynamically adjusted according to the overall abnormal bending probability to obtain the corresponding dynamic detection threshold of the endoscope distal end in the movement process.

[0068] Step S4012, whether the displacement difference and the displacement difference change rate of the continuous N sampling points are higher than the corresponding threshold value or not is judged. Step S4013, when the displacement difference and the displacement difference change rate of the continuous N sampling points are lower than the corresponding threshold value, it is determined that the endoscope has abnormal bending, and the position-velocity difference detection result is obtained.

[0069] In the embodiment, according to the displacement difference and the displacement difference change rate between the endoscope distal end and the instrument box obtained in step S100, the remote-proximal motion difference detection module is used to perform abnormal bending detection on the continuous N sampling points to obtain the position-velocity difference detection result.

[0070] In the process of abnormal bending detection, when the displacement difference value at a certain moment satisfies , and the displacement difference change rate of the endoscope distal end satisfies , it indicates that the endoscope distal end may be in a motion stagnation state, which implies the risk of abnormal bending or jamming. , are respectively a displacement difference threshold and a displacement difference change rate threshold, and the two thresholds are used to distinguish between normal or abnormal bending of the endoscope.

[0071] In the embodiment, the endoscope can abnormally bend at any one bronchial branch junction, and the more tortuous the path is, the more likely the abnormal bending occurs. Considering the motion characteristics of the endoscope at different bronchial branches and the differences in anatomical structure, the probability of abnormal bending at each bronchial branch junction is denoted as , as shown in the formula (1), wherein, Figure 6 is the number of the bronchial branch junction, and the corresponding bronchial branch path is , is the angle of the branch pipeline, is the size of the branch pipe diameter. Based on the cumulative abnormal probability of the multiple branch openings in the path length L, the overall abnormal bending probability when passing through the path L can be calculated as ; By counting the overall abnormal bending probability accumulated on the endoscope distal path , the threshold value at the current time is dynamically calculated and adjusted, and the threshold value : ; ; wherein, , are the displacement difference threshold value and the displacement difference change rate threshold value of the abnormal bending obtained by multiple experiments at a single branch junction.

[0072] Although single-point determination can prompt abnormalities, it is easy to be disturbed by accidental measurement errors or noise signals, thereby causing false positives. To enhance the robustness of detection and the stability of judgment, a threshold determination mechanism of consecutive sampling points is introduced in the embodiment: only when the displacement difference and the displacement difference change rate of the consecutive N sampling points (the interval between adjacent sampling points is ) of the endoscope distal end are both lower than the corresponding dynamic detection threshold value, the occurrence of abnormal bending is confirmed. The specific judgment conditions are as follows: ; The threshold determination mechanism of the consecutive sampling points can further enhance the accuracy and anti-interference ability of the abnormal bending state recognition in combination with auxiliary indicators such as acceleration, thereby effectively avoiding false judgments caused by accidental abnormal signal interference, and ensuring the safety and accuracy of endoscope operation.

[0073] Specifically, in one implementation manner of the embodiment, the step S400 further includes the following steps: Step S402: performing propelling resistance detection according to the propelling force change data to obtain a propelling resistance detection result; In one implementation manner of the embodiment, the step S402 includes the following steps: ​In step S4021, the mechanical statistical data under normal propulsion condition and prior knowledge are acquired, and a double threshold condition of the propulsion force and the propulsion force change rate is determined according to the mechanical statistical data and the prior knowledge. In step S4022, the propulsion resistance detection is performed by using the threshold judgment mechanism of the continuous sampling points according to the double threshold condition, and the propulsion resistance detection result is obtained.

[0074] In an implementation manner of the embodiment, the propulsion resistance detection is performed by using the threshold judgment mechanism of the continuous sampling points according to the double threshold condition, and the propulsion resistance detection result is obtained, including: judging whether the propulsion force and the propulsion force change rate of the continuous N sampling points in the propulsion force change data satisfy the double threshold condition according to the threshold judgment mechanism of the continuous sampling points; when the propulsion force and the propulsion force change rate of the continuous N sampling points satisfy the double threshold condition, it is determined that the endoscope is abnormally bent, and the propulsion resistance detection result is obtained.

[0075] In the embodiment, the propulsion resistance detection is performed on N sampling points by the propulsion resistance detection module for the propulsion force change data obtained in step S300, and the propulsion resistance detection result is obtained.

[0076] In the process of performing the propulsion resistance detection, if the propulsion force at a single time satisfies , and the propulsion force change rate satisfies , it is regarded as a potential abnormal state. Herein, and are threshold values set in advance according to the mechanical data statistics under normal propulsion condition and clinical experience (i.e., prior knowledge), and are used to distinguish the normal and abnormal propulsion states. Although a single sampling can prompt an abnormality, the single-point judgment condition is easily disturbed by noise or transient abnormal signals, and may produce false positives.

[0077] To improve the robustness of detection and the reliability of judgment, the threshold judgment mechanism of the continuous sampling points is introduced in the embodiment. If the double threshold condition of the propulsion force and the propulsion force change rate is satisfied in the continuous N sampling points, it is determined that the endoscope is abnormally bent. That is, ; Among them, is the time interval between two continuous sampling points.

[0078] This continuous judgment mechanism can effectively avoid the influence of incidental noise and transient abnormalities, improve the accuracy and stability of abnormal event discrimination. At the same time, the setting of parameter N needs to consider the response timeliness and robustness. Larger N value can reduce false positives, but may delay detection. Smaller N value can improve detection sensitivity, but may increase false positive rate. Generally, N can be optimized and adjusted through experimental data and clinical feedback.

[0079] In this embodiment, the threshold judgment mechanism based on continuous sampling points detects the propulsion force and the change rate of the propulsion force of the continuous N sampling points, improves the accuracy of the propulsion resistance detection, and thus guarantees the safety and accuracy of the endoscope operation.

[0080] Specifically, in one implementation manner of the embodiment, step S400 further includes the following steps: Step S403, according to the position-velocity difference detection result, the endoscope motion state and the propulsion resistance detection result, performing fusion analysis by using a preset module weight coefficient to obtain the endoscope abnormal bending detection result.

[0081] In order to improve the detection accuracy and reliability of the endoscope abnormal bending, a multi-information fusion judgment method is used in this embodiment, which comprehensively utilizes the endoscope distal position and velocity difference, visual image change, propulsion resistance signal and other multi-modal data to realize intelligent evaluation and dynamic evaluation of the abnormal state.

[0082] Specifically, in this embodiment, according to the sensitivity, accuracy and false positive rate of each detection module (i.e. distal-proximal motion difference detection module, image-based distal motion detection module and propulsion resistance detection module), a corresponding module weight coefficient is preset in the system to reflect the importance and credibility of different sensors in the abnormal bending judgment.

[0083] As an example, in this embodiment, the accuracy and false positive rate of each module are obtained through 100 endoscope abnormal bending experiments, and the weight of each module is determined combined with experience. For example, the distal-proximal motion difference detection module has good and stable detection effect, and the weight of the position-velocity difference detection module is set to 0.5 combined with experimental data and experience. The image-based distal motion detection module is easily affected by factors such as mucus and light, resulting in insufficient accuracy of visual image detection. Therefore, the weight of the image detection module is set to 0.3 combined with experimental data and experience. The propulsion resistance detection module is easily affected by the contact of the endoscope, and the weight of the propulsion resistance detection module is set to 0.3 combined with experimental data and experience. .

[0084] In the fusion stage, the scores of each module are multiplied by the corresponding weight and then accumulated by weighted summation, so as to obtain the overall score of the abnormal bending of the endoscope.

[0085] Specifically, in the present embodiment, there are three detection modules, and the abnormal score of the first detection module is (normalized to the interval [0, 1]), and the module weight coefficient is preset according to the performance indicators such as sensitivity, accuracy and false positive rate , and the overall abnormal score calculation formula is: ; Among them, the weight coefficient satisfies the normalization condition: ; wherein, , , .

[0086] In the present embodiment, a weighted fusion decision model is designed according to the sensitivity and false positive rate of each detection indicator, different weights are given to the results from different detection modules, and weighted scoring is realized. By fusing the information of multiple sensors and detection modules, the actual running state of the endoscope can be more comprehensively reflected, and the misjudgment risk of a single data source can be reduced.

[0087] It is worth mentioning that, in addition to the above-mentioned method (fusion analysis using preset module weight coefficients), the data collected by the position sensor, image sensor and force sensor can also be directly fused and analyzed by a deep neural network model, so as to quickly obtain the abnormal bending detection result of the endoscope. These deep neural network models can be a CLIP model (a multi-modal pre-training model based on contrastive learning) and a SAM model (an image segmentation model) after training.

[0088] The present embodiment achieves the following technical effects through the above technical solutions: The present embodiment monitors the relative motion data between the distal end of the endoscope and the instrument box by the sensor, processes the endoscope image sequence by the optical flow analysis method, analyzes the endoscope motion state in real time according to the endoscope image sequence, detects the change data of the pushing force by the force sensor, and intelligently evaluates whether the endoscope has abnormal bending in a multi-modal information fusion analysis manner. The present embodiment combines the multi-modal information of displacement, vision and pushing force sensor, and effectively improves the sensitivity and accuracy of endoscope detection through real-time data monitoring and analysis.

[0089] Exemplary device Based on the above embodiment, the present application also provides an endoscope abnormal bending detection system, comprising: ​a distal-proximal motion difference detection module configured to obtain relative motion data between a distal end of the endoscope and the instrument box; an image-based distal motion detection module configured to obtain an endoscope image sequence and analyze the endoscope motion state based on the endoscope image sequence; a pushing resistance detection module configured to obtain pushing force change data detected by the force sensor; a fusion analysis module configured to perform fusion analysis based on the relative motion data, the endoscope motion state, and the pushing force change data to obtain an endoscope abnormal bending detection result.

[0090] The embodiment achieves the following technical effects through the above technical solutions: The embodiment monitors the relative motion data between the distal end of the endoscope and the instrument box through the sensor, processes the endoscope image sequence by using the optical flow analysis method, analyzes the endoscope motion state in real time based on the endoscope image sequence, detects the pushing force change data through the force sensor, and intelligently evaluates whether the endoscope has abnormal bending in a multi-modal information fusion analysis manner. The embodiment combines the multi-modal information of displacement, vision, and pushing force sensor, performs real-time data monitoring and analysis, and effectively improves the sensitivity and accuracy of endoscope detection.

[0091] Based on the above embodiment, the application further provides a terminal, and a principle block diagram thereof can be as shown in Figure 8 .

[0092] The terminal includes a processor, a memory, an interface, a display screen, and a communication module connected through a system bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal includes a computer readable storage medium and an internal memory. The computer readable storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the computer readable storage medium. The interface is configured to connect external devices. The display screen is configured to display corresponding information. The communication module is configured to communicate with a cloud server or other devices.

[0093] The computer program is executed by the processor to implement the operations of the endoscope abnormal bending detection method.

[0094] Those skilled in the art can understand that, Figure 8 The principle block diagram shown in the above

[0095] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing an endoscope abnormal bending detection program, the endoscope abnormal bending detection program being used to implement the operations of the endoscope abnormal bending detection method as above when executed by the processor.

[0096] In one embodiment, a computer readable storage medium is provided, wherein the computer readable storage medium stores an endoscope abnormal bending detection program, the endoscope abnormal bending detection program being used to implement the operations of the endoscope abnormal bending detection method as above when executed by the processor.

[0097] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments of the present application can include non-volatile and volatile memory.

[0098] In summary, the present application provides an endoscope abnormal bending detection method, system, terminal and storage medium, comprising: acquiring relative motion data between the distal end of the endoscope and the instrument box; acquiring an endoscope image sequence, and analyzing the endoscope motion state according to the endoscope image sequence; acquiring the change data of the pushing force detected by the force sensor; and performing fusion analysis according to the relative motion data, the endoscope motion state and the change data of the pushing force to obtain an endoscope abnormal bending detection result. The present application combines multi-modal information of displacement, vision and pushing force sensor, and accurately identifies the abnormal bending condition of the endoscope through real-time data monitoring and analysis.

[0099] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes should belong to the protection scope of the appended claims of the present application.

Claims

1. An endoscope abnormal bending detection method characterized by comprising: The method comprises the following steps: acquiring relative motion data between the distal end of the endoscope and the instrument box; acquiring an endoscope image sequence and analyzing the endoscope motion state according to the endoscope image sequence; acquiring push force change data detected by a force sensor; performing fusion analysis according to the relative motion data, the endoscope motion state and the push force change data to obtain an endoscope abnormal bending detection result.

2. The endoscope abnormal bending detection method according to claim 1, characterized by, The acquisition of the relative motion data between the distal end of the endoscope and the instrument box comprises: collecting and recording position data of the distal end of the endoscope; performing dynamic processing and analysis on the collected position data to obtain the relative motion data between the distal end of the endoscope and the instrument box.

3. The endoscope abnormal bending detection method according to claim 2, characterized by, The dynamic processing and analysis on the collected position data to obtain the relative motion data between the distal end of the endoscope and the instrument box comprises: performing filtering processing on the continuously collected position data to obtain a denoised continuous displacement trajectory curve; calculating the displacement difference between the distal end of the endoscope and the instrument box according to the continuous displacement trajectory curve, and calculating the displacement difference change rate between the distal end of the endoscope and the instrument box to obtain the relative motion data between the distal end of the endoscope and the instrument box.

4. The endoscope abnormal bending detection method according to claim 1, characterized by, The acquisition of the endoscope image sequence and the analysis of the endoscope motion state according to the endoscope image sequence comprise: acquiring the endoscope image sequence, and calculating the optical flow vector of each pixel point in each frame of image in the endoscope image sequence by using the optical flow method; statistically analyzing the optical flow vectors of all pixels in each frame of image to obtain the average optical flow amplitude of each frame of image; determining the optical flow vector change trend between two or more consecutive frames of endoscope image according to the average optical flow amplitudes of the images; determining the endoscope motion state according to the optical flow vector change trend; wherein the endoscope motion state comprises a normal pushing state and a motion stagnation state.

5. The endoscope abnormal bending detection method according to claim 1, characterized by, The acquisition of the push force change data detected by the force sensor comprises: acquiring force sensor data connected to the bottom of the driving box to obtain push force data; performing filtering processing on the push force of the continuous sampling points in the push force data, and calculating the average force value at each time according to the filtered push force data; calculating the difference value of the average force values of adjacent time points, and calculating the change speed of the push force according to the difference value and the sampling interval to obtain the push force change data.

6. The endoscope abnormal bending detection method according to claim 1, characterized by, The fusion analysis according to the relative motion data, the endoscope motion state and the push force change data to obtain the endoscope abnormal bending detection result comprises: performing abnormal bending detection according to the displacement difference and the displacement difference change rate of the continuous N sampling points in the relative motion data to obtain a position-velocity difference detection result; wherein N is an integer greater than 1; performing push resistance detection according to the push force change data to obtain a push resistance detection result; performing fusion analysis according to the position-velocity difference detection result, the endoscope motion state and the push resistance detection result by using a preset module weight coefficient to obtain the endoscope abnormal bending detection result.

7. The endoscope abnormal bending detection method according to claim 6, characterized by, The abnormal bending detection according to the displacement difference and the displacement difference change rate of the continuous N sampling points in the relative motion data to obtain a position-velocity difference detection result comprises: acquire a probability of abnormal bending of each bronchial branch junction based on prior knowledge setting, and determine a corresponding dynamic detection threshold of the endoscope distal end in the movement process according to the acquired probability; wherein the dynamic detection threshold includes a displacement difference threshold and a displacement difference change rate threshold; determine whether the displacement difference and the displacement difference change rate of the continuous N sampling points are both higher than the corresponding threshold; when the displacement difference and the displacement difference change rate of the continuous N sampling points are both lower than the corresponding threshold, it is determined that the endoscope has abnormal bending, and the position-velocity difference detection result is obtained.

8. The endoscope abnormal bending detection method according to claim 7, characterized by, The acquiring of the probability of abnormal bending of each bronchial branch junction based on prior knowledge setting, and the determination of the corresponding dynamic detection threshold of the endoscope distal end in the movement process according to the acquired probability, include: acquiring the probability of abnormal bending of each bronchial branch junction in the path L, and cumulatively calculating the overall abnormal bending probability when passing through the path L according to the acquired probability; dynamically adjusting the displacement difference threshold and the displacement difference change rate threshold according to the overall abnormal bending probability, to obtain the corresponding dynamic detection threshold of the endoscope distal end in the movement process.

9. The endoscope abnormal bending detection method according to claim 6, characterized by, The propelling resistance detection according to the propelling force change data to obtain the propelling resistance detection result, includes: acquiring mechanical statistical data and prior knowledge under normal propelling conditions, and determining a double threshold condition of propelling force and propelling force change speed according to the mechanical statistical data and the prior knowledge; performing propelling resistance detection according to the double threshold condition and using a threshold judgment mechanism of continuous sampling points, to obtain the propelling resistance detection result.

10. The endoscope abnormal bending detection method according to claim 9, characterized by, The propelling resistance detection according to the double threshold condition and using a threshold judgment mechanism of continuous sampling points to obtain the propelling resistance detection result, includes: determining whether the propelling force and the propelling force change speed of the continuous N sampling points in the propelling force change data both satisfy the double threshold condition according to the threshold judgment mechanism of the continuous sampling points; when the propelling force and the propelling force change speed of the continuous N sampling points both satisfy the double threshold condition, it is determined that the endoscope has abnormal bending, and the propelling resistance detection result is obtained.

11. An endoscope abnormal bending detection system characterized by comprising: It includes: a distal-proximal movement difference detection module for acquiring relative movement data between the endoscope distal end and the instrument box; an image-based distal movement detection module for acquiring an endoscope image sequence, and analyzing the endoscope movement state according to the endoscope image sequence; a propelling resistance detection module for acquiring propelling force change data detected by a force sensor; a fusion analysis module for performing fusion analysis according to the relative movement data, the endoscope movement state, and the propelling force change data, to obtain an endoscope abnormal bending detection result.

12. A terminal, characterized by comprising: It includes: a processor and a memory, the memory stores an endoscope abnormal bending detection program, and the endoscope abnormal bending detection program is used to implement the operations of the endoscope abnormal bending detection method in any one of claims 1-10 when executed by the processor.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores an endoscope abnormal bending detection program, and the endoscope abnormal bending detection program, when executed by the processor, is used to implement operations of the endoscope abnormal bending detection method in any one of claims 1-10.

Images