An endoscope abnormal bending detection method, system, terminal and storage medium

Abstract

The application belongs to the technical field of medical robots, and discloses an endoscope abnormal bending detection method, system, terminal and storage medium, which comprises the following steps: acquiring relative motion data between an endoscope distal end and an instrument box; acquiring an endoscope image sequence, and analyzing an endoscope motion state according to the endoscope image sequence; acquiring push force change data detected by a force sensor; and performing fusion analysis according to the relative motion data, the endoscope motion state and the push force change data to obtain an endoscope abnormal bending detection result. The application combines multi-modal information of displacement, vision and push force sensors, and accurately identifies the abnormal bending condition of the endoscope through real-time data monitoring and analysis.

Description

Technical Field

[0001] This invention relates to the field of medical robot technology, and in particular to a method, system, terminal, and storage medium for detecting abnormal curvature of an endoscope. Background Technology

[0002] In a bronchoscopic surgical robot system, the endoscope's motion system typically involves structures such as a feed slide, a drive unit, and the endoscope itself. The endoscope comprises an instrument box, an insertion cannula, and a stylus. The drive unit controls the stylus's movement via propulsion, enabling precise navigation of the endoscope along complex paths within the body.

[0003] When the endoscope moves through the winding bronchus, the applied thrust may be insufficient to overcome the friction between the bronchial bone and the bronchial wall, causing the distal endoscope to stop moving. When the endoscope bends abnormally, the insertion tube may exert excessive contact force on the tracheal wall, leading to bronchial tissue damage or even rupture. Current surgical robot systems cannot sense the endoscope's propulsion force or obstacle feedback, and lack real-time anomaly monitoring methods.

[0004] Existing methods for detecting abnormal endoscopic curvature primarily compare the displacement difference between the distal and proximal ends of the endoscope. If this displacement difference exceeds a threshold, the insertion tube is considered to have abnormally curved. However, this method is not accurate enough. Firstly, the distal endoscope movement is a combination of feeding and serpentine bending, and the displacement changes caused by the serpentine bending can affect the judgment. Secondly, selecting an appropriate threshold for different scenarios is a significant challenge. While shape sensors can simplify this problem, their generally high cost limits their application.

[0005] In summary, existing methods for detecting abnormal bending of endoscopes are relatively simple, usually comparing the displacement difference between the instrument box and the distal end of the endoscope to determine whether the insertion tube has become abnormally bent. This method is not accurate enough and faces problems such as difficulty in threshold selection and insufficient sensitivity.

[0006] Therefore, existing technologies still need improvement. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method, system, terminal and storage medium for detecting abnormal curvature of endoscopes, in order to solve the problem of poor detection accuracy in existing methods for detecting abnormal curvature of endoscopes.

[0008] The technical solution adopted by this invention to solve the technical problem is as follows:

[0009] In a first aspect, the present invention provides a method for detecting abnormal curvature of an endoscope, comprising:

[0010] Acquire relative motion data between the distal endoscope and the instrument box;

[0011] Acquire endoscopic image sequences and analyze the endoscopic motion state based on the endoscopic image sequences;

[0012] Acquire propulsion force change data detected by the force sensor;

[0013] The abnormal bending detection result of the endoscope is obtained by fusing and analyzing the relative motion data, the endoscope motion state, and the propulsion force change data.

[0014] In one implementation, acquiring the relative motion data between the distal endoscope and the instrument box includes:

[0015] Collect and record the position data of the distal endoscope;

[0016] The collected position data is dynamically processed and analyzed to obtain the relative motion data between the distal endoscope and the instrument box.

[0017] In one implementation, the step of dynamically processing and analyzing the collected position data to obtain the relative motion data between the distal endoscope and the instrument box includes:

[0018] The continuously acquired position data is filtered to obtain the denoised continuous displacement trajectory curve.

[0019] The displacement difference between the distal end of the endoscope and the instrument box is calculated based on the continuous displacement trajectory curve, and the rate of change of the displacement difference between the distal end of the endoscope and the instrument box is calculated to obtain the relative motion data between the distal end of the endoscope and the instrument box.

[0020] In one implementation, acquiring the endoscopic image sequence and analyzing the endoscopic motion state based on the endoscopic image sequence includes:

[0021] The endoscopic image sequence is acquired, and the optical flow vector of each pixel in each frame of the endoscopic image sequence is calculated using the optical flow method.

[0022] Statistical analysis is performed on the optical flow vectors of all pixels in each frame of the image to obtain the average optical flow amplitude of each frame.

[0023] Based on the average optical flow amplitude of each image, determine the trend of optical flow vector change between two or more consecutive endoscopic images;

[0024] The endoscope motion state is determined based on the trend of optical flow vector change; wherein, the endoscope motion state includes: normal advancement state and motion stagnation state.

[0025] In one implementation, acquiring the propulsion force change data detected by the force sensor includes:

[0026] Obtain propulsion force data by acquiring data from the force sensor connected to the bottom of the drive box;

[0027] The thrust force of the continuously sampled points in the thrust force data is filtered, and the average force value at each moment is calculated based on the filtered thrust force data;

[0028] The difference between the average force values ​​at adjacent time points is calculated, and the rate of change of propulsion force is calculated based on the difference and the sampling interval to obtain the propulsion force change data.

[0029] In one implementation, the step of fusing and analyzing the relative motion data, the endoscope motion state, and the propulsion force change data to obtain the endoscope abnormal bending detection result includes:

[0030] Abnormal bending is detected based on the displacement difference and the rate of change of displacement difference of N consecutive sampling points in the relative motion data, and the position-velocity difference detection result is obtained; where N is an integer greater than 1.

[0031] Propulsion resistance is detected based on the propulsion force change data to obtain the propulsion resistance detection result;

[0032] Based on the position-velocity difference detection results, the endoscope motion state, and the propulsion resistance detection results, a fusion analysis is performed using preset module weighting coefficients to obtain the endoscope abnormal bending detection results.

[0033] In one implementation, the abnormal bending detection based on the displacement difference and the rate of change of displacement difference among N consecutive sampling points in the relative motion data, to obtain the position-velocity difference detection result, includes:

[0034] The probability of abnormal bending at each bronchial branch intersection, set based on prior knowledge, is obtained, and the dynamic detection threshold corresponding to the distal endoscope during movement is determined based on the obtained probability; wherein, the dynamic detection threshold includes: displacement difference threshold and displacement difference change rate threshold.

[0035] Determine whether the displacement difference and the rate of change of displacement difference of N consecutive sampling points are both higher than the corresponding threshold;

[0036] When the displacement difference and the rate of change of displacement difference of N consecutive sampling points are both lower than the corresponding threshold, it is determined that the endoscope has abnormally bent, and the position-velocity difference detection result is obtained.

[0037] In one implementation, the step of obtaining the probability of abnormal bending at each bronchial branch intersection based on prior knowledge, and determining the dynamic detection threshold corresponding to the distal endoscope during movement based on the obtained probability, includes:

[0038] The probability of abnormal bending at each bronchial branch intersection within path L is obtained, and the overall probability of abnormal bending when passing through path L is calculated cumulatively based on the obtained probabilities.

[0039] Based on the overall abnormal bending probability, the displacement difference threshold and the displacement difference change rate threshold are dynamically adjusted to obtain the dynamic detection threshold corresponding to the distal end of the endoscope during movement.

[0040] In one implementation, the step of detecting propulsion resistance based on the propulsion force change data to obtain a propulsion resistance detection result includes:

[0041] Obtain mechanical statistics and prior knowledge under normal propulsion conditions, and determine dual threshold conditions for propulsion force and propulsion force change rate based on the mechanical statistics and prior knowledge;

[0042] Based on the aforementioned dual threshold conditions, a threshold judgment mechanism using continuous sampling points is employed to detect propulsion resistance, thereby obtaining the propulsion resistance detection result.

[0043] In one implementation, the step of performing propulsion resistance detection using a threshold judgment mechanism based on continuous sampling points according to the dual threshold conditions, and obtaining the propulsion resistance detection result, includes:

[0044] Based on the threshold judgment mechanism of the continuous sampling points, it is determined whether the propulsion force and the propulsion force change rate of N consecutive sampling points in the propulsion force change data all satisfy the dual threshold conditions.

[0045] When the propulsion force and the rate of change of propulsion force at N consecutive sampling points both meet the dual threshold conditions, it is determined that the endoscope has abnormally bent, and the propulsion resistance detection result is obtained.

[0046] In a second aspect, the present invention provides an endoscope abnormal curvature detection system, comprising:

[0047] The distal-proximal motion difference detection module is used to acquire relative motion data between the distal end of the endoscope and the instrument box;

[0048] The image-based remote motion detection module is used to acquire endoscopic image sequences and analyze the endoscopic image sequences to obtain the endoscopic motion state.

[0049] The thrust resistance detection module is used to acquire thrust force change data detected by the force sensor;

[0050] The fusion analysis module is used to perform fusion analysis based on the relative motion data, the endoscope motion state, and the propulsion force change data to obtain the endoscope abnormal bending detection results.

[0051] Thirdly, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores an endoscope abnormal curvature detection program, and the endoscope abnormal curvature detection program, when executed by the processor, is used to implement the operation of the endoscope abnormal curvature detection method as described in the first aspect.

[0052] Fourthly, the present invention also provides a computer-readable storage medium storing an endoscope abnormal curvature detection program, which, when executed by a processor, is used to implement the operation of the endoscope abnormal curvature detection method as described in the first aspect.

[0053] The present invention, by employing the above technical solution, has the following effects:

[0054] This invention monitors the relative motion data between the distal endoscope and the instrument box using sensors, and processes the endoscope image sequence using optical flow analysis. The motion state of the endoscope is obtained in real time based on the image sequence. Simultaneously, force sensors detect changes in propulsion force, and the invention intelligently assesses whether abnormal bending of the endoscope has occurred through multimodal information fusion analysis. By combining multimodal information from displacement, vision, and propulsion force sensors, and through real-time data monitoring and analysis, this invention effectively improves the sensitivity and accuracy of endoscopic detection. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0056] Figure 1 This is a flowchart of the method for detecting abnormal curvature of endoscopes in this invention.

[0057] Figure 2 This is a schematic diagram of the bronchoscopic movement of the bronchoscopic surgical robot in this invention.

[0058] Figure 3 This is a schematic diagram of the movement of the endoscope in the bronchus in this invention.

[0059] Figure 4 This is a schematic diagram of the displacement of the drive box in this invention.

[0060] Figure 5This is a schematic diagram of the displacement of the distal endoscope in this invention.

[0061] Figure 6 This is a schematic diagram of thresholds on different paths in this invention.

[0062] Figure 7 This is a schematic diagram of the endoscope propulsion resistance detection in this invention.

[0063] Figure 8 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0064] In the diagram: 1. Feed slide; 2. Drive box; 3. Endoscope; 4. Instrument box (proximal end of endoscope); 5. Insertion tube; 6. Snake bone; 7. Distal end of endoscope; 8. Bronchus; 10. Force sensor.

[0065] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0067] Exemplary methods

[0068] Existing methods for detecting abnormal endoscopic curvature primarily compare the displacement difference between the distal and proximal ends of the endoscope. If this displacement difference exceeds a threshold, the insertion tube is considered to have abnormally curved. However, this method is not accurate enough. Firstly, the distal endoscope movement is a combination of feeding and serpentine bending, and the displacement changes caused by the serpentine bending can affect the judgment. Secondly, selecting an appropriate threshold for different scenarios is a significant challenge. While shape sensors can simplify this problem, their generally high cost limits their application.

[0069] In summary, existing methods for detecting abnormal bending of endoscopes are relatively simple, usually comparing the displacement difference between the instrument box and the distal end of the endoscope to determine whether the insertion tube has become abnormally bent. This method is not accurate enough and faces problems such as difficulty in threshold selection and insufficient sensitivity.

[0070] To address the above-mentioned technical problems, this invention provides a method for detecting abnormal endoscope bending, comprising: acquiring relative motion data between the distal end of the endoscope and the instrument box; acquiring an endoscope image sequence and analyzing the endoscope image sequence to obtain the endoscope motion state; acquiring propulsion force change data detected by a force sensor; and performing fusion analysis based on the relative motion data, the endoscope motion state, and the propulsion force change data to obtain an endoscope abnormal bending detection result. This invention combines multimodal information from displacement, vision, and propulsion force sensors, and through real-time data monitoring and analysis, accurately identifies abnormal endoscope bending.

[0071] like Figure 1 As shown, this embodiment of the invention provides a method for detecting abnormal curvature of an endoscope, comprising the following steps:

[0072] Step S100: Obtain relative motion data between the distal endoscope and the instrument box.

[0073] In this embodiment, the endoscope abnormal bending detection method is applied to the endoscope abnormal bending detection scenario of the bronchoscopic surgical robot system; or it is applied to the abnormal bending detection scenario of other detection systems, such as the bending detection scenario of industrial endoscopes, for abnormal bending detection in complex environments such as deep cavities and narrow welds.

[0074] As an example, such as Figure 2 As shown, in a bronchoscopic surgical robot system, the structure responsible for the movement of the endoscope (endoscopy) typically includes a feed slide 1, a drive unit 2, and an endoscope 3. The endoscope 3 is composed of components such as an instrument box 4, an insertion tube 5, and a serpentine frame 6. The active movement of the endoscope 3 consists of two parts: the feed motion provided by the feed slide 1 and the bending motion provided by the drive unit 2 towards the serpentine frame 6. The drive unit 2 can achieve the feed motion along the feed slide 1.

[0075] The working principle of the aforementioned bronchoscopic surgical robot system is as follows: During use, the drive unit 2 applies a pushing or pulling force to the instrument box 4 to control the advance and retreat of the serpentine 6. However, when the endoscope 3 extends into the lesion along the tortuous path of the bronchus, the force transmission path of the endoscope 3 is a complex curve. When the frictional force between the serpentine 6 and the bronchial wall is greater than the pushing force transmitted to the serpentine 6, the distal end 7 of the endoscope stops moving, and the insertion tube 5 exhibits an undesirable abnormal bend; the normal bend of the insertion tube 5 is illustrated as follows. Figure 3 As shown in (a), for comparison, the abnormal bending of the insertion tube 5 is illustrated as follows: Figure 3 As shown in (b), the abnormal bending of the insertion tube 5 exerts a large contact force on the tracheal wall, which may cause damage or even rupture of the bronchial tissue. Continuing to advance the endoscope after the insertion tube 5 has an abnormal bend may be dangerous.

[0076] In this embodiment, the method for detecting abnormal endoscope curvature is a multi-information fusion-based scheme. This scheme mainly determines whether abnormal endoscope curvature has occurred through the following methods:

[0077] 1) Use the displacement difference and the rate of change of displacement difference between the distal endoscope and the instrument box to determine whether abnormal bending of the endoscope has occurred;

[0078] 2) Use visual information from the distal end of the endoscope to determine if there is abnormal curvature of the endoscope; if the change between adjacent frames is small, it means that the lens is no longer moving, which can be used to determine if there is abnormal curvature of the endoscope.

[0079] 3) The propulsion force is monitored by a force sensor to determine whether there is abnormal bending of the endoscope; if the magnitude of the force and the rate of change of the force reach the threshold, the endoscope is determined to be abnormally bent.

[0080] As the first method of multi-information fusion-based endoscope abnormal bending detection scheme, this embodiment refers to it as the distal-proximal motion difference detection method. This detection method mainly acquires the relative motion data between the distal end of the endoscope and the instrument box, including the displacement difference and the rate of change of the displacement difference, and uses this relative motion data to further determine whether abnormal endoscope bending has occurred. Specifically, the relative motion data between the distal end of the endoscope and the instrument box refers to the displacement difference between the distal end of the endoscope and the instrument box caused by motion, and the rate of change of the displacement difference between the distal end of the endoscope and the instrument box.

[0081] Specifically, in one implementation of this embodiment, step S100 includes the following steps:

[0082] Step S101: Collect and record the position data of the distal end of the endoscope;

[0083] Step S102: Dynamically process and analyze the collected position data to obtain the relative motion data between the distal end of the endoscope and the instrument box.

[0084] In this embodiment, the distal end of the endoscope is equipped with a high-precision position sensor, which can collect and record the three-dimensional spatial position of the distal end of the endoscope in real time. In order to achieve timely and accurate detection of abnormal bending of the endoscope, it is necessary to continuously and dynamically monitor and analyze the collected position data to obtain the relative motion data between the distal end of the endoscope and the instrument box.

[0085] Because the raw location data collected is limited by various factors such as sensor measurement errors and environmental interference, it usually contains a certain amount of high-frequency noise and occasional abnormal fluctuations. If the unprocessed data is used directly for judgment, it is very easy to cause false alarms and missed alarms. Therefore, it is necessary to further preprocess the collected location data.

[0086] In one implementation of this embodiment, step S102 includes the following steps:

[0087] Step S1021: Filter the continuously acquired position data to obtain the denoised continuous displacement trajectory curve.

[0088] Step S1022: Calculate the displacement difference between the distal end of the endoscope and the instrument box based on the continuous displacement trajectory curve, and calculate the rate of change of the displacement difference between the distal end of the endoscope and the instrument box to obtain the relative motion data between the distal end of the endoscope and the instrument box.

[0089] Specifically, during the preprocessing of the acquired location data, a set window M (i.e., window size M) is used to perform smoothing filtering on the continuously acquired location data to obtain the denoised continuous displacement trajectory curve. The filtering method can be any one or a combination of moving average filtering, median filtering, and adaptive mean filtering techniques.

[0090] As an example, this embodiment employs adaptive mean filtering. For continuously acquired location data, the variation amplitude (i.e., local standard deviation) within the neighborhood of each data point is calculated, and then the size of the window M is adjusted according to the variation amplitude. For instance, when the variation is small, the window M is decreased to preserve trajectory details; when the variation is large, the window M is increased to enhance smoothing and noise suppression capabilities.

[0091] In this embodiment, after obtaining the denoised continuous displacement trajectory curve, the displacement difference between the distal endoscope and the instrument box can be calculated based on the continuous displacement trajectory curve, and the rate of change of the displacement difference between the distal endoscope and the instrument box can also be calculated. Since the instrument box is directly driven by the drive box, in this embodiment, the displacement difference between the distal endoscope and the instrument box can be determined by calculating the displacements of the drive box and the distal endoscope; wherein, the displacements of the drive box and the distal endoscope are respectively as follows: Figure 4 and Figure 5 As shown.

[0092] In this embodiment, This difference is the difference between the displacement of the drive box and the displacement of the distal endoscope during the same time period. This refers to the displacement difference between the distal endoscope and the instrument case, and this difference... Calculated using Euclidean distance; and The relative motion data between the distal end of the endoscope and the instrument box is obtained by measuring the rate of change of the displacement difference between them over a continuous period of time. This data is then used to detect abnormal bending and obtain the position-velocity difference detection result.

[0093] like Figure 1As shown, this embodiment of the invention provides a method for detecting abnormal curvature of an endoscope, comprising the following steps:

[0094] Step S200: Obtain the endoscope image sequence and analyze the endoscope motion state based on the endoscope image sequence.

[0095] In this embodiment, as the second method of the multi-information fusion endoscope abnormal bending detection scheme, referred to as the endoscope image detection method, this method performs image detection through an image-based distal motion detection module and uses optical flow to detect whether the endoscope is in a normal advancing state or a stationary state. Optical flow is an important technique in the field of computer vision, mainly used to estimate the motion state of objects in an image sequence. Its basic principle is: assuming that the image brightness remains constant for a short period of time, by comparing the brightness changes of adjacent pixels in two consecutive frames, the corresponding motion vector is calculated to determine whether the object is in motion. Optical flow can effectively capture subtle motion information in images and is suitable for various applications such as dynamic scene analysis, target tracking, and motion detection.

[0096] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0097] Step S201: Obtain the endoscopic image sequence, and calculate the optical flow vector of each pixel in each frame of the endoscopic image sequence using the optical flow method;

[0098] Step S202: Perform statistical analysis on the optical flow vectors of all pixels in each frame of the image to obtain the average optical flow amplitude of each frame of the image;

[0099] Step S203: Determine the trend of optical flow vector change between two or more consecutive endoscopic images based on the average optical flow amplitude of each image;

[0100] Step S204: Determine the endoscope motion state based on the trend of optical flow vector change; wherein, the endoscope motion state includes: normal advancement state and motion stagnation state.

[0101] In this embodiment, after acquiring the endoscopic image sequence, the horizontal and vertical components of each pixel, i.e., the optical flow vector, are calculated by combining the pixel brightness changes of adjacent frames with the optical flow constraint equation. Specifically, the optical flow vector of each pixel in each frame of the endoscopic image sequence is calculated using the optical flow method; then, statistical analysis is performed on the optical flow vectors of all pixels in each frame, i.e., the average optical flow magnitude is calculated for all pixels in the entire image to reflect the overall motion intensity and obtain the overall motion pattern of the image. This is particularly important for endoscopic image analysis because the motion of the distal end of the endoscope directly affects the variation characteristics of the image sequence.

[0102] As an example, in practical applications, optical flow can be used to detect the motion state of the distal endoscope image to determine whether there is motion stagnation or abnormality. This is because during normal advancement, there will be significant changes in optical flow between two or more consecutive frames; conversely, if the change in optical flow vector is extremely small or approaches zero, it usually means that the distal end of the endoscope has not moved effectively, and there may be abnormal bending or blockage. By setting a reasonable optical flow threshold, abnormal states can be detected in a timely manner. Specifically, the optical flow threshold can be determined by collecting a large amount of optical flow data under normal advancement and abnormal bending conditions, and statistically analyzing the range of optical flow vector changes under normal conditions.

[0103] Based on the analysis of optical flow detection results, this embodiment enables the system to dynamically determine the actual movement of the endoscope and take corresponding response measures accordingly.

[0104] like Figure 1 As shown, this embodiment of the invention provides a method for detecting abnormal curvature of an endoscope, comprising the following steps:

[0105] Step S300: Obtain propulsion force change data detected by the force sensor.

[0106] In this embodiment, as the third method of the multi-information fusion endoscope abnormal bending detection scheme, it is called the propulsion resistance detection method. This method first requires the acquisition of real-time propulsion force change data.

[0107] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0108] Step S301: Obtain data from the force sensor connected to the bottom of the drive box to obtain propulsion force data;

[0109] Step S302: Filter the propulsion force of the continuous sampling points in the propulsion force data, and calculate the average force value at each moment based on the filtered propulsion force data;

[0110] Step S303: Calculate the difference in average force values ​​between adjacent time points, and calculate the rate of change of propulsion force based on the difference and the sampling interval to obtain the propulsion force change data.

[0111] In this embodiment, as Figure 7 As shown, a force sensor 10 is connected to the bottom of the drive box 2. During the endoscope's advancement, the force sensor 10 monitors the propulsion force. To ensure the accuracy and stability of the propulsion force signal, real-time propulsion force data must first be collected. .

[0112] Because the raw data from force sensors contains noise and abrupt changes, directly using the raw data can easily lead to misjudgments. Therefore, filtering the acquired propulsion force data is essential. Commonly used filtering methods include moving average filtering and low-pass filtering. These methods can effectively suppress high-frequency noise and short-term abnormal fluctuations, thereby extracting a smoother and more representative force curve.

[0113] Specifically, using a set window N (window size N, number of sampling points N, where N is an integer greater than 1), a moving average operation is performed on the propulsion force of consecutive sampling points to calculate the average force value at each time step, thereby reducing the influence of random noise. Then, by calculating the difference between the moving averages of adjacent time steps, the calculated difference is divided by the sampling interval. The rate of change of propulsion can then be obtained. (i.e., propulsion change data) provides accurate and stable parameter basis for subsequent threshold judgment.

[0114] like Figure 1 As shown, this embodiment of the invention provides a method for detecting abnormal curvature of an endoscope, comprising the following steps:

[0115] Step S400: Based on the relative motion data, the endoscope motion state, and the propulsion force change data, a fusion analysis is performed to obtain the endoscope abnormal bending detection result.

[0116] In this embodiment, by combining relative motion data, endoscope motion state, and propulsion force change data, and using preset module weight coefficients for fusion analysis, the endoscope abnormal bending detection results can be obtained quickly; wherein, the preset module weight coefficients are weights obtained based on corresponding experimental data and experience (i.e., prior knowledge).

[0117] Specifically, in one implementation of this embodiment, step S400 includes the following steps:

[0118] Step S401: Based on the displacement difference and displacement difference change rate of N consecutive sampling points in the relative motion data, abnormal bending detection is performed to obtain the position-velocity difference detection result.

[0119] In one implementation of this embodiment, step S401 includes the following steps:

[0120] Step S4011: Obtain the probability of abnormal bending at each bronchial branch intersection based on prior knowledge, and determine the dynamic detection threshold corresponding to the distal endoscope during movement based on the obtained probability; wherein, the dynamic detection threshold includes: displacement difference threshold and displacement difference change rate threshold.

[0121] In one implementation of this embodiment, the step of obtaining the probability of abnormal bending at each bronchial branch intersection and determining the dynamic detection threshold corresponding to the distal endoscope during movement based on the obtained probability includes: obtaining the probability of abnormal bending at each bronchial branch intersection within path L, and cumulatively calculating the overall abnormal bending probability when passing through path L based on the obtained probability; dynamically adjusting the displacement difference threshold and the displacement difference change rate threshold based on the overall abnormal bending probability to obtain the dynamic detection threshold corresponding to the distal endoscope during movement.

[0122] Step S4012: Determine whether the displacement difference and the rate of change of displacement difference of N consecutive sampling points are both higher than the corresponding threshold.

[0123] Step S4013: When the displacement difference and displacement difference change rate of N consecutive sampling points are both lower than the corresponding threshold, it is determined that the endoscope has abnormally bent, and the position-velocity difference detection result is obtained.

[0124] In this embodiment, based on the displacement difference and displacement difference change rate between the distal end of the endoscope and the instrument box obtained in step S100 above, the distal-proximal motion difference detection module is used to perform abnormal bending detection on N consecutive sampling points to obtain the position-velocity difference detection result.

[0125] During abnormal bending detection, when the displacement difference at a certain moment satisfies... And the rate of change of displacement difference at the distal end of the endoscope satisfies This indicates that the distal endoscope may be in a state of motion stagnation, suggesting a risk of abnormal bending or entrapment. , These are the displacement difference threshold and the displacement difference change rate threshold, which are used to distinguish between normal and abnormal curvature of the endoscope.

[0126] In this embodiment, the endoscope may experience abnormal bending at any bronchial branch junction, and the more tortuous the path, the more likely abnormal bending will occur. Considering the differences in the motion characteristics and anatomical structures of the endoscope at different bronchial branches, the probability of abnormal bending at each bronchial branch junction is denoted as . ,like Figure 6 As shown, where, This is the number of the bronchial branch intersection, and the corresponding bronchial branch path is... , The angle between the branch pipes, Let L be the diameter of the branch pipe. Based on the cumulative probability of anomalies at multiple branch points within the path length L, the overall probability of abnormal bending when passing through path L can be calculated:

[0127] ;

[0128] By statistically analyzing the overall abnormal tortuosity probability accumulated along the distal path of the endoscope Dynamically calculate and adjust the threshold at the current time. Threshold :

[0129] ;

[0130] ;

[0131] in, , These are the displacement difference threshold and displacement difference change rate threshold, respectively, obtained through multiple experiments at a single branch intersection.

[0132] Although single-point detection can indicate anomalies, it is easily affected by accidental measurement errors or noise signals, leading to false alarms. To enhance the robustness of detection and the stability of judgment, this embodiment introduces a threshold determination mechanism for continuous sampling points: only when N consecutive sampling points at the distal end of the endoscope (with an interval of 1 / N) are detected... Abnormal bending is confirmed only when both the displacement difference and the rate of change of displacement difference are below the corresponding dynamic detection threshold. The specific judgment conditions are as follows:

[0133] ;

[0134] The threshold determination mechanism of this continuous sampling point, combined with auxiliary indicators such as acceleration, can further enhance the accuracy and anti-interference ability of abnormal bending state identification, thereby effectively avoiding misjudgment caused by occasional abnormal signal interference and ensuring the safety and accuracy of endoscopic operation.

[0135] Specifically, in one implementation of this embodiment, step S400 further includes the following steps:

[0136] Step S402: Detect propulsion resistance based on the propulsion force change data to obtain propulsion resistance detection results;

[0137] In one implementation of this embodiment, step S402 includes the following steps:

[0138] Step S4021: Obtain mechanical statistics and prior knowledge under normal propulsion conditions, and determine the dual threshold conditions of propulsion force and propulsion force change rate based on the mechanical statistics and prior knowledge.

[0139] Step S4022: Based on the dual threshold conditions, propulsion resistance is detected using a threshold judgment mechanism based on continuous sampling points to obtain the propulsion resistance detection result.

[0140] In one implementation of this embodiment, the step of using a threshold judgment mechanism of continuous sampling points to detect propulsion resistance based on the dual threshold conditions and obtain the propulsion resistance detection result includes: determining whether the propulsion force and propulsion force change rate of N consecutive sampling points in the propulsion force change data all satisfy the dual threshold conditions according to the threshold judgment mechanism of continuous sampling points; when the propulsion force and propulsion force change rate of N consecutive sampling points all satisfy the dual threshold conditions, determining that the endoscope has abnormally bent, and obtaining the propulsion resistance detection result.

[0141] In this embodiment, for the thrust change data obtained in step S300, the thrust resistance detection module is used to detect the thrust resistance at N sampling points to obtain the thrust resistance detection result.

[0142] During the propulsion resistance detection process, if the magnitude of the propulsion force at a single moment meets the following conditions... And the rate of change of propulsion satisfies If so, it is considered a potential abnormal state. Here, and A threshold is pre-set based on statistical data of mechanical properties under normal propulsion conditions and clinical experience (i.e., prior knowledge) to distinguish between normal and abnormal propulsion states. Although a single sample can indicate an abnormality, single-point judgment conditions are easily affected by noise or transient abnormal signals, which may produce false alarms.

[0143] To improve the robustness of detection and the reliability of judgment, this embodiment introduces a threshold judgment mechanism based on continuous sampling points. If the dual threshold conditions of propulsion force and the rate of change of propulsion force are met in N consecutive sampling points, then the endoscope is determined to have abnormal curvature. That is:

[0144] ;

[0145] in, The time interval between two consecutive sampling points.

[0146] This continuous judgment mechanism effectively avoids the influence of sporadic noise and transient anomalies, improving the accuracy and stability of abnormal event detection. Simultaneously, the setting of parameter N needs to balance response timeliness and robustness. A larger N value reduces false alarms but may lead to delayed detection, while a smaller N value can improve detection sensitivity but may increase the false alarm rate. N can generally be optimized and adjusted based on experimental data and clinical feedback.

[0147] In this embodiment, a threshold judgment mechanism based on continuous sampling points is used to perform dual detection of the propulsion force and the rate of change of propulsion force at N consecutive sampling points, which improves the accuracy of propulsion resistance detection and thus ensures the safety and precision of endoscopic operations.

[0148] Specifically, in one implementation of this embodiment, step S400 further includes the following steps:

[0149] Step S403: Based on the position-velocity difference detection result, the endoscope motion state, and the propulsion resistance detection result, a fusion analysis is performed using preset module weighting coefficients to obtain the endoscope abnormal bending detection result.

[0150] To improve the accuracy and reliability of detecting abnormal endoscope curvature, this embodiment adopts a multi-information fusion judgment method, which comprehensively utilizes multi-modal data such as endoscope distal position and velocity differences, visual image changes, and propulsion resistance signals to achieve intelligent and dynamic assessment of abnormal states.

[0151] Specifically, in this embodiment, based on the sensitivity, accuracy and false alarm rate of each detection module (i.e., the distal-proximal motion difference detection module, the image-based distal motion detection module, and the propulsion resistance detection module), corresponding module weight coefficients are pre-set in the system to reflect the importance and reliability of different sensors in abnormal bending judgment.

[0152] As an example, in this embodiment, the accuracy and false alarm rate of each module were obtained through 100 endoscopic abnormal bending experiments beforehand, and the weight of each module was determined based on experience. For instance, the distal-proximal motion difference detection module showed better and more stable detection performance, and the weight of the position-velocity difference detection module was set based on experimental data and experience. Image-based remote motion detection modules are easily affected by factors such as mucus and lighting, leading to insufficient accuracy in visual image detection. Therefore, based on experimental data and experience, the weight of the image detection module is set as follows: For the propulsion resistance detection module, since propulsion resistance is easily affected by endoscope contact, the weight of the propulsion resistance detection module is set based on experimental data and experience. .

[0153] During the fusion phase, a weighted summation method is used, where the scores of each module are multiplied by their corresponding weights and then summed to obtain the overall score for abnormal endoscopic curvature.

[0154] Specifically, in this embodiment, there are three detection modules, the first being... The anomaly score for each detection module is: (Normalized to the [0,1] interval), the module weight coefficients are pre-set based on performance indicators such as sensitivity, accuracy, and false alarm rate. The overall anomaly score calculation formula is as follows:

[0155] ;

[0156] The weighting coefficients satisfy the normalization condition:

[0157] ;in, , , .

[0158] In this embodiment, a weighted fusion decision model was designed based on the sensitivity and false alarm rate of each detection indicator. Different weights are assigned to results from different detection modules to achieve weighted scoring. By fusing information from multiple sensors and detection modules, the actual operating status of the endoscope can be more comprehensively reflected, reducing the risk of misjudgment from a single data source.

[0159] It is worth mentioning that, in addition to the above-mentioned method (using preset module weight coefficients for fusion analysis), this embodiment can also directly perform fusion analysis on real-time data collected by position sensors, image sensors, and force sensors using deep neural network models, thereby quickly obtaining the detection results of abnormal curvature of the endoscope. These deep neural network models can be trained CLIP models (a multimodal pre-trained model based on contrastive learning), SAM models (an image segmentation model), etc.

[0160] This embodiment achieves the following technical effects through the above technical solution:

[0161] This embodiment monitors the relative motion data between the distal endoscope and the instrument box using sensors, and processes the endoscope image sequence using optical flow analysis. The motion state of the endoscope is obtained in real time based on the image sequence. Simultaneously, force sensors detect changes in propulsion force, and the system intelligently assesses whether abnormal bending of the endoscope has occurred through multimodal information fusion analysis. This embodiment combines multimodal information from displacement, vision, and propulsion force sensors, and through real-time data monitoring and analysis, effectively improves the sensitivity and accuracy of endoscopic detection.

[0162] Exemplary device

[0163] Based on the above embodiments, the present invention also provides an endoscope abnormal curvature detection system, comprising:

[0164] The distal-proximal motion difference detection module is used to acquire relative motion data between the distal end of the endoscope and the instrument box;

[0165] The image-based remote motion detection module is used to acquire endoscopic image sequences and analyze the endoscopic image sequences to obtain the endoscopic motion state.

[0166] The thrust resistance detection module is used to acquire thrust force change data detected by the force sensor;

[0167] The fusion analysis module is used to perform fusion analysis based on the relative motion data, the endoscope motion state, and the propulsion force change data to obtain the endoscope abnormal bending detection results.

[0168] This embodiment achieves the following technical effects through the above technical solution:

[0169] This embodiment monitors the relative motion data between the distal endoscope and the instrument box using sensors, and processes the endoscope image sequence using optical flow analysis. The motion state of the endoscope is obtained in real time based on the image sequence. Simultaneously, force sensors detect changes in propulsion force, and the system intelligently assesses whether abnormal bending of the endoscope has occurred through multimodal information fusion analysis. This embodiment combines multimodal information from displacement, vision, and propulsion force sensors, and through real-time data monitoring and analysis, effectively improves the sensitivity and accuracy of endoscopic detection.

[0170] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 8 As shown.

[0171] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.

[0172] When executed by the processor, this computer program is used to implement the method for detecting abnormal curvature of the endoscope.

[0173] It will be understood by those skilled in the art that Figure 8 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0174] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing an endoscope abnormal curvature detection program, which, when executed by the processor, is used to implement the operation of the endoscope abnormal curvature detection method described above.

[0175] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores an endoscope abnormal curvature detection program, which, when executed by a processor, is used to implement the operation of the endoscope abnormal curvature detection method described above.

[0176] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.

[0177] In summary, this invention provides a method, system, terminal, and storage medium for detecting abnormal endoscope bending, comprising: acquiring relative motion data between the distal end of the endoscope and the instrument cassette; acquiring an endoscope image sequence and analyzing the endoscope image sequence to obtain the endoscope motion state; acquiring propulsion force change data detected by a force sensor; and performing fusion analysis based on the relative motion data, the endoscope motion state, and the propulsion force change data to obtain the endoscope abnormal bending detection result. This invention combines multimodal information from displacement, vision, and propulsion force sensors, and through real-time data monitoring and analysis, accurately identifies abnormal endoscope bending.

[0178] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for detecting abnormal curvature of an endoscope, characterized in that, include: Acquire relative motion data between the distal endoscope and the instrument cartridge, including: Collect and record the position data of the distal endoscope; The collected position data is dynamically processed and analyzed to obtain the relative motion data between the distal endoscope and the instrument box; wherein, the relative motion data includes: the displacement difference and the rate of change of the displacement difference between the distal endoscope and the instrument box; Acquire endoscopic image sequences and analyze the endoscopic motion state based on the endoscopic image sequences; Acquire propulsion force change data detected by the force sensor; Based on the fusion analysis of the relative motion data, the endoscope motion state, and the propulsion force change data, the abnormal bending detection results of the endoscope are obtained, including: Based on the displacement difference and the rate of change of the displacement difference, it is determined whether abnormal bending of the endoscope has occurred; based on the visual information of the distal end of the endoscope, it is determined whether abnormal bending of the endoscope has occurred; based on the propulsion force monitored by the force sensor, it is determined whether abnormal bending of the endoscope has occurred. The abnormal curvature detection result of the endoscope is obtained by performing fusion analysis using preset module weight coefficients.

2. The method for detecting abnormal curvature of an endoscope according to claim 1, characterized in that, The dynamic processing and analysis of the collected position data to obtain the relative motion data between the distal endoscope and the instrument box includes: The continuously acquired position data is filtered to obtain the denoised continuous displacement trajectory curve. The displacement difference between the distal end of the endoscope and the instrument box is calculated based on the continuous displacement trajectory curve, and the rate of change of the displacement difference between the distal end of the endoscope and the instrument box is calculated to obtain the relative motion data between the distal end of the endoscope and the instrument box.

3. The method for detecting abnormal curvature of an endoscope according to claim 1, characterized in that, The step of acquiring the endoscopic image sequence and analyzing the endoscopic image sequence to obtain the endoscopic motion state includes: The endoscopic image sequence is acquired, and the optical flow vector of each pixel in each frame of the endoscopic image sequence is calculated using the optical flow method. Statistical analysis is performed on the optical flow vectors of all pixels in each frame of the image to obtain the average optical flow amplitude of each frame. Based on the average optical flow amplitude of each image, determine the trend of optical flow vector change between two or more consecutive endoscopic images; The endoscope motion state is determined based on the trend of optical flow vector change; wherein, the endoscope motion state includes: normal advancement state and motion stagnation state.

4. The method for detecting abnormal curvature of an endoscope according to claim 1, characterized in that, The propulsion force change data detected by the force sensor includes: Obtain propulsion force data by acquiring data from the force sensor connected to the bottom of the drive box; The thrust force of the continuously sampled points in the thrust force data is filtered, and the average force value at each moment is calculated based on the filtered thrust force data; The difference between the average force values ​​at adjacent time points is calculated, and the rate of change of propulsion force is calculated based on the difference and the sampling interval to obtain the propulsion force change data.

5. The method for detecting abnormal curvature of an endoscope according to claim 1, characterized in that, The step of fusing and analyzing the relative motion data, the endoscope motion state, and the propulsion force change data to obtain the endoscope abnormal bending detection result includes: Abnormal bending is detected based on the displacement difference and the rate of change of displacement difference of N consecutive sampling points in the relative motion data, and the position-velocity difference detection result is obtained; where N is an integer greater than 1. Propulsion resistance is detected based on the propulsion force change data to obtain the propulsion resistance detection result; Based on the position-velocity difference detection results, the endoscope motion state, and the propulsion resistance detection results, a fusion analysis is performed using preset module weighting coefficients to obtain the endoscope abnormal bending detection results.

6. The method for detecting abnormal curvature of an endoscope according to claim 5, characterized in that, The abnormal bending detection based on the displacement difference and the rate of change of displacement difference of N consecutive sampling points in the relative motion data, to obtain the position-velocity difference detection result, includes: The probability of abnormal bending at each bronchial branch intersection, set based on prior knowledge, is obtained, and the dynamic detection threshold corresponding to the distal endoscope during movement is determined based on the obtained probability; wherein, the dynamic detection threshold includes: displacement difference threshold and displacement difference change rate threshold. Determine whether the displacement difference and the rate of change of displacement difference of N consecutive sampling points are both higher than the corresponding threshold; When the displacement difference and the rate of change of displacement difference of N consecutive sampling points are both lower than the corresponding threshold, it is determined that the endoscope has abnormally bent, and the position-velocity difference detection result is obtained.

7. The method for detecting abnormal curvature of an endoscope according to claim 6, characterized in that, The step of obtaining the probability of abnormal bending at each bronchial branch intersection based on prior knowledge, and determining the dynamic detection threshold corresponding to the distal endoscope during movement based on the obtained probability, includes: The probability of abnormal bending at each bronchial branch intersection within path L is obtained, and the overall probability of abnormal bending when passing through path L is calculated cumulatively based on the obtained probabilities. Based on the overall abnormal bending probability, the displacement difference threshold and the displacement difference change rate threshold are dynamically adjusted to obtain the dynamic detection threshold corresponding to the distal end of the endoscope during movement.

8. The method for detecting abnormal curvature of an endoscope according to claim 5, characterized in that, The step of detecting propulsion resistance based on the propulsion force change data to obtain propulsion resistance detection results includes: Obtain mechanical statistics and prior knowledge under normal propulsion conditions, and determine dual threshold conditions for propulsion force and propulsion force change rate based on the mechanical statistics and prior knowledge; Based on the aforementioned dual threshold conditions, a threshold judgment mechanism using continuous sampling points is employed to detect propulsion resistance, thereby obtaining the propulsion resistance detection result.

9. The method for detecting abnormal curvature of an endoscope according to claim 8, characterized in that, The step of detecting propulsion resistance using a threshold judgment mechanism based on continuous sampling points according to the dual threshold conditions, and obtaining the propulsion resistance detection result, includes: Based on the threshold judgment mechanism of the continuous sampling points, it is determined whether the propulsion force and the propulsion force change rate of N consecutive sampling points in the propulsion force change data all satisfy the dual threshold conditions. When the propulsion force and the rate of change of propulsion force at N consecutive sampling points both meet the dual threshold conditions, it is determined that the endoscope has abnormally bent, and the propulsion resistance detection result is obtained.

10. An endoscope abnormal curvature detection system, used to implement the endoscope abnormal curvature detection method as described in any one of claims 1-9, characterized in that, include: The distal-proximal motion difference detection module is used to acquire relative motion data between the distal end of the endoscope and the instrument box; The image-based remote motion detection module is used to acquire endoscopic image sequences and analyze the endoscopic image sequences to obtain the endoscopic motion state. The thrust resistance detection module is used to acquire thrust force change data detected by the force sensor; The fusion analysis module is used to perform fusion analysis based on the relative motion data, the endoscope motion state, and the propulsion force change data to obtain the endoscope abnormal bending detection results.

11. A terminal, characterized in that, include: The processor and memory, the memory storing an endoscope abnormal curvature detection program, which, when executed by the processor, is used to implement the operation of the endoscope abnormal curvature detection method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an endoscope abnormal curvature detection program, which, when executed by a processor, is used to implement the operation of the endoscope abnormal curvature detection method as described in any one of claims 1-9.

Images