A magnetic driven enteroscope autonomous biopsy method, system, storage medium and robot

Abstract

The application discloses a kind of magnetic drive enteroscope autonomous biopsy method, system, storage medium and robot, the method includes the following steps: step S1: by artificial intelligence image recognition technology, detect polyp in enteroscope field of view;Step S2: inflatable balloon at the front end of enteroscope is inflated, so that balloon is expanded and is attached to the inner wall of colon, anchor the position of enteroscope in intestinal tract;Step S3: the polyp is aimed at to the field of view preset biopsy area;Step S4: by binocular camera synchronous acquisition image, calculate the parallax of polyp feature point, based on triangulation principle, obtain the depth distance between the polyp and enteroscope lens;Step S5: control biopsy forceps to extend corresponding length, complete polyp clamping, sampling and retraction, realize autonomous biopsy.The application can realize from polyp identification to biopsy sampling Full autonomous operation, to improve biopsy accuracy, safety and operation efficiency, reduce the dependence on manual operation of doctor.

Description

Technical Field

[0001] This invention relates to the field of medical robotics, specifically to a method, system, storage medium, and robot for autonomous biopsy using a magnetically driven colonoscope. Background Technology

[0002] Colonoscopy and polyp biopsy are core methods for early screening and pathological diagnosis of colorectal cancer, and are also key steps in reducing the incidence and mortality of colorectal cancer. In traditional colonoscopy biopsy procedures, the entire process relies entirely on the doctor's manual operation and clinical experience: the doctor needs to manually identify the polyp target in the intestine through the monocular visual feedback of the colonoscope, manually operate the colonoscope to adjust its position to align with the polyp, and then manually control the extension and sampling of the biopsy forceps. A high degree of concentration is required throughout the entire process.

[0003] However, this traditional manual operation mode has many inherent drawbacks, making it difficult to meet the needs of clinical precision and efficiency. First, it is extremely labor-intensive for doctors; prolonged operation can easily lead to visual and physical fatigue, resulting in missed or misdiagnosed polyps and affecting diagnostic accuracy. Second, the intestines are characterized by dynamic peristalsis, complex structure, and narrow space, making manual adjustment of the colonoscope's position difficult, resulting in low polyp localization accuracy and a risk of intestinal perforation due to excessive extension of the biopsy forceps, or sampling failure due to insufficient extension. Third, the operation has a high barrier to entry; novice doctors need extensive training to master precise operating skills, resulting in a long and costly training period for professional physicians. Fourth, the operation is inefficient; polyp localization and sampling require repeated adjustments to the colonoscope's position, significantly prolonging the examination time and increasing patient discomfort.

[0004] To address these issues, semi-automated and automated colonoscopy technologies have been gradually developed both domestically and internationally in recent years, attempting to reduce reliance on manual operation. Current automated colonoscopy technologies often incorporate visual-assisted tracking and magnetic drive technologies, which alleviate the burden of manual operation to some extent. However, significant shortcomings remain, making it difficult to achieve stable, accurate, and safe fully autonomous biopsy throughout the entire process. (1) The problem of positioning drift is prominent: most systems lack a reliable colonoscope posture anchoring mechanism. When the intestines move, the colonoscope position is easily shifted, which makes the polyp positioning inaccurate and affects the biopsy accuracy.

[0005] (2) Insufficient depth measurement accuracy: Existing technologies mostly use monocular vision estimation or simple distance measurement methods, which cannot accurately obtain the three-dimensional spatial distance between the polyp and the colonoscope lens, resulting in improper control of the biopsy forceps extension length and increasing surgical risks.

[0006] (3) Poor system robustness: The visual servo control algorithm is poorly designed and lacks an effective constraint mechanism. It is prone to control instability in the dynamic environment of the intestine and cannot achieve stable tracking and alignment of polyps.

[0007] (4) Low functional integration: Most systems only achieve single-function automation and do not integrate polyp identification, anchoring, positioning, depth detection and autonomous biopsy to form a closed loop. Doctors still need to intervene and cannot achieve autonomous operation throughout the entire process. Summary of the Invention

[0008] In order to overcome the shortcomings of the prior art, one of the objectives of this invention is to provide a magnetically driven colonoscopy-based autonomous biopsy method, which can realize fully autonomous operation from polyp identification to biopsy sampling, thereby improving biopsy accuracy, safety and operational efficiency, and reducing reliance on manual operation by doctors.

[0009] The second objective of this invention is to provide a magnetically driven colonoscopy autonomous biopsy system that can achieve fully autonomous operation from polyp identification to biopsy sampling, thereby improving biopsy accuracy, safety and operational efficiency, and reducing reliance on manual operation by doctors.

[0010] The third objective of this invention is to provide a storage medium that enables fully autonomous operation from polyp identification to biopsy sampling, thereby improving biopsy accuracy, safety, and operational efficiency, and reducing reliance on manual operation by doctors.

[0011] The fourth objective of this invention is to provide a robot that can perform fully autonomous operations from polyp identification to biopsy sampling, thereby improving biopsy accuracy, safety and operational efficiency, and reducing reliance on manual operation by doctors.

[0012] To achieve one of the objectives of this invention, the following solution is adopted: A method for autonomous biopsy using a magnetically driven colonoscope includes the following steps: Step S1: Using artificial intelligence image recognition technology, detect polyps in the colonoscopy field of view and obtain the pixel position coordinates of the polyps on the image plane; Step S2: Inflate the inflatable balloon at the tip of the colonoscope to make the balloon expand and adhere to the inner wall of the colon, anchoring the colonoscope in the intestine and obtaining a stable colonoscope posture. Step S3: Under the stable colonoscope posture, using the pixel position coordinates, based on robot kinematics modeling and visual servo control, drive the robotic arm and end effector permanent magnet to move, control the colonoscope posture through magnetic coupling, align the polyp to the preset biopsy area in the field of view, and obtain the aligned polyp position. Step S4: Using the aligned polyp location, images are simultaneously acquired via binocular cameras. The disparity of the polyp's feature points is calculated. Based on the triangulation principle, the depth distance between the polyp and the colonoscope lens is obtained. The expression for the depth distance is: Where f is the focal length of the camera, b is the baseline distance, and d is the parallax; Step S5: Using the depth distance, control the biopsy forceps to extend to the corresponding length, complete the polyp clamping, sampling and retraction, and realize autonomous biopsy.

[0013] Furthermore, the kinematic modeling in step S3 includes: Establish the relationship between the speed of the permanent magnet driving the robotic arm's end effector and the joint speed: ,in For the linear velocity and angular velocity of the EPM, For the Jacobian matrix of the robotic arm, It is joint velocity; Motion mapping between the endoscope and the permanent magnet at the end of the robotic arm, based on the principle of magnetic drive: ,in For colonoscopy speed, The Jacobian matrix is ​​driven by the magnetic field; Image spatial pixel speed and colonoscopy speed satisfy the following: ,in It is the image Jacobian matrix. It is a coordinate system transformation matrix; The overall kinematics of the system is: The kinematics of the integrated system is ,in It is the total Jacobian matrix of the system.

[0014] Furthermore, the visual servo control in step S3 employs a null neural network controller based on error dynamics: in, It is an adaptive gain function. It is an array of activation functions; The control law is obtained by combining the kinematic model: This causes the pixel error index to converge to zero.

[0015] Furthermore, safety constraints on joint position and velocity are introduced in step S3: And the constraints are uniformly represented at the joint velocity layer: in, These are soft constraint adjustment parameters; Based on the aforementioned joint constraints, the control problem is transformed into a constrained quadratic programming problem: in, , , , The optimal joint velocity is solved by using ZNN combined with projection operators.

[0016] Furthermore, the solution to the quadratic programming problem employs an inverse-independent ZNN model to avoid direct matrix inversion; the model is as follows: Output optimal joint velocity: in, .

[0017] Furthermore, in step S5, the biopsy forceps extend to the corresponding length. and depth distance Satisfies a linear relationship: in, This is a proportionality coefficient to avoid extending too far or too little.

[0018] To achieve the second objective of this invention, the following solution is adopted: A magnetically driven colonoscope autonomous biopsy system, comprising: The polyp detection module is used to detect polyps in the colonoscopy field of view using artificial intelligence image recognition technology, and to obtain the pixel position coordinates of the polyps in the image plane; The balloon fixation module is used to inflate the expandable balloon at the tip of the colonoscope, causing the balloon to expand and adhere to the inner wall of the colon, thus anchoring the colonoscope in the intestine and obtaining a stable colonoscope posture. The visual servo control module is used to drive the robotic arm and the end effector permanent magnet to move by using the pixel position coordinates under the stable colonoscope posture, based on robot kinematics modeling and visual servo control, and control the colonoscope posture through magnetic coupling to align the polyp to the preset biopsy area in the field of view, thereby obtaining the aligned polyp position. The depth detection module is used to simultaneously acquire images via binocular cameras using the aligned polyp location, calculate the disparity of polyp feature points, and obtain the depth distance between the polyp and the colonoscope lens based on the triangulation principle. The expression for the depth distance is: Where f is the focal length of the camera, b is the baseline distance, and d is the parallax; The biopsy forceps control module is used to control the biopsy forceps to extend to the corresponding length using the depth distance, so as to complete the polyp clamping, sampling and retraction, and realize autonomous biopsy.

[0019] Furthermore, the robotic arm is a robot with at least 6 degrees of freedom, and the end of the colonoscope integrates a binocular camera, an inflatable balloon, and a biopsy forceps channel. The binocular camera is calibrated and its baseline is horizontally aligned.

[0020] To achieve the third objective of this invention, the following solution is adopted: A storage medium, which is a computer-readable storage medium, stores a computer program thereon, the computer program being a magnetically driven colonoscope autonomous biopsy program, characterized in that, when the magnetically driven colonoscope autonomous biopsy program is executed by a processor, it implements the steps of the magnetically driven colonoscope autonomous biopsy method as described in one of the objectives of this invention.

[0021] To achieve the fourth objective of this invention, the following solution is adopted: A robot includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being a magnetically driven colonoscope autonomous biopsy program, characterized in that, when the processor executes the magnetically driven colonoscope autonomous biopsy program, it implements the steps of the magnetically driven colonoscope autonomous biopsy method as described in one of the objectives of this invention.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves fully autonomous operation, significantly reducing reliance on manual labor. By integrating polyp identification, colonoscopy anchoring, visual servo alignment, binocular depth measurement, and automatic control of biopsy forceps into a closed loop, manual operation by doctors is eliminated, greatly reducing their workload, shortening the training cycle for specialist physicians, and facilitating the clinical promotion and application of the technology.

[0023] 2. This invention effectively suppresses positioning drift and ensures the stability of the colonoscope's posture. Utilizing an inflatable balloon anchoring mechanism, the balloon adheres tightly to the colonic wall, maintaining colonoscope posture stability even under dynamic conditions such as intestinal peristalsis. This avoids the problem of inaccurate polyp positioning caused by colonoscope displacement in traditional methods, significantly improving the controllability and accuracy of subsequent alignment and biopsy procedures.

[0024] 3. This invention achieves precise visual servo alignment with strong robustness. Combining robot kinematic modeling and visual servo control, and utilizing pixel position coordinates detected by artificial intelligence, the robotic arm and end effector permanent magnet are driven to move. Through magnetic coupling, the colonoscope's posture is precisely controlled, automatically aligning the polyp to the preset biopsy area. This solves the problems of high difficulty and low accuracy in traditional manual adjustment, ensuring the speed and stability of the alignment process.

[0025] 4. This invention significantly improves depth measurement accuracy and ensures the safety of biopsy procedures. It employs a binocular camera to acquire images and calculates the depth distance between the polyp and the lens based on the triangulation principle, overcoming the shortcomings of insufficient accuracy in monocular vision estimation. This provides an accurate distance reference for extending the biopsy forceps. Furthermore, by controlling the extension length of the biopsy forceps according to this depth distance, it effectively avoids the risks of intestinal perforation due to excessive extension or sampling failure due to insufficient extension, significantly improving the safety of biopsy procedures.

[0026] 5. This invention features high integration, forming a closed-loop autonomous biopsy process. From polyp identification, colonoscopy anchoring, visual alignment, depth detection, to biopsy execution, each step is interconnected, requiring no doctor intervention. This overcomes the shortcomings of existing technologies, such as fragmented functions and low integration, improving examination efficiency, shortening operation time, and enhancing the patient's examination experience. Attached Figure Description

[0027] Figure 1 This is a flowchart of the autonomous biopsy method using a magnetically driven colonoscope in an embodiment of the present invention; Figure 2 This is a hardware system diagram of the magnetically driven binocular colonoscope autonomous biopsy robot in an embodiment of the present invention; Figure 3 This is a diagram of a balloon anchoring model in an embodiment of the present invention; Figure 4 This is a kinematic modeling diagram of the robot system in an embodiment of the present invention; Figure 5 This is a diagram of the colonoscopy device and biopsy forceps actuator in an embodiment of the present invention; Figure 6 This is a design diagram of the end of the colonoscope in an embodiment of the present invention; Figure 7 This is a diagram of the CoppeliaSim simulation environment in an embodiment of the present invention; Figure 8 This is a schematic diagram of the motion trajectory of feature points on the end-effector camera plane of the simulated UR12e robotic arm in an embodiment of the present invention; wherein, starting from the initial position, the point eventually reaches the center of the camera plane; Figure 9 This is a pixel error map of feature points in the field of view plane in an embodiment of the present invention; Figure 10 This is a diagram showing the joint angle changes of the simulated UR12e robotic arm in an embodiment of the present invention; Figure 11 This is a diagram showing the joint velocity changes of the simulated UR12e robotic arm in an embodiment of the present invention; Figure 12 This is a schematic diagram of a depth ranging comparison table in an embodiment of the present invention; Figure 13 This is a block diagram of the magnetically driven colonoscope autonomous biopsy system in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0029] Example 1 like Figure 1-12As shown, this embodiment of the invention provides a magnetically driven colonoscope for autonomous biopsy, which is suitable for automatic identification, stable anchoring, visual servo alignment, binocular depth measurement and automatic biopsy sampling of colon polyps. The entire process can be completed autonomously without manual intervention by a doctor.

[0030] The magnetically driven colonoscopy-based autonomous biopsy method of this invention achieves automatic detection and localization of polyps through artificial intelligence image recognition technology. Combined with an inflatable balloon anchoring mechanism, it ensures colonoscopy stability. Binocular visual depth detection is used to obtain the precise distance between the polyp and the lens. With an optimized visual servo control algorithm and biopsy forceps control mechanism, precise distance control of the biopsy forceps and automated sampling are achieved. This effectively improves the automation level and operational safety of colonoscopy, reduces the workload of doctors, shortens examination time, improves patient comfort, and significantly enhances the accuracy and efficiency of biopsies, providing reliable technical support for early colorectal cancer screening.

[0031] like Figure 1 As shown, the magnetically driven colonoscope self-biopsy method of this invention includes the following steps: Step S1: Using artificial intelligence image recognition technology, detect polyps in the colonoscopy field of view and obtain the pixel position coordinates of the polyps on the image plane.

[0032] Step S2: Inflate the inflatable balloon at the tip of the colonoscope to make the balloon expand and adhere to the inner wall of the colon, anchoring the colonoscope in the intestine and obtaining a stable colonoscope posture.

[0033] Step S3: Under the stable colonoscope posture, using the pixel position coordinates, based on robot kinematics modeling and visual servo control, drive the robotic arm and end effector permanent magnet to move, control the colonoscope posture through magnetic coupling, align the polyp to the preset biopsy area in the field of view, and obtain the aligned polyp position.

[0034] Step S4: Using the aligned polyp location, images are simultaneously acquired via binocular cameras. The disparity of the polyp's feature points is calculated. Based on the triangulation principle, the depth distance between the polyp and the colonoscope lens is obtained. The expression for the depth distance is: Where f is the focal length of the camera, b is the baseline distance, and d is the parallax.

[0035] Step S5: Using the depth distance, control the biopsy forceps to extend to the corresponding length, complete the polyp clamping, sampling and retraction, and realize autonomous biopsy.

[0036] The following is a more detailed description of the magnetically driven colonoscope-based autonomous biopsy method according to an embodiment of the present invention.

[0037] like Figure 2As shown, the robotic hardware system used in the magnetically driven colonoscopy autonomous biopsy method of this invention mainly includes: a robotic arm, an added rotating mechanism, a driving permanent magnet, a colonoscopy device, a patient bed, a biopsy forceps execution unit, and a control host.

[0038] In this embodiment, step S1 is the intelligent polyp detection step, the specific content of which is as follows: The polyp detection module uses artificial intelligence image recognition technology to detect polyps in images acquired in real time by binocular colonoscopy, extracts the pixel coordinates of the polyps in the image, and provides initial input for subsequent visual servo control; In this embodiment, step S2 is the colonoscopy balloon anchoring step, the details of which are as follows: After a polyp is detected, the balloon fixation module activates the air supply device to inflate the flexible balloon at the tip of the colonoscope, causing it to expand and adhere to the inner wall of the colon, thus anchoring the colonoscope stably in the intestine and preventing intestinal peristalsis or operative disturbance from causing the colonoscope to shift, ensuring a stable and reliable biopsy process. Figure 3 The balloon anchoring model shown illustrates the inflation mechanism of the balloon at the end of the colonoscope. The model inflates the balloon using an air supply device, causing it to expand within the colon and make close contact with the intestinal wall, thus anchoring the colonoscope in place. This design effectively prevents colonoscope displacement during the procedure, ensuring stability during biopsies. The balloon is made of a flexible material, allowing it to adapt to different intestinal shapes after inflation, reducing the risk of tissue damage.

[0039] In this embodiment, step S3 is the visual servo control and QP optimization solution step, which includes three parts: kinematic modeling, visual servo control, and QP problem solving. The specific contents are as follows: Specifically, kinematic modeling: like Figure 4 As shown, based on the principle of differential kinematics, the mapping relationship between the external driving permanent magnet (EPM) at the end of the robotic arm and the colonoscope is established. The system kinematic equation is: ,in This is the system's Jacobian matrix. Specifically: According to differential kinematics, the relationship between the EPM velocity attached to the end effector of the robotic arm and the joint velocity is as follows: in, Represented as a vector consisting of the linear velocity and angular velocity of the EPM. It is the Jacobian matrix of the robotic arm ( (The number of degrees of freedom of the robotic arm). It is the joint velocity vector of the robotic arm.

[0040] Based on the principle of magnetic drive, the motion mapping relationship between the colonoscope and the EPM can be highly linearized. Considering that the colonoscope mainly adjusts its field of view through rotation in the narrow intestinal environment, and that the rotational motion around the magnetization direction of the colonoscope magnet is uncontrollable, the motion mapping relationship between the EPM and the colonoscope is constructed as follows: in, It's the speed of the colonoscopy. The Jacobian matrix is ​​driven by the magnetic field.

[0041] In image space, imagine target features in three-dimensional space. The projection on the image plane of one of the cameras is Its pixel speed With colonoscopy speed The motion mapping relationship is as follows: in, It is the image Jacobian matrix, which reflects the mapping relationship between colonoscopy motion and the change of pixel position of polyps in image space. It is a coordinate system mapping matrix used to realize the transformation between the colonoscopy coordinate system and the image coordinate system.

[0042] By combining the above mapping relationships, we finally obtain the complete kinematic equations of the system: here, It is the system Jacobian matrix.

[0043] Specifically, visual servo control: The actual location of polyp feature points in image space Drive to the preset desired position (i.e., the pre-defined biopsy area within the field of view), the error vector is defined as follows: ( ), and ensure error It converges rapidly to zero.

[0044] The error dynamics equation based on ZNN is designed as follows: in, It is an adaptive gain function. This is the activation function.

[0045] Combining the system's kinematic equations The equations for the visual servo controller are derived as follows: in, Let be the velocity vector of the desired position of the polyp feature point. Here, we set it to zero vector to achieve stable dwell of the polyp feature point at the desired position.

[0046] To further improve the robustness and convergence speed of the system, an adaptive gain function is designed as follows: in, , For minimum gain, For maximum gain, This is the gain adjustment coefficient. Through this adaptive design, the gain can be dynamically adjusted according to the magnitude of the error. When the error is large, the gain is increased to speed up convergence, and when the error is small, the gain is decreased to ensure control stability.

[0047] The activation function employs a multi-parameter composite activation function to improve the speed and robustness of error convergence. The expression is as follows: in, This ensures the effective functioning of the nonlinear term; The weights of the linear terms can be adjusted according to actual control requirements; This is used to adjust the convergence characteristics of nonlinear terms; , , The value is a natural number to ensure the smoothness and differentiability of the activation function, which facilitates the solution and implementation of the controller.

[0048] Specifically, the construction and solution of multi-constraint QP problems: First, clarify the joint safety constraints of the robotic arm, including joint angle constraints and joint speed constraints: in These are the minimum and maximum angle limits for the robotic arm joints, respectively. These are the minimum and maximum speed limits for the robotic arm joints, respectively.

[0049] To avoid impact when the joint moves to its limit, a soft constraint design is adopted, which transforms the hard limit into a smooth soft constraint output. The soft constraint expression is as follows: in, For soft constraint adjustment parameters, typical parameter settings are: It can be fine-tuned according to the actual parameters of the robotic arm to ensure the smoothness and safety of joint movement.

[0050] Based on the above constraints, the joint velocity problem is constructed as a QP problem. The goal is to find the optimal joint velocity while satisfying the constraints, thereby minimizing the control error. The QP problem is constructed as follows: in The joint velocity vector to be solved is... It is a positive definite weight matrix, which can be set as an identity matrix to ensure the rationality of the objective function. It is the system Jacobian matrix. It is the target vector output by the visual servo controller.

[0051] To improve the real-time performance of the solution and avoid the high computational complexity and numerical instability caused by directly inverting the Jacobian matrix, a ZNN is used to solve this QP problem. For this QP problem, the KKT conditions are equivalent to: in, These are the Lagrange multipliers corresponding to equality constraints. These are the Lagrange multipliers corresponding to inequality constraints. It is a projection operator, defined as follows (for vectors) The (number of elements) in, Let this be the scaling factor. Transform the equation... Left multiplication : use Simplified, we get: Substitute into the equation : in Assuming If the row rank is full (feasible for visual servoing tasks), then Reversible, solution: Substitution equation We can obtain:

[0052] Summarized as follows: Define matrix: The decision variables can then be expressed as:

[0053] Substitute into the projection equation: After simplification, the error function is defined as follows: The error is reduced based on the ZNN design formula. Converging along the negative gradient direction: in, For adaptive gain, The activation function, and the design of its two parameters, are as described in the controller design submodule. The dynamic equation updates the multipliers online. This ensures that the inequality constraints are satisfied.

[0054] Define the state of a neural network The goal is to The error dynamics equation based on ZNN is then: As time goes on... converged to Therefore, it is available replace This avoids directly inverting the values ​​and improves real-time performance.

[0055] In summary, we obtain the inversely independent ZNN model: Where the matrix and Updated to: The final output joint speed is: In this embodiment, step S4 is the binocular depth detection step, the details of which are as follows: The depth detection module acquires images simultaneously using binocular cameras and calculates parallax. And obtain depth based on the principle of triangulation. ,in, It's the camera's focal length. It is the baseline distance between the left and right cameras.

[0056] In this embodiment, step S5 is the automatic biopsy execution step, the specific content of which is as follows: like Figure 5As shown, the biopsy forceps control module extends the forceps to the corresponding length based on depth information, accurately reaching the polyp location and automatically completing the grasping, sampling, and retraction actions. This avoids excessive or insufficient insertion, improving the success rate and safety of the biopsy. After the biopsy is completed, the balloon is deflated, the anchoring is released, and the colonoscope can continue the examination.

[0057] like Figure 6 As shown in the diagram, the colonoscope tip design model details the integrated structure of the binocular cameras, balloon, and biopsy forceps channel. The colonoscope tip features a compact design with symmetrically arranged left and right cameras and a fixed baseline distance to ensure accurate depth detection. The biopsy forceps extend from the central channel, and their extension length is precisely controlled by the depth detection module.

[0058] like Figure 7 As shown, the UR12e robotic arm, driving magnet, colonoscopy balloon and end cap were deployed in the CoppeliaSim simulation environment to realistically recreate a clinical colonoscopy and to verify the method proposed in this invention.

[0059] like Figure 8 As shown, the polyp feature points starting from the initial position smoothly converge to the center of the image under the ZNN visual servo control, with no oscillation in the trajectory, verifying the controller's fast convergence and strong robustness.

[0060] like Figure 9 As shown, the pixel error of the feature points smoothly converges to 0 under ZNN visual servo control, and the trajectory has no oscillation.

[0061] like Figure 10 As shown, The angles of the six joints of the UR12e robotic arm and the angle of the seventh rotary joint added to the end effector are respectively represented. During the task execution, the changes of each joint angle are continuous and stable, without sudden changes or exceeding the limits, proving that the constraints are effective.

[0062] like Figure 11 As shown, The θ represents the joint angular velocity of the robot system. During task execution, the rate of change of each joint angle can be guaranteed to vary within a certain range, demonstrating the effectiveness of this invention in handling double-ended constraints and joint limit avoidance in quadratic programming problems.

[0063] like Figure 12 As shown, this embodiment provides the accuracy verification results of the binocular depth detection module. Through high-precision experimental statistics, the binocular ranging method exhibits excellent performance, with an average absolute error of 0.066 cm and a relative error of 0.77%. These results demonstrate that the binocular vision ranging method employed in this invention possesses sub-millimeter-level accuracy and high consistency, far exceeding traditional monocular or other ranging methods, thus providing reliable distance information for the precise extension and retraction control of biopsy forceps.

[0064] The magnetically driven colonoscope autonomous biopsy method of this invention has the following advantages: 1. This invention achieves fully autonomous operation, significantly reducing reliance on manual labor. This invention integrates core technologies such as intelligent polyp detection, balloon anchoring, visual servo alignment, binocular depth measurement, and autonomous biopsy, forming a complete autonomous biopsy closed loop. The entire process from polyp identification to biopsy sampling requires no manual operation by the doctor, significantly reducing reliance on the doctor's clinical experience, alleviating the doctor's workload, and shortening the training cycle for professional physicians, facilitating the clinical application of the technology. 2. This invention effectively suppresses positioning drift, ensuring the stability of the colonoscope's posture. This invention uses an inflatable balloon to anchor the colonoscope's posture within the intestine, effectively counteracting positional drift caused by intestinal peristalsis, ensuring stable colonoscope posture, providing a stable foundation for polyp alignment, depth detection, and biopsy operations, significantly improving the accuracy of biopsy positioning and operational stability, and avoiding missed detections, false detections, and intestinal tissue damage caused by colonoscope deviation. 3. This invention achieves precise and reliable visual servo control with high robustness. This invention employs a zero-dimensional neural network (ZNN) to design a visual servo controller, combining an adaptive gain function and a composite activation function to achieve exponential convergence of polyp pixel errors, resulting in strong anti-interference capabilities. Simultaneously, through quadratic programming (QP) constraints and inverse-independent ZNN solutions, it ensures the robotic arm moves within a safe range, avoiding joint over-limits and improving system robustness and control precision, enabling it to adapt to the complex environment of dynamic intestinal peristalsis. 4. This invention significantly improves depth detection accuracy, ensuring the safety of biopsy procedures. This invention uses binocular vision stereo matching combined with triangulation principles. Compared to existing monocular vision estimation and simple ranging methods, depth measurement accuracy is higher, enabling precise acquisition of the three-dimensional spatial distance between the polyp and the lens. This allows for precise control of the biopsy forceps extension length, effectively avoiding the risk of intestinal perforation due to excessive forceps extension or sampling failure due to insufficient extension, significantly improving the safety of biopsy procedures. 5. This invention adopts a modular design, offering strong real-time performance and facilitating maintenance and upgrades. This invention employs a modular design, with each module functioning independently yet collaboratively, resulting in a clear structure that facilitates later maintenance, debugging, and functional upgrades. Simultaneously, the inverse-independent ZNN model avoids direct matrix inversion, resulting in low computational complexity and significantly improving the system's real-time control performance, meeting the real-time requirements of clinical surgery. 6. This invention significantly improves operational safety and patient comfort. Through soft constraint of the robotic arm joints, precise control of the biopsy forceps, and non-invasive magnetic drive technology, this invention ensures that the entire operation is performed within a safe range, reducing medical risks. The magnetic drive method eliminates mechanical contact, reducing the mechanical stimulation of the intestines caused by traditional manual operations, minimizing patient discomfort during the examination, and enhancing the patient's examination experience.

[0065] Example 2 like Figure 13As shown, this embodiment of the invention also provides a magnetically driven colonoscope autonomous biopsy system, comprising: The polyp detection module is used to detect polyps in the colonoscopy field of view using artificial intelligence image recognition technology, and to obtain the pixel position coordinates of the polyps in the image plane; The balloon fixation module is used to inflate the expandable balloon at the tip of the colonoscope, causing the balloon to expand and adhere to the inner wall of the colon, thus anchoring the colonoscope in the intestine and obtaining a stable colonoscope posture. The visual servo control module is used to drive the robotic arm and the end effector permanent magnet to move by using the pixel position coordinates under the stable colonoscope posture, based on robot kinematics modeling and visual servo control, and control the colonoscope posture through magnetic coupling to align the polyp to the preset biopsy area in the field of view, thereby obtaining the aligned polyp position. The depth detection module is used to simultaneously acquire images via binocular cameras using the aligned polyp location, calculate the disparity of polyp feature points, and obtain the depth distance between the polyp and the colonoscope lens based on the triangulation principle. The expression for the depth distance is: Where f is the focal length of the camera, b is the baseline distance, and d is the parallax; The biopsy forceps control module is used to control the biopsy forceps to extend to the corresponding length using the depth distance, so as to complete the polyp clamping, sampling and retraction, and realize autonomous biopsy.

[0066] Furthermore, the robotic arm is a robot with at least 6 degrees of freedom, and the end of the colonoscope integrates a binocular camera, an inflatable balloon, and a biopsy forceps channel. The binocular camera is calibrated and its baseline is horizontally aligned.

[0067] The magnetically driven colonoscope autonomous biopsy system of this invention will be further described in detail below.

[0068] The magnetically driven colonoscope autonomous biopsy system of this invention includes: a polyp detection module, a balloon fixation module, a visual servo control module, a depth detection module, and a biopsy forceps control module. These modules work together to form a complete autonomous biopsy closed-loop system. The specific structure and function of each module are as follows: Specifically, the polyp detection module is used to detect and locate polyps within the surgical field of view using artificial intelligence image recognition technology. It includes an image acquisition unit and a recognition and processing unit. The image acquisition unit is connected to the binocular vision component at the front end of the colonoscope, acquiring binocular image data of the intestine in real time and transmitting it to the recognition and processing unit. The recognition and processing unit, based on a pre-trained deep learning model (such as an improved YOLO model), performs real-time preprocessing, feature extraction, and target recognition on the acquired images, automatically detecting polyp targets within the field of view and accurately extracting the pixel coordinates of the polyps. This coordinate information is used as the initial input signal for the visual servo control module, providing basic data support for subsequent polyp alignment and localization. Specifically, the balloon fixation module is used to stably anchor the colonoscope within the intestine, preventing displacement due to intestinal peristalsis during the procedure and ensuring the stability of subsequent positioning, depth detection, and biopsy operations. It includes an inflatable balloon, an air supply interface, and an air supply device. The inflatable balloon is installed on the outer side of the colonoscope's tip and connects to an external air supply device via the air supply interface. When the polyp detection module detects a polyp and outputs positioning information, the air supply device inflates the inflatable balloon through the air supply interface, causing the balloon to expand and fit tightly against the intestinal wall. The reaction force of the intestinal wall is used to fix the colonoscope's position and orientation within the colon. After the biopsy is completed, the air supply device stops supplying air and deflates, causing the balloon to contract, releasing the colonoscope from anchorage, and facilitating further advancement of the colonoscope for subsequent intestinal examinations. Specifically, the visual servo control module is used to drive the robotic arm to move an external permanent magnet (EPM) based on robot kinematic modeling and visual servo algorithms. This drives the colonoscope to adjust its posture through magnetic drive, achieving automatic alignment and stable tracking of polyps. The module includes a kinematic modeling submodule, a controller design submodule, and an optimization solution submodule. These submodules work together to ensure the accuracy, robustness, and safety of the control.

[0069] The kinematic modeling submodule is used to establish the kinematic relationships of the entire system, clarifying the motion mapping between the robotic arm, the externally driven permanent magnet (EPM), the colonoscope, and the image space, providing a theoretical basis for controller design. Based on the principle of differential kinematics, the mapping relationship between the externally driven permanent magnet (EPM) at the end of the robotic arm and the colonoscope is established, and the system kinematic equations are as follows: ,in This is the system Jacobian matrix. The specific details regarding kinematic mapping relationships, visual servo control laws, constraint transformations, and solving the inverse independent ZNN model are the same as those described in step S3 of this embodiment.

[0070] The controller design submodule is used to design a visual servo controller to achieve accurate tracking and alignment of polyp feature points, ensuring rapid error convergence and improving system robustness. This invention uses a zero-sum neural network (ZNN) to design the visual servo controller, based on the principle of error dynamics, enabling the polyp pixel error to converge exponentially to zero, thus achieving precise control.

[0071] The optimization solution submodule is used to optimize the joint speed output by the controller. It transforms the joint limit constraints of the robotic arm into a quadratic programming (QP) problem and uses the ZNN algorithm to solve for the optimal joint speed online. This ensures that the robotic arm moves within a safe range, avoids mechanical damage caused by joint over-limit, and improves the real-time performance of the control.

[0072] Specifically, the depth detection module is used to accurately obtain the three-dimensional spatial distance between the polyp and the colonoscope lens, providing precise data support for controlling the extension length of the biopsy forceps and avoiding operational risks caused by inaccurate distance measurement. This module is based on the principle of binocular vision, simultaneously acquiring two images of the intestine through a binocular camera mounted at the front of the colonoscope. After image preprocessing and stereo matching, the disparity of the polyp's feature points in the left and right images is calculated. Then, combined with the intrinsic (focal length) and extrinsic (baseline distance) parameters of the binocular camera, the precise depth between the polyp and the lens is calculated using the principle of triangulation. Specifically: Using a stereo matching algorithm, feature points were matched between the left and right images captured by the binocular camera to calculate the x-coordinate of the polyp feature point in the left image. The x-coordinate in the right image The difference between the two is the parallax. : Based on the principle of triangulation, combined with the focal length of a binocular camera... Distance from binocular baseline (i.e., the distance between the optical centers of the two cameras), the depth of the polyp relative to the endoscope lens is calculated. : Among them, focal length Distance from baseline The fixed parameters for the binocular camera were determined during the system calibration phase to ensure the accuracy of depth measurement; parallax. The depth measurement is calculated in real time by a stereo matching algorithm, which ensures the real-time nature of the depth measurement and ultimately achieves accurate and real-time detection of polyp depth.

[0073] Specifically, the biopsy forceps control module automatically controls the extension length, jaw opening and closing, and sampling action of the biopsy forceps based on the distance information between the polyp and the lens output by the depth detection module, achieving precise and safe autonomous biopsy. This module includes a biopsy forceps actuator and a control unit. The biopsy forceps actuator is integrated into the colonoscope and can extend and retract along the colonoscope axis; the jaws can open and close. The control unit communicates with the depth detection module and the visual servo control module, receiving polyp depth and positioning information in real time.

[0074] Once the visual servo control module aligns the polyp with the preset biopsy area and the depth detection module outputs a precise depth distance, the control unit automatically calculates the required extension length of the biopsy forceps based on this depth distance. It then drives the biopsy forceps to extend the corresponding length along the colonoscope axis, ensuring the forceps accurately reach the polyp location. Subsequently, it controls the opening and closing of the jaws to clamp the polyp tissue and complete the sampling. After sampling, the control unit drives the biopsy forceps back into the colonoscope and simultaneously sends a signal to the air supply device to deflate the inflatable balloon, releasing the colonoscope from its anchor and facilitating further advancement for subsequent intestinal examinations and biopsy procedures. Based on the distance information output by the depth detection module, the control unit automatically extends the biopsy forceps to the appropriate length, then automatically opens and closes the forceps, precisely completing the biopsy operation and avoiding excessive or insufficient insertion.

[0075] The workflow of the magnetically driven colonoscope autonomous biopsy system according to this invention is as follows: 1. Intelligent Polyp Detection: The colonoscope is slowly inserted into the human intestine. The binocular camera at the front of the colonoscope acquires binocular image data of the intestine in real time and transmits it to the polyp detection module. The recognition and processing unit of the polyp detection module is based on a pre-trained deep learning model. It performs real-time preprocessing (denoising, enhancement, normalization), feature extraction and target recognition on the acquired images, automatically detects polyp targets in the colonoscope's field of view, and accurately extracts the pixel position coordinates of the polyp in the image. The coordinate information is transmitted to the visual servo control module to complete the intelligent detection and localization of the polyp. 2. Colonoscopy Balloon Anchoring: After the polyp detection module successfully detects a polyp and outputs positioning information, the air supply device of the balloon fixation module receives a control signal and inflates the inflatable balloon at the front end of the colonoscope through the air supply interface. The balloon gradually expands during the inflation process until it fits tightly against the intestinal wall. The reaction force of the intestinal wall is used to anchor the colonoscope in the intestinal posture, preventing intestinal peristalsis from causing the colonoscope to shift position. This provides a stable foundation for subsequent polyp alignment, depth detection, and biopsy operations. 3. Visual Servo Alignment: The visual servo control module receives the polyp pixel position coordinates output by the polyp detection module, and calculates the error between the current position and the desired position of the polyp by combining the preset biopsy area position (usually the center of the colonoscope's field of view); the kinematic modeling submodule establishes the motion mapping between the robotic arm, EPM, colonoscope, and image space based on the system's kinematic equations; the controller design submodule calculates the control signal based on a zero-form neural network (ZNN) combined with an adaptive gain function and a composite activation function; the optimization solution submodule transforms the joint constraint of the robotic arm into a quadratic programming (QP) problem, solves the optimal joint speed online through an inverse independent ZNN model, drives the robotic arm to move the EPM, and then adjusts the posture and position of the colonoscope through magnetic drive, accurately moving the polyp to the preset biopsy area in the field of view, thus completing the precise alignment of the polyp. 4. Binocular Depth Measurement: After the polyp is aligned, the depth detection module initiates the binocular depth detection process: the binocular cameras simultaneously acquire two images of the polyp area from the left and right sides. After image preprocessing, feature point matching, and other steps, the disparity of the polyp feature points is calculated. Combining the focal length of the binocular cameras with the baseline distance, the three-dimensional spatial distance between the polyp and the colonoscope lens is accurately calculated using the triangulation principle. This depth distance information is transmitted to the biopsy forceps control module, providing precise data support for controlling the extension length of the biopsy forceps. 5. Autonomous Biopsy Execution: The biopsy forceps control module receives depth distance information output by the depth detection module, automatically calculates the required extension length of the biopsy forceps, and drives the biopsy forceps to extend the corresponding length along the colonoscope axis to ensure that the biopsy forceps accurately reach the polyp location; then, it controls the opening and closing of the biopsy forceps jaws to grasp the polyp tissue and complete the sampling; after sampling, it controls the biopsy forceps to retract into the colonoscope and simultaneously sends a deflation signal to the air supply device, which stops supplying air and releases the gas in the balloon, causing the balloon to contract and releasing the colonoscope from its anchoring state; the colonoscope returns to a mobile state and continues to advance to the next area of ​​the intestine, repeating the above steps to complete the autonomous biopsy examination of the entire intestine.

[0076] Example 3 This invention also provides a storage medium, which is a computer-readable storage medium storing a computer program thereon. The computer program is a magnetically driven colonoscope autonomous biopsy program, characterized in that, when the magnetically driven colonoscope autonomous biopsy program is executed by a processor, it implements the steps of the magnetically driven colonoscope autonomous biopsy method as described in Embodiment 1.

[0077] Example 4 This invention also provides a robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is a magnetically driven colonoscope autonomous biopsy program. The robot is characterized in that when the processor executes the magnetically driven colonoscope autonomous biopsy program, it implements the steps of the magnetically driven colonoscope autonomous biopsy method as described in Embodiment 1.

[0078] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for autonomous biopsy using a magnetically driven colonoscope, characterized in that, Includes the following steps: Step S1: Using artificial intelligence image recognition technology, detect polyps in the colonoscopy field of view and obtain the pixel position coordinates of the polyps on the image plane; Step S2: Inflate the inflatable balloon at the tip of the colonoscope to make the balloon expand and adhere to the inner wall of the colon, anchoring the colonoscope in the intestine and obtaining a stable colonoscope posture. Step S3: Under the stable colonoscope posture, using the pixel position coordinates, based on robot kinematics modeling and visual servo control, drive the robotic arm and end effector permanent magnet to move, control the colonoscope posture through magnetic coupling, align the polyp to the preset biopsy area in the field of view, and obtain the aligned polyp position. The kinematic modeling includes: Establish the relationship between the speed of the permanent magnet driving the robotic arm's end effector and the joint speed: ,in For the linear velocity and angular velocity of the EPM, For the Jacobian matrix of the robotic arm, It is joint velocity; Motion mapping between the endoscope and the permanent magnet at the end of the robotic arm, based on the principle of magnetic drive: ,in For colonoscopy speed, The Jacobian matrix is ​​driven by the magnetic field; Image spatial pixel speed and colonoscopy speed satisfy the following: ,in It is the image Jacobian matrix. It is a coordinate system transformation matrix; The overall kinematics of the system is: The kinematics of the integrated system is ,in It is the total Jacobian matrix of the system; In step S3, joint position and velocity safety constraints are introduced and uniformly represented at the joint velocity layer. Based on the joint position and velocity safety constraints, the control problem is transformed into a constrained quadratic programming problem, and the optimal joint velocity is solved by ZNN combined with projection operators. Step S4: Using the aligned polyp location, images are simultaneously acquired via binocular cameras. The disparity of the polyp's feature points is calculated. Based on the triangulation principle, the depth distance between the polyp and the colonoscope lens is obtained. The expression for the depth distance is: Where f is the focal length of the camera, b is the baseline distance, and d is the parallax; Step S5: Using the depth distance, control the biopsy forceps to extend to the corresponding length, complete the polyp clamping, sampling and retraction, and realize autonomous biopsy.

2. The method for autonomous biopsy using a magnetically driven colonoscope according to claim 1, characterized in that, The visual servo control in step S3 employs a null neural network controller based on error dynamics. in, It is an adaptive gain function. It is an array of activation functions; The control law is obtained by combining the kinematic model: This causes the pixel error index to converge to zero.

3. The method for autonomous biopsy using a magnetically driven colonoscope according to claim 1, characterized in that, The quadratic programming problem is solved using the inverse-independent ZNN model, avoiding direct matrix inversion.

4. The method for autonomous biopsy using a magnetically driven colonoscope according to claim 1, characterized in that, In step S5, the biopsy forceps extend to the corresponding length. and depth distance Satisfies a linear relationship: in, This is a proportionality coefficient to avoid extending too far or too little.

5. A magnetically driven colonoscope autonomous biopsy system, characterized in that, include: The polyp detection module is used to detect polyps in the colonoscopy field of view using artificial intelligence image recognition technology, and to obtain the pixel position coordinates of the polyps in the image plane; The balloon fixation module is used to inflate the expandable balloon at the tip of the colonoscope, causing the balloon to expand and adhere to the inner wall of the colon, thus anchoring the colonoscope in the intestine and obtaining a stable colonoscope posture. The visual servo control module is used to drive the robotic arm and the end effector permanent magnet to move by using the pixel position coordinates under the stable colonoscope posture, based on robot kinematics modeling and visual servo control, and control the colonoscope posture through magnetic coupling to align the polyp to the preset biopsy area in the field of view, thereby obtaining the aligned polyp position. The kinematic modeling includes: Establish the relationship between the speed of the permanent magnet driving the robotic arm's end effector and the joint speed: ,in For the linear velocity and angular velocity of the EPM, For the Jacobian matrix of the robotic arm, It is joint velocity; Motion mapping between the endoscope and the permanent magnet at the end of the robotic arm, based on the principle of magnetic drive: ,in For colonoscopy speed, The Jacobian matrix is ​​driven by the magnetic field; Image spatial pixel speed and colonoscopy speed satisfy the following: ,in It is the image Jacobian matrix. It is a coordinate system transformation matrix; The overall kinematics of the system is: The kinematics of the integrated system is ,in It is the total Jacobian matrix of the system; The visual servo control module also introduces joint position and velocity safety constraints, and represents these constraints uniformly at the joint velocity layer. Based on the joint position and velocity safety constraints, the control problem is transformed into a constrained quadratic programming problem, and the optimal joint velocity is solved by using ZNN combined with projection operators. The depth detection module is used to simultaneously acquire images via binocular cameras using the aligned polyp location, calculate the disparity of polyp feature points, and obtain the depth distance between the polyp and the colonoscope lens based on the triangulation principle. The expression for the depth distance is: Where f is the focal length of the camera, b is the baseline distance, and d is the parallax; The biopsy forceps control module is used to control the biopsy forceps to extend to the corresponding length using the depth distance, so as to complete the polyp clamping, sampling and retraction, and realize autonomous biopsy.

6. The system according to claim 5, characterized in that, The robotic arm is a robot with at least 6 degrees of freedom. The end of the colonoscope integrates a binocular camera, an inflatable balloon, and a biopsy forceps channel. The binocular camera is calibrated and its baseline is horizontally aligned.

7. A storage medium, said storage medium being a computer-readable storage medium, having stored thereon a computer program, said computer program being a magnetically driven colonoscope autonomous biopsy program, characterized in that, When the magnetically driven colonoscope autonomous biopsy program is executed by the processor, it implements the steps of the magnetically driven colonoscope autonomous biopsy method as described in any one of claims 1-4.

8. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, said computer program being a magnetically driven colonoscope autonomous biopsy program, characterized in that, When the processor executes the autonomous biopsy procedure of the magnetically driven colonoscope, it implements the steps of the autonomous biopsy method of the magnetically driven colonoscope as described in any one of claims 1-4.

Images