Intelligent choledochoscope assisted minimally invasive calculus removal method and system for bile duct calculus

Abstract

The invention provides an intelligent choledochoscope-assisted bile duct calculus minimally invasive calculus removal method and system, and relates to the technical field of minimally invasive calculus removal, and the method comprises the steps: collecting a video image in a bile duct through a choledochoscope, extracting bile duct calculus feature information through a deep learning target detection network, and carrying out the enhancement processing of the image to obtain bile duct cavity wall features; constructing a three-dimensional bile duct blood vessel tree model with focus marks; calculating a risk coefficient based on bile duct calculus characteristics, inputting the three-dimensional model, an anatomy knowledge base and expert experience data into a deep reinforcement learning network, and generating an optimal calculus taking path; the system establishes a real-time three-dimensional navigation map, and real-time path correction is realized through dynamic registration; a visual servo and force feedback control method is adopted, the choledochoscope robot is controlled to move to the target position along the corrected path, the grabbing parameters of the end effector are adjusted according to the morphological characteristics of the calculus, and accurate calculus taking is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of minimally invasive lithotripsy, and in particular to a method and system for minimally invasive lithotripsy of bile duct stones assisted by an intelligent choledochoscope. Background Art

[0002] Bile duct stones are one of the common diseases in hepatobiliary surgery. Traditional surgical methods are highly traumatic and have many postoperative complications. Currently, minimally invasive choledochoscopy and stone removal surgery has become the main method for treating bile duct stones. With the development of robotics and artificial intelligence, the intelligent choledochoscopy-assisted system has shown important value in improving surgical accuracy and safety. This technology uses a choledochoscope to obtain real-time images of the bile duct, combined with robotic control technology to achieve precise positioning and removal of stones, greatly reducing surgical trauma and the risk of complications.

[0003] There are still some problems with the existing choledochoscope stone removal technology. Traditional image processing methods are difficult to accurately identify the location and characteristics of stones in a complex bile duct environment; there is a lack of an intelligent surgical path planning system, and doctors need to rely on experience to operate, which can easily cause bile duct damage; the existing choledochoscope robot has insufficient control accuracy and is unable to adjust the operation strategy in real time according to the elastic deformation of the bile duct, increasing the risk of surgery; the existing system lacks intraoperative real-time navigation and path optimization functions, and is difficult to cope with dynamic changes in a complex bile duct environment.

[0004] In summary, there is an urgent need for an intelligent choledochoscope-assisted stone removal system based on deep learning, which can accurately identify and extract features of stones through a deep learning network, establish a dynamic three-dimensional bile duct model, use a deep reinforcement learning algorithm, and calculate the optimal stone removal path in combination with expert experience data; realize accurate navigation and dynamic path correction of the choledochoscope through visual servo and force feedback control; and adaptively adjust the grasping parameters according to the stone characteristics to ensure the safety and effectiveness of the operation. The present invention can solve the problems in the prior art. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for minimally invasive bile duct stone removal assisted by an intelligent choledochoscope, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provided is a minimally invasive method for removing bile duct stones assisted by an intelligent choledochoscope, comprising:

[0008] Using the bile duct video images collected in real time by a choledochoscope, a deep learning target detection network is constructed to process the bile duct video images to obtain characteristic information of bile duct stones; image enhancement processing is performed on the bile duct video images to extract characteristic information of the bile duct cavity wall and establish a three-dimensional bile duct vascular tree model; the characteristic information of the bile duct stones is annotated in the three-dimensional bile duct vascular tree model to generate a three-dimensional bile duct structure model with lesion annotations, and the risk coefficient value of each bile duct stone is calculated based on the characteristic information of the bile duct stones;

[0009] The three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data are input into the deep reinforcement learning network, and the optimal stone removal path is calculated based on the risk coefficient value, and the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; a real-time three-dimensional navigation map is established according to the optimal stone removal path, and the real-time spatial position information of the choledochoscope is collected, and the real-time spatial position information is dynamically aligned with the real-time three-dimensional navigation map to generate real-time alignment error data, and the optimal stone removal path is compensated and corrected based on the real-time alignment error data to obtain a corrected optimal stone removal path;

[0010] The corrected optimal stone removal path is input into the predictive control model, and the motion control parameters of the choledochoscope robot are calculated using the visual servo control method and the force feedback control method. Based on the motion control parameters, the choledochoscope robot is controlled to move along the corrected optimal stone removal path to the target bile duct stone position; the end effector of the choledochoscope robot is controlled to adjust the grasping force value and the grasping posture angle based on the morphological feature information of the bile duct stone to complete the grasping action and removal operation of the bile duct stone.

[0011] In an optional embodiment,

[0012] The video image of the bile duct collected in real time by a choledochoscope is used to construct a deep learning target detection network to process the video image of the bile duct to obtain characteristic information of bile duct stones; the video image of the bile duct is enhanced to extract characteristic information of the bile duct cavity wall and establish a three-dimensional bile duct vascular tree model; the characteristic information of the bile duct stones is annotated in the three-dimensional bile duct vascular tree model to generate a three-dimensional bile duct structure model with lesion annotations, and the risk coefficient value of each bile duct stone is calculated based on the characteristic information of the bile duct stones, including:

[0013] The intra-biliary video image is input into a deep learning target detection network, the deep learning target detection network extracts image depth features through multi-level residual convolution units, the image depth features are input into a cross-layer feature fusion module, the cross-layer feature fusion module uses an adaptive weight allocation mechanism to perform multi-scale feature aggregation on the image depth features to obtain a target feature map; the target feature map is deconvoluted and reconstructed with dense features to obtain spatial position information, morphological contour information, and density distribution information of bile duct stones;

[0014] Based on the gray value distribution of each pixel in the intra-biliary video image, a pixel-level gain matrix is ​​constructed, and based on the edge gradient information in the intra-biliary video image, a regional contrast enhancement matrix is ​​constructed; the pixel-level gain matrix is ​​multiplied with the intra-biliary video image to obtain a first enhanced image, and the regional contrast enhancement matrix is ​​weightedly superimposed with the first enhanced image to obtain an enhanced bile duct image;

[0015] The enhanced bile duct image is input into a semantic segmentation network, and multi-scale feature information of the enhanced bile duct image is extracted through a multi-layer spatial pyramid pooling module, and input into a feature progressive fusion module, and the multi-scale feature information is upsampled and reconstructed to obtain bile duct lumen wall feature information;

[0016] The bile duct wall feature information is voxel-divided using a quadtree structure, and surface continuity constraints and topology preservation constraints are imposed on the divided grid structure to generate a three-dimensional bile duct vascular tree model; the spatial position information, the morphological contour information, and the density distribution information of the bile duct stone are mapped to corresponding positions of the three-dimensional bile duct vascular tree model to generate a three-dimensional bile duct structure model with lesion annotations;

[0017] A multidimensional feature set of bile duct stones is extracted based on the three-dimensional bile duct structure model, and the multidimensional feature set includes size feature values, location hazard values, density distribution values ​​and tissue damage risk values. The multidimensional feature set is input into the risk prediction module, and feature fusion weight parameters are obtained through training based on preset expert annotation data. The multidimensional feature set is weightedly fused using the attention mechanism to output the risk coefficient value of bile duct stones.

[0018] In an optional embodiment,

[0019] The enhanced bile duct image is input into a semantic segmentation network, and multi-scale feature information of the enhanced bile duct image is extracted through a multi-layer spatial pyramid pooling module, and input into a feature progressive fusion module, and the multi-scale feature information is upsampled and reconstructed to obtain bile duct wall feature information including:

[0020] The enhanced bile duct image is input into a shallow feature extraction pathway and a deep feature extraction pathway respectively; in the shallow feature extraction pathway, local texture features are extracted through a series of small-size convolution operations and a dense connection structure based on a preset small-size convolution kernel; in the deep feature extraction pathway, global semantic features are extracted through a cascade of dilated convolution operations and a skip connection structure;

[0021] Performing adaptive pyramid pooling decomposition on the local texture features and the global semantic features respectively to generate feature subgraphs of multiple scales, performing channel grouping and feature reorganization on the feature subgraphs to form a multi-scale feature tensor;

[0022] Based on the multi-scale feature tensor, first-order moment distribution parameters and second-order moment distribution parameters of the feature channel are calculated; the first-order moment distribution parameters and the second-order moment distribution parameters are input into a nonlinear mapping network to generate feature channel attention weights; according to the feature channel attention weights, the multi-scale feature tensor is weighted reconstructed to obtain an enhanced feature tensor;

[0023] Calculating the mutual correlation coefficient matrix between different scale features in the enhanced feature tensor, constructing a feature affinity network, performing random walk propagation in the feature affinity network, updating the association weights of feature nodes, performing spectral decomposition and normalization processing on the updated association weights, and obtaining a cross-scale feature fusion coefficient;

[0024] Based on the cross-scale feature fusion coefficient, the enhanced feature tensor is adaptively weighted combined to obtain a fused feature tensor, and progressive upsampling is performed to obtain an upsampled feature, a residual learning unit is inserted into the upsampled feature, a local structure feature is extracted, and the local structure feature is superimposed with the upsampled feature to obtain a structure enhancement feature;

[0025] Constructing a feature compensation network, wherein the feature compensation network comprises a multi-layer deconvolution structure and a dense connection structure; inputting the structural enhancement feature into the feature compensation network to obtain a reconstructed feature map;

[0026] The reconstructed feature map is residually connected and feature fused with features of corresponding scales in the multi-scale feature tensor, and bile duct lumen wall feature information is output through a nonlinear activation function.

[0027] In an optional embodiment,

[0028] The three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data are input into the deep reinforcement learning network, and the optimal stone removal path is calculated based on the risk coefficient value, and the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; a real-time three-dimensional navigation map is established according to the optimal stone removal path, and the real-time spatial position information of the choledochoscope is collected, and the real-time spatial position information is dynamically aligned with the real-time three-dimensional navigation map to generate real-time alignment error data, and the optimal stone removal path is compensated and corrected based on the real-time alignment error data, and the corrected optimal stone removal path includes:

[0029] voxel-encode the three-dimensional bile duct structure model to obtain bile duct structure feature data; extract anatomical marker information from the bile duct anatomy knowledge base data, and map the anatomical marker information to the bile duct structure feature data; perform temporal encoding on the expert surgical experience data to obtain surgical operation sequence data; input the bile duct structure feature data, the anatomical marker information and the surgical operation sequence data into the attention-guided knowledge distillation unit to generate a multimodal feature representation;

[0030] Based on the multimodal feature representation, a path planning network is constructed, and the path planning network outputs a choledochoscope position coordinate sequence and a bile duct stone removal position coordinate sequence; based on the choledochoscope position coordinate sequence and the bile duct stone removal position coordinate sequence, a choledochoscope motion trajectory parameter is calculated to generate an initial motion trajectory;

[0031] Based on the bile duct structure feature data and the initial motion trajectory, a spatial deformation risk coefficient is calculated; based on the surgical operation sequence data and the initial motion trajectory, an operation difficulty coefficient is calculated; the spatial deformation risk coefficient and the operation difficulty coefficient are input into a risk assessment network to generate a comprehensive risk score;

[0032] Constructing a motion trajectory evaluation function, calculating a first spatial distance value between the choledochoscope position coordinate sequence and a preset target position, calculating a second spatial distance value between the bile duct stone removal position coordinate sequence and a preset stone removal position, and calculating a reward value based on the first spatial distance value and the second spatial distance value; constructing a safety evaluation function, calculating a penalty value based on the comprehensive risk score; performing a weighted combination of the reward value and the penalty value to obtain a composite reward signal;

[0033] The composite reward signal is used to train the path planning network to generate an optimal motion trajectory; the optimal motion trajectory is converted into a spatial position parameter and a posture angle parameter of a choledochoscope to obtain an optimal stone removal path; real-time spatial position information of the choledochoscope is collected, and the real-time spatial position information is dynamically aligned with the spatial position parameter and the posture angle parameter of the choledochoscope to obtain position error data;

[0034] Based on the position error data, the real-time position offset and the real-time angle offset are calculated; based on the continuous multi-frame recording of the position error data, the motion trajectory correction is calculated; the real-time position offset, the real-time angle offset and the motion trajectory correction are input into the adaptive fusion network to generate correction parameters, and the optimal stone removal path is compensated and corrected according to the correction parameters to obtain a corrected optimal stone removal path.

[0035] In an optional embodiment,

[0036] Using the composite reward signal to train the path planning network to generate an optimal motion trajectory includes:

[0037] Constructing a proximal strategy optimization objective function, wherein the optimization reward item in the proximal strategy optimization objective function is obtained by a weighted combination of the reward value and the penalty value, and the constraint item limits the strategy update step length by the trust domain range;

[0038] Based on the proximal strategy optimization objective function and the constraint violation penalty function, a Lagrangian dual function is constructed, gradient information of the Lagrangian dual function is extracted, and a dual gradient descent direction is generated;

[0039] Inputting the choledochoscope position coordinate sequence into a first training branch of a path planning network, and inputting the bile duct stone removal position coordinate sequence into a second training branch of the path planning network;

[0040] Generate a sampling trajectory based on the parameters of the current path planning network, calculate a first spatial distance value between the sampling trajectory and a preset target position, and calculate a second spatial distance value between the sampling trajectory and a preset stone removal position;

[0041] Calculate the reward value according to the first spatial distance value and the second spatial distance value, calculate the penalty value according to the comprehensive risk score, and update the composite reward signal by weighted combination of the reward value and the penalty value;

[0042] Setting a constraint violation penalty coefficient based on the training rounds, wherein the constraint violation penalty coefficient increases as the training rounds increase, and inputting the constraint violation penalty coefficient into the constraint violation penalty function;

[0043] According to the dual gradient descent direction, the composite reward signal and the output of the constraint violation penalty function, the parameter distribution of the path planning network is updated to generate the optimal motion trajectory.

[0044] In an optional embodiment,

[0045] Inputting the modified optimal stone removal path into a predictive control model, using a visual servo control method and a force feedback control method, calculating motion control parameters of a choledochoscope robot, and controlling the choledochoscope robot to move to a target bile duct stone position along the modified optimal stone removal path based on the motion control parameters; controlling the end effector of the choledochoscope robot to adjust a grasping force value and a grasping posture angle based on the morphological feature information of the bile duct stone, and completing the grasping action and removal operation of the bile duct stone includes:

[0046] The corrected optimal stone removal path is input into the predictive control model, and the position coordinates and posture angles of multiple sampling points of the optimal stone removal path in the rolling time domain are extracted as reference trajectories;

[0047] Using recursive least squares method to identify the dynamic parameters of the choledochoscope robot, establishing a state prediction model based on the dynamic parameters, and inputting the state prediction model into a hybrid optimization controller;

[0048] The image sequence is collected by a choledochoscope image sensor and image feature points are extracted. Based on the image feature points, the image feature error between the current position and the reference trajectory is calculated. According to the image feature error, a mapping matrix from the feature point to the camera speed is established. The joint speed instruction is calculated based on the mapping matrix and the image feature error. The interactive force data is collected by a terminal force sensor of the choledochoscope robot, and the interactive force data is compared with a preset force threshold to obtain a force deviation. Based on the force deviation, the position deviation between the end effector and the reference trajectory is calculated, and the position deviation and the force deviation are converted into a joint torque instruction.

[0049] The joint speed command and the joint torque command are used as constraints, input into the hybrid optimization controller, and the optimal control sequence is solved under the actuator saturation constraint and the safety boundary constraint. The choledochoscope robot is controlled to move to the target bile duct stone position according to the optimal control sequence, and the image data of the target bile duct stone is collected; geometric features and surface texture features are extracted from the image data, and the grasping force value and the grasping posture angle are calculated;

[0050] The grasping force value and the grasping posture angle are input into an adaptive force tracking controller, and the adaptive force tracking controller is used to control the end effector to complete the stone grasping.

[0051] In an optional embodiment,

[0052] Other methods for removing bile duct stones include:

[0053] Collecting contact force components of the end force sensor of the choledochoscope robot in three orthogonal directions and torque components in three rotation directions, combining the contact force components and the torque components into a six-dimensional contact force vector, calculating the modulus of the six-dimensional contact force vector, obtaining a composite contact force value, and comparing the composite contact force value with a preset safety threshold;

[0054] When the synthetic contact force value is greater than the preset safety threshold, a force buffer coefficient is calculated based on the difference between the synthetic contact force value and the preset safety threshold; the force buffer coefficient is multiplied by the current movement speed value of the choledochoscope robot to obtain a movement speed value after deceleration;

[0055] Decomposing the six-dimensional contact force vector to obtain a force direction unit vector, and calculating an avoidance posture angle based on the force direction unit vector; inputting the decelerated motion speed value and the avoidance posture angle into a motion planning module to generate a new motion control instruction;

[0056] The new motion control instruction is input into the driving system of the choledochoscope robot to control the choledochoscope robot to move according to the decelerated motion speed value and the avoidance posture angle until the synthetic contact force value is less than the preset safety threshold.

[0057] According to a second aspect of the embodiments of the present invention,

[0058] Provided is an intelligent choledochoscope-assisted minimally invasive stone removal system for bile duct stones, comprising:

[0059] The first unit is used to construct a deep learning target detection network to process the bile duct video image acquired in real time by a choledochoscope, and obtain bile duct stone feature information; perform image enhancement processing on the bile duct video image, extract bile duct lumen wall feature information, and establish a three-dimensional bile duct vascular tree model; annotate the bile duct stone feature information in the three-dimensional bile duct vascular tree model, generate a three-dimensional bile duct structure model with lesion annotations, and calculate the risk coefficient value of each bile duct stone based on the bile duct stone feature information;

[0060] The second unit is used to input the three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data into the deep reinforcement learning network, and calculate the optimal stone removal path based on the risk coefficient value, wherein the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; establish a real-time three-dimensional navigation map according to the optimal stone removal path, collect the real-time spatial position information of the choledochoscope, dynamically align the real-time spatial position information with the real-time three-dimensional navigation map, generate real-time alignment error data, and compensate and correct the optimal stone removal path based on the real-time alignment error data to obtain a corrected optimal stone removal path;

[0061] The third unit is used to input the corrected optimal stone removal path into the predictive control model, adopt the visual servo control method and the force feedback control method to calculate the motion control parameters of the choledochoscope robot, and control the choledochoscope robot to move along the corrected optimal stone removal path to the target bile duct stone position based on the motion control parameters; control the end effector of the choledochoscope robot to adjust the grasping force value and grasping posture angle based on the morphological feature information of the bile duct stone, so as to complete the grasping action and removal operation of the bile duct stone.

[0062] According to a third aspect of the embodiments of the present invention,

[0063] An electronic device is provided, comprising:

[0064] processor;

[0065] a memory for storing processor-executable instructions;

[0066] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0067] A fourth aspect of the embodiments of the present invention is:

[0068] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0069] In an embodiment of the present invention, the bile duct stones are accurately identified and their characteristic information is extracted through the deep learning target detection network. Combined with the three-dimensional bile duct vascular tree model and risk coefficient assessment, the internal structure of the bile duct and the distribution of stones can be more clearly understood, thereby improving the accuracy of diagnosis and the efficiency of surgical plan formulation. The precise positioning and risk assessment of the lesion help doctors choose the best surgical path and method to reduce surgical risks; the optimal stone removal path is calculated based on the deep reinforcement learning network and expert experience data, and the path compensation correction is performed through real-time three-dimensional navigation and dynamic registration, which can guide the choledochoscope robot to accurately reach the target position, avoid unnecessary exploration and operation, shorten the operation time, and improve the efficiency of the operation. At the same time, accurate path planning and control can reduce damage to surrounding tissues and improve surgical safety; using visual servo control and force feedback control, the choledochoscope robot can automatically adjust the grasping force and posture according to the morphological characteristics of the stones to realize automated stone removal operations. Minimally invasive operation methods can reduce surgical incisions, reduce patient pain, shorten postoperative recovery time, and improve the quality of life of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of the process of a minimally invasive bile duct stone removal method assisted by an intelligent choledochoscope according to an embodiment of the present invention;

[0071] Figure 2 This is a schematic diagram of the structure of the intelligent choledochoscope-assisted minimally invasive bile duct stone removal system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0074] Figure 1 FIG. 1 is a flow chart of a minimally invasive method for removing bile duct stones assisted by an intelligent choledochoscope according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0075] S101. Using the bile duct video image acquired in real time by the choledochoscope, construct a deep learning target detection network to process the bile duct video image to obtain bile duct stone feature information; perform image enhancement processing on the bile duct video image to extract bile duct lumen wall feature information and establish a three-dimensional bile duct vascular tree model; annotate the bile duct stone feature information in the three-dimensional bile duct vascular tree model to generate a three-dimensional bile duct structure model with lesion annotations, and calculate the risk coefficient value of each bile duct stone based on the bile duct stone feature information;

[0076] S102. Input the three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data into the deep reinforcement learning network, and calculate the optimal stone removal path based on the risk coefficient value, wherein the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; establish a real-time three-dimensional navigation map according to the optimal stone removal path, collect the real-time spatial position information of the choledochoscope, dynamically align the real-time spatial position information with the real-time three-dimensional navigation map, generate real-time alignment error data, and compensate and correct the optimal stone removal path based on the real-time alignment error data to obtain a corrected optimal stone removal path;

[0077] S103. Input the corrected optimal stone removal path into the predictive control model, adopt the visual servo control method and the force feedback control method to calculate the motion control parameters of the choledochoscope robot, and control the choledochoscope robot to move to the target bile duct stone position along the corrected optimal stone removal path based on the motion control parameters; control the end effector of the choledochoscope robot to adjust the grasping force value and grasping posture angle based on the morphological feature information of the bile duct stone, so as to complete the grasping action and removal operation of the bile duct stone.

[0078] Obtain bile duct video images. Use a high-definition choledochoscope to acquire bile duct video images in RGB format at a rate of 30 frames per second with a resolution of 1920x1080 pixels.

[0079] The deep learning target detection network is used to process the video images in the bile duct. The target detection network based on YOLOv5 is used to process the collected video images in real time. The network is pre-trained on a large-scale dataset containing various types of bile duct stones (such as cholesterol stones, pigment stones, and mixed stones), and can accurately identify and locate stones in the bile duct. For example, three stones are detected in one frame of image, and their position information is (x1, y1, w1, h1), (x2, y2, w2, h2), and (x3, y3, w3, h3), respectively, where (x, y) represents the coordinates of the upper left corner of the bounding box, and (w, h) represents the width and height of the bounding box. At the same time, the color, texture, shape and other feature information of each stone are extracted. For example, the color feature of stone 1 is RGB (128, 80, 50), the texture feature is rough, and the shape feature is elliptical.

[0080] The video image of the bile duct is enhanced and the characteristic information of the bile duct wall is extracted. An image enhancement algorithm, such as contrast-limited adaptive histogram equalization (CLAHE), is used to enhance the contrast of the bile duct wall. Then, an edge detection algorithm, such as the Canny edge detector, is used to extract the edge information of the bile duct wall. Taking a frame of image as an example, the extracted edge information of the bile duct wall is a series of pixel coordinates, such as (100, 200), (101, 201), (102, 202)..., and these coordinate points are connected to form the contour of the bile duct wall.

[0081] A three-dimensional bile duct vascular tree model is established. Based on the extracted bile duct lumen wall edge information, a three-dimensional reconstruction algorithm, such as the marching cubes algorithm, is used to construct a three-dimensional bile duct vascular tree model. The model represents the geometric structure of the bile duct in the form of triangular facets. For example, the model contains 1000 triangular facets, each of which is defined by three vertex coordinates.

[0082] Generate a three-dimensional bile duct structure model with lesion annotations. Annotate the characteristic information (position, size, morphology, etc.) of bile duct stones detected by the deep learning network in the constructed three-dimensional bile duct vascular tree model. For example, map the location information (x1, y1, w1, h1) of stone 1 to the corresponding position in the three-dimensional model, and mark an ellipse at the position to represent stone 1.

[0083] Calculate the risk coefficient value for each bile duct stone. Based on the characteristic information of bile duct stones (such as size, location, number, and morphology) and the preset risk assessment rules (such as stones with a diameter greater than 5 mm have a higher risk, and stones located at the lower end of the common bile duct have a higher risk), calculate the risk coefficient value for each bile duct stone. For example, the risk coefficient value of stone 1 is 0.8, the risk coefficient value of stone 2 is 0.5, and the risk coefficient value of stone 3 is 0.2.

[0084] The optimal stone removal path is calculated based on a deep reinforcement learning network. The three-dimensional bile duct structure model, the preset bile duct anatomical knowledge base data (such as bile duct branch structure, vascular distribution), the expert surgical experience data (such as the best stone removal method for different types of stones), and the risk coefficient value of each bile duct stone are input into the deep reinforcement learning network. The network finds an optimal stone removal path through trial and error learning, which can minimize the surgical risk and improve the efficiency of stone removal. The optimal stone removal path includes the choledochoscope motion path parameters (such as the position sequence of the choledochoscope in three-dimensional space) and the bile duct stone removal path parameters (such as the optimal angle and strength for grabbing stones). For example, the calculated optimal path is a series of choledochoscope positions, such as (x1, y1, z1, rx1, ry1, rz1), (x2, y2, z2, rx2, ry2, rz2)..., where (x, y, z) represents the position coordinates and (rx, ry, rz) represents the rotation angle.

[0085] Establish a real-time three-dimensional navigation map and perform dynamic registration. According to the calculated optimal stone removal path, establish a real-time three-dimensional navigation map, and display the current position of the choledochoscope and the position of the target stone in the navigation map. At the same time, collect the real-time spatial position information of the choledochoscope, for example, use an optical tracking system to obtain the position and posture of the choledochoscope. Dynamically align the collected real-time spatial position information with the real-time three-dimensional navigation map, and calculate the real-time registration error data. For example, the deviation between the actual position of the current choledochoscope and the planned position is (dx, dy, dz).

[0086] Compensate and correct the optimal stone removal path. Based on the real-time registration error data, compensate and correct the optimal stone removal path to eliminate the impact of the registration error on the path planning. For example, adjust the movement direction and speed of the choledochoscope according to the deviation (dx, dy, dz).

[0087] Control the choledochoscope robot to move to the target position. Input the corrected optimal stone removal path into the predictive control model, and use the visual servo control method and force feedback control method to calculate the motion control parameters of the choledochoscope robot, such as speed, acceleration, and torque. Based on the calculated motion control parameters, control the choledochoscope robot to move along the corrected optimal stone removal path to the target bile duct stone position.

[0088] Control the choledochoscope robot to grasp and remove stones. Control the end effector of the choledochoscope robot, adjust the grasping force value and grasping posture angle based on the morphological characteristics of the bile duct stones (such as size and shape), and complete the grasping action and removal of bile duct stones. For example, for a spherical stone with a diameter of 3mm, set the grasping force to 0.5N and the grasping angle to 45 degrees.

[0089] In this embodiment, through deep learning target detection and three-dimensional reconstruction technology, bile duct stones can be accurately identified and located, and an accurate three-dimensional bile duct structure model can be established, thereby improving surgical accuracy and reducing surgical risks; through deep reinforcement learning and path planning algorithms, the optimal stone removal path can be calculated, reducing operation time and trauma; through visual servo control and force feedback control, precise control of the choledochoscope robot can be achieved, improving surgical efficiency and reducing the operating burden of doctors.

[0090] In an optional embodiment, a deep learning target detection network is constructed to process the bile duct video image acquired in real time by a choledochoscope to obtain characteristic information of bile duct stones; image enhancement processing is performed on the bile duct video image to extract characteristic information of the bile duct cavity wall and establish a three-dimensional bile duct vascular tree model; the bile duct stone characteristic information is annotated in the three-dimensional bile duct vascular tree model to generate a three-dimensional bile duct structure model with lesion annotations, and the risk coefficient value of each bile duct stone is calculated based on the bile duct stone characteristic information, including:

[0091] The intra-biliary video image is input into a deep learning target detection network, the deep learning target detection network extracts image depth features through multi-level residual convolution units, the image depth features are input into a cross-layer feature fusion module, the cross-layer feature fusion module uses an adaptive weight allocation mechanism to perform multi-scale feature aggregation on the image depth features to obtain a target feature map; the target feature map is deconvoluted and reconstructed with dense features to obtain spatial position information, morphological contour information, and density distribution information of bile duct stones;

[0092] Based on the gray value distribution of each pixel in the intra-biliary video image, a pixel-level gain matrix is ​​constructed, and based on the edge gradient information in the intra-biliary video image, a regional contrast enhancement matrix is ​​constructed; the pixel-level gain matrix is ​​multiplied with the intra-biliary video image to obtain a first enhanced image, and the regional contrast enhancement matrix is ​​weightedly superimposed with the first enhanced image to obtain an enhanced bile duct image;

[0093] The enhanced bile duct image is input into a semantic segmentation network, and multi-scale feature information of the enhanced bile duct image is extracted through a multi-layer spatial pyramid pooling module, and input into a feature progressive fusion module, and the multi-scale feature information is upsampled and reconstructed to obtain bile duct lumen wall feature information;

[0094] The bile duct wall feature information is voxel-divided using a quadtree structure, and surface continuity constraints and topology preservation constraints are imposed on the divided grid structure to generate a three-dimensional bile duct vascular tree model; the spatial position information, the morphological contour information, and the density distribution information of the bile duct stone are mapped to corresponding positions of the three-dimensional bile duct vascular tree model to generate a three-dimensional bile duct structure model with lesion annotations;

[0095] A multidimensional feature set of bile duct stones is extracted based on the three-dimensional bile duct structure model, and the multidimensional feature set includes size feature values, location hazard values, density distribution values ​​and tissue damage risk values. The multidimensional feature set is input into the risk prediction module, and feature fusion weight parameters are obtained through training based on preset expert annotation data. The multidimensional feature set is weightedly fused using the attention mechanism to output the risk coefficient value of bile duct stones.

[0096] Specifically, the video image inside the bile duct is obtained. A real-time video image sequence of the inside of the patient's bile duct is collected through a choledochoscope and converted into a digital format for storage, such as AVI or MP4 format, with a frame rate set to 30 frames per second and a resolution set to 1920x1080 pixels.

[0097] Then, the bile duct stone feature extraction is performed. The acquired bile duct video image is input into a pre-trained deep learning target detection network, such as YOLOv5 or Faster R-CNN. The network uses multi-level residual convolution units to extract image depth features. Taking ResNet50 as an example, the image is subjected to multi-layer convolution and residual connection to extract feature maps of different scales. Then, these image depth features are input into the cross-layer feature fusion module, which uses an adaptive weight allocation mechanism to perform multi-scale feature aggregation on the image depth features. For example, the feature pyramid network (FPN) structure is used to upsample and fuse the feature maps of different levels to obtain the target feature map. Finally, the target feature map is deconvoluted and reconstructed with dense features to obtain the spatial position information, morphological contour information and density distribution information of the bile duct stones. For example, the non-maximum suppression (NMS) algorithm is used to remove overlapping detection boxes, and the bounding box coordinates, pixel area and average pixel gray value of each stone are output, representing the spatial position, morphological contour and density distribution information, respectively. Suppose a stone is detected with bounding box coordinates (100, 100, 200, 200), pixel area 10,000, and average pixel grayscale value 150.

[0098] Next, the bile duct wall feature extraction and three-dimensional reconstruction are performed. Based on the gray value distribution of each pixel in the bile duct video image, a pixel-level gain matrix is ​​constructed. For example, the gray value of each pixel is nonlinearly transformed to enhance the contrast of the image. Based on the edge gradient information in the bile duct video image, for example, the image gradient is calculated using the Sobel operator to construct a regional contrast enhancement matrix. The pixel-level gain matrix is ​​multiplied with the bile duct video image to obtain a first enhanced image. The regional contrast enhancement matrix is ​​weightedly superimposed with the first enhanced image to obtain an enhanced bile duct image. The enhanced bile duct image is input into a semantic segmentation network, such as U-Net or DeepLabv3+. The multi-scale feature information of the enhanced bile duct image is extracted through a multi-layer spatial pyramid pooling module. The multi-scale feature information is input into the feature progressive fusion module, and the multi-scale feature information is upsampled and reconstructed to obtain the bile duct wall feature information, such as the segmentation result of the bile duct wall represented by a binary image. The bile duct wall feature information is voxel-divided using a quadtree structure, such as converting the image segmentation result into three-dimensional volume data. Surface continuity constraints and topology preservation constraints are imposed on the divided mesh structure to generate a three-dimensional bile duct vascular tree model. For example, a Marching Cubes algorithm is used to generate a three-dimensional mesh model.

[0099] Then, lesion annotation is performed. The spatial position information, morphological contour information, and density distribution information of the bile duct stones are mapped to the corresponding positions of the three-dimensional bile duct vascular tree model to generate a three-dimensional bile duct structure model with lesion annotation. For example, the two-dimensional bounding box coordinates are converted into three-dimensional coordinates, and the stone information is marked on the three-dimensional model with different colors or textures.

[0100] Finally, the risk of bile duct stones is assessed. Based on the three-dimensional bile duct structure model, a multidimensional feature set of bile duct stones is extracted, including size feature values ​​(such as volume), location hazard values ​​(such as distance from the bile duct stenosis), density distribution values ​​(such as average grayscale value) and tissue damage risk values ​​(such as contact area with the bile duct wall). The multidimensional feature set is input into a risk prediction module, such as a support vector machine or a random forest. Based on the preset expert annotation data, the feature fusion weight parameters are trained. The attention mechanism is used to perform weighted fusion on the multidimensional feature set, and the risk coefficient value of bile duct stones is output, such as a value between 0 and 1, indicating the risk level of the stone. Assume that the size of a stone is 10mm 3 , the position hazard is 0.8, the density distribution value is 150, and the tissue damage risk value is 5mm 2 After calculation by the risk prediction module, the risk coefficient value is 0.75.

[0101] In this embodiment, through deep learning target detection and image enhancement technology, bile duct stones can be more accurately identified and located, and their multidimensional feature information can be extracted, thereby improving the accuracy of diagnosis; constructing a three-dimensional bile duct structure model with lesion annotations can provide an intuitive visualization of the bile duct structure and stone distribution, which helps doctors better understand the condition and formulate a more accurate treatment plan; based on a multidimensional feature set and machine learning algorithm, a personalized risk assessment can be performed for each bile duct stone, providing a more reliable basis for clinical decision-making, and helping to guide postoperative follow-up and management.

[0102] In an optional implementation, the enhanced bile duct image is input into a semantic segmentation network, and multi-scale feature information of the enhanced bile duct image is extracted through a multi-layer spatial pyramid pooling module, and input into a feature progressive fusion module, and the multi-scale feature information is upsampled and reconstructed to obtain bile duct wall feature information including:

[0103] The enhanced bile duct image is input into a shallow feature extraction pathway and a deep feature extraction pathway respectively; in the shallow feature extraction pathway, local texture features are extracted through a series of small-size convolution operations and a dense connection structure based on a preset small-size convolution kernel; in the deep feature extraction pathway, global semantic features are extracted through a cascade of dilated convolution operations and a skip connection structure;

[0104] Performing adaptive pyramid pooling decomposition on the local texture features and the global semantic features respectively to generate feature subgraphs of multiple scales, performing channel grouping and feature reorganization on the feature subgraphs to form a multi-scale feature tensor;

[0105] Based on the multi-scale feature tensor, first-order moment distribution parameters and second-order moment distribution parameters of the feature channel are calculated; the first-order moment distribution parameters and the second-order moment distribution parameters are input into a nonlinear mapping network to generate feature channel attention weights; according to the feature channel attention weights, the multi-scale feature tensor is weighted reconstructed to obtain an enhanced feature tensor;

[0106] Calculating the mutual correlation coefficient matrix between different scale features in the enhanced feature tensor, constructing a feature affinity network, performing random walk propagation in the feature affinity network, updating the association weights of feature nodes, performing spectral decomposition and normalization processing on the updated association weights, and obtaining a cross-scale feature fusion coefficient;

[0107] Based on the cross-scale feature fusion coefficient, the enhanced feature tensor is adaptively weighted combined to obtain a fused feature tensor, and progressive upsampling is performed to obtain an upsampled feature, a residual learning unit is inserted into the upsampled feature, a local structure feature is extracted, and the local structure feature is superimposed with the upsampled feature to obtain a structure enhancement feature;

[0108] Constructing a feature compensation network, wherein the feature compensation network comprises a multi-layer deconvolution structure and a dense connection structure; inputting the structural enhancement feature into the feature compensation network to obtain a reconstructed feature map;

[0109] The reconstructed feature map is residually connected and feature fused with features of corresponding scales in the multi-scale feature tensor, and bile duct lumen wall feature information is output through a nonlinear activation function.

[0110] Enhanced bile duct image semantic segmentation method is used to accurately extract bile duct wall feature information. This method uses multi-scale feature extraction, attention mechanism, cross-scale feature fusion and feature compensation strategy to improve segmentation accuracy.

[0111] Specifically, the enhanced bile duct image is input into a semantic segmentation network. The network contains a multi-layer spatial pyramid pooling module to extract multi-scale feature information of the input image. These multi-scale features are then input into a feature progressive fusion module for upsampling and reconstruction, and finally the bile duct wall feature information is obtained.

[0112] Specifically, the enhanced bile duct image is simultaneously input into the shallow and deep feature extraction pathways. The shallow feature extraction pathway uses a preset small-size convolution kernel (e.g., 3x3) to extract local texture features of the image through a series of serially connected small-size convolution operations and dense connection structures. The deep feature extraction pathway uses cascaded dilated convolution operations and jump connection structures to extract global semantic features of the image. Here, the dilation rate of the dilated convolution can be set to 2, 4, 8, etc., and the jump connection can be implemented by splicing shallow features with deep features.

[0113] Next, the extracted local texture features and global semantic features are adaptively decomposed by pyramid pooling. For example, pooling kernels of different sizes (such as 1x1, 2x2, 4x4) can be used for pooling operations to generate feature sub-maps of multiple scales. Then, these feature sub-maps are channel grouped and feature reorganized to form a multi-scale feature tensor. Assuming that the number of channels of local texture features and global semantic features is 256, each channel can be divided into 4 groups, each with 64 channels, and then feature sub-maps of different scales are spliced ​​according to channel groups to form a new multi-scale feature tensor.

[0114] Based on the generated multi-scale feature tensor, the first-order moment (mean) and second-order moment (variance) of the feature channel are calculated. The calculated first-order moment and second-order moment are input into a nonlinear mapping network (e.g., a two-layer fully connected network) to generate the attention weights of the feature channels. Then, according to these attention weights, the multi-scale feature tensor is weighted reconstructed to obtain the enhanced feature tensor.

[0115] The mutual correlation coefficient matrix between different scale features in the enhanced feature tensor is calculated to construct a feature affinity network. Random walk propagation is performed in the network to update the association weights of feature nodes. The updated association weights are spectrally decomposed and normalized to obtain the cross-scale feature fusion coefficients.

[0116] Based on the obtained cross-scale feature fusion coefficient, the enhanced feature tensor is adaptively weighted and combined to obtain a fused feature tensor. The fused feature tensor is then progressively upsampled, for example, using a bilinear interpolation method, to obtain an upsampled feature. A residual learning unit (for example, a residual block containing two 3x3 convolutional layers) is inserted into the upsampled feature to extract local structural features. The extracted local structural features are superimposed with the upsampled features to obtain structural enhancement features.

[0117] A feature compensation network is constructed, which includes a multi-layer deconvolution structure and a dense connection structure. The structure enhancement feature is input into the feature compensation network to obtain a reconstructed feature map.

[0118] Finally, the reconstructed feature map is residually connected and feature fused with the features of the corresponding scale in the multi-scale feature tensor, for example, by element-wise addition, and then a nonlinear activation function (such as the ReLU function) is used to output the final bile duct wall feature information.

[0119] For example, a 256x256 enhanced bile duct image is input. After the above steps, a 256x256 bile duct wall feature map can be obtained, in which the value of each pixel represents the probability that the pixel belongs to the bile duct wall.

[0120] In this embodiment, through multi-scale feature extraction, attention mechanism and cross-scale feature fusion, this method can more effectively capture the subtle structure and global context information of the bile duct wall, thereby improving the segmentation accuracy; the introduction of the feature compensation network can effectively compensate for the information lost in the feature extraction process and improve the robustness of the model; the design of progressive upsampling and residual learning units can reduce the amount of calculation and improve the operation efficiency of the model.

[0121] In an optional embodiment, the three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data are input into the deep reinforcement learning network, and the optimal stone removal path is calculated based on the risk coefficient value, and the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; a real-time three-dimensional navigation map is established according to the optimal stone removal path, and the real-time spatial position information of the choledochoscope is collected, and the real-time spatial position information is dynamically aligned with the real-time three-dimensional navigation map to generate real-time alignment error data, and the optimal stone removal path is compensated and corrected based on the real-time alignment error data, and the corrected optimal stone removal path includes:

[0122] voxel-encode the three-dimensional bile duct structure model to obtain bile duct structure feature data; extract anatomical marker information from the bile duct anatomy knowledge base data, and map the anatomical marker information to the bile duct structure feature data; perform temporal encoding on the expert surgical experience data to obtain surgical operation sequence data; input the bile duct structure feature data, the anatomical marker information and the surgical operation sequence data into the attention-guided knowledge distillation unit to generate a multimodal feature representation;

[0123] Based on the multimodal feature representation, a path planning network is constructed, and the path planning network outputs a choledochoscope position coordinate sequence and a bile duct stone removal position coordinate sequence; based on the choledochoscope position coordinate sequence and the bile duct stone removal position coordinate sequence, a choledochoscope motion trajectory parameter is calculated to generate an initial motion trajectory;

[0124] Based on the bile duct structure feature data and the initial motion trajectory, a spatial deformation risk coefficient is calculated; based on the surgical operation sequence data and the initial motion trajectory, an operation difficulty coefficient is calculated; the spatial deformation risk coefficient and the operation difficulty coefficient are input into a risk assessment network to generate a comprehensive risk score;

[0125] Constructing a motion trajectory evaluation function, calculating a first spatial distance value between the choledochoscope position coordinate sequence and a preset target position, calculating a second spatial distance value between the bile duct stone removal position coordinate sequence and a preset stone removal position, and calculating a reward value based on the first spatial distance value and the second spatial distance value; constructing a safety evaluation function, calculating a penalty value based on the comprehensive risk score; performing a weighted combination of the reward value and the penalty value to obtain a composite reward signal;

[0126] The composite reward signal is used to train the path planning network to generate an optimal motion trajectory; the optimal motion trajectory is converted into a spatial position parameter and a posture angle parameter of a choledochoscope to obtain an optimal stone removal path; real-time spatial position information of the choledochoscope is collected, and the real-time spatial position information is dynamically aligned with the spatial position parameter and the posture angle parameter of the choledochoscope to obtain position error data;

[0127] Based on the position error data, the real-time position offset and the real-time angle offset are calculated; based on the continuous multi-frame recording of the position error data, the motion trajectory correction is calculated; the real-time position offset, the real-time angle offset and the motion trajectory correction are input into the adaptive fusion network to generate correction parameters, and the optimal stone removal path is compensated and corrected according to the correction parameters to obtain a corrected optimal stone removal path.

[0128] Specifically, a three-dimensional bile duct structure model is constructed. The patient's CT or MRI image data can be used to construct a patient-specific three-dimensional bile duct structure model through three-dimensional reconstruction technology. For example, a threshold-based segmentation method is used to extract the bile duct area, and then a Marching Cubes algorithm is used to generate a three-dimensional mesh model. The model is voxelized, for example, the model space is discretized into 1mm 3 Each voxel is assigned a value of 0 or 1 according to whether it belongs to the bile duct, thereby obtaining the bile duct structure feature data.

[0129] Next, prepare the preset bile duct anatomy knowledge base data and expert surgical experience data. The bile duct anatomy knowledge base data contains the anatomical structure information of the bile duct system, such as the position, morphology and connection relationship of the common bile duct, intrahepatic bile duct, gallbladder, etc., as well as important anatomical landmarks, such as the cystic duct opening, liver hilum, etc. The expert surgical experience data records the operation process of the expert during choledochoscope surgery, such as the movement trajectory of the choledochoscope, the operation method of the stone removal tool, etc. The expert surgical video data can be analyzed to extract the spatial position and posture information of the choledochoscope and the stone removal tool, as well as the time series information of the surgical operation, and then perform time series encoding, for example, encoding different types of surgical operations into different numbers to form surgical operation sequence data.

[0130] Then, the prepared data is input into the deep reinforcement learning network. Specifically, the bile duct structure feature data, the anatomical landmark information extracted from the bile duct anatomy knowledge base data (for example, the position coordinates of the anatomical landmarks are mapped to the bile duct structure feature data), and the surgical operation sequence data are input into the attention-guided knowledge distillation unit. This unit can learn the association between the bile duct anatomical structure, anatomical landmarks, and expert surgical experience, and generate a multimodal feature representation. For example, the Transformer network can be used to implement the attention mechanism to fuse information from different modalities.

[0131] Based on the generated multimodal feature representation, a path planning network is constructed. The network can output a choledochoscope position coordinate sequence and a bile duct stone removal position coordinate sequence. For example, a recurrent neural network (RNN) or a convolutional neural network (CNN) can be used to construct a path planning network. Assuming that the choledochoscope position coordinate sequence output by the network is {(1, 2, 3), (2, 3, 4), (3, 4, 5)}, and the bile duct stone removal position coordinate sequence is {(4, 5, 6), (5, 6, 7)}, the motion trajectory parameters of the choledochoscope, such as speed, acceleration, etc., can be calculated based on these coordinate sequences to generate an initial motion trajectory.

[0132] Next, perform risk assessment. Based on the bile duct structure feature data and the initial motion trajectory, calculate the spatial deformation risk coefficient. For example, the spatial deformation risk can be assessed based on the distance between the choledochoscope motion trajectory and the bile duct wall. Based on the surgical operation sequence data and the initial motion trajectory, calculate the operation difficulty coefficient. For example, the operation difficulty can be assessed based on the complexity of the surgical operation and the time required. Input the spatial deformation risk coefficient and the operation difficulty coefficient into the risk assessment network to generate a comprehensive risk score.

[0133] In order to train the path planning network, it is necessary to construct a reward function and a penalty function. The reward function calculates the reward value based on the first spatial distance value between the choledochoscope position coordinate sequence and the preset target position (e.g., stone position), and the second spatial distance value between the bile duct stone removal position coordinate sequence and the preset stone removal position. The smaller the distance, the higher the reward value. The penalty function calculates the penalty value based on the comprehensive risk score. The higher the risk score, the higher the penalty value. The reward value and the penalty value are weighted and combined to obtain a composite reward signal.

[0134] The composite reward signal is used to train the path planning network to generate the optimal motion trajectory. The optimal motion trajectory is converted into the spatial position parameters and posture angle parameters of the choledochoscope to obtain the optimal stone removal path. For example, assuming that the optimal motion trajectory is a cubic spline curve, it can be converted into a series of position coordinates and posture angles.

[0135] During the operation, the real-time spatial position information of the choledochoscope is collected, for example, the position and posture information of the choledochoscope is obtained using an electromagnetic tracking system or an optical tracking system. The real-time spatial position information is dynamically aligned with the optimal stone removal path to obtain position error data.

[0136] Based on the position error data, the real-time position offset, real-time angle offset and motion trajectory correction are calculated. For example, the offset trend of the motion trajectory can be calculated based on the position error data of multiple consecutive frames. These data are input into the adaptive fusion network to generate correction parameters. The optimal stone removal path is compensated and corrected according to the correction parameters to obtain the corrected optimal stone removal path.

[0137] In this embodiment, the risk of complications in choledochoscopy, such as bile duct perforation and bleeding, can be effectively reduced through risk assessment and path optimization; the operation time can be shortened and the patient's pain can be reduced by planning the optimal stone removal path; this method can provide doctors with real-time navigation and path planning suggestions to assist doctors in making surgical decisions, which is of great guiding significance, especially for inexperienced doctors.

[0138] In an optional implementation, using the composite reward signal to train the path planning network to generate an optimal motion trajectory includes:

[0139] Constructing a proximal strategy optimization objective function, wherein the optimization reward item in the proximal strategy optimization objective function is obtained by a weighted combination of the reward value and the penalty value, and the constraint item limits the strategy update step length by the trust domain range;

[0140] Based on the proximal strategy optimization objective function and the constraint violation penalty function, a Lagrangian dual function is constructed, gradient information of the Lagrangian dual function is extracted, and a dual gradient descent direction is generated;

[0141] Inputting the choledochoscope position coordinate sequence into a first training branch of a path planning network, and inputting the bile duct stone removal position coordinate sequence into a second training branch of the path planning network;

[0142] Generate a sampling trajectory based on the parameters of the current path planning network, calculate a first spatial distance value between the sampling trajectory and a preset target position, and calculate a second spatial distance value between the sampling trajectory and a preset stone removal position;

[0143] Calculate the reward value according to the first spatial distance value and the second spatial distance value, calculate the penalty value according to the comprehensive risk score, and update the composite reward signal by weighted combination of the reward value and the penalty value;

[0144] Setting a constraint violation penalty coefficient based on the training rounds, wherein the constraint violation penalty coefficient increases as the training rounds increase, and inputting the constraint violation penalty coefficient into the constraint violation penalty function;

[0145] According to the dual gradient descent direction, the composite reward signal and the output of the constraint violation penalty function, the parameter distribution of the path planning network is updated to generate the optimal motion trajectory.

[0146] In order to generate the optimal motion trajectory in choledochoscopy surgery, a method for training a path planning network based on a composite reward signal is proposed. The core of this method is to use the proximal strategy optimization algorithm and combine it with the constraint violation penalty mechanism to train the path planning network, and finally generate the optimal motion trajectory that takes into account both the surgical target position and the stone removal position.

[0147] Specifically, a proximal policy optimization objective function is constructed. The objective function consists of two parts: an optimization reward term and a constraint term. The optimization reward term is obtained by a weighted combination of a reward value and a penalty value, and is used to guide the path planning network to generate a better trajectory. The higher the reward value, the closer the trajectory is to the target location and the stone removal location; the higher the penalty value, the greater the risk of the trajectory. The constraint term limits the policy update step size by the trust domain range to prevent excessive parameter updates from causing unstable training. For example, the trust domain range can be set to 0.1, which means that the amplitude of each parameter update cannot exceed 10% of the current parameter value.

[0148] Then, based on the proximal strategy, the objective function and the constraint violation penalty function are optimized to construct the Lagrangian dual function. By extracting the gradient information of the Lagrangian dual function, the dual gradient descent direction is generated to guide the update of the path planning network parameters.

[0149] Next, the choledochoscope position coordinate sequence and the bile duct stone removal position coordinate sequence are respectively input into the first training branch and the second training branch of the path planning network. The path planning network can adopt a deep neural network structure, such as a recurrent neural network or a convolutional neural network.

[0150] Generate sampling trajectories based on the parameters of the current path planning network. For example, 10 trajectories can be sampled from the strategy distribution of the path planning network. Calculate the first spatial distance value between each sampling trajectory and the preset target position, and the second spatial distance value between each sampling trajectory and the preset stone removal position. Spatial distance can be calculated using Euclidean distance. Assume that a sampling trajectory is 2 mm away from the target position and 3 mm away from the stone removal position.

[0151] The reward value is calculated based on the first spatial distance value and the second spatial distance value. For example, the reward value can be set to the negative of the distance, i.e., -2 and -3. The penalty value is calculated based on the comprehensive risk score. The comprehensive risk score can take into account factors such as the smoothness and safety of the trajectory. Assuming that the comprehensive risk score of the trajectory is 1, the penalty value is 1. The reward value and the penalty value are weighted and combined to update the composite reward signal. For example, the weight of the reward value is 0.8 and the weight of the penalty value is 0.2, then the composite reward signal is 0.8*(-2)+0.8*(-3)+0.2*1=-3.8.

[0152] Set the constraint violation penalty coefficient based on the number of training rounds. The constraint violation penalty coefficient increases with the number of training rounds and is used to punish the behavior of violating the constraint. For example, the initial constraint violation penalty coefficient is 0.1, and it increases by 0.1 every 100 training rounds. Input the constraint violation penalty coefficient into the constraint violation penalty function.

[0153] Finally, according to the dual gradient descent direction, the composite reward signal and the output of the constraint violation penalty function, the parameter distribution of the path planning network is updated to generate the optimal motion trajectory.

[0154] In this embodiment, through the guidance of the composite reward signal, the path planning network can generate a trajectory that is closer to the target position and the stone removal position, thereby improving the accuracy of the operation; through the penalty value mechanism, the surgical risk can be effectively reduced, such as avoiding damage to surrounding tissues; by optimizing the path planning, the operation time can be shortened and the surgical efficiency can be improved.

[0155] In an optional implementation, the modified optimal stone removal path is input into a predictive control model, a visual servo control method and a force feedback control method are used to calculate motion control parameters of a choledochoscope robot, and based on the motion control parameters, the choledochoscope robot is controlled to move along the modified optimal stone removal path to a target bile duct stone position; the end effector of the choledochoscope robot is controlled to adjust a grasping force value and a grasping posture angle based on the morphological feature information of the bile duct stone, and the grasping action and removal operation of the bile duct stone are completed, including:

[0156] The corrected optimal stone removal path is input into the predictive control model, and the position coordinates and posture angles of multiple sampling points of the optimal stone removal path in the rolling time domain are extracted as reference trajectories;

[0157] Using recursive least squares method to identify the dynamic parameters of the choledochoscope robot, establishing a state prediction model based on the dynamic parameters, and inputting the state prediction model into a hybrid optimization controller;

[0158] The image sequence is collected by a choledochoscope image sensor and image feature points are extracted. Based on the image feature points, the image feature error between the current position and the reference trajectory is calculated. According to the image feature error, a mapping matrix from the feature point to the camera speed is established. The joint speed instruction is calculated based on the mapping matrix and the image feature error. The interactive force data is collected by a terminal force sensor of the choledochoscope robot, and the interactive force data is compared with a preset force threshold to obtain a force deviation. Based on the force deviation, the position deviation between the end effector and the reference trajectory is calculated, and the position deviation and the force deviation are converted into a joint torque instruction.

[0159] The joint speed command and the joint torque command are used as constraints, input into the hybrid optimization controller, and the optimal control sequence is solved under the actuator saturation constraint and the safety boundary constraint. The choledochoscope robot is controlled to move to the target bile duct stone position according to the optimal control sequence, and the image data of the target bile duct stone is collected; geometric features and surface texture features are extracted from the image data, and the grasping force value and the grasping posture angle are calculated;

[0160] The grasping force value and the grasping posture angle are input into an adaptive force tracking controller, and the adaptive force tracking controller is used to control the end effector to complete the stone grasping.

[0161] In order to achieve accurate grasping and removal of bile duct stones by a choledochoscope robot, a motion control method of a choledochoscope robot based on predictive control and visual servoing was proposed.

[0162] Specifically, the corrected optimal stone removal path is preprocessed. The path is discretized to obtain the position coordinates and attitude angle information of a series of sampling points in the prediction time domain, which will be used as the reference trajectory of the robot movement. For example, the path is divided into 100 sampling points, each of which contains three-dimensional coordinates (x, y, z) and rotation angles (α, β, γ).

[0163] Then, the dynamic parameters of the choledochoscope robot are identified and the state prediction model is established. The recursive least squares method is used to identify the dynamic parameters of the robot online, for example, the robot's mass, moment of inertia, friction coefficient and other parameters are identified. Based on the identified dynamic parameters, the robot's state prediction model is established, which can predict the robot's state in the future, such as position, velocity, acceleration, etc.

[0164] Next, a hybrid optimization controller is constructed. The controller receives joint velocity commands and joint torque commands as inputs and solves the optimal control sequence under actuator saturation constraints and safety boundary constraints, such as joint angle limits, joint velocity limits, and joint torque limits, to control the motion of the choledochoscope robot.

[0165] In order to obtain the joint velocity command, visual servo control is required. The image sequence is collected by the choledochoscope image sensor, and the image feature points are extracted, for example, using the SIFT algorithm to extract the image feature points. Then, the image feature error between the current position and the reference trajectory is calculated. A mapping matrix from the feature point to the camera velocity is established, for example, the mapping relationship is represented by the Jacobian matrix. Based on the mapping matrix and the image feature error, the joint velocity command is calculated.

[0166] At the same time, force feedback control is required to obtain joint torque instructions. The interaction force data is collected by the end force sensor of the choledochoscope robot, and the collected interaction force data is compared with the preset force threshold, for example, the preset force threshold is 0.1N, to obtain the force deviation. Based on the force deviation, for example, using the proportional-differential control algorithm, the position deviation between the end effector and the reference trajectory is calculated. The position deviation and force deviation are converted into joint torque instructions.

[0167] The calculated joint speed command and joint torque command are input into the hybrid optimization controller to solve the optimal control sequence. According to the optimal control sequence, the choledochoscope robot is controlled to move to the target bile duct stone position, for example, the robot is controlled to move to the target position at a speed of 0.1 mm / s.

[0168] After reaching the target position, the image data of the target bile duct stone is collected. Geometric features and surface texture features are extracted from the image data, for example, the length, width, height, surface roughness and other features of the stone are extracted. Based on the extracted features, the grasping force value and grasping posture angle are calculated. For example, based on the size and surface texture of the stone, the grasping force value is calculated to be 0.5N and the grasping posture angle is 30 degrees.

[0169] Finally, the calculated grasping force value and grasping posture angle are input into the adaptive force tracking controller, which controls the end effector to complete the stone grasping. For example, the mechanical claw is controlled to grasp the stone with a force of 0.2N and rotate the mechanical claw at an angle of 30 degrees to complete the stone grasping and removal operation.

[0170] In this embodiment, by combining visual servo control and force feedback control, the robot's motion trajectory and grasping force can be adjusted in real time, thereby improving the grasping accuracy and stability, and effectively preventing stones from slipping or damaging the bile duct; by analyzing the morphological feature information of the stones and adaptively adjusting the grasping force value and grasping posture angle, it can better adapt to stones of different sizes, shapes and surface textures, and improve the grasping success rate; through predictive control models and hybrid optimization controllers, a better stone removal path can be planned, and the robot can be controlled to reach the target position in a shorter time, thereby improving surgical efficiency and reducing trauma to patients.

[0171] In an optional embodiment, the method for removing bile duct stones comprises:

[0172] Collecting contact force components of the end force sensor of the choledochoscope robot in three orthogonal directions and torque components in three rotation directions, combining the contact force components and the torque components into a six-dimensional contact force vector, calculating the modulus of the six-dimensional contact force vector, obtaining a composite contact force value, and comparing the composite contact force value with a preset safety threshold;

[0173] When the synthetic contact force value is greater than the preset safety threshold, a force buffer coefficient is calculated based on the difference between the synthetic contact force value and the preset safety threshold; the force buffer coefficient is multiplied by the current movement speed value of the choledochoscope robot to obtain a movement speed value after deceleration;

[0174] Decomposing the six-dimensional contact force vector to obtain a force direction unit vector, and calculating an avoidance posture angle based on the force direction unit vector; inputting the decelerated motion speed value and the avoidance posture angle into a motion planning module to generate a new motion control instruction;

[0175] The new motion control instruction is input into the driving system of the choledochoscope robot to control the choledochoscope robot to move according to the decelerated motion speed value and the avoidance posture angle until the synthetic contact force value is less than the preset safety threshold.

[0176] In order to improve the safety and operation efficiency of choledochoscope robot in the process of bile duct stone removal, a choledochoscope robot motion control method based on contact force feedback is proposed. The core of this method is to monitor the contact force between the robot and the bile duct wall in real time, and adjust the robot's movement speed and posture according to the size and direction of the contact force to avoid damage to the bile duct.

[0177] Specifically, it is necessary to obtain real-time data of the end force sensor of the choledochoscope robot. The force sensor is set on the end operating platform of the robot to sense the interaction force between the robot and the surrounding environment. The sensor can collect contact force components in three orthogonal directions (such as X, Y, and Z axes) and torque components in three rotation directions (such as rotation around the X, Y, and Z axes). For example, at a certain moment, the data collected by the force sensor may be: the contact force in the X direction is 0.1N, the contact force in the Y direction is 0.2N, the contact force in the Z direction is 0.3N, the torque around the X axis is 0.01Nm, the torque around the Y axis is 0.02Nm, and the torque around the Z axis is 0.03Nm.

[0178] Next, the six collected force / torque components are combined into a six-dimensional contact force vector. For example, the collected data can be represented as a six-dimensional vector: [0.1, 0.2, 0.3, 0.01, 0.02, 0.03]. Then, the modulus of the six-dimensional vector is calculated to obtain the composite contact force value. For example, the modulus of the above vector is about 0.41N, which represents the total contact force between the robot end and the bile duct wall.

[0179] The calculated composite contact force value is compared with the preset safety threshold. The safety threshold is a value preset according to the tolerance of the bile duct tissue, such as 0.3 N. If the composite contact force value is greater than the preset safety threshold, it means that the contact force between the robot and the bile duct wall is too large and needs to be adjusted.

[0180] When the composite contact force exceeds the safety threshold, the force buffer coefficient needs to be calculated. The force buffer coefficient is calculated by dividing the difference between the composite contact force and the preset safety threshold by the safety threshold. For example, in the above example, the composite contact force is 0.41N and the safety threshold is 0.3N, then the force buffer coefficient is (0.41-0.3) / 0.3≈0.37. This coefficient reflects the degree to which the contact force exceeds the safety threshold.

[0181] Then, the calculated force buffer coefficient is multiplied by the current movement speed value of the choledochoscope robot to obtain the movement speed value after deceleration. For example, assuming that the current movement speed of the robot is 1 mm / s, the movement speed after deceleration is 1*(1-0.37)=0.63 mm / s.

[0182] At the same time, the avoidance posture angle needs to be calculated. First, the six-dimensional contact force vector is decomposed to obtain the unit vector of the force direction. The unit vector of the force direction represents the direction of the contact force. Then, based on the unit vector of the force direction, combined with the kinematic model of the choledochoscope robot and the geometry of the bile duct, the avoidance posture angle that needs to be adjusted is calculated. For example, if the force direction is mainly in the X direction, it can be calculated that a certain angle needs to be rotated around the Y axis to avoid obstacles. Assume that the calculated avoidance posture angle is 5 degrees clockwise around the Y axis.

[0183] The decelerated motion speed value (0.63 mm / s) and the avoidance posture angle (5 degrees clockwise around the Y axis) are input into the motion planning module to generate new motion control instructions. The motion planning module generates a series of control instructions based on the new speed and posture requirements and the kinematic model of the choledochoscope robot to control the movement of each joint of the robot.

[0184] Finally, the new motion control instructions are input into the drive system of the choledochoscope robot to control the robot to move at the new speed and posture until the synthetic contact force value is less than the preset safety threshold. The robot will continue to monitor the contact force and adjust the movement according to the contact force feedback until the stone removal operation is completed.

[0185] In this embodiment, by real-time monitoring of the contact force and performing feedback control, the choledochoscope robot can be effectively prevented from causing damage to the bile duct tissue, thereby improving the safety of the operation; by adjusting the movement speed and posture according to the contact force, the robot can reach the target position more quickly and smoothly, thereby improving operational efficiency; through precise force control, the accuracy of the choledochoscope robot's operation in a narrow bile duct can be improved, reducing the occurrence of misoperation.

[0186] Figure 2 FIG. 1 is a schematic diagram of the structure of the intelligent choledochoscope-assisted minimally invasive stone removal system for bile duct stones according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0187] The first unit is used to construct a deep learning target detection network to process the bile duct video image acquired in real time by a choledochoscope, and obtain bile duct stone feature information; perform image enhancement processing on the bile duct video image, extract bile duct lumen wall feature information, and establish a three-dimensional bile duct vascular tree model; annotate the bile duct stone feature information in the three-dimensional bile duct vascular tree model, generate a three-dimensional bile duct structure model with lesion annotations, and calculate the risk coefficient value of each bile duct stone based on the bile duct stone feature information;

[0188] The second unit is used to input the three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data into the deep reinforcement learning network, and calculate the optimal stone removal path based on the risk coefficient value, wherein the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; establish a real-time three-dimensional navigation map according to the optimal stone removal path, collect the real-time spatial position information of the choledochoscope, dynamically align the real-time spatial position information with the real-time three-dimensional navigation map, generate real-time alignment error data, and compensate and correct the optimal stone removal path based on the real-time alignment error data to obtain a corrected optimal stone removal path;

[0189] The third unit is used to input the corrected optimal stone removal path into the predictive control model, adopt the visual servo control method and the force feedback control method to calculate the motion control parameters of the choledochoscope robot, and control the choledochoscope robot to move along the corrected optimal stone removal path to the target bile duct stone position based on the motion control parameters; control the end effector of the choledochoscope robot to adjust the grasping force value and grasping posture angle based on the morphological feature information of the bile duct stone, so as to complete the grasping action and removal operation of the bile duct stone.

[0190] According to a third aspect of the embodiments of the present invention,

[0191] An electronic device is provided, comprising:

[0192] processor;

[0193] a memory for storing processor-executable instructions;

[0194] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0195] According to a fourth aspect of the embodiments of the present invention,

[0196] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0197] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. Intelligent choledochoscope-assisted minimally invasive stone removal method for bile duct stones, characterized in that: include: Using the real-time video images of the bile duct collected by the choledochoscope, a deep learning target detection network is constructed to process the video images of the bile duct to obtain characteristic information of bile duct stones; Performing image enhancement processing on the intra-biliary video image, extracting bile duct lumen wall feature information, and establishing a three-dimensional bile duct vascular tree model; marking the bile duct stone feature information in the three-dimensional bile duct vascular tree model, generating a three-dimensional bile duct structure model with lesion annotations, and calculating the risk coefficient value of each bile duct stone based on the bile duct stone feature information; The three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data are input into the deep reinforcement learning network, and the optimal stone removal path is calculated based on the risk coefficient value, and the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; a real-time three-dimensional navigation map is established according to the optimal stone removal path, and the real-time spatial position information of the choledochoscope is collected, and the real-time spatial position information is dynamically aligned with the real-time three-dimensional navigation map to generate real-time alignment error data, and the optimal stone removal path is compensated and corrected based on the real-time alignment error data to obtain a corrected optimal stone removal path; The corrected optimal stone removal path is input into the predictive control model, and the motion control parameters of the choledochoscope robot are calculated using the visual servo control method and the force feedback control method. Based on the motion control parameters, the choledochoscope robot is controlled to move along the corrected optimal stone removal path to the target bile duct stone position; the end effector of the choledochoscope robot is controlled to adjust the grasping force value and the grasping posture angle based on the morphological feature information of the bile duct stone to complete the grasping action and removal operation of the bile duct stone.

2. The method according to claim 1, characterized in that Using the real-time video images of the bile duct collected by the choledochoscope, a deep learning target detection network is constructed to process the video images of the bile duct to obtain characteristic information of bile duct stones; Performing image enhancement processing on the intra-biliary video image, extracting bile duct lumen wall feature information, and establishing a three-dimensional bile duct vascular tree model; marking the bile duct stone feature information in the three-dimensional bile duct vascular tree model, generating a three-dimensional bile duct structure model with lesion annotations, and calculating the risk coefficient value of each bile duct stone based on the bile duct stone feature information includes: The intra-biliary video image is input into a deep learning target detection network, the deep learning target detection network extracts image depth features through multi-level residual convolution units, the image depth features are input into a cross-layer feature fusion module, the cross-layer feature fusion module uses an adaptive weight allocation mechanism to perform multi-scale feature aggregation on the image depth features to obtain a target feature map; the target feature map is deconvoluted and reconstructed with dense features to obtain spatial position information, morphological contour information, and density distribution information of bile duct stones; Based on the gray value distribution of each pixel in the intra-biliary video image, a pixel-level gain matrix is ​​constructed, and based on the edge gradient information in the intra-biliary video image, a regional contrast enhancement matrix is ​​constructed; the pixel-level gain matrix is ​​multiplied with the intra-biliary video image to obtain a first enhanced image, and the regional contrast enhancement matrix is ​​weightedly superimposed with the first enhanced image to obtain an enhanced bile duct image; The enhanced bile duct image is input into a semantic segmentation network, and multi-scale feature information of the enhanced bile duct image is extracted through a multi-layer spatial pyramid pooling module, and input into a feature progressive fusion module, and the multi-scale feature information is upsampled and reconstructed to obtain bile duct lumen wall feature information; The bile duct wall feature information is voxel-divided using a quadtree structure, and surface continuity constraints and topology preservation constraints are imposed on the divided grid structure to generate a three-dimensional bile duct vascular tree model; the spatial position information, the morphological contour information, and the density distribution information of the bile duct stone are mapped to corresponding positions of the three-dimensional bile duct vascular tree model to generate a three-dimensional bile duct structure model with lesion annotations; A multidimensional feature set of bile duct stones is extracted based on the three-dimensional bile duct structure model, and the multidimensional feature set includes size feature values, location hazard values, density distribution values ​​and tissue damage risk values. The multidimensional feature set is input into the risk prediction module, and feature fusion weight parameters are obtained through training based on preset expert annotation data. The multidimensional feature set is weightedly fused using the attention mechanism to output the risk coefficient value of bile duct stones.

3. The method according to claim 2, characterized in that The enhanced bile duct image is input into a semantic segmentation network, and multi-scale feature information of the enhanced bile duct image is extracted through a multi-layer spatial pyramid pooling module, and input into a feature progressive fusion module, and the multi-scale feature information is upsampled and reconstructed to obtain bile duct wall feature information including: The enhanced bile duct image is input into a shallow feature extraction pathway and a deep feature extraction pathway respectively; in the shallow feature extraction pathway, local texture features are extracted through a series of small-size convolution operations and a dense connection structure based on a preset small-size convolution kernel; in the deep feature extraction pathway, global semantic features are extracted through a cascade of dilated convolution operations and a skip connection structure; Performing adaptive pyramid pooling decomposition on the local texture features and the global semantic features respectively to generate feature subgraphs of multiple scales, performing channel grouping and feature reorganization on the feature subgraphs to form a multi-scale feature tensor; Based on the multi-scale feature tensor, first-order moment distribution parameters and second-order moment distribution parameters of the feature channel are calculated; the first-order moment distribution parameters and the second-order moment distribution parameters are input into a nonlinear mapping network to generate feature channel attention weights; according to the feature channel attention weights, the multi-scale feature tensor is weighted reconstructed to obtain an enhanced feature tensor; Calculating the mutual correlation coefficient matrix between different scale features in the enhanced feature tensor, constructing a feature affinity network, performing random walk propagation in the feature affinity network, updating the association weights of feature nodes, performing spectral decomposition and normalization processing on the updated association weights, and obtaining a cross-scale feature fusion coefficient; Based on the cross-scale feature fusion coefficient, the enhanced feature tensor is adaptively weighted combined to obtain a fused feature tensor, and progressive upsampling is performed to obtain an upsampled feature, a residual learning unit is inserted into the upsampled feature, a local structure feature is extracted, and the local structure feature is superimposed with the upsampled feature to obtain a structure enhancement feature; Constructing a feature compensation network, wherein the feature compensation network comprises a multi-layer deconvolution structure and a dense connection structure; inputting the structural enhancement feature into the feature compensation network to obtain a reconstructed feature map; The reconstructed feature map is residually connected and feature fused with features of corresponding scales in the multi-scale feature tensor, and bile duct lumen wall feature information is output through a nonlinear activation function.

4. The method according to claim 1, characterized in that The three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data are input into the deep reinforcement learning network, and the optimal stone removal path is calculated based on the risk coefficient value, and the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; a real-time three-dimensional navigation map is established according to the optimal stone removal path, and the real-time spatial position information of the choledochoscope is collected, and the real-time spatial position information is dynamically aligned with the real-time three-dimensional navigation map to generate real-time alignment error data, and the optimal stone removal path is compensated and corrected based on the real-time alignment error data, and the corrected optimal stone removal path includes: voxel-encode the three-dimensional bile duct structure model to obtain bile duct structure feature data; extract anatomical marker information from the bile duct anatomy knowledge base data, and map the anatomical marker information to the bile duct structure feature data; perform temporal encoding on the expert surgical experience data to obtain surgical operation sequence data; input the bile duct structure feature data, the anatomical marker information and the surgical operation sequence data into the attention-guided knowledge distillation unit to generate a multimodal feature representation; Based on the multimodal feature representation, a path planning network is constructed, and the path planning network outputs a choledochoscope position coordinate sequence and a bile duct stone removal position coordinate sequence; based on the choledochoscope position coordinate sequence and the bile duct stone removal position coordinate sequence, a choledochoscope motion trajectory parameter is calculated to generate an initial motion trajectory; Based on the bile duct structure feature data and the initial motion trajectory, a spatial deformation risk coefficient is calculated; based on the surgical operation sequence data and the initial motion trajectory, an operation difficulty coefficient is calculated; the spatial deformation risk coefficient and the operation difficulty coefficient are input into a risk assessment network to generate a comprehensive risk score; Constructing a motion trajectory evaluation function, calculating a first spatial distance value between the choledochoscope position coordinate sequence and a preset target position, calculating a second spatial distance value between the bile duct stone removal position coordinate sequence and a preset stone removal position, and calculating a reward value based on the first spatial distance value and the second spatial distance value; constructing a safety evaluation function, calculating a penalty value based on the comprehensive risk score; performing a weighted combination of the reward value and the penalty value to obtain a composite reward signal; The composite reward signal is used to train the path planning network to generate an optimal motion trajectory; the optimal motion trajectory is converted into a spatial position parameter and a posture angle parameter of a choledochoscope to obtain an optimal stone removal path; real-time spatial position information of the choledochoscope is collected, and the real-time spatial position information is dynamically aligned with the spatial position parameter and the posture angle parameter of the choledochoscope to obtain position error data; Based on the position error data, the real-time position offset and the real-time angle offset are calculated; based on the continuous multi-frame recording of the position error data, the motion trajectory correction is calculated; the real-time position offset, the real-time angle offset and the motion trajectory correction are input into the adaptive fusion network to generate correction parameters, and the optimal stone removal path is compensated and corrected according to the correction parameters to obtain a corrected optimal stone removal path.

5. The method according to claim 4, characterized in that Using the composite reward signal to train the path planning network to generate an optimal motion trajectory includes: Constructing a proximal strategy optimization objective function, wherein the optimization reward item in the proximal strategy optimization objective function is obtained by a weighted combination of the reward value and the penalty value, and the constraint item limits the strategy update step length by the trust domain range; Based on the proximal strategy optimization objective function and the constraint violation penalty function, a Lagrangian dual function is constructed, gradient information of the Lagrangian dual function is extracted, and a dual gradient descent direction is generated; Inputting the choledochoscope position coordinate sequence into a first training branch of a path planning network, and inputting the bile duct stone removal position coordinate sequence into a second training branch of the path planning network; Generate a sampling trajectory based on the parameters of the current path planning network, calculate a first spatial distance value between the sampling trajectory and a preset target position, and calculate a second spatial distance value between the sampling trajectory and a preset stone removal position; Calculate the reward value according to the first spatial distance value and the second spatial distance value, calculate the penalty value according to the comprehensive risk score, and update the composite reward signal by weighted combination of the reward value and the penalty value; Setting a constraint violation penalty coefficient based on the training rounds, wherein the constraint violation penalty coefficient increases as the training rounds increase, and inputting the constraint violation penalty coefficient into the constraint violation penalty function; According to the dual gradient descent direction, the composite reward signal and the output of the constraint violation penalty function, the parameter distribution of the path planning network is updated to generate the optimal motion trajectory.

6. The method according to claim 1, characterized in that Inputting the modified optimal stone removal path into a predictive control model, using a visual servo control method and a force feedback control method, calculating motion control parameters of a choledochoscope robot, and controlling the choledochoscope robot to move to a target bile duct stone position along the modified optimal stone removal path based on the motion control parameters; controlling the end effector of the choledochoscope robot to adjust a grasping force value and a grasping posture angle based on the morphological feature information of the bile duct stone, and completing the grasping action and removal operation of the bile duct stone includes: The corrected optimal stone removal path is input into the predictive control model, and the position coordinates and posture angles of multiple sampling points of the optimal stone removal path in the rolling time domain are extracted as reference trajectories; Using recursive least squares method to identify the dynamic parameters of the choledochoscope robot, establishing a state prediction model based on the dynamic parameters, and inputting the state prediction model into a hybrid optimization controller; The image sequence is collected by a choledochoscope image sensor and image feature points are extracted. Based on the image feature points, the image feature error between the current position and the reference trajectory is calculated. According to the image feature error, a mapping matrix from the feature point to the camera speed is established. The joint speed instruction is calculated based on the mapping matrix and the image feature error. The interactive force data is collected by a terminal force sensor of the choledochoscope robot, and the interactive force data is compared with a preset force threshold to obtain a force deviation. Based on the force deviation, the position deviation between the end effector and the reference trajectory is calculated, and the position deviation and the force deviation are converted into a joint torque instruction. The joint speed command and the joint torque command are used as constraints, input into the hybrid optimization controller, and the optimal control sequence is solved under the actuator saturation constraint and the safety boundary constraint. The choledochoscope robot is controlled to move to the target bile duct stone position according to the optimal control sequence, and the image data of the target bile duct stone is collected; geometric features and surface texture features are extracted from the image data, and the grasping force value and the grasping posture angle are calculated; The grasping force value and the grasping posture angle are input into an adaptive force tracking controller, and the adaptive force tracking controller is used to control the end effector to complete the stone grasping.

7. The method according to claim 6, characterized in that Other methods for removing bile duct stones include: Collecting contact force components of the end force sensor of the choledochoscope robot in three orthogonal directions and torque components in three rotation directions, combining the contact force components and the torque components into a six-dimensional contact force vector, calculating the modulus of the six-dimensional contact force vector, obtaining a composite contact force value, and comparing the composite contact force value with a preset safety threshold; When the synthetic contact force value is greater than the preset safety threshold, a force buffer coefficient is calculated based on the difference between the synthetic contact force value and the preset safety threshold; the force buffer coefficient is multiplied by the current movement speed value of the choledochoscope robot to obtain a movement speed value after deceleration; Decomposing the six-dimensional contact force vector to obtain a force direction unit vector, and calculating an avoidance posture angle based on the force direction unit vector; inputting the decelerated motion speed value and the avoidance posture angle into a motion planning module to generate a new motion control instruction; The new motion control instruction is input into the driving system of the choledochoscope robot to control the choledochoscope robot to move according to the decelerated motion speed value and the avoidance posture angle until the synthetic contact force value is less than the preset safety threshold.

8. An intelligent choledochoscope-assisted minimally invasive bile duct stone removal system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to construct a deep learning target detection network to process the bile duct video image collected in real time by a choledochoscope to obtain bile duct stone feature information; Performing image enhancement processing on the intra-biliary video image, extracting bile duct lumen wall feature information, and establishing a three-dimensional bile duct vascular tree model; marking the bile duct stone feature information in the three-dimensional bile duct vascular tree model, generating a three-dimensional bile duct structure model with lesion annotations, and calculating the risk coefficient value of each bile duct stone based on the bile duct stone feature information; The second unit is used to input the three-dimensional bile duct structure model, the preset bile duct anatomy knowledge base data, and the expert surgical experience data into the deep reinforcement learning network, and calculate the optimal stone removal path based on the risk coefficient value, wherein the optimal stone removal path includes the choledochoscope motion path parameters and the bile duct stone removal path parameters; establish a real-time three-dimensional navigation map according to the optimal stone removal path, collect the real-time spatial position information of the choledochoscope, dynamically align the real-time spatial position information with the real-time three-dimensional navigation map, generate real-time alignment error data, and compensate and correct the optimal stone removal path based on the real-time alignment error data to obtain a corrected optimal stone removal path; The third unit is used to input the corrected optimal stone removal path into the predictive control model, adopt the visual servo control method and the force feedback control method to calculate the motion control parameters of the choledochoscope robot, and control the choledochoscope robot to move along the corrected optimal stone removal path to the target bile duct stone position based on the motion control parameters; control the end effector of the choledochoscope robot to adjust the grasping force value and grasping posture angle based on the morphological feature information of the bile duct stone, so as to complete the grasping action and removal operation of the bile duct stone.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images