Robot-assisted wound suture method and system based on artificial intelligence

Abstract

The invention provides a robot-assisted wound suturing method and system based on artificial intelligence, and relates to the technical field of artificial intelligence, and the robot-assisted wound suturing method comprises the steps: obtaining multi-modal perception data of a wound area, constructing multi-modal three-dimensional representation of a wound, and determining a plurality of suturing points based on the multi-modal three-dimensional representation of the wound; for each suture point, calculating a candidate suture point, and for the adjacent suture points, generating a needle tip track of a needle tip in the wound, and sampling on the needle tip track to obtain a suture scheme adapted to the wound form and the tissue attribute; based on the obtained stitching scheme, an optimal control sequence of the stitching robot is obtained, meanwhile, a robust compensation control law is constructed through coordinate transformation and a feedback control law, and the optimal control sequence and the robust compensation control law are integrated into a stitching robot real-time control system; the output of a real-time control system of the suturing robot is converted into an actuator control instruction, and the suturing robot is controlled to assist in wound suturing.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to an artificial intelligence-based robot-assisted wound suturing method and system. Background Art

[0002] Wound suturing is an indispensable part of surgical operations, which directly affects the postoperative recovery effect and patient prognosis. Traditional manual suturing relies on the doctor's experience and skills, and has problems such as difficulty in operation, low efficiency, and unstable suturing quality. In order to improve the accuracy and repeatability of wound suturing, people have begun to study robot-assisted suturing technology. At present, some robot-assisted suturing systems have been put into clinical application, such as the da Vinci surgical robot system. The system maps the doctor's hand movements to the end of the robot through master-slave operation to achieve remote precision control. However, this type of system still relies on the doctor's subjective operation, and the suturing quality is limited by the operator's proficiency. At the same time, perception decision-making and planning control in complex wound environments also place higher requirements on operators.

[0003] Therefore, it is urgent to develop an intelligent robot-assisted wound suturing system that integrates perception, decision-making, and control. The system should be able to accurately perceive wound characteristics and suturing status, autonomously plan and optimize needle tip movement, and accurately control the suturing process to improve suturing efficiency and quality. Summary of the invention

[0004] The embodiments of the present invention provide a robot-assisted wound suturing method and system based on artificial intelligence, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provided is an artificial intelligence-based robot-assisted wound suturing method, comprising:

[0007] Acquire multimodal sensing data of the wound area, the multimodal sensing data including RGB images, force tactile data and ultrasonic imaging data, map the multimodal sensing data to a unified spatial coordinate system, construct a multimodal three-dimensional representation of the wound, and perform equal-interval sampling on the wound edge according to the wound shape and suturing requirements based on the multimodal three-dimensional representation of the wound to determine multiple suturing points;

[0008] For each suture point, the direction of needle tip movement, tissue thickness and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated, and sampling is performed on the needle tip trajectory to calculate the optimal needle posture at each sampling point. According to the curvature of the needle tip trajectory and tissue properties, the needle tip movement speed is adaptively adjusted to obtain a suture plan that is compatible with the wound morphology and tissue properties.

[0009] Based on the obtained suturing scheme, combined with the position, speed, and torque sensor information of the suturing robot's joints, the optimal control sequence of the suturing robot is obtained. At the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and the robust compensation control law are integrated into the real-time control system of the suturing robot. The output of the real-time control system of the suturing robot is converted into actuator control instructions to control the suturing robot to assist in wound suturing.

[0010] In an optional embodiment,

[0011] Acquiring multimodal sensing data of the wound area, the multimodal sensing data including RGB images, force tactile data and ultrasonic imaging data, mapping the multimodal sensing data to a unified spatial coordinate system, and constructing a multimodal three-dimensional representation of the wound includes:

[0012] A convolutional neural network is used to extract pixel-level disparity maps from the left and right views of the RGB image. At the same time, an attention mechanism and multi-scale feature fusion are introduced to convert the pixel-level disparity map into three-dimensional point cloud data of the wound surface.

[0013] The surface reconstruction method based on implicit function is used to transform the obtained 3D point cloud data into a signed distance field, and the implicit neural representation network is used to learn the mapping of the coordinates of the 3D point cloud data to the signed distance field to obtain the wound implicit surface. The wound triangulation network model is obtained by extracting the zero-isovalue surface of the wound implicit surface.

[0014] According to the gradient changes of force tactile data and ultrasound imaging data, the voxel resolution is adaptively adjusted, and sparse voxel representation is achieved through the hierarchical structure of the octree. The force tactile data and ultrasound imaging data are attached to the vertices of the pre-built visual model using voxel interpolation and nearest neighbor search algorithms to obtain the final wound visual model.

[0015] A physics-based wound tissue modeling technology is introduced, the wound area is divided using a three-dimensional tetrahedral mesh, the parameters of the three-dimensional tetrahedral mesh are initialized using the histological characteristics extracted from ultrasound images, the deformation and stress distribution of the wound are calculated using the finite element method, and the parameters of the three-dimensional tetrahedral mesh are iteratively updated based on the comparison results of the stress distribution and the force tactile data to obtain the final wound biomechanical model;

[0016] The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound.

[0017] In an optional embodiment,

[0018] The wound triangulated network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound, including:

[0019] The wound triangle network model, wound visual model and wound biomechanical model are integrated into a unified graph data structure, where the nodes in the graph data structure represent sampling points and the edges represent the connection relationship between nodes.

[0020] Adaptively determine the convolution kernel size, shape and weight distribution parameters of the node according to the local geometric features and neighborhood distribution of the node, wherein the local geometric features include the curvature and detail complexity of the node neighborhood, and the neighborhood distribution includes the geometric structure and spatial distribution of the node neighborhood;

[0021] According to the convolution kernel size, shape and weight distribution parameters of the node, a dynamic graph convolution kernel of the node is constructed, wherein the dynamic graph convolution kernel is used to extract features of the node, and the weight of the dynamic graph convolution kernel is adaptively adjusted through an attention mechanism;

[0022] Extract node features at multiple scales, apply the corresponding dynamic graph convolution kernel at each scale to obtain node features at different scales, and input the features at different scales into the feature fusion layer;

[0023] In the feature fusion layer, the aggregation weight of the node is adaptively determined according to the connection strength between the nodes and the importance of the nodes, and the features of the neighboring nodes are weightedly aggregated according to the aggregation weight to obtain the final multimodal three-dimensional representation of the wound.

[0024] In an optional embodiment,

[0025] For each suture point, the direction of needle tip movement, tissue thickness and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated, and sampling is performed on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties. The suture schemes adapted to the wound morphology and tissue properties are obtained, including:

[0026] For each suture point, the tissue thickness, hardness and vascular nerve density in its local neighborhood are extracted, and the suture points that do not meet the threshold conditions are eliminated according to the preset threshold, and the number and index of suture points are updated to finally obtain the screened candidate suture point set;

[0027] Calculate the spatial distance, curvature and tangent direction between adjacent stitching points in the candidate stitching point set;

[0028] The wound closure effect evaluation function and tissue stress distribution evaluation function were constructed as optimization targets, and the spacing between adjacent suture points, curvature and the number of suture points were used as constraints. The optimal number and position of suture points were obtained through optimization and solution as the final suture point sequence.

[0029] Based on each pair of adjacent suture points in the final suture point sequence, combined with the local curvature and thickness of the wound, the center line of the needle tip trajectory is fitted in the wound cross section, and the trajectory sampling point sequence is obtained by uniformly sampling along the center line of the needle tip trajectory. The tangent direction and normal vector of the normal plane at each trajectory sampling point are calculated and used as the optimal needle posture at the corresponding trajectory sampling point.

[0030] According to the tissue hardness, trajectory curvature and total trajectory length at the trajectory sampling point, the needle tip movement speed at each trajectory sampling point is adaptively calculated through a predefined speed adjustment function to form a needle tip trajectory speed distribution curve;

[0031] Based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve, the optimization problem of needle tip motion planning is constructed, and the optimization problem is solved using an adaptive optimization algorithm. Combined with the obtained optimal needle posture, a suturing plan that is compatible with the wound morphology and tissue properties is finally obtained.

[0032] In an optional embodiment,

[0033] Constructing wound closure effect evaluation function and tissue stress distribution evaluation function as optimization objectives includes:

[0034] The calculation formula of the wound closure effect evaluation function is as follows:

[0035]

[0036] Among them, E c represents the wound closure effect, ω1 represents the weight coefficient of shape error, N represents the number of suture points, R i represents the local curvature radius of the i-th suture point, p i represents the coordinates of the i-th suture point, r(u i ) represents the parameterized representation of the wound edge curve, u i represents the parameter coordinates of the i-th suture point on the wound edge curve, ω2 represents the weight coefficient of curvature continuity, t i+1 represents the tangent direction of the i+1th suture point, t i represents the tangent direction of the i-th suture point, ||·|| represents the norm of the vector;

[0037] The calculation formula of the tissue stress distribution evaluation function is as follows:

[0038]

[0039] Among them, E s represents tissue stress distribution, ω3 represents the weight coefficient of stress distribution smoothness, A represents the area of ​​wound and surrounding tissue, ▽ represents the gradient operator, σ(a) represents the tissue stress tensor at tissue a, ω4 represents the weight coefficient of peak stress, ||·|| F represents the Frobenius norm;

[0040] The optimization problems of needle tip motion planning based on the obtained needle tip trajectory center line and needle tip trajectory velocity distribution curve include:

[0041] According to the distance between the needle tip trajectory sampling point and the wound centerline, the speed and direction of the needle tip trajectory, the objective function of the optimization problem is constructed, and the calculation formula is as follows:

[0042]

[0043] Among them, J represents the objective function, q 1:M represents the sampling point sequence of the needle tip trajectory, v 1:M represents the velocity distribution of the needle tip trajectory, θ 1:M represents the direction sequence of the needle tip trajectory, M represents the number of sampling points, and q j represents the coordinates of the jth sampling point, c(u j ) represents the center line of the needle tip trajectory fitted in the wound cross section at parameter u j The coordinates at the position, λ1 represents the weight coefficient of the distance, μ t represents the weight coefficient of tangential velocity, v t，j represents the tangential velocity at the jth sampling point, μ n Represents the weight coefficient of normal velocity, v n，j represents the normal velocity at the jth sampling point, △t j represents the time interval between the jth sampling point and the next sampling point, λ2 represents the directional weight coefficient, θ j+1 represents the direction at the j+1th sampling point, θ j represents the direction at the jth sampling point, λ3 represents the weight coefficient of the speed, and v j+1 represents the velocity at the j+1th sampling point, v j represents the velocity at the jth sampling point.

[0044] In an optional embodiment,

[0045] Based on the obtained suturing scheme, combined with the position, speed, and torque sensor information of the suturing robot joints, the optimal control sequence of the suturing robot is obtained. At the same time, the robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the real-time control system of the suturing robot, including:

[0046] The mechanical structure parameters of the suturing robot are obtained, and the robot coordinate system including the base coordinate system, joint coordinate system, manipulator coordinate system and tissue surface coordinate system is established. The position and posture of the suturing robot relative to the base coordinate system are solved by coordinate transformation, and the kinematic positive solution relationship between the robot joint angle and the needle tip position and posture is established; the robot dynamics model describing the relationship between the joint torque and the needle tip force is derived by using the Lagrange equation, combined with the robot joint angle, angular velocity and torque;

[0047] The robot joint angle and angular velocity are used as state variables, the joint torque is used as the control variable, the needle tip posture tracking error and the control input smoothness are used as performance indicators, and the optimization problem of the suture robot motion control is constructed. The state equation, boundary conditions and penalty factors corresponding to the joint constraints are introduced to transform the optimization problem into an unconstrained optimization problem. The sequential convex optimization method is used to divide the unconstrained optimization problem into multiple convex quadratic sub-problems. Through penalty factor adjustment and trust region control, each sub-problem is iteratively solved to obtain the optimal control torque sequence of the suture robot joint.

[0048] A position sensor is set at the end of the suturing robot to measure the posture deviation of the needle tip relative to the tissue surface in real time. The obtained posture deviation of the needle tip is mapped to the joint space through coordinate transformation. The joint angle compensation for compensating the posture deviation of the needle tip is generated by combining the pre-built time series prediction model and the robot dynamics model.

[0049] The optimal control torque sequence is used as the feedforward control input, and the joint angle compensation is used as the feedback control input. The composite control torque of the robot joint is generated by weighted summation, and the final composite control torque is input into the real-time control system of the suture robot.

[0050] In an optional embodiment,

[0051] Combining the pre-built time series prediction model and the robot dynamics model, the steps before generating the joint angle compensation for compensating the needle tip posture deviation also include:

[0052] Constructing a time series data set of needle tip posture deviation, wherein the time series data set includes historical observation values ​​and corresponding timestamps;

[0053] Preprocessing the time series data set, and dividing the preprocessed time series data set into a training set, a validation set, and a test set;

[0054] A multi-scale attention recurrent neural network structure of a needle tip posture deviation prediction model is designed, wherein the network structure includes an input layer, a multi-scale feature extraction layer, an attention fusion layer, a bidirectional recurrent neural network layer, and an output layer; wherein the multi-scale feature extraction layer uses a one-dimensional convolutional neural network to extract local features of the needle tip posture deviation at different time scales with different convolution kernel sizes and step sizes; the attention fusion layer uses a self-attention mechanism to adaptively learn the correlation between different modal features and fuse them into a unified feature representation; the bidirectional recurrent neural network layer uses a bidirectional long short-term memory network to respectively learn the forward and backward dependencies of the needle tip posture deviation in the time dimension and capture its long-term trend characteristics;

[0055] Construct the loss function of multi-task joint learning, and update the network parameters through the back propagation algorithm, repeat the iteration until the model converges, and obtain the trained needle tip posture deviation prediction model;

[0056] The trained needle tip posture deviation prediction model is used to predict the needle tip posture deviation sequence within a certain time range in the future, and the predicted posture deviation sequence is converted into motion compensation instructions in the joint space through kinematic inverse solution and trajectory optimization algorithm.

[0057] According to a second aspect of the embodiments of the present invention,

[0058] Provided is an artificial intelligence-based robot-assisted wound suturing system, comprising:

[0059] The first unit is used to obtain multimodal sensing data of the wound area, wherein the multimodal sensing data includes RGB images, force tactile data and ultrasonic imaging data, and map the multimodal sensing data to a unified spatial coordinate system to construct a multimodal three-dimensional representation of the wound. Based on the multimodal three-dimensional representation of the wound, the wound edge is sampled at equal intervals according to the wound shape and suturing requirements to determine multiple suturing points;

[0060] The second unit is used to calculate candidate suture points for each suture point by comprehensively considering the needle tip movement direction, tissue thickness and blood vessel nerve distribution. At the same time, for adjacent suture points, it generates a needle tip trajectory inside the wound, samples on the needle tip trajectory, calculates the optimal needle posture at each sampling point, and adaptively adjusts the needle tip movement speed according to the curvature of the needle tip trajectory and tissue properties to obtain a suture solution that is compatible with the wound morphology and tissue properties.

[0061] The third unit is used to obtain the optimal control sequence of the suture robot based on the obtained suturing plan and the position, speed and torque sensor information of the suture robot joints, and to construct a robust compensation control law through coordinate transformation and feedback control law, integrate the optimal control sequence and the robust compensation control law into the real-time control system of the suture robot, and convert the output of the real-time control system of the suture robot into actuator control instructions to control the suture robot to assist in wound suturing.

[0062] According to a third aspect of the embodiments of the present invention,

[0063] An electronic device is provided, comprising:

[0064] processor;

[0065] a memory for storing processor-executable instructions;

[0066] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0067] A fourth aspect of the embodiments of the present invention is:

[0068] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0069] In this embodiment, by acquiring RGB images, force tactile data and ultrasonic imaging data and mapping them into a unified spatial coordinate system, a multimodal three-dimensional representation of the wound can be constructed. This three-dimensional representation not only contains the geometric shape information of the wound, but also integrates multimodal information such as mechanical properties and tissue structure, providing comprehensive wound information support for subsequent suture planning. Based on the three-dimensional representation of the wound, the system can automatically plan a reasonable suture point position, needle tip motion trajectory and needle posture according to multiple factors such as wound shape, tissue thickness, vascular nerve distribution, etc., and dynamically adjust the needle tip motion speed considering the trajectory curvature and tissue properties to generate an overall suture plan that is compatible with the wound morphology and tissue properties. This intelligent planning can greatly improve the quality and efficiency of suture. The system maps the planned optimal suture plan into a control sequence of the robot joint space, and constructs robust compensation through feedback control law, realizing accurate real-time control of the suture robot. This method of planning first and then real-time control can ensure the accuracy and robustness of suture movement. The whole process automation from wound perception to suture planning and then to robot control is realized, which greatly reduces the labor intensity of doctors in surgery. At the same time, through multimodal information fusion and intelligent planning, the suturing quality and efficiency are also significantly improved, which is conducive to improving the level of clinical surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1Schematic diagram of the process of a robot-assisted wound suturing method based on artificial intelligence according to an embodiment of the present invention;

[0071] Figure 2 The present invention is a schematic structural diagram of an artificial intelligence-based robot-assisted wound suturing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0074] Figure 1 FIG. 1 is a flow chart of a robot-assisted wound suturing method based on artificial intelligence according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0075] S101. Acquire multimodal perception data of the wound area, wherein the multimodal perception data includes RGB images, force tactile data, and ultrasonic imaging data, map the multimodal perception data into a unified spatial coordinate system, construct a multimodal three-dimensional representation of the wound, and based on the multimodal three-dimensional representation of the wound, perform equally spaced sampling on the wound edge according to the wound shape and suturing requirements to determine multiple suturing points.

[0076] Among them, a high-resolution industrial camera equipped with a fixed-focus lens is used to realize the acquisition of RGB images and depth images of the wound area. The camera should have a sufficient field of view to cover the entire wound area. A miniature six-axis force sensor is selected to measure the three-dimensional contact force and torque between the needle tip and the tissue. The sensor should have high sensitivity and low noise characteristics to meet the needs of small force measurement. At the same time, a tactile sensor array is arranged around the needle tip to collect texture and slip information on the tissue surface. The electrical impedance tactile sensing technology is introduced to measure the conductivity change between the needle tip and the tissue to evaluate the insertion depth and contact state of the needle tip in real time. A miniaturized and flexible electrode array is designed to ensure a close fit between the electrode and the needle tip.

[0077] The data collected by multimodal sensors are heterogeneous in time and space, and need to be synchronized in time and space and the coordinate system is unified. Specifically, a timestamp is introduced into each sensor data to record the absolute time of sampling. The clocks of different sensors are synchronized through methods such as the Network Time Protocol (NTP). Through hand-eye calibration and external parameter calibration, the spatial transformation relationship between the visual sensor, force tactile sensor, and robot base coordinate system is established. High-precision calibration data is obtained by using checkerboard calibration plates, precision measurement tools, etc. Align the timestamps and unify the data of different sensors to the same time reference. Using spatial coordinate transformation, the force tactile and electrical impedance data are mapped to the visual coordinate system to achieve spatial registration of the data.

[0078] In an optional embodiment,

[0079] Acquiring multimodal sensing data of the wound area, the multimodal sensing data including RGB images, force tactile data and ultrasonic imaging data, mapping the multimodal sensing data to a unified spatial coordinate system, and constructing a multimodal three-dimensional representation of the wound includes:

[0080] A convolutional neural network is used to extract pixel-level disparity maps from the left and right views of the RGB image. At the same time, an attention mechanism and multi-scale feature fusion are introduced to convert the pixel-level disparity map into three-dimensional point cloud data of the wound surface.

[0081] The surface reconstruction method based on implicit function is used to transform the obtained 3D point cloud data into a signed distance field, and the implicit neural representation network is used to learn the mapping of the coordinates of the 3D point cloud data to the signed distance field to obtain the wound implicit surface. The wound triangulation network model is obtained by extracting the zero-isovalue surface of the wound implicit surface.

[0082] According to the gradient changes of force tactile data and ultrasound imaging data, the voxel resolution is adaptively adjusted, and sparse voxel representation is achieved through the hierarchical structure of the octree. The force tactile data and ultrasound imaging data are attached to the vertices of the pre-built visual model using voxel interpolation and nearest neighbor search algorithms to obtain the final wound visual model.

[0083] A physics-based wound tissue modeling technology is introduced, the wound area is divided using a three-dimensional tetrahedral mesh, the parameters of the three-dimensional tetrahedral mesh are initialized using the histological characteristics extracted from ultrasound images, the deformation and stress distribution of the wound are calculated using the finite element method, and the parameters of the three-dimensional tetrahedral mesh are iteratively updated based on the comparison results of the stress distribution and the force tactile data to obtain the final wound biomechanical model;

[0084] The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound.

[0085] For example, first, a pair of RGB images of the wound area is collected using a binocular camera, and stereo matching is performed through a deep learning method to estimate pixel-level disparity information. An end-to-end convolutional neural network is used to directly regress the disparity map from the left and right views, and an attention mechanism and a multi-scale feature fusion strategy are introduced to improve the accuracy and robustness of disparity estimation. After obtaining the disparity map, it is converted into dense three-dimensional point cloud data of the wound surface according to the binocular camera parameters.

[0086] Then, in view of the irregularity and complex topology of the wound surface, a surface reconstruction method based on implicit functions is used to transform the scattered point cloud into a continuous and smooth implicit surface. By mapping the point cloud coordinates to the signed distance field and using the implicit neural representation network to learn this mapping relationship, the implicit function representing the wound shape is obtained. On this basis, by extracting the zero isosurface of the implicit surface, a topologically correct and geometrically accurate triangular mesh model of the wound is generated.

[0087] Next, in order to register and fuse the force tactile and ultrasound imaging data with the visual model, an adaptive octree voxelization method is proposed. According to the local gradient changes of the force tactile and ultrasound data, the resolution of the voxels is dynamically adjusted, the voxels are subdivided in the area with large gradients, and the voxels are merged in the area with small gradients. The octree hierarchical structure is used to achieve sparse and efficient voxel representation. Using voxel interpolation and nearest neighbor search algorithms, the force tactile and ultrasound information are attached to the vertices of the pre-built visual model to obtain a wound visual model that integrates multimodal data.

[0088] In order to further obtain the physical properties and internal structure of wound tissue, a physics-based biomechanical modeling technology was introduced. The wound area was divided by a three-dimensional tetrahedral mesh, and the material parameters of the tetrahedral unit were initialized using the tissue acoustic properties extracted from ultrasound images. The finite element method was used to solve the deformation and stress distribution of the wound under the action of external force. By comparing the deformation results predicted by the finite element with the actual deformation measured by the force tactile sensor, the material parameters were iteratively optimized and updated, and finally a biomechanical model that accurately reflects the physical properties of the wound tissue was obtained.

[0089] Finally, the wound triangular mesh model, visual model and biomechanical model are integrated into a unified data structure. Through graph convolution operations, while retaining the spatial structure of the model, the propagation and fusion of different modal features in the mesh surface and voxel space can be achieved, and a multimodal three-dimensional representation of the wound containing geometric, texture, mechanical and anatomical information can be obtained. This representation can be used to guide subsequent wound repair and rehabilitation, such as suture planning, healing monitoring and surgical training.

[0090] In this embodiment, a convolutional neural network is used to extract a pixel-level disparity map, and combined with an attention mechanism and multi-scale feature fusion, the three-dimensional point cloud data of the wound can be accurately reconstructed from the RGB binocular view. The point cloud is then converted into a wound triangular mesh model through a surface reconstruction method based on an implicit function, and the reconstruction accuracy and quality are significantly improved. Multimodal fusion not only enhances the information integrity of the three-dimensional representation of the wound, but also provides more comprehensive data support for subsequent suture planning. The physics-based wound tissue modeling technology is introduced to obtain a biomechanical model of the wound. This biomechanical model has important guiding significance for tissue behavior during the suturing process. The three-dimensional representation of the wound based on multimodal fusion can provide rich information input for intelligent suture planning, making the planning more personalized and optimized, and improving the quality and efficiency of suture. At the same time, high-precision three-dimensional reconstruction and modeling of the wound also lays the foundation for the automated control of the suture robot.

[0091] In an optional embodiment,

[0092] The wound triangulated network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound, including:

[0093] The wound triangle network model, wound visual model and wound biomechanical model are integrated into a unified graph data structure, where the nodes in the graph data structure represent sampling points and the edges represent the connection relationship between nodes.

[0094] Adaptively determine the convolution kernel size, shape and weight distribution parameters of the node according to the local geometric features and neighborhood distribution of the node, wherein the local geometric features include the curvature and detail complexity of the node neighborhood, and the neighborhood distribution includes the geometric structure and spatial distribution of the node neighborhood;

[0095] According to the convolution kernel size, shape and weight distribution parameters of the node, a dynamic graph convolution kernel of the node is constructed, wherein the dynamic graph convolution kernel is used to extract features of the node, and the weight of the dynamic graph convolution kernel is adaptively adjusted through an attention mechanism;

[0096] Extract node features at multiple scales, apply the corresponding dynamic graph convolution kernel at each scale to obtain node features at different scales, and input the features at different scales into the feature fusion layer;

[0097] In the feature fusion layer, the aggregation weight of the node is adaptively determined according to the connection strength between the nodes and the importance of the nodes, and the features of the neighboring nodes are weightedly aggregated according to the aggregation weight to obtain the final multimodal three-dimensional representation of the wound.

[0098] For example, the wound model data of different modalities are first converted into a representation of a graph data structure, where the nodes in the graph represent the sampling points on the wound surface, and the edges represent the topological connection relationship between the nodes. In this way, heterogeneous data of different modalities can be mapped to a unified graph domain representation, which is convenient for subsequent feature extraction and fusion processing.

[0099] After constructing the graph data structure of wounds, a design method of adaptive dynamic graph convolution kernel is proposed to extract multi-scale and multi-modal wound features from graph structure data. Among them, the local geometric features of nodes include the curvature and detail complexity of the node neighborhood, which reflects the local shape characteristics of the wound surface; the neighborhood distribution of nodes includes the geometric structure and spatial distribution of the node neighborhood, which reflects the local topological characteristics of the wound surface. By comprehensively considering these features, the optimal convolution kernel parameters can be designed for each node to improve the adaptability and accuracy of feature extraction.

[0100] After determining the convolution kernel parameters of the node, the dynamic graph convolution method is used to construct the corresponding dynamic graph convolution kernel according to the convolution kernel size, shape and weight distribution parameters of each node. Unlike the traditional convolution kernel that uses fixed weights, the weight of the dynamic graph convolution kernel is adaptively adjusted according to the characteristics of the node, which can better adapt to the local changes of the wound surface. When extracting features, the dynamic graph convolution kernel performs convolution operations in the neighborhood of the node to extract the local features of the node. At the same time, the attention mechanism is introduced to adaptively adjust the weight of the dynamic graph convolution kernel according to the importance of the node features, further improving the accuracy and robustness of feature extraction.

[0101] Considering that the shape and texture features of the wound surface have different performances at different scales, a multi-scale feature extraction strategy is adopted. Specifically, in the graph data structure, with the node as the center, neighborhoods of different sizes are selected as receptive fields at different scales. At each scale, the corresponding dynamic graph convolution kernel is used to extract the neighborhood features of the node to obtain the feature representation of the node at that scale. By extracting the features of the nodes at multiple scales, the shape and texture information of the wound surface at different levels and granularities can be obtained, which improves the richness and completeness of the feature representation.

[0102] After obtaining the features of nodes at different scales, a feature fusion layer was designed to fuse node features of different scales and modalities to obtain the final three-dimensional representation of the wound. In the feature fusion layer, the aggregation weight of each node is adaptively determined according to the connection strength between nodes and the importance of nodes. Among them, the connection strength between nodes reflects the topological relationship of nodes in the graph structure, which can be measured by the weight of the edge or the similarity between nodes; the importance of the node reflects the contribution of the node to the overall representation of the wound, which can be measured by graph theory indicators such as the centrality and clustering coefficient of the node. By comprehensively considering these factors, the optimal aggregation weight is assigned to each node for weighted aggregation of the features of neighboring nodes.

[0103] In the process of feature aggregation, an adaptive weight adjustment strategy based on the attention mechanism is adopted. Specifically, for each node, an attention weight matrix is ​​calculated based on the feature correlation between it and its neighboring nodes. This matrix reflects the importance of the neighboring node features to the current node features and can be used to adjust the aggregation weight. By introducing the attention mechanism, important neighboring node features can be adaptively highlighted, noise and redundant information can be suppressed, and the accuracy and robustness of feature aggregation can be improved.

[0104] Finally, through the feature aggregation layer, the node features of different scales and modalities are weighted and fused to obtain a multimodal three-dimensional representation of the wound. This representation includes both the geometric shape features of the wound surface and the visual texture features and biomechanical property features of the wound. It is a comprehensive and compact three-dimensional representation of the wound. Based on this representation, tasks such as wound segmentation, suture planning, and surgical navigation can be further carried out to provide key wound information for the intelligent assisted surgery system.

[0105] In this embodiment, by mapping the multimodal data of the wound into a unified graph data structure, using adaptive dynamic convolution kernels and attention mechanisms to extract multi-scale, multi-modal wound features, and obtaining the final three-dimensional representation of the wound through a feature fusion layer, the feature extraction and fusion strategies can be adaptively adjusted according to the local shape and topological structure of the wound surface, thereby improving the accuracy and robustness of representation learning. At the same time, the use of graph data structures and graph convolutional neural networks can effectively process the non-Euclidean geometric characteristics and unstructured characteristics of wound data, overcoming the limitations of traditional grid-based and voxel-based representation methods. It provides new ideas and technical means for three-dimensional reconstruction of wounds and intelligent assisted surgery, which has important theoretical significance and application value.

[0106] S102. For each suture point, the direction of needle tip movement, tissue thickness and blood vessel nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, a needle tip trajectory inside the wound is generated, and sampling is performed on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties to obtain a suture plan that is compatible with the wound morphology and tissue properties.

[0107] For example, the wound edge curve is first reparameterized, and the sampling interval is adaptively adjusted according to the wound size and suture density requirements. Several sampling points are arranged at equal distances along the wound edge as candidate suture points. Then the thickness, hardness and vascular nerve distribution of the tissue at the candidate point are evaluated, and the points that are not suitable for suture are eliminated. Then the spatial distance, curvature and tangent direction of adjacent suture points are calculated for subsequent generation of needle tip trajectory. Finally, the wound closure effect and tissue stress distribution are comprehensively considered to optimize the number and position of suture points to obtain the final suture point sequence. With the suture point as the endpoint, a smooth curve is fitted in the wound cross section according to the wound curvature and thickness distribution as the center line of the needle tip trajectory. In order to keep the needle tip at a suitable contact angle with the tissue, we uniformly sample on the center line of the trajectory, calculate the tangent direction and normal plane at each sampling point, and use it as a reference for the posture of the needle tip. At the same time, considering the curvature change of the needle tip trajectory and the difference in tissue properties, the needle tip movement speed of each point on the trajectory is adaptively adjusted. The basis for speed adjustment includes ensuring stable contact between the needle tip and the tissue, reducing needle tip slippage, and avoiding tearing caused by excessive speed changes. Through iterative optimization, the needle tip motion trajectory and its velocity distribution that meet the requirements of suturing accuracy and safety are obtained.

[0108] In an optional embodiment,

[0109] For each suture point, the direction of needle tip movement, tissue thickness and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated, and sampling is performed on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties. The suture schemes adapted to the wound morphology and tissue properties are obtained, including:

[0110] For each suture point, the tissue thickness, hardness and vascular nerve density in its local neighborhood are extracted, and the suture points that do not meet the threshold conditions are eliminated according to the preset threshold, and the number and index of suture points are updated to finally obtain the screened candidate suture point set;

[0111] Calculate the spatial distance, curvature and tangent direction between adjacent stitching points in the candidate stitching point set;

[0112] The wound closure effect evaluation function and tissue stress distribution evaluation function were constructed as optimization targets, and the spacing between adjacent suture points, curvature and the number of suture points were used as constraints. The optimal number and position of suture points were obtained through optimization and solution as the final suture point sequence.

[0113] Based on each pair of adjacent suture points in the final suture point sequence, combined with the local curvature and thickness of the wound, the center line of the needle tip trajectory is fitted in the wound cross section, and the trajectory sampling point sequence is obtained by uniformly sampling along the center line of the needle tip trajectory. The tangent direction and normal vector of the normal plane at each trajectory sampling point are calculated and used as the optimal needle posture at the corresponding trajectory sampling point.

[0114] According to the tissue hardness, trajectory curvature and total trajectory length at the trajectory sampling point, the needle tip movement speed at each trajectory sampling point is adaptively calculated through a predefined speed adjustment function to form a needle tip trajectory speed distribution curve;

[0115] Based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve, the optimization problem of needle tip motion planning is constructed, and the optimization problem is solved using an adaptive optimization algorithm. Combined with the obtained optimal needle posture, a suturing plan that is compatible with the wound morphology and tissue properties is finally obtained.

[0116] Exemplarily, for each suture point in the candidate suture point set, biomechanical and anatomical features such as tissue thickness, hardness, and vascular nerve density in its local neighborhood are extracted. By comparing the extracted feature values ​​with the preset tissue thickness range threshold, tissue hardness threshold, and vascular nerve density threshold, it is determined whether the suture point meets the suture requirements. If any feature does not meet the corresponding threshold condition, the suture point is marked as unsuitable for suturing and removed from the candidate suture point set. After completing the suitability evaluation of all candidate suture points, the number of remaining suitable suture points is updated, and they are renumbered and indexed, and finally the screened candidate suture point set is obtained.

[0117] Next, the spatial distance between each pair of adjacent suture points in the selected candidate suture point set, the curvature of the connected line segment, and the tangent direction of the wound edge curve at each suture point are calculated. Among them, the spatial distance between adjacent suture points reflects the local suture density, the curvature reflects the smoothness of the wound edge after suture, and the tangent direction at the suture point will be used as a posture reference for subsequent needle tip trajectory planning.

[0118] In order to take into account both the suturing effect and the risk of tissue damage when selecting suture points, wound closure effect evaluation function and tissue stress distribution evaluation function were constructed as cost functions for multi-objective optimization. The wound closure effect evaluation function uses the weighted sum of the shape error and coincidence error of the wound edge after suturing as quantitative indicators to measure the appearance and wound closure quality of the suture scheme. The tissue stress distribution evaluation function uses the uniformity of local tissue stress distribution and the magnitude of peak stress after suturing as the objects of investigation to evaluate the tissue damage risk of the suture scheme. On this basis, with the spacing between adjacent suture points, the curvature of the connecting line segment and the total number of suture points as constraints, the optimal number and spatial position of suture points that take into account both suture quality and tissue protection are obtained through a multi-objective optimization algorithm as the final suture point sequence.

[0119] For each pair of adjacent suture points in the final suture point sequence, a cubic B-spline curve is fitted in the wound cross section, combined with the local curvature radius and tissue thickness of the wound, as the center line of the needle tip trajectory from one suture point to another. In order to ensure that the needle tip trajectory maintains a safe distance from the inner wall of the wound and adapts to the tissue thickness, the suture point is set as the trajectory endpoint, and under the constraint of the continuity of the tangent direction at the trajectory endpoint, the position of the intermediate control point is adaptively adjusted to make it symmetrical about the suture point line, and the minimum distance from the inner surface of the wound is proportional to the local tissue thickness. After completing the fitting of the needle tip trajectory centerline, uniform sampling is performed along it to generate a trajectory sampling point sequence. For each trajectory sampling point, the tangent direction of the centerline at its location and the normal vector of the local normal plane are calculated, and used as the reference value of the needle tip posture at the corresponding sampling point to achieve the conformity of the needle tip trajectory to the wound morphology.

[0120] In order to adapt the needle tip movement speed to the local tissue properties of the wound, the optimal speed value of the needle tip movement at each trajectory sampling point is adaptively calculated using the predefined speed adjustment function according to the tissue hardness at the trajectory sampling point, the local trajectory curvature and the length of the entire trajectory, thereby obtaining the speed distribution curve covering the entire needle tip trajectory. Among them, the greater the tissue hardness, the slower the speed of the needle tip should move in it; the greater the trajectory curvature, the needle tip speed should also be reduced accordingly; the longer the total trajectory length, the average speed should be appropriately increased to control the movement time. By establishing a quantitative relationship between the above factors and the needle tip speed through the speed adjustment function, the adaptive matching of the needle tip speed and the wound properties can be achieved.

[0121] Finally, the optimization model of the needle tip motion planning problem is established with the spatial position, attitude angle and motion speed of the needle tip trajectory sampling points as optimization variables, and the weighted sum of the distance error between the entire needle tip trajectory and the inner surface of the wound, the change in the needle tip attitude between adjacent sampling points, and the change in speed as the optimization target. At the same time, the optimization conditions are to meet the needle tip start and end state constraints, speed range constraints, and total motion time constraints. On this basis, the optimization problem is solved using adaptive optimization algorithms such as sequential quadratic programming and gradient descent. After multiple iterations and convergence, the optimal needle tip trajectory and speed distribution that are compatible with the local morphology and tissue properties of the wound can be obtained. By repeating the above needle tip motion planning process for each pair of adjacent suture points in the final suture point sequence, a complete suture solution is finally obtained.

[0122] In an optional embodiment,

[0123] Constructing wound closure effect evaluation function and tissue stress distribution evaluation function as optimization objectives includes:

[0124] The calculation formula of the wound closure effect evaluation function is as follows:

[0125]

[0126] Among them, E c represents the wound closure effect, ω1 represents the weight coefficient of shape error, N represents the number of suture points, R i represents the local curvature radius of the i-th suture point, p i represents the coordinates of the i-th suture point, r(u i ) represents the parameterized representation of the wound edge curve, u i represents the parameter coordinates of the i-th suture point on the wound edge curve, ω2 represents the weight coefficient of curvature continuity, t i+1 represents the tangent direction of the i+1th suture point, t i represents the tangent direction of the i-th suture point, ||·|| represents the norm of the vector;

[0127] The calculation formula of the tissue stress distribution evaluation function is as follows:

[0128]

[0129] Among them, E s represents tissue stress distribution, ω3 represents the weight coefficient of stress distribution smoothness, A represents the area of ​​wound and surrounding tissue, ▽ represents the gradient operator, σ(a) represents the tissue stress tensor at tissue a, ω4 represents the weight coefficient of peak stress, ||·|| F represents the Frobenius norm;

[0130] The optimization problems of needle tip motion planning based on the obtained needle tip trajectory center line and needle tip trajectory velocity distribution curve include:

[0131] According to the distance between the needle tip trajectory sampling point and the wound centerline, the speed and direction of the needle tip trajectory, the objective function of the optimization problem is constructed, and the calculation formula is as follows:

[0132]

[0133] Among them, J represents the objective function, q 1:M represents the sampling point sequence of the needle tip trajectory, v 1:M represents the velocity distribution of the needle tip trajectory, θ 1:M represents the direction sequence of the needle tip trajectory, M represents the number of sampling points, and q j represents the coordinates of the jth sampling point, c(u j ) represents the center line of the needle tip trajectory fitted in the wound cross section at parameter u j The coordinates at the position, λ1 represents the weight coefficient of the distance, μ t represents the weight coefficient of tangential velocity, v t，j represents the tangential velocity at the jth sampling point, μ n Represents the weight coefficient of normal velocity, v n，j represents the normal velocity at the jth sampling point, △t j represents the time interval between the jth sampling point and the next sampling point, λ2 represents the directional weight coefficient, θ j+1 represents the direction at the j+1th sampling point, θ j represents the direction at the jth sampling point, λ3 represents the weight coefficient of the speed, and v j+1 represents the velocity at the j+1th sampling point, v j represents the velocity at the jth sampling point.

[0134] In this embodiment, the closing effect and tissue stress evaluation function are used as optimization targets, and the optimal number and position of suture points are solved under constraints, so that the suture scheme can be well adapted to the personalized characteristics of the wound. The trajectory planning method based on local geometric features can effectively avoid the needle tip from colliding with the wound edge and improve the smoothness and safety of the suture movement. Combined with various factors such as tissue hardness, curvature and total trajectory length at the trajectory sampling point, the optimal movement speed of the needle tip at each sampling point can be adaptively calculated to form a speed distribution curve of the needle tip trajectory. Speed ​​adaptive adjustment can dynamically optimize suture motion parameters according to tissue properties, improve suture quality, and avoid tissue damage. Through personalized suture point selection, reasonable needle tip trajectory planning, needle tip motion adaptive control, and overall optimization solution and other innovative technical means, a suture scheme that is highly matched with specific wound morphology and tissue properties can be generated, which lays a technical foundation for improving suture quality and safety and reducing tissue damage, and has important theoretical value and application prospects.

[0135] S103. Based on the obtained suturing scheme, combined with the position, speed, and torque sensor information of the suturing robot joints, the optimal control sequence of the suturing robot is obtained, and at the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and the robust compensation control law are integrated into the real-time control system of the suturing robot, and the output of the real-time control system of the suturing robot is converted into actuator control instructions to control the suturing robot to assist in wound suturing.

[0136] For example, the kinematic and dynamic models of the joints of the robot manipulator are first established, and the joint angle, velocity and acceleration limits are introduced as constraints. Then, with the suture operation accuracy and energy consumption minimization as the optimization goals, the optimal joint space control sequence of the manipulator is derived by using the variational method, Pontryagin minimum principle and other methods. Considering the uncertainty of robot parameters and environmental disturbances, a robust compensation control law is designed on the basis of optimal control. Using the feedback information of the position, velocity and torque sensors of the robot joints, the posture error of the manipulator end is mapped to the joint space through coordinate transformation, and the joint angle / velocity feedback and integral term are constructed to compensate the system error in real time. Finally, the optimal control sequence is combined with the compensation control law to form a closed-loop control system. The control system uses the planned needle tip trajectory as the reference input, solves the control instructions of the manipulator joints in real time, and converts them into control instructions of the robot actuator after coordinate transformation, so as to realize the precise control of the robot-assisted wound suturing process.

[0137] In an optional embodiment,

[0138] Based on the obtained suturing scheme, combined with the position, speed, and torque sensor information of the suturing robot joints, the optimal control sequence of the suturing robot is obtained. At the same time, the robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the real-time control system of the suturing robot, including:

[0139] The mechanical structure parameters of the suturing robot are obtained, and the robot coordinate system including the base coordinate system, joint coordinate system, manipulator coordinate system and tissue surface coordinate system is established. The position and posture of the suturing robot relative to the base coordinate system are solved by coordinate transformation, and the kinematic positive solution relationship between the robot joint angle and the needle tip position and posture is established; the robot dynamics model describing the relationship between the joint torque and the needle tip force is derived by using the Lagrange equation, combined with the robot joint angle, angular velocity and torque;

[0140] The robot joint angle and angular velocity are used as state variables, the joint torque is used as the control variable, the needle tip posture tracking error and the control input smoothness are used as performance indicators, and the optimization problem of the suture robot motion control is constructed. The state equation, boundary conditions and penalty factors corresponding to the joint constraints are introduced to transform the optimization problem into an unconstrained optimization problem. The sequential convex optimization method is used to divide the unconstrained optimization problem into multiple convex quadratic sub-problems. Through penalty factor adjustment and trust region control, each sub-problem is iteratively solved to obtain the optimal control torque sequence of the suture robot joint.

[0141] A position sensor is set at the end of the suturing robot to measure the posture deviation of the needle tip relative to the tissue surface in real time. The obtained posture deviation of the needle tip is mapped to the joint space through coordinate transformation. The joint angle compensation for compensating the posture deviation of the needle tip is generated by combining the pre-built time series prediction model and the robot dynamics model.

[0142] The optimal control torque sequence is used as the feedforward control input, and the joint angle compensation is used as the feedback control input. The composite control torque of the robot joint is generated by weighted summation, and the final composite control torque is input into the real-time control system of the suture robot.

[0143] For example, first, it is necessary to obtain the mechanical structure parameters of the suture robot, such as the length of the connecting rod, the type of joints, etc. Based on these parameters, the coordinate system of the robot is established, including the base coordinate system, the joint coordinate system, the manipulator coordinate system, and the tissue surface coordinate system. Through homogeneous coordinate transformation, the position and posture of the robot manipulator needle tip relative to the base coordinate system can be solved. Furthermore, using the forward kinematics theory in robotics, the mapping relationship between the robot joint angle and the needle tip position and posture is established, and the kinematic forward solution equation describing the robot motion is obtained.

[0144] In order to describe the dynamic characteristics of the robot, it is necessary to use the Lagrangian mechanics equations, combined with the robot's joint angles, angular velocities and joint torques, to derive the robot's dynamic model. This model describes the dynamic relationship between the robot's joint torque and the force on the needle tip, and is the basis for building a robot motion controller.

[0145] Based on the established robot kinematics and dynamics model, the optimization problem of suture robot motion control can be constructed. The robot joint angle and angular velocity are used as state variables, the joint torque is used as the control variable, the needle tip posture tracking error and the control input smoothness are used as performance indicators, and the objective functional of the optimal control problem is formed. At the same time, the robot dynamics equation is used as the state equation, and the penalty function corresponding to the boundary conditions and joint constraints is introduced to transform the original constrained optimization problem into an unconstrained optimization problem.

[0146] The method of sequential convex optimization is used to solve the transformed unconstrained optimization problem. The entire optimization problem is divided into multiple convex quadratic sub-problems. By introducing penalty factors and trust region control technology, each sub-problem is solved iteratively in turn. In each iteration, the optimal solution of the current sub-problem is obtained by solving the convex quadratic programming, and the next sub-problem is updated according to the penalty factor and trust region parameters. After multiple iterations, the approximate optimal solution of the original optimization problem is finally obtained, that is, the optimal control torque sequence of the robot joints.

[0147] In order to further improve the control accuracy of the suture robot, a position sensor is set at the end of the robot manipulator to measure the posture deviation of the needle tip relative to the tissue surface in real time. Through coordinate transformation, the measured posture deviation of the needle tip is mapped from the manipulator coordinate system to the joint space to obtain the joint angle deviation. At the same time, the pre-built time series prediction model, such as the autoregressive model or the Kalman filter, is used to predict the posture deviation of the needle tip and obtain the deviation estimate in the future. Combined with the robot dynamics model, the posture deviation and the deviation prediction value are converted into joint angle compensation as the input of feedback control.

[0148] Finally, the optimal control torque sequence is used as the feedforward control input, and the joint angle compensation corresponding to the needle tip posture deviation is used as the feedback control input. The composite control torque of the robot joint is generated by weighted summation. By adjusting the weight coefficients of feedforward and feedback, a trade-off can be made between tracking accuracy and robustness. The composite control torque is input into the real-time control system of the suture robot, and through the servo drive and actuator, the precise motion control and real-time trajectory compensation of the robot operator's needle tip are realized.

[0149] In this embodiment, the optimal control theory and sensor feedback technology are comprehensively utilized to effectively improve the accuracy and robustness of robot-assisted surgery. By constructing kinematic and dynamic models, the robot control problem is transformed into an optimization problem for solution, and the optimal control torque sequence is obtained as the feedforward control input. At the same time, the position sensor is used to measure the needle tip posture deviation in real time, and combined with the time series prediction model, the joint angle compensation is generated as the feedback control input. Finally, through the feedforward-feedback composite control, high-precision tracking and real-time compensation of the needle tip movement are achieved.

[0150] In an optional embodiment,

[0151] Combining the pre-built time series prediction model and the robot dynamics model, the steps before generating the joint angle compensation for compensating the needle tip posture deviation also include:

[0152] Constructing a time series data set of needle tip posture deviation, wherein the time series data set includes historical observation values ​​and corresponding timestamps;

[0153] Preprocessing the time series data set, and dividing the preprocessed time series data set into a training set, a validation set, and a test set;

[0154] A multi-scale attention recurrent neural network structure of a needle tip posture deviation prediction model is designed, wherein the network structure includes an input layer, a multi-scale feature extraction layer, an attention fusion layer, a bidirectional recurrent neural network layer, and an output layer; wherein the multi-scale feature extraction layer uses a one-dimensional convolutional neural network to extract local features of the needle tip posture deviation at different time scales with different convolution kernel sizes and step sizes; the attention fusion layer uses a self-attention mechanism to adaptively learn the correlation between different modal features and fuse them into a unified feature representation; the bidirectional recurrent neural network layer uses a bidirectional long short-term memory network to respectively learn the forward and backward dependencies of the needle tip posture deviation in the time dimension and capture its long-term trend characteristics;

[0155] Construct the loss function of multi-task joint learning, and update the network parameters through the back propagation algorithm, repeat the iteration until the model converges, and obtain the trained needle tip posture deviation prediction model;

[0156] The trained needle tip posture deviation prediction model is used to predict the needle tip posture deviation sequence within a certain time range in the future, and the predicted posture deviation sequence is converted into motion compensation instructions in the joint space through kinematic inverse solution and trajectory optimization algorithm.

[0157] Exemplarily, a time series dataset of needle tip posture deviation is first constructed, which contains the historical observation values ​​of needle tip posture deviation and the corresponding timestamp information during the robot suturing process. After obtaining the time series dataset of needle tip posture deviation, preprocessing is performed. Specifically, the needle tip posture deviation data is cleaned and standardized by using common time series preprocessing techniques such as data smoothing, outlier detection and removal, and data normalization. At the same time, in order to evaluate the generalization performance of the model, the preprocessed time series dataset is divided into three non-overlapping subsets: training set, validation set, and test set using the holdout method. Among them, the training set is used for learning and optimizing model parameters, the validation set is used for selecting and tuning model hyperparameters, and the test set is used to evaluate the final performance and generalization ability of the model.

[0158] According to the characteristics of the needle tip posture deviation prediction problem, a multi-scale attention recurrent neural network model is designed. The network structure of the model includes an input layer, a multi-scale feature extraction layer, an attention fusion layer, a bidirectional recurrent neural network layer and an output layer. The input layer receives a fixed-length time series fragment of the needle tip posture deviation and passes it to the subsequent feature extraction layer. The multi-scale feature extraction layer uses a one-dimensional convolutional neural network to extract features from the input sequence. By setting convolution kernels of different sizes and steps, the local features of the needle tip posture deviation are extracted at different time scales.

[0159] After extracting multi-scale local features, the self-attention mechanism is used to adaptively fuse features of different scales. Specifically, the attention fusion layer automatically learns the correlation and interaction between features of different scales by calculating the correlation and importance weights between them. Then, based on the learned attention weights, features of different scales are weightedly fused to obtain a unified feature representation.

[0160] After fusing multi-scale local features, a bidirectional recurrent neural network is used to model the long-term dependency of the needle tip posture deviation. Specifically, the bidirectional recurrent neural network includes forward and backward recurrent neural networks, which propagate information forward and backward in time, respectively. In the forward propagation, the bidirectional recurrent neural network gradually accumulates the historical state information of the needle tip posture deviation and learns its forward causal relationship and trend characteristics. In the backward propagation, the bidirectional recurrent neural network reversely summarizes the future state information of the needle tip posture deviation and learns its backward influence relationship and constraint conditions. Through the bidirectional recurrent structure, the contextual information of the needle tip posture deviation in the time dimension can be fully utilized to capture its long-term evolution law and change trend. At the same time, in order to alleviate the gradient vanishing problem of the recurrent neural network, the long short-term memory network is used as the recurrent unit, and the gating mechanism and memory unit are introduced to enhance the model's ability to process long sequence data.

[0161] After designing the network structure of the needle tip posture deviation prediction model, a loss function of multi-task joint learning is constructed to guide the optimization and learning of model parameters. This loss function not only takes into account the prediction accuracy of the needle tip posture deviation, but also introduces the supervision information of other auxiliary tasks, such as needle tip velocity prediction and needle tip acceleration prediction. By jointly learning multiple related tasks, feature sharing and knowledge transfer between different tasks can be promoted, and the generalization performance and robustness of the model can be improved. In the model training process, the back propagation algorithm and gradient descent optimization method are used to iteratively calculate the gradient of the loss function and use it to update the model parameters. Repeat this process until the model converges on the training set to obtain the final needle tip posture deviation prediction model.

[0162] After obtaining the trained needle tip posture deviation prediction model, it is applied to the real-time control and compensation of the robot suturing process. Specifically, during the suturing process, the prediction model receives the needle tip posture deviation observation values ​​within a certain time range in real time in the form of a sliding window, and predicts the needle tip posture deviation sequence in the future. Then, the predicted posture deviation sequence is converted into the motion compensation amount in the robot joint space using the kinematic inverse solution algorithm. Finally, the trajectory optimization algorithm is used to smooth and optimize the compensated joint motion trajectory under the premise of meeting the suturing task constraints, and the final motion compensation instruction is obtained. The instruction is passed to the robot control system to realize the predictive active compensation for the needle tip posture deviation and improve the accuracy and quality of suturing.

[0163] In this embodiment, by constructing a time series data set, designing a multi-scale attention recurrent neural network model, and combining a multi-task joint learning strategy, an accurate prediction of the long-term and short-term trends and change patterns of the needle tip posture deviation is achieved. This method makes full use of the contextual information and associated features of the needle tip posture deviation in the spatiotemporal dimension, and improves the representation ability and generalization performance of the prediction model by introducing an attention mechanism and a bidirectional loop structure. At the same time, this method combines the prediction results with the kinematic inverse solution and trajectory optimization algorithm to achieve predictive active compensation for the needle tip posture deviation, providing new ideas and methods for improving the accuracy and stability of robot suturing.

[0164] Figure 2 FIG. 1 is a schematic diagram of a robot-assisted wound suturing system based on artificial intelligence according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0165] Acquire multimodal sensing data of the wound area, the multimodal sensing data including RGB images, force tactile data and ultrasonic imaging data, map the multimodal sensing data to a unified spatial coordinate system, construct a multimodal three-dimensional representation of the wound, and perform equal-interval sampling on the wound edge according to the wound shape and suturing requirements based on the multimodal three-dimensional representation of the wound to determine multiple suturing points;

[0166] For each suture point, the direction of needle tip movement, tissue thickness and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated, and sampling is performed on the needle tip trajectory to calculate the optimal needle posture at each sampling point. According to the curvature of the needle tip trajectory and tissue properties, the needle tip movement speed is adaptively adjusted to obtain a suture plan that is compatible with the wound morphology and tissue properties.

[0167] Based on the obtained suturing scheme, combined with the position, speed, and torque sensor information of the suturing robot's joints, the optimal control sequence of the suturing robot is obtained. At the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and the robust compensation control law are integrated into the real-time control system of the suturing robot. The output of the real-time control system of the suturing robot is converted into actuator control instructions to control the suturing robot to assist in wound suturing.

[0168] According to a third aspect of the embodiments of the present invention,

[0169] An electronic device is provided, comprising:

[0170] processor;

[0171] a memory for storing processor-executable instructions;

[0172] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0173] A fourth aspect of the embodiments of the present invention is:

[0174] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0175] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot-assisted wound suturing method based on artificial intelligence, characterized in that: include: Acquire multimodal sensing data of the wound area, the multimodal sensing data including RGB images, force tactile data and ultrasonic imaging data, map the multimodal sensing data to a unified spatial coordinate system, construct a multimodal three-dimensional representation of the wound, and perform equal-interval sampling on the wound edge according to the wound shape and suturing requirements based on the multimodal three-dimensional representation of the wound to determine multiple suturing points; For each suture point, the direction of needle tip movement, tissue thickness and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated, and sampling is performed on the needle tip trajectory to calculate the optimal needle posture at each sampling point. According to the curvature of the needle tip trajectory and tissue properties, the needle tip movement speed is adaptively adjusted to obtain a suture plan that is compatible with the wound morphology and tissue properties. Based on the obtained suturing scheme, combined with the position, speed, and torque sensor information of the suturing robot's joints, the optimal control sequence of the suturing robot is obtained. At the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and the robust compensation control law are integrated into the real-time control system of the suturing robot. The output of the real-time control system of the suturing robot is converted into actuator control instructions to control the suturing robot to assist in wound suturing.

2. The method according to claim 1, characterized in that Acquiring multimodal sensing data of the wound area, the multimodal sensing data including RGB images, force tactile data and ultrasonic imaging data, mapping the multimodal sensing data to a unified spatial coordinate system, and constructing a multimodal three-dimensional representation of the wound includes: A convolutional neural network is used to extract pixel-level disparity maps from the left and right views of the RGB image. At the same time, an attention mechanism and multi-scale feature fusion are introduced to convert the pixel-level disparity map into three-dimensional point cloud data of the wound surface. The surface reconstruction method based on implicit function is used to transform the obtained 3D point cloud data into a signed distance field, and the implicit neural representation network is used to learn the mapping of the coordinates of the 3D point cloud data to the signed distance field to obtain the wound implicit surface. The wound triangulation network model is obtained by extracting the zero-isovalue surface of the wound implicit surface. According to the gradient changes of force tactile data and ultrasound imaging data, the voxel resolution is adaptively adjusted, and sparse voxel representation is achieved through the hierarchical structure of the octree. The force tactile data and ultrasound imaging data are attached to the vertices of the pre-built visual model using voxel interpolation and nearest neighbor search algorithms to obtain the final wound visual model. A physics-based wound tissue modeling technology is introduced, the wound area is divided using a three-dimensional tetrahedral mesh, the parameters of the three-dimensional tetrahedral mesh are initialized using the histological characteristics extracted from ultrasound images, the deformation and stress distribution of the wound are calculated using the finite element method, and the parameters of the three-dimensional tetrahedral mesh are iteratively updated based on the comparison results of the stress distribution and the force tactile data to obtain the final wound biomechanical model; The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound.

3. The method according to claim 2, characterized in that The wound triangulated network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound, including: The wound triangle network model, wound visual model and wound biomechanical model are integrated into a unified graph data structure, where the nodes in the graph data structure represent sampling points and the edges represent the connection relationship between nodes. Adaptively determine the convolution kernel size, shape and weight distribution parameters of the node according to the local geometric features and neighborhood distribution of the node, wherein the local geometric features include the curvature and detail complexity of the node neighborhood, and the neighborhood distribution includes the geometric structure and spatial distribution of the node neighborhood; According to the convolution kernel size, shape and weight distribution parameters of the node, a dynamic graph convolution kernel of the node is constructed, wherein the dynamic graph convolution kernel is used to extract features of the node, and the weight of the dynamic graph convolution kernel is adaptively adjusted through an attention mechanism; Extract node features at multiple scales, apply the corresponding dynamic graph convolution kernel at each scale to obtain node features at different scales, and input the features at different scales into the feature fusion layer; In the feature fusion layer, the aggregation weight of the node is adaptively determined according to the connection strength between the nodes and the importance of the nodes, and the features of the neighboring nodes are weightedly aggregated according to the aggregation weight to obtain the final multimodal three-dimensional representation of the wound.

4. The method according to claim 1, characterized in that For each suture point, the direction of needle tip movement, tissue thickness and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated, and sampling is performed on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties. The suture schemes adapted to the wound morphology and tissue properties are obtained, including: For each suture point, the tissue thickness, hardness and vascular nerve density in its local neighborhood are extracted, and the suture points that do not meet the threshold conditions are eliminated according to the preset threshold, and the number and index of suture points are updated to finally obtain the screened candidate suture point set; Calculate the spatial distance, curvature and tangent direction between adjacent stitching points in the candidate stitching point set; The wound closure effect evaluation function and tissue stress distribution evaluation function were constructed as optimization targets, and the spacing between adjacent suture points, curvature and the number of suture points were used as constraints. The optimal number and position of suture points were obtained through optimization and solution as the final suture point sequence. Based on each pair of adjacent suture points in the final suture point sequence, combined with the local curvature and thickness of the wound, the center line of the needle tip trajectory is fitted in the wound cross section, and the trajectory sampling point sequence is obtained by uniformly sampling along the center line of the needle tip trajectory. The tangent direction and normal vector of the normal plane at each trajectory sampling point are calculated and used as the optimal needle posture at the corresponding trajectory sampling point. According to the tissue hardness, trajectory curvature and total trajectory length at the trajectory sampling point, the needle tip movement speed at each trajectory sampling point is adaptively calculated through a predefined speed adjustment function to form a needle tip trajectory speed distribution curve; Based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve, the optimization problem of needle tip motion planning is constructed, and the optimization problem is solved using an adaptive optimization algorithm. Combined with the obtained optimal needle posture, a suturing plan that is compatible with the wound morphology and tissue properties is finally obtained.

5. The method according to claim 4, characterized in that Constructing wound closure effect evaluation function and tissue stress distribution evaluation function as optimization objectives includes: The calculation formula of the wound closure effect evaluation function is as follows: Among them, E c represents the wound closure effect, ω1 represents the weight coefficient of shape error, N represents the number of suture points, R i represents the local curvature radius of the i-th suture point, p i represents the coordinates of the i-th suture point, r(u i ) represents the parameterized representation of the wound edge curve, u i represents the parameter coordinates of the i-th suture point on the wound edge curve, ω2 represents the weight coefficient of curvature continuity, t i+1 represents the tangent direction of the i+1th suture point, t i represents the tangent direction of the i-th suture point, ||·|| represents the norm of the vector; The calculation formula of the tissue stress distribution evaluation function is as follows: Among them, E s represents tissue stress distribution, ω3 represents the weight coefficient of stress distribution smoothness, A represents the area of ​​wound and surrounding tissue, ▽ represents the gradient operator, σ(a) represents the tissue stress tensor at tissue a, ω4 represents the weight coefficient of peak stress, ||·|| F represents the Frobenius norm; The optimization problems of needle tip motion planning based on the obtained needle tip trajectory center line and needle tip trajectory velocity distribution curve include: According to the distance between the needle tip trajectory sampling point and the wound centerline, the speed and direction of the needle tip trajectory, the objective function of the optimization problem is constructed, and the calculation formula is as follows: Among them, J represents the objective function, q 1:M represents the sampling point sequence of the needle tip trajectory, v 1:M represents the velocity distribution of the needle tip trajectory, θ 1:M represents the direction sequence of the needle tip trajectory, M represents the number of sampling points, q j represents the coordinates of the jth sampling point, c(u j ) represents the center line of the needle tip trajectory fitted in the wound cross section at parameter u j The coordinates at the position, λ1 represents the weight coefficient of the distance, μ t represents the weight coefficient of tangential velocity, v t，j represents the tangential velocity at the jth sampling point, μ n Represents the weight coefficient of normal velocity, v n，j represents the normal velocity at the jth sampling point, △t j represents the time interval between the jth sampling point and the next sampling point, λ2 represents the directional weight coefficient, θ j+1 represents the direction at the j+1th sampling point, θ j represents the direction at the jth sampling point, λ3 represents the weight coefficient of the speed, and v j+1 represents the velocity at the j+1th sampling point, v j represents the velocity at the jth sampling point.

6. The method according to claim 1, characterized in that Based on the obtained suturing scheme, combined with the position, speed, and torque sensor information of the suturing robot joints, the optimal control sequence of the suturing robot is obtained. At the same time, the robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the real-time control system of the suturing robot, including: The mechanical structure parameters of the suturing robot are obtained, and the robot coordinate system including the base coordinate system, joint coordinate system, manipulator coordinate system and tissue surface coordinate system is established. The position and posture of the suturing robot relative to the base coordinate system are solved by coordinate transformation, and the kinematic positive solution relationship between the robot joint angle and the needle tip position and posture is established; the robot dynamics model describing the relationship between the joint torque and the needle tip force is derived by using the Lagrange equation, combined with the robot joint angle, angular velocity and torque; The robot joint angle and angular velocity are used as state variables, the joint torque is used as the control variable, the needle tip posture tracking error and the control input smoothness are used as performance indicators, and the optimization problem of the suture robot motion control is constructed. The state equation, boundary conditions and penalty factors corresponding to the joint constraints are introduced to transform the optimization problem into an unconstrained optimization problem. The sequential convex optimization method is used to divide the unconstrained optimization problem into multiple convex quadratic sub-problems. Through penalty factor adjustment and trust region control, each sub-problem is iteratively solved to obtain the optimal control torque sequence of the suture robot joint. A position sensor is set at the end of the suturing robot to measure the posture deviation of the needle tip relative to the tissue surface in real time. The obtained posture deviation of the needle tip is mapped to the joint space through coordinate transformation. The joint angle compensation for compensating the posture deviation of the needle tip is generated by combining the pre-built time series prediction model and the robot dynamics model. The optimal control torque sequence is used as the feedforward control input, and the joint angle compensation is used as the feedback control input. The composite control torque of the robot joint is generated by weighted summation, and the final composite control torque is input into the real-time control system of the suture robot.

7. The method according to claim 6, characterized in that Combining the pre-built time series prediction model and the robot dynamics model, the steps before generating the joint angle compensation for compensating the needle tip posture deviation also include: Constructing a time series data set of needle tip posture deviation, wherein the time series data set includes historical observation values ​​and corresponding timestamps; Preprocessing the time series data set, and dividing the preprocessed time series data set into a training set, a validation set, and a test set; A multi-scale attention recurrent neural network structure of a needle tip posture deviation prediction model is designed, wherein the network structure includes an input layer, a multi-scale feature extraction layer, an attention fusion layer, a bidirectional recurrent neural network layer, and an output layer; wherein the multi-scale feature extraction layer uses a one-dimensional convolutional neural network to extract local features of the needle tip posture deviation at different time scales with different convolution kernel sizes and step sizes; the attention fusion layer uses a self-attention mechanism to adaptively learn the correlation between different modal features and fuse them into a unified feature representation; the bidirectional recurrent neural network layer uses a bidirectional long short-term memory network to respectively learn the forward and backward dependencies of the needle tip posture deviation in the time dimension and capture its long-term trend characteristics; Construct the loss function of multi-task joint learning, and update the network parameters through the back propagation algorithm, repeat the iteration until the model converges, and obtain the trained needle tip posture deviation prediction model; The trained needle tip posture deviation prediction model is used to predict the needle tip posture deviation sequence within a certain time range in the future, and the predicted posture deviation sequence is converted into motion compensation instructions in the joint space through kinematic inverse solution and trajectory optimization algorithm.

8. An artificial intelligence-based robot-assisted wound suturing system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain multimodal sensing data of the wound area, wherein the multimodal sensing data includes RGB images, force tactile data and ultrasonic imaging data, and map the multimodal sensing data to a unified spatial coordinate system to construct a multimodal three-dimensional representation of the wound. Based on the multimodal three-dimensional representation of the wound, the wound edge is sampled at equal intervals according to the wound shape and suturing requirements to determine multiple suturing points; The second unit is used to calculate candidate suture points for each suture point by comprehensively considering the needle tip movement direction, tissue thickness and blood vessel nerve distribution. At the same time, for adjacent suture points, it generates a needle tip trajectory inside the wound, samples on the needle tip trajectory, calculates the optimal needle posture at each sampling point, and adaptively adjusts the needle tip movement speed according to the curvature of the needle tip trajectory and tissue properties to obtain a suture solution that is compatible with the wound morphology and tissue properties. The third unit is used to obtain the optimal control sequence of the suture robot based on the obtained suturing plan and the position, speed and torque sensor information of the suture robot joints, and to construct a robust compensation control law through coordinate transformation and feedback control law, integrate the optimal control sequence and the robust compensation control law into the real-time control system of the suture robot, and convert the output of the real-time control system of the suture robot into actuator control instructions to control the suture robot to assist in wound suturing.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images