Artificial intelligence-based robot-assisted wound suturing method and system

Abstract

The present invention provides an artificial intelligence-based robot-assisted wound suturing method and system, which relates to the field of artificial intelligence technology. The method comprises the following steps: obtaining multimodal perception data of a wound area, constructing a multimodal three-dimensional representation of the wound, and determining a plurality of suturing points based on the multimodal three-dimensional representation of the wound; calculating candidate suturing points for each suturing point, and generating a needle tip trajectory of the needle tip inside the wound for adjacent suturing points, sampling the needle tip trajectory, and obtaining a suturing scheme adapted to the wound morphology and tissue properties; obtaining an optimal control sequence of a suturing robot based on the obtained suturing scheme, constructing a robust compensation control law through coordinate transformation and feedback control law, integrating the optimal control sequence and the robust compensation control law into a real-time control system of the suturing robot, and converting the output of the real-time control system of the suturing robot into an actuator control instruction to control the suturing robot to assist in wound suturing.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to an artificial intelligence-based robot-assisted wound suturing method and system. Background Art

[0002] Wound suturing is an essential step in surgical procedures, directly impacting postoperative recovery and patient outcomes. Traditional manual suturing relies on the surgeon's experience and skill, resulting in operational difficulty, low efficiency, and inconsistent suturing quality. To improve the precision and repeatability of wound suturing, research has begun on robotic-assisted suturing technology. Currently, several robotic-assisted suturing systems have been put into clinical use, such as the da Vinci surgical robot system. This system uses a master-slave system to map the surgeon's hand movements to the end of the robot, enabling precise remote control. However, these systems still rely on the surgeon's subjective control, and suturing quality is limited by the operator's proficiency. Furthermore, perception, decision-making, and planning control in complex wound environments place higher demands on the operator.

[0003] Therefore, there is an urgent need to develop an intelligent robot-assisted wound suturing system that integrates perception, decision-making, and control. The system should be able to accurately perceive wound characteristics and suturing status, autonomously plan and optimize needle tip motion, and precisely control the suturing process to improve suturing efficiency and quality. Summary of the Invention

[0004] The embodiments of the present invention provide a robot-assisted wound suturing method and system based on artificial intelligence, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provided is an artificial intelligence-based robot-assisted wound suturing method, comprising:

[0007] Acquiring multimodal sensory data of the wound area, the multimodal sensory data including RGB images, force tactile data, and ultrasonic imaging data, mapping the multimodal sensory data into a unified spatial coordinate system, constructing a multimodal three-dimensional representation of the wound, and determining multiple suturing points by sampling the wound edge at equal intervals based on the wound shape and suturing requirements based on the multimodal three-dimensional representation of the wound;

[0008] For each suture point, the needle tip movement direction, tissue thickness, and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated and samples are taken on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties to obtain a suture plan that is adapted to the wound morphology and tissue properties.

[0009] Based on the obtained suturing scheme and combined with the position, speed, and torque sensor information of the suturing robot's joints, the optimal control sequence of the suturing robot is obtained. At the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the suturing robot's real-time control system, and the output of the suturing robot's real-time control system is converted into actuator control instructions to control the suturing robot to assist in wound suturing.

[0010] In an optional embodiment,

[0011] Acquiring multimodal sensory data of the wound area, the multimodal sensory data including RGB images, force tactile data, and ultrasonic imaging data, mapping the multimodal sensory data into a unified spatial coordinate system, and constructing a multimodal three-dimensional representation of the wound includes:

[0012] A convolutional neural network is used to extract pixel-level disparity maps from the left and right views of the RGB image. At the same time, an attention mechanism and multi-scale feature fusion are introduced to convert the pixel-level disparity map into 3D point cloud data of the wound surface.

[0013] The surface reconstruction method based on implicit function is used to convert the obtained 3D point cloud data into a signed distance field. The implicit neural representation network is used to learn the mapping from the coordinates of the 3D point cloud data to the signed distance field, and the wound implicit surface is obtained. By extracting the zero-isovalue surface of the wound implicit surface, the wound triangulation network model is obtained.

[0014] Based on the gradient changes of force-tactile data and ultrasound imaging data, the voxel resolution is adaptively adjusted, and a sparse voxel representation is achieved through an octree hierarchy. Voxel interpolation and nearest neighbor search algorithms are used to append the force-tactile data and ultrasound imaging data to the vertices of the pre-built visual model to obtain the final wound visual model.

[0015] A physics-based wound tissue modeling technique was introduced. The wound area was divided into three-dimensional tetrahedral meshes. The parameters of the three-dimensional tetrahedral meshes were initialized using histological characteristics extracted from ultrasound images. The deformation and stress distribution of the wound were calculated using the finite element method. Based on the comparison results of the stress distribution and force tactile data, the parameters of the three-dimensional tetrahedral meshes were iteratively updated to obtain the final wound biomechanical model.

[0016] The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound.

[0017] In an optional embodiment,

[0018] The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound including:

[0019] The wound triangulation network model, wound visual model and wound biomechanics model are integrated into a unified graph data structure, where nodes represent sampling points and edges represent the connection relationship between nodes.

[0020] Adaptively determine the convolution kernel size, shape, and weight distribution parameters of the node based on the node's local geometric features and neighborhood distribution, where the local geometric features include the curvature and detail complexity of the node's neighborhood, and the neighborhood distribution includes the geometric structure and spatial distribution of the node's neighborhood;

[0021] According to the convolution kernel size, shape and weight distribution parameters of the node, a dynamic graph convolution kernel of the node is constructed, the dynamic graph convolution kernel is used to extract features of the node, and the weight of the dynamic graph convolution kernel is adaptively adjusted through the attention mechanism;

[0022] Extract node features at multiple scales, apply the corresponding dynamic graph convolution kernel at each scale to obtain node features at different scales, and input the features at different scales into the feature fusion layer;

[0023] In the feature fusion layer, the aggregation weight of the node is adaptively determined according to the connection strength between the nodes and the importance of the nodes, and the features of the neighboring nodes are weightedly aggregated according to the aggregation weight to obtain the final multimodal three-dimensional representation of the wound.

[0024] In an optional embodiment,

[0025] For each suture point, the needle tip movement direction, tissue thickness, and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated and samples are taken on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties. The suture scheme adapted to the wound morphology and tissue properties is obtained, including:

[0026] For each suture point, the tissue thickness, hardness, and vascular nerve density in its local neighborhood are extracted. Suture points that do not meet the threshold conditions are eliminated according to the preset threshold, and the number and index of suture points are updated to finally obtain the filtered candidate suture point set;

[0027] Calculate the spatial distance, curvature and tangent direction between adjacent stitching points in the candidate stitching point set;

[0028] The wound closure effect evaluation function and tissue stress distribution evaluation function were constructed as optimization objectives. The spacing between adjacent suture points, curvature, and the number of suture points were used as constraints. The optimal number and position of suture points were obtained through optimization and solved as the final suture point sequence.

[0029] Based on each pair of adjacent suture points in the final suture point sequence, combined with the local curvature and thickness of the wound, the centerline of the needle tip trajectory is fitted within the wound cross section. The trajectory sampling point sequence is obtained by uniform sampling along the centerline of the needle tip trajectory. The tangent direction and normal vector at each trajectory sampling point are calculated and used as the optimal needle posture at the corresponding trajectory sampling point.

[0030] According to the tissue hardness, trajectory curvature and total trajectory length at the trajectory sampling point, the needle tip movement speed at each trajectory sampling point is adaptively calculated through a predefined speed adjustment function to form a needle tip trajectory speed distribution curve;

[0031] Based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve, an optimization problem of needle tip motion planning is constructed, and the optimization problem is solved using an adaptive optimization algorithm. Combined with the obtained optimal needle posture, a suturing scheme that is adapted to the wound morphology and tissue properties is finally obtained.

[0032] In an optional embodiment,

[0033] Constructing wound closure effect evaluation function and tissue stress distribution evaluation function as optimization objectives includes:

[0034] The calculation formula of the wound closure effect evaluation function is as follows:

[0035]

[0036] Among them, E c represents the wound closure effect, ω1 represents the weight coefficient of shape error, N represents the number of suture points, R i represents the local curvature radius of the i-th suture point, p i represents the coordinates of the i-th suture point, r(u i ) represents the parameterized representation of the wound edge curve, u i represents the parameter coordinates of the i-th suture point on the wound edge curve, ω2 represents the weight coefficient of curvature continuity, t i+1 Indicates the tangent direction of the i+1th suture point, t i represents the tangent direction of the i-th suture point, ||·|| represents the norm of the vector;

[0037] The calculation formula of the tissue stress distribution evaluation function is as follows:

[0038]

[0039] Among them, E s represents the tissue stress distribution, ω3 represents the weight coefficient of the smoothness of the stress distribution, A represents the area of ​​the wound and surrounding tissues, ▽ represents the gradient operator, σ(a) represents the tissue stress tensor at tissue a, ω4 represents the weight coefficient of the peak stress, ||·|| F represents the Frobenius norm;

[0040] The optimization problems of needle tip motion planning based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve include:

[0041] According to the distance between the needle tip trajectory sampling point and the wound centerline, the speed and direction of the needle tip trajectory, the objective function of the optimization problem is constructed, and the calculation formula is as follows:

[0042]

[0043] Among them, J represents the objective function, q 1:M represents the sampling point sequence of the needle tip trajectory, v 1:M represents the velocity distribution of the needle tip trajectory, θ 1:M represents the direction sequence of the needle tip trajectory, M represents the number of sampling points, q j represents the coordinates of the jth sampling point, c(u j ) represents the center line of the needle tip trajectory fitted in the wound cross section at parameter u j The coordinates at the position, λ1 represents the weight coefficient of the distance, μ t Represents the weight coefficient of tangential velocity, v t，j represents the tangential velocity at the jth sampling point, μ n Represents the weight coefficient of normal velocity, v n，j represents the normal velocity at the jth sampling point, △t j represents the time interval between the jth sampling point and the next sampling point, λ2 represents the directional weight coefficient, θ j+1 Indicates the direction of the j+1th sampling point, θ j represents the direction at the jth sampling point, λ3 represents the weight coefficient of the speed, v j+1 represents the velocity at the j+1th sampling point, v j represents the velocity at the jth sampling point.

[0044] In an optional embodiment,

[0045] Based on the obtained suturing scheme and combined with the position, velocity, and torque sensor information of the suturing robot's joints, the optimal control sequence of the suturing robot is obtained. At the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the real-time control system of the suturing robot, including:

[0046] The mechanical structural parameters of the suturing robot were obtained, and the robot coordinate system, including the base coordinate system, joint coordinate system, manipulator coordinate system, and tissue surface coordinate system, was established. The position of the suturing robot relative to the base coordinate system was solved through coordinate transformation, and the kinematic positive relationship between the robot joint angle and the needle tip position was established. The robot dynamics model describing the relationship between the joint torque and the force on the needle tip was derived by combining the Lagrange equations with the robot joint angle, angular velocity, and torque.

[0047] The robot's joint angles and angular velocities are used as state variables, joint torques are used as control variables, and needle tip posture tracking error and control input smoothness are used as performance indicators. The optimization problem of suture robot motion control is constructed. The state equations, boundary conditions, and penalty factors corresponding to joint constraints are introduced to transform the optimization problem into an unconstrained optimization problem. A sequential convex optimization method is used to divide the unconstrained optimization problem into multiple convex quadratic subproblems. Through penalty factor adjustment and trust region control, the subproblems are iteratively solved to obtain the optimal control torque sequence of the suture robot's joints.

[0048] A position sensor is installed at the end of the suturing robot to measure the position deviation of the needle tip relative to the tissue surface in real time. The obtained needle tip position deviation is mapped to the joint space through coordinate transformation. Combined with the pre-built time series prediction model and the robot dynamics model, the joint angle compensation value is generated to compensate for the needle tip position deviation.

[0049] The optimal control torque sequence is used as the feedforward control input, and the joint angle compensation is used as the feedback control input. The composite control torque of the robot joint is generated by weighted summation, and the final composite control torque is input into the real-time control system of the suture robot.

[0050] In an optional embodiment,

[0051] Combining the pre-built time series prediction model and the robot dynamics model, the steps before generating the joint angle compensation to compensate for the needle tip posture deviation also include:

[0052] Constructing a time series dataset of needle tip posture deviation, wherein the time series dataset includes historical observations and corresponding timestamps;

[0053] Preprocessing the time series data set, and dividing the preprocessed time series data set into a training set, a validation set, and a test set;

[0054] A multi-scale attention recurrent neural network structure for the needle tip posture deviation prediction model is designed. The network structure includes an input layer, a multi-scale feature extraction layer, an attention fusion layer, a bidirectional recurrent neural network layer, and an output layer. The multi-scale feature extraction layer uses a one-dimensional convolutional neural network to extract local features of the needle tip posture deviation at different time scales with different convolution kernel sizes and step sizes. The attention fusion layer uses a self-attention mechanism to adaptively learn the correlation between different modal features and fuse them into a unified feature representation. The bidirectional recurrent neural network layer uses a bidirectional long short-term memory network to learn the forward and backward dependencies of the needle tip posture deviation in the time dimension and capture its long-term trend characteristics.

[0055] Construct a loss function for multi-task joint learning, and update the network parameters through the back-propagation algorithm. Repeat the iteration until the model converges to obtain a trained needle tip posture deviation prediction model.

[0056] The trained needle tip posture deviation prediction model is used to predict the needle tip posture deviation sequence within a certain time range in the future, and the predicted posture deviation sequence is converted into motion compensation instructions in the joint space through kinematic inverse solution and trajectory optimization algorithm.

[0057] According to a second aspect of the embodiments of the present invention,

[0058] Provided is an artificial intelligence-based robot-assisted wound suturing system, comprising:

[0059] The first unit is configured to acquire multimodal sensory data of the wound area, the multimodal sensory data including RGB images, force tactile data, and ultrasonic imaging data, map the multimodal sensory data into a unified spatial coordinate system, construct a multimodal three-dimensional representation of the wound, and perform equally spaced sampling at the wound edge based on the wound shape and suturing requirements to determine multiple suturing points.

[0060] The second unit is used to calculate candidate suture points for each suture point by comprehensively considering the needle tip movement direction, tissue thickness, and vascular nerve distribution. At the same time, for adjacent suture points, it generates a needle tip trajectory inside the wound, samples the needle tip trajectory, calculates the optimal needle posture at each sampling point, and adaptively adjusts the needle tip movement speed according to the curvature of the needle tip trajectory and tissue properties to obtain a suture plan that is adapted to the wound morphology and tissue properties;

[0061] The third unit is used to obtain the optimal control sequence of the suturing robot based on the obtained suturing plan and combined with the position, speed, and torque sensor information of the suturing robot joints, and at the same time construct a robust compensation control law through coordinate transformation and feedback control law, integrate the optimal control sequence and robust compensation control law into the suturing robot real-time control system, and convert the output of the suturing robot real-time control system into actuator control instructions to control the suturing robot to assist in wound suturing.

[0062] According to a third aspect of the embodiments of the present invention,

[0063] An electronic device is provided, comprising:

[0064] processor;

[0065] a memory for storing processor-executable instructions;

[0066] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0067] According to a fourth aspect of the embodiments of the present invention,

[0068] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0069] In this embodiment, a multimodal three-dimensional representation of the wound is constructed by acquiring RGB images, force and tactile data, and ultrasound imaging data and mapping them into a unified spatial coordinate system. This three-dimensional representation not only incorporates the wound's geometric shape but also incorporates multimodal information such as mechanical properties and tissue structure, providing comprehensive wound information support for subsequent suture planning. Based on this three-dimensional wound representation, the system automatically plans optimal suture point locations, needle tip trajectory, and needle posture based on factors such as wound shape, tissue thickness, and vascular and neural distribution. It also dynamically adjusts needle tip velocity based on trajectory curvature and tissue properties, generating a comprehensive suture plan tailored to the wound's morphology and tissue properties. This intelligent planning significantly improves suture quality and efficiency. The system maps the planned optimal suture plan into a control sequence in the robot's joint space and, through feedback control laws, establishes robust compensation, enabling precise real-time control of the suture robot. This planning-first, real-time control approach ensures accurate and robust suture motion. This fully automated process, from wound perception to suture planning to robot control, significantly reduces the surgeon's surgical workload. At the same time, through multimodal information fusion and intelligent planning, the suturing quality and efficiency are also significantly improved, which is conducive to improving the level of clinical surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1Schematic diagram of the process of the robot-assisted wound suturing method based on artificial intelligence according to an embodiment of the present invention;

[0071] Figure 2 Schematic diagram of the structure of an artificial intelligence-based robot-assisted wound suturing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0073] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0074] Figure 1 FIG is a flow chart of a robot-assisted wound suturing method based on artificial intelligence according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0075] S101. Acquire multimodal perception data of the wound area, wherein the multimodal perception data includes RGB images, force tactile data, and ultrasonic imaging data; map the multimodal perception data into a unified spatial coordinate system; construct a multimodal three-dimensional representation of the wound; based on the multimodal three-dimensional representation of the wound, perform equally spaced sampling on the wound edge according to the wound shape and suturing requirements, and determine multiple suturing points.

[0076] Among them, a high-resolution industrial camera equipped with a fixed-focus lens is used to realize the acquisition of RGB images and depth images of the wound area. The camera should have a sufficient field of view to cover the entire wound area. A miniature six-axis force sensor is selected to measure the three-dimensional contact force and torque between the needle tip and the tissue. The sensor should have high sensitivity and low noise characteristics to meet the needs of small force measurement. At the same time, a tactile sensor array is arranged around the needle tip to collect texture and slip information on the tissue surface. Electrical impedance tactile sensing technology is introduced to measure the change in electrical conductivity between the needle tip and the tissue to evaluate the insertion depth and contact status of the needle tip in real time. A miniaturized and flexible electrode array is designed to ensure a close fit between the electrode and the needle tip.

[0077] Data collected by multimodal sensors is heterogeneous in time and space, necessitating spatiotemporal synchronization and coordinate system unification. Specifically, a timestamp is introduced into each sensor data to record the absolute time of sampling. Synchronization of the clocks of different sensors is achieved through methods such as the Network Time Protocol (NTP). Through hand-eye calibration and external parameter calibration, the spatial transformation relationship between the visual sensor, force and tactile sensor, and the robot's base coordinate system is established. High-precision calibration data is acquired using a checkerboard calibration plate and precision measurement tools. Timestamps are aligned to unify the data from different sensors to the same time base. Using spatial coordinate transformation, force and tactile and electrical impedance data are mapped to the visual coordinate system to achieve spatial data registration.

[0078] In an optional embodiment,

[0079] Acquiring multimodal sensory data of the wound area, the multimodal sensory data including RGB images, force tactile data, and ultrasonic imaging data, mapping the multimodal sensory data into a unified spatial coordinate system, and constructing a multimodal three-dimensional representation of the wound includes:

[0080] A convolutional neural network is used to extract pixel-level disparity maps from the left and right views of the RGB image. At the same time, an attention mechanism and multi-scale feature fusion are introduced to convert the pixel-level disparity map into 3D point cloud data of the wound surface.

[0081] The surface reconstruction method based on implicit function is used to convert the obtained 3D point cloud data into a signed distance field. The implicit neural representation network is used to learn the mapping from the coordinates of the 3D point cloud data to the signed distance field, and the wound implicit surface is obtained. By extracting the zero-isovalue surface of the wound implicit surface, the wound triangulation network model is obtained.

[0082] Based on the gradient changes of force-tactile data and ultrasound imaging data, the voxel resolution is adaptively adjusted, and a sparse voxel representation is achieved through an octree hierarchy. Voxel interpolation and nearest neighbor search algorithms are used to append the force-tactile data and ultrasound imaging data to the vertices of the pre-built visual model to obtain the final wound visual model.

[0083] A physics-based wound tissue modeling technique was introduced. The wound area was divided into three-dimensional tetrahedral meshes. The parameters of the three-dimensional tetrahedral meshes were initialized using histological characteristics extracted from ultrasound images. The deformation and stress distribution of the wound were calculated using the finite element method. Based on the comparison results of the stress distribution and force tactile data, the parameters of the three-dimensional tetrahedral meshes were iteratively updated to obtain the final wound biomechanical model.

[0084] The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound.

[0085] For example, a binocular camera is used to capture a pair of RGB images of the wound area. Stereo matching is performed using deep learning methods to estimate pixel-level disparity information. An end-to-end convolutional neural network is used to directly regress the disparity map from the left and right views. An attention mechanism and multi-scale feature fusion strategy are introduced to improve the accuracy and robustness of disparity estimation. Once the disparity map is obtained, it is converted into dense 3D point cloud data of the wound surface based on the binocular camera parameters.

[0086] Then, to address the irregularities and complex topology of the wound surface, an implicit function-based surface reconstruction method was employed to transform the scattered point cloud into a continuous, smooth implicit surface. By mapping the point cloud coordinates to a signed distance field and learning this mapping using an implicit neural representation network, an implicit function representing the wound shape was derived. Based on this, a topologically correct and geometrically accurate triangulated mesh model of the wound was generated by extracting the zero-isosurface of the implicit surface.

[0087] Next, an adaptive octree voxelization method is proposed to align and fuse force tactile and ultrasound imaging data with the visual model. Based on the local gradient changes in the force tactile and ultrasound data, the voxel resolution is dynamically adjusted, subdividing voxels in areas with large gradients and merging voxels in areas with small gradients. This achieves a sparse and efficient voxel representation through the octree hierarchy. Using voxel interpolation and a nearest neighbor search algorithm, the force tactile and ultrasound information is appended to the vertices of the pre-built visual model, resulting in a visual model of the wound that integrates the multimodal data.

[0088] To further understand the physical properties and internal structure of wound tissue, a physics-based biomechanical modeling technique was introduced. The wound area was divided into three-dimensional tetrahedral meshes, and the material parameters of the tetrahedral elements were initialized using tissue acoustic properties extracted from ultrasound images. The finite element method was used to determine the deformation and stress distribution of the wound under external forces. By comparing the deformation predicted by the finite element method with the actual deformation measured by force tactile sensors, the material parameters were iteratively optimized and updated, ultimately resulting in a biomechanical model that accurately reflects the physical properties of the wound tissue.

[0089] Finally, the wound triangulated mesh model, visual model, and biomechanical model are integrated into a unified data structure. Graph convolution allows for the propagation and fusion of different modal features across the mesh surface and voxel space while preserving the spatial structure of the model. This results in a multimodal 3D representation of the wound that incorporates geometric, textural, mechanical, and anatomical information. This representation can be used to guide subsequent wound repair and rehabilitation, such as suture planning, healing monitoring, and surgical training.

[0090] In this embodiment, a convolutional neural network is used to extract pixel-level disparity maps, and combined with an attention mechanism and multi-scale feature fusion, it is possible to accurately reconstruct three-dimensional point cloud data of the wound from the RGB binocular view. The point cloud is then converted into a triangular mesh model of the wound using an implicit function-based surface reconstruction method, significantly improving both reconstruction accuracy and quality. Multimodal fusion not only enhances the information integrity of the three-dimensional wound representation but also provides more comprehensive data support for subsequent suture planning. Physics-based wound tissue modeling technology is introduced to obtain a biomechanical model of the wound. This biomechanical model has important guiding significance for tissue behavior during the suturing process. The three-dimensional wound representation based on multimodal fusion can provide rich information input for intelligent suture planning, making the planning more personalized and optimized, and improving suture quality and efficiency. At the same time, high-precision three-dimensional wound reconstruction and modeling also lay the foundation for the automated control of the suture robot.

[0091] In an optional embodiment,

[0092] The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound including:

[0093] The wound triangulation network model, wound visual model and wound biomechanics model are integrated into a unified graph data structure, where nodes represent sampling points and edges represent the connection relationship between nodes.

[0094] Adaptively determine the convolution kernel size, shape, and weight distribution parameters of the node based on the node's local geometric features and neighborhood distribution, where the local geometric features include the curvature and detail complexity of the node's neighborhood, and the neighborhood distribution includes the geometric structure and spatial distribution of the node's neighborhood;

[0095] According to the convolution kernel size, shape and weight distribution parameters of the node, a dynamic graph convolution kernel of the node is constructed, the dynamic graph convolution kernel is used to extract features of the node, and the weight of the dynamic graph convolution kernel is adaptively adjusted through the attention mechanism;

[0096] Extract node features at multiple scales, apply the corresponding dynamic graph convolution kernel at each scale to obtain node features at different scales, and input the features at different scales into the feature fusion layer;

[0097] In the feature fusion layer, the aggregation weight of the node is adaptively determined according to the connection strength between the nodes and the importance of the nodes, and the features of the neighboring nodes are weightedly aggregated according to the aggregation weight to obtain the final multimodal three-dimensional representation of the wound.

[0098] For example, wound model data from different modalities is first converted into a graph data structure, where nodes represent sampling points on the wound surface, and edges represent the topological connections between nodes. This approach maps heterogeneous data from different modalities into a unified graph domain representation, facilitating subsequent feature extraction and fusion processing.

[0099] After constructing a wound graph data structure, a method for designing adaptive dynamic graph convolution kernels was proposed to extract multi-scale, multi-modal wound features from this graph data. The local geometric features of a node, including the curvature and detail complexity of its neighborhood, reflect the local shape characteristics of the wound surface. The neighborhood distribution of a node, including its geometric structure and spatial distribution, reflects the local topological characteristics of the wound surface. By comprehensively considering these features, optimal convolution kernel parameters can be designed for each node, improving the adaptability and accuracy of feature extraction.

[0100] After determining the convolution kernel parameters for each node, a dynamic graph convolution approach is used to construct a corresponding dynamic graph convolution kernel based on the kernel size, shape, and weight distribution parameters for each node. Unlike traditional convolution kernels that use fixed weights, the weights of the dynamic graph convolution kernel are adaptively adjusted based on the node's features, better adapting to local variations in the wound surface. During feature extraction, the dynamic graph convolution kernel performs convolution operations within the node's neighborhood to extract the node's local features. An attention mechanism is also introduced to adaptively adjust the weights of the dynamic graph convolution kernel based on the importance of the node's features, further improving the accuracy and robustness of feature extraction.

[0101] Considering that the shape and texture features of wound surfaces vary at different scales, a multi-scale feature extraction strategy was adopted. Specifically, within a graph data structure, neighborhoods of varying sizes are selected, centered around a node, as receptive fields at different scales. At each scale, the corresponding dynamic graph convolution kernel is used to extract features from the node's neighborhood, yielding a feature representation for the node at that scale. By extracting node features at multiple scales, we can obtain shape and texture information of the wound surface at different levels and granularities, improving the richness and completeness of the feature representation.

[0102] After obtaining node features at different scales, a feature fusion layer was designed to fuse node features from different scales and modalities to obtain the final three-dimensional representation of the wound. In this feature fusion layer, the aggregation weight of each node is adaptively determined based on the connection strength and importance of the nodes. The connection strength between nodes reflects the topological relationship of the nodes in the graph structure and can be measured by edge weights or node similarity. The importance of a node reflects its contribution to the overall wound representation and can be measured using graph-theoretic metrics such as node centrality and clustering coefficient. By comprehensively considering these factors, an optimal aggregation weight is assigned to each node, which is used to perform weighted aggregation of the features of neighboring nodes.

[0103] During feature aggregation, an adaptive weight adjustment strategy based on the attention mechanism is employed. Specifically, for each node, an attention weight matrix is ​​calculated based on the feature correlations between it and its neighboring nodes. This matrix reflects the importance of neighboring node features to the current node's features and can be used to adjust aggregation weights. By introducing the attention mechanism, important neighboring node features can be adaptively highlighted, noise and redundant information can be suppressed, and the accuracy and robustness of feature aggregation can be improved.

[0104] Finally, a feature aggregation layer performs a weighted fusion of node features from different scales and modalities, resulting in a multimodal 3D representation of the wound. This representation incorporates both the geometric shape characteristics of the wound surface as well as its visual texture and biomechanical properties, providing a comprehensive and compact 3D representation of the wound. This representation can be used to perform tasks such as wound segmentation, suture planning, and surgical navigation, providing critical wound information for intelligent assisted surgery systems.

[0105] In this embodiment, by mapping the multimodal wound data into a unified graph data structure, using adaptive dynamic convolution kernels and an attention mechanism to extract multi-scale, multimodal wound features, and obtaining the final three-dimensional representation of the wound through a feature fusion layer, the feature extraction and fusion strategies can be adaptively adjusted according to the local shape and topological structure of the wound surface, thereby improving the accuracy and robustness of representation learning. Furthermore, the use of a graph data structure and graph convolutional neural network can effectively process the non-Euclidean and unstructured properties of wound data, overcoming the limitations of traditional grid- and voxel-based representation methods. This provides new ideas and technical means for three-dimensional wound reconstruction and intelligent assisted surgery, and has important theoretical significance and application value.

[0106] S102. For each suture point, the needle tip movement direction, tissue thickness, and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, a needle tip trajectory inside the wound is generated, and sampling is performed on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties to obtain a suture solution that is adapted to the wound morphology and tissue properties.

[0107] For example, the wound edge curve is first reparameterized. The sampling interval is adaptively adjusted based on the wound size and suture density requirements, and several sampling points are equidistantly spaced along the wound edge as candidate suture points. Tissue thickness, stiffness, and vascular and neural distribution at the candidate points are then evaluated, eliminating points unsuitable for suturing. The spatial distance, curvature, and tangent direction of adjacent suture points are then calculated for subsequent needle tip trajectory generation. Finally, the number and location of suture points are optimized, taking into account wound closure and tissue stress distribution, to obtain the final suture point sequence. With the suture points as endpoints, a smooth curve is fitted within the wound cross-section based on the wound curvature and thickness distribution, serving as the centerline of the needle tip trajectory. To maintain an appropriate contact angle between the needle tip and the tissue, uniform sampling is performed along the trajectory centerline. The tangent direction and normal plane at each sampling point are calculated and used as a reference for the needle tip's posture. The needle tip's velocity is adaptively adjusted at each point along the trajectory, taking into account variations in the needle tip's trajectory curvature and differences in tissue properties. Velocity adjustment is based on factors such as ensuring stable contact between the needle tip and the tissue, minimizing tip slippage, and preventing tearing caused by excessive velocity variations. Through iterative optimization, the needle tip motion trajectory and its velocity distribution that meet the requirements of suturing accuracy and safety are obtained.

[0108] In an optional embodiment,

[0109] For each suture point, the needle tip movement direction, tissue thickness, and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated and samples are taken on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties. The suture scheme adapted to the wound morphology and tissue properties is obtained, including:

[0110] For each suture point, the tissue thickness, hardness, and vascular nerve density in its local neighborhood are extracted. Suture points that do not meet the threshold conditions are eliminated according to the preset threshold, and the number and index of suture points are updated to finally obtain the filtered candidate suture point set;

[0111] Calculate the spatial distance, curvature and tangent direction between adjacent stitching points in the candidate stitching point set;

[0112] The wound closure effect evaluation function and tissue stress distribution evaluation function were constructed as optimization objectives. The spacing between adjacent suture points, curvature, and the number of suture points were used as constraints. The optimal number and position of suture points were obtained through optimization and solved as the final suture point sequence.

[0113] Based on each pair of adjacent suture points in the final suture point sequence, combined with the local curvature and thickness of the wound, the centerline of the needle tip trajectory is fitted within the wound cross section. The trajectory sampling point sequence is obtained by uniform sampling along the centerline of the needle tip trajectory. The tangent direction and normal vector at each trajectory sampling point are calculated and used as the optimal needle posture at the corresponding trajectory sampling point.

[0114] According to the tissue hardness, trajectory curvature and total trajectory length at the trajectory sampling point, the needle tip movement speed at each trajectory sampling point is adaptively calculated through a predefined speed adjustment function to form a needle tip trajectory speed distribution curve;

[0115] Based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve, an optimization problem of needle tip motion planning is constructed, and the optimization problem is solved using an adaptive optimization algorithm. Combined with the obtained optimal needle posture, a suturing scheme that is adapted to the wound morphology and tissue properties is finally obtained.

[0116] For example, for each suture point in the candidate suture point set, biomechanical and anatomical features such as tissue thickness, hardness, and vascular and nerve density are extracted from its local neighborhood. The extracted feature values ​​are compared with preset thresholds for tissue thickness, tissue hardness, and vascular and nerve density to determine whether the suture point meets the suturing requirements. If any feature fails to meet the corresponding threshold, the suture point is marked as unsuitable for suturing and removed from the candidate suture point set. After the suitability assessment of all candidate suture points is completed, the number of remaining suitable suture points is updated and renumbered, resulting in a final set of filtered candidate suture points.

[0117] Next, the spatial distance between each pair of adjacent suture points in the selected candidate suture point set, the curvature of the connecting line segment, and the tangent direction of the wound edge curve at each suture point are calculated. The spatial distance between adjacent suture points reflects the local suture density, while the curvature reflects the smoothness of the wound edge after suture. The tangent direction at the suture point will be used as a posture reference for subsequent needle tip trajectory planning.

[0118] To balance suture effectiveness and tissue damage risk when selecting suture points, a wound closure evaluation function and a tissue stress distribution evaluation function were constructed as cost functions for a multi-objective optimization approach. The wound closure evaluation function uses the weighted sum of the shape error and overlap error of the wound edge after suture as quantitative indicators to measure the aesthetic appearance and wound closure quality of the suture scheme. The tissue stress distribution evaluation function considers the uniformity of the local tissue stress distribution and the magnitude of the peak stress after suture to assess the tissue damage risk of the suture scheme. Based on this, a multi-objective optimization algorithm was used to solve the problem, using the spacing between adjacent suture points, the curvature of the connecting line segments, and the total number of suture points as constraints. The optimal number and spatial location of suture points that balance suture quality and tissue protection were obtained as the final suture point sequence.

[0119] For each pair of adjacent suture points in the final suture point sequence, a cubic B-spline curve is fitted within the wound cross-section, taking into account the local wound curvature radius and tissue thickness. This curve serves as the centerline of the needle tip trajectory from one suture point to another. To ensure that the needle tip trajectory maintains a safe distance from the wound inner wall and adapts to tissue thickness, the suture points are set as trajectory endpoints. Under the constraint of tangent continuity at the trajectory endpoints, the positions of intermediate control points are adaptively adjusted to be symmetrical about the line connecting the suture points, with the minimum distance from the wound inner surface proportional to the local tissue thickness. After fitting the needle tip trajectory centerline, uniform sampling is performed along it to generate a sequence of trajectory sampling points. For each trajectory sampling point, the tangent direction of the centerline at that point and the normal vector of the local normal plane are calculated. These are used as reference values ​​for the needle tip posture at the corresponding sampling point to ensure that the needle tip trajectory conforms to the wound morphology.

[0120] To adapt the needle tip velocity to the local tissue properties of the wound, a predefined velocity adjustment function is used to adaptively calculate the optimal velocity for each sampling point based on the tissue hardness, local trajectory curvature, and the entire trajectory length. This results in a velocity distribution curve covering the entire needle tip trajectory. The greater the tissue hardness, the slower the needle tip velocity should be; the greater the trajectory curvature, the lower the needle tip velocity should be; and the longer the total trajectory length, the more appropriate the average velocity should be to control the motion time. By establishing a quantitative relationship between these factors and the needle tip velocity through the velocity adjustment function, adaptive matching of the needle tip velocity to the wound properties can be achieved.

[0121] Finally, an optimization model for the needle tip motion planning problem is established, using the spatial position, attitude angle, and velocity of the needle tip trajectory sampling points as optimization variables, and the weighted sum of the distance error between the entire needle tip trajectory and the inner surface of the wound, the change in the needle tip attitude between adjacent sampling points, and the change in velocity as optimization objectives. Furthermore, constraints such as the needle tip's starting and ending states, velocity range, and total motion time are considered as optimization conditions. Based on this, adaptive optimization algorithms such as sequential quadratic programming and gradient descent are used to solve the established optimization problem. After multiple iterations and convergence, the optimal needle tip trajectory and velocity distribution that are compatible with the local wound morphology and tissue properties are obtained. By repeating this needle tip motion planning process for each pair of adjacent suture points in the final suture point sequence, a complete suture plan is ultimately obtained.

[0122] In an optional embodiment,

[0123] Constructing wound closure effect evaluation function and tissue stress distribution evaluation function as optimization objectives includes:

[0124] The calculation formula of the wound closure effect evaluation function is as follows:

[0125]

[0126] Among them, E c represents the wound closure effect, ω1 represents the weight coefficient of shape error, N represents the number of suture points, R i represents the local curvature radius of the i-th suture point, p i represents the coordinates of the i-th suture point, r(u i ) represents the parameterized representation of the wound edge curve, u i represents the parameter coordinates of the i-th suture point on the wound edge curve, ω2 represents the weight coefficient of curvature continuity, t i+1 Indicates the tangent direction of the i+1th suture point, t i represents the tangent direction of the i-th suture point, ||·|| represents the norm of the vector;

[0127] The calculation formula of the tissue stress distribution evaluation function is as follows:

[0128]

[0129] Among them, E s represents the tissue stress distribution, ω3 represents the weight coefficient of the smoothness of the stress distribution, A represents the area of ​​the wound and surrounding tissues, ▽ represents the gradient operator, σ(a) represents the tissue stress tensor at tissue a, ω4 represents the weight coefficient of the peak stress, ||·|| F represents the Frobenius norm;

[0130] The optimization problems of needle tip motion planning based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve include:

[0131] According to the distance between the needle tip trajectory sampling point and the wound centerline, the speed and direction of the needle tip trajectory, the objective function of the optimization problem is constructed, and the calculation formula is as follows:

[0132]

[0133] Among them, J represents the objective function, q 1:M represents the sampling point sequence of the needle tip trajectory, v 1:M represents the velocity distribution of the needle tip trajectory, θ 1:M represents the direction sequence of the needle tip trajectory, M represents the number of sampling points, q j represents the coordinates of the jth sampling point, c(u j ) represents the center line of the needle tip trajectory fitted in the wound cross section at parameter u j The coordinates at the position, λ1 represents the weight coefficient of the distance, μ t Represents the weight coefficient of tangential velocity, v t，j represents the tangential velocity at the jth sampling point, μ n Represents the weight coefficient of normal velocity, v n，j represents the normal velocity at the jth sampling point, △t j represents the time interval between the jth sampling point and the next sampling point, λ2 represents the directional weight coefficient, θ j+1 Indicates the direction of the j+1th sampling point, θ j represents the direction at the jth sampling point, λ3 represents the weight coefficient of the speed, v j+1 represents the velocity at the j+1th sampling point, v j represents the velocity at the jth sampling point.

[0134] In this embodiment, closure effect and tissue stress evaluation functions are used as optimization objectives to determine the optimal number and location of suture points within constraints, enabling the suture plan to be tailored to the individual characteristics of the wound. A trajectory planning method based on local geometric features can effectively prevent the needle tip from colliding with the wound edge, improving the smoothness and safety of the suturing motion. By combining multiple factors, such as tissue hardness, curvature, and total trajectory length at the trajectory sampling point, the optimal needle tip velocity at each sampling point can be adaptively calculated to construct a velocity distribution curve for the needle tip trajectory. Adaptive velocity adjustment dynamically optimizes suturing parameters based on tissue properties, improving suture quality and minimizing tissue damage. Through a variety of innovative technical approaches, including personalized suture point selection, rational needle tip trajectory planning, adaptive needle tip motion control, and overall optimization, a suture plan can be generated that is highly compatible with specific wound morphology and tissue properties. This lays a technical foundation for improving suture quality and safety and reducing tissue damage, and possesses significant theoretical value and application prospects.

[0135] S103. Based on the obtained suturing scheme, combined with the position, speed, and torque sensor information of the suturing robot joints, the optimal control sequence of the suturing robot is obtained, and at the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and the robust compensation control law are integrated into the real-time control system of the suturing robot, and the output of the real-time control system of the suturing robot is converted into actuator control instructions to control the suturing robot to assist in wound suturing.

[0136] For example, the kinematic and dynamic models of each joint of the robotic manipulator are first established, introducing joint angle, velocity, and acceleration limits as constraints. Then, with suturing accuracy and minimizing energy consumption as optimization objectives, the optimal joint space control sequence of the manipulator is derived using methods such as the calculus of variations and the Pontryagin minimum principle. Taking into account robot parameter uncertainty and environmental perturbations, a robust compensation control law is designed based on optimal control. Using feedback from the position, velocity, and torque sensors of the robot joints, the pose error of the manipulator end-point is mapped to the joint space through coordinate transformation. Joint angle / velocity feedback and integral terms are then constructed to compensate for system errors in real time. Finally, the optimal control sequence is combined with the compensation control law to form a closed-loop control system. Using the planned needle tip trajectory as a reference input, the control system calculates control commands for the manipulator joints in real time. After coordinate transformation, these commands are converted into control commands for the robot actuators, achieving precise control of the robot-assisted wound suturing process.

[0137] In an optional embodiment,

[0138] Based on the obtained suturing scheme and combined with the position, velocity, and torque sensor information of the suturing robot's joints, the optimal control sequence of the suturing robot is obtained. At the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the real-time control system of the suturing robot, including:

[0139] The mechanical structural parameters of the suturing robot were obtained, and the robot coordinate system, including the base coordinate system, joint coordinate system, manipulator coordinate system, and tissue surface coordinate system, was established. The position of the suturing robot relative to the base coordinate system was solved through coordinate transformation, and the kinematic positive relationship between the robot joint angle and the needle tip position was established. The robot dynamics model describing the relationship between the joint torque and the force on the needle tip was derived by combining the Lagrange equations with the robot joint angle, angular velocity, and torque.

[0140] The robot's joint angles and angular velocities are used as state variables, joint torques are used as control variables, and needle tip posture tracking error and control input smoothness are used as performance indicators. The optimization problem of suture robot motion control is constructed. The state equations, boundary conditions, and penalty factors corresponding to joint constraints are introduced to transform the optimization problem into an unconstrained optimization problem. A sequential convex optimization method is used to divide the unconstrained optimization problem into multiple convex quadratic subproblems. Through penalty factor adjustment and trust region control, the subproblems are iteratively solved to obtain the optimal control torque sequence of the suture robot's joints.

[0141] A position sensor is installed at the end of the suturing robot to measure the position deviation of the needle tip relative to the tissue surface in real time. The obtained needle tip position deviation is mapped to the joint space through coordinate transformation. Combined with the pre-built time series prediction model and the robot dynamics model, the joint angle compensation value is generated to compensate for the needle tip position deviation.

[0142] The optimal control torque sequence is used as the feedforward control input, and the joint angle compensation is used as the feedback control input. The composite control torque of the robot joint is generated by weighted summation, and the final composite control torque is input into the real-time control system of the suture robot.

[0143] For example, first, the mechanical structural parameters of the suture robot, such as link length and joint type, need to be obtained. Based on these parameters, the robot's coordinate system is established, including the base coordinate system, joint coordinate system, manipulator coordinate system, and tissue surface coordinate system. Through homogeneous coordinate transformation, the position of the robot's manipulator needle tip relative to the base coordinate system can be solved. Furthermore, using the theory of forward kinematics in robotics, a mapping relationship is established between the robot's joint angles and the needle tip position, resulting in the kinematic equations describing the robot's motion.

[0144] To describe the robot's dynamic characteristics, it's necessary to use the Lagrangian mechanics equations, combined with the robot's joint angles, angular velocities, and joint torques, to derive a robot dynamics model. This model describes the dynamic relationship between the robot's joint torques and the forces acting on the needle tip, and serves as the basis for building a robot motion controller.

[0145] Based on the established robot kinematic and dynamic models, the optimization problem for the suture robot's motion control can be constructed. Using the robot's joint angles and angular velocities as state variables, joint torques as control variables, and needle tip pose tracking error and control input smoothness as performance indicators, the objective functional for the optimal control problem is formed. Furthermore, the robot's dynamics equations are used as state equations, and penalty functions corresponding to boundary conditions and joint constraints are introduced to transform the original constrained optimization problem into an unconstrained one.

[0146] A sequential convex optimization method is used to solve the transformed unconstrained optimization problem. The entire optimization problem is divided into multiple convex quadratic subproblems. By introducing penalty factors and trust region control techniques, each subproblem is solved iteratively. In each iteration, the optimal solution to the current subproblem is obtained by solving a convex quadratic program, and the next subproblem is updated based on the penalty factor and trust region parameters. After multiple iterations, a near-optimal solution to the original optimization problem is finally obtained, namely, the optimal control torque sequence for the robot joints.

[0147] To further improve the control accuracy of the suture robot, a position sensor is installed at the end of the robot's manipulator to measure the needle tip's posture deviation relative to the tissue surface in real time. Through coordinate transformation, the measured needle tip posture deviation is mapped from the manipulator coordinate system to the joint space, resulting in joint angle deviation. Simultaneously, a pre-built time series prediction model, such as an autoregressive model or a Kalman filter, is used to predict the needle tip posture deviation, obtaining an estimated deviation for a period of time in the future. Combined with the robot's dynamic model, the posture deviation and predicted deviation values ​​are converted into joint angle compensation values, which serve as input for feedback control.

[0148] Finally, the optimal control torque sequence is used as the feedforward control input, and the joint angle compensation corresponding to the needle tip posture deviation is used as the feedback control input. Through a weighted summation approach, a composite control torque for the robot joint is generated. By adjusting the weight coefficients of the feedforward and feedback, a trade-off between tracking accuracy and robustness can be achieved. This composite control torque is input into the real-time control system of the suture robot. Through the servo drive and actuators, precise motion control and real-time trajectory compensation of the robot's needle tip are achieved.

[0149] In this embodiment, optimal control theory and sensor feedback technology are integrated to effectively improve the precision and robustness of robot-assisted surgery. By constructing kinematic and dynamic models, the robot control problem is transformed into an optimization problem and solved, resulting in an optimal control torque sequence that serves as the feedforward control input. Simultaneously, a position sensor measures the needle tip posture deviation in real time, and combined with a time series prediction model, generates joint angle compensation values ​​that serve as feedback control input. Ultimately, through feedforward-feedback hybrid control, high-precision tracking and real-time compensation of the needle tip motion are achieved.

[0150] In an optional embodiment,

[0151] Combining the pre-built time series prediction model and the robot dynamics model, the steps before generating the joint angle compensation to compensate for the needle tip posture deviation also include:

[0152] Constructing a time series dataset of needle tip posture deviation, wherein the time series dataset includes historical observations and corresponding timestamps;

[0153] Preprocessing the time series data set, and dividing the preprocessed time series data set into a training set, a validation set, and a test set;

[0154] A multi-scale attention recurrent neural network structure for the needle tip posture deviation prediction model is designed. The network structure includes an input layer, a multi-scale feature extraction layer, an attention fusion layer, a bidirectional recurrent neural network layer, and an output layer. The multi-scale feature extraction layer uses a one-dimensional convolutional neural network to extract local features of the needle tip posture deviation at different time scales with different convolution kernel sizes and step sizes. The attention fusion layer uses a self-attention mechanism to adaptively learn the correlation between different modal features and fuse them into a unified feature representation. The bidirectional recurrent neural network layer uses a bidirectional long short-term memory network to learn the forward and backward dependencies of the needle tip posture deviation in the time dimension and capture its long-term trend characteristics.

[0155] Construct a loss function for multi-task joint learning, and update the network parameters through the back-propagation algorithm. Repeat the iteration until the model converges to obtain a trained needle tip posture deviation prediction model.

[0156] The trained needle tip posture deviation prediction model is used to predict the needle tip posture deviation sequence within a certain time range in the future, and the predicted posture deviation sequence is converted into motion compensation instructions in the joint space through kinematic inverse solution and trajectory optimization algorithm.

[0157] For example, a time series dataset of needle tip posture deviation is first constructed, which contains historical observations of needle tip posture deviation during robotic suturing and the corresponding timestamp information. After obtaining the needle tip posture deviation time series dataset, preprocessing is performed. Specifically, common time series preprocessing techniques such as data smoothing, outlier detection and removal, and data normalization are used to clean and standardize the needle tip posture deviation data. At the same time, to evaluate the generalization performance of the model, the holdout method is used to divide the preprocessed time series dataset into three non-overlapping subsets: training set, validation set, and test set. The training set is used to learn and optimize model parameters, the validation set is used to select and tune model hyperparameters, and the test set is used to evaluate the final performance and generalization ability of the model.

[0158] Based on the characteristics of the needle tip posture deviation prediction problem, a multi-scale attention recurrent neural network model is designed. The model's network structure consists of an input layer, a multi-scale feature extraction layer, an attention fusion layer, a bidirectional recurrent neural network layer, and an output layer. The input layer receives fixed-length time series segments of the needle tip posture deviation and passes them to the subsequent feature extraction layer. The multi-scale feature extraction layer uses a one-dimensional convolutional neural network to extract features from the input sequence. By setting convolution kernels of different sizes and strides, local features of the needle tip posture deviation are extracted at different time scales.

[0159] After extracting multi-scale local features, a self-attention mechanism is used to adaptively fuse features at different scales. Specifically, the attention fusion layer automatically learns the correlations and interactions between features at different scales by calculating their correlations and importance weights. Then, based on the learned attention weights, features at different scales are weightedly fused to produce a unified feature representation.

[0160] After integrating multi-scale local features, a bidirectional recurrent neural network is used to model the long-term dependencies of the needle tip posture deviation. Specifically, the bidirectional recurrent neural network includes forward and backward recurrent neural networks, which propagate information forward and backward in time, respectively. In the forward propagation, the bidirectional recurrent neural network gradually accumulates historical state information of the needle tip posture deviation, learning its forward causal relationships and trend characteristics. In the backward propagation, the bidirectional recurrent neural network reversely summarizes the future state information of the needle tip posture deviation, learning its backward influence relationships and constraints. The bidirectional recurrent structure can fully utilize the contextual information of the needle tip posture deviation in the time dimension and capture its long-term evolution and change trends. At the same time, to alleviate the vanishing gradient problem of the recurrent neural network, a long short-term memory network is used as the recurrent unit. By introducing a gating mechanism and memory units, the model's ability to process long sequences of data is enhanced.

[0161] After designing the network structure of the needle tip pose deviation prediction model, a loss function for multi-task joint learning was constructed to guide the optimization and learning of model parameters. This loss function not only considers the prediction accuracy of the needle tip pose deviation but also incorporates supervisory information from other auxiliary tasks, such as needle tip velocity prediction and needle tip acceleration prediction. By jointly learning multiple related tasks, feature sharing and knowledge transfer between different tasks can be promoted, improving the generalization performance and robustness of the model. During model training, a backpropagation algorithm and gradient descent optimization method are used to iteratively calculate the gradient of the loss function and use it to update the model parameters. This process is repeated until the model converges on the training set, resulting in the final needle tip pose deviation prediction model.

[0162] After obtaining a trained needle tip pose deviation prediction model, it is applied to the real-time control and compensation of the robotic suturing process. Specifically, during the suturing process, the prediction model receives real-time needle tip pose deviation observations within a certain time range using a sliding window method and predicts the needle tip pose deviation sequence for a period of time in the future. Then, using an inverse kinematics algorithm, the predicted pose deviation sequence is converted into motion compensation in the robot joint space. Finally, using a trajectory optimization algorithm, the compensated joint motion trajectory is smoothed and optimized while satisfying the suturing task constraints, resulting in the final motion compensation command. This command is then transmitted to the robot control system, enabling predictive and active compensation for needle tip pose deviation, improving suturing accuracy and quality.

[0163] In this example, by constructing a time series dataset, designing a multi-scale attention recurrent neural network model, and incorporating a multi-task joint learning strategy, we achieve accurate prediction of both long-term and short-term trends and patterns of needle tip posture deviation. This method fully leverages the contextual information and associated features of needle tip posture deviation in both temporal and spatial dimensions. By introducing an attention mechanism and a bidirectional recurrent structure, it improves the representational power and generalization performance of the prediction model. Furthermore, this method combines the prediction results with an inverse kinematic solution and trajectory optimization algorithm to achieve predictive and active compensation for needle tip posture deviation, providing new insights and methods for improving the accuracy and stability of robotic suturing.

[0164] Figure 2 FIG is a schematic diagram of the structure of an artificial intelligence-based robot-assisted wound suturing system according to an embodiment of the present invention. Figure 2 As shown, the system includes:

[0165] Acquiring multimodal sensory data of the wound area, the multimodal sensory data including RGB images, force tactile data, and ultrasonic imaging data, mapping the multimodal sensory data into a unified spatial coordinate system, constructing a multimodal three-dimensional representation of the wound, and determining multiple suturing points by sampling the wound edge at equal intervals based on the wound shape and suturing requirements based on the multimodal three-dimensional representation of the wound;

[0166] For each suture point, the needle tip movement direction, tissue thickness, and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated and samples are taken on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties to obtain a suture plan that is adapted to the wound morphology and tissue properties.

[0167] Based on the obtained suturing scheme and combined with the position, speed, and torque sensor information of the suturing robot's joints, the optimal control sequence of the suturing robot is obtained. At the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the suturing robot's real-time control system, and the output of the suturing robot's real-time control system is converted into actuator control instructions to control the suturing robot to assist in wound suturing.

[0168] According to a third aspect of the embodiments of the present invention,

[0169] An electronic device is provided, comprising:

[0170] processor;

[0171] a memory for storing processor-executable instructions;

[0172] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0173] According to a fourth aspect of the embodiments of the present invention,

[0174] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0175] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot-assisted wound suturing system based on artificial intelligence, characterized in that: include: The first unit is configured to acquire multimodal sensory data of the wound area, the multimodal sensory data including RGB images, force tactile data, and ultrasonic imaging data, map the multimodal sensory data into a unified spatial coordinate system, construct a multimodal three-dimensional representation of the wound, and perform equally spaced sampling at the wound edge based on the wound shape and suturing requirements to determine multiple suturing points. The second unit is used to calculate candidate suture points for each suture point by comprehensively considering the needle tip movement direction, tissue thickness, and vascular nerve distribution. At the same time, for adjacent suture points, it generates a needle tip trajectory inside the wound, samples the needle tip trajectory, calculates the optimal needle posture at each sampling point, and adaptively adjusts the needle tip movement speed according to the curvature of the needle tip trajectory and tissue properties to obtain a suture plan that is adapted to the wound morphology and tissue properties; The third unit is used to obtain the optimal control sequence of the suturing robot based on the obtained suturing plan and the position, speed, and torque sensor information of the suturing robot joints, and to construct a robust compensation control law through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the real-time control system of the suturing robot, and the output of the real-time control system of the suturing robot is converted into actuator control instructions to control the suturing robot to assist in wound suturing; For each suture point, the needle tip movement direction, tissue thickness, and vascular nerve distribution are comprehensively considered to calculate candidate suture points. At the same time, for adjacent suture points, the needle tip trajectory inside the wound is generated and samples are taken on the needle tip trajectory. The optimal needle posture at each sampling point is calculated, and the needle tip movement speed is adaptively adjusted according to the curvature of the needle tip trajectory and tissue properties. The suture scheme adapted to the wound morphology and tissue properties is obtained, including: For each suture point, the tissue thickness, hardness, and vascular nerve density in its local neighborhood are extracted. Suture points that do not meet the threshold conditions are eliminated according to the preset threshold, and the number and index of suture points are updated to finally obtain the filtered candidate suture point set; Calculate the spatial distance, curvature and tangent direction between adjacent stitching points in the candidate stitching point set; The wound closure effect evaluation function and tissue stress distribution evaluation function were constructed as optimization objectives. The spacing between adjacent suture points, curvature, and the number of suture points were used as constraints. The optimal number and position of suture points were obtained through optimization and solved as the final suture point sequence. Based on each pair of adjacent suture points in the final suture point sequence, combined with the local curvature and thickness of the wound, the centerline of the needle tip trajectory is fitted within the wound cross section. The trajectory sampling point sequence is obtained by uniformly sampling along the centerline of the needle tip trajectory. The tangent direction and normal vector at each trajectory sampling point are calculated and used as the optimal needle posture at the corresponding trajectory sampling point. According to the tissue hardness, trajectory curvature and total trajectory length at the trajectory sampling point, the needle tip movement speed at each trajectory sampling point is adaptively calculated through a predefined speed adjustment function to form a needle tip trajectory speed distribution curve; An optimization problem for needle tip motion planning is constructed based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve, and the optimization problem is solved using an adaptive optimization algorithm. Combined with the obtained optimal needle posture, a suturing plan that is adapted to the wound morphology and tissue properties is ultimately obtained; Constructing wound closure effect evaluation function and tissue stress distribution evaluation function as optimization objectives includes: The calculation formula of the wound closure effect evaluation function is as follows: Among them, E c represents the wound closure effect, ω1 represents the weight coefficient of shape error, N represents the number of suture points, R i represents the local curvature radius of the i-th suture point, p i represents the coordinates of the i-th suture point, r(u i ) represents the parameterized representation of the wound edge curve, u i represents the parameter coordinates of the i-th suture point on the wound edge curve, ω2 represents the weight coefficient of curvature continuity, t i+1 Indicates the tangent direction of the i+1th suture point, t i represents the tangent direction of the i-th suture point, ||·|| represents the norm of the vector; The calculation formula of the tissue stress distribution evaluation function is as follows: Among them, E s represents the tissue stress distribution, ω3 represents the weight coefficient of stress distribution smoothness, A represents the area of ​​the wound and surrounding tissues, represents the gradient operator, σ(a) represents the tissue stress tensor at tissue a, ω4 represents the weight coefficient of the peak stress, ||·|| F represents the Frobenius norm; The optimization problems of needle tip motion planning based on the obtained needle tip trajectory centerline and needle tip trajectory velocity distribution curve include: According to the distance between the needle tip trajectory sampling point and the wound centerline, the speed and direction of the needle tip trajectory, the objective function of the optimization problem is constructed, and the calculation formula is as follows: Among them, J represents the objective function, q 1:M represents the sampling point sequence of the needle tip trajectory, v 1:M represents the velocity distribution of the needle tip trajectory, θ 1:M represents the direction sequence of the needle tip trajectory, M represents the number of sampling points, q j represents the coordinates of the jth sampling point, c(u j ) represents the center line of the needle tip trajectory fitted in the wound cross section at parameter u j The coordinates at the position, λ1 represents the weight coefficient of the distance, μ t Represents the weight coefficient of tangential velocity, v t，j represents the tangential velocity at the jth sampling point, μ n Represents the weight coefficient of normal velocity, v n，j represents the normal velocity at the jth sampling point, △t j represents the time interval between the jth sampling point and the next sampling point, λ2 represents the directional weight coefficient, θ j+1 Indicates the direction of the j+1th sampling point, θ j represents the direction at the jth sampling point, λ3 represents the weight coefficient of the speed, v j+1 represents the velocity at the j+1th sampling point, v j represents the velocity at the jth sampling point.

2. The system according to claim 1, wherein: Acquiring multimodal sensory data of the wound area, the multimodal sensory data including RGB images, force tactile data, and ultrasonic imaging data, mapping the multimodal sensory data into a unified spatial coordinate system, and constructing a multimodal three-dimensional representation of the wound includes: A convolutional neural network is used to extract pixel-level disparity maps from the left and right views of the RGB image. At the same time, an attention mechanism and multi-scale feature fusion are introduced to convert the pixel-level disparity map into 3D point cloud data of the wound surface. The surface reconstruction method based on implicit function is used to convert the obtained 3D point cloud data into a signed distance field. The implicit neural representation network is used to learn the mapping from the coordinates of the 3D point cloud data to the signed distance field, and the wound implicit surface is obtained. By extracting the zero-isovalue surface of the wound implicit surface, the wound triangulation network model is obtained. Based on the gradient changes of force-tactile data and ultrasound imaging data, the voxel resolution is adaptively adjusted, and a sparse voxel representation is achieved through an octree hierarchy. Voxel interpolation and nearest neighbor search algorithms are used to append the force-tactile data and ultrasound imaging data to the vertices of the pre-built visual model to obtain the final wound visual model. A physics-based wound tissue modeling technique was introduced. The wound area was divided into three-dimensional tetrahedral meshes. The parameters of the three-dimensional tetrahedral meshes were initialized using histological characteristics extracted from ultrasound images. The deformation and stress distribution of the wound were calculated using the finite element method. Based on the comparison results of the stress distribution and force tactile data, the parameters of the three-dimensional tetrahedral meshes were iteratively updated to obtain the final wound biomechanical model. The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound.

3. The system according to claim 2, characterized in that The wound triangulation network model, wound visual model and wound biomechanical model are integrated into a unified data structure to obtain a multimodal three-dimensional representation of the wound including: The wound triangulation network model, wound visual model and wound biomechanics model are integrated into a unified graph data structure, where nodes represent sampling points and edges represent the connection relationship between nodes. Adaptively determine the convolution kernel size, shape, and weight distribution parameters of the node based on the node's local geometric features and neighborhood distribution, where the local geometric features include the curvature and detail complexity of the node's neighborhood, and the neighborhood distribution includes the geometric structure and spatial distribution of the node's neighborhood; According to the convolution kernel size, shape and weight distribution parameters of the node, a dynamic graph convolution kernel of the node is constructed, the dynamic graph convolution kernel is used to extract features of the node, and the weight of the dynamic graph convolution kernel is adaptively adjusted through the attention mechanism; Extract node features at multiple scales, apply the corresponding dynamic graph convolution kernel at each scale to obtain node features at different scales, and input the features at different scales into the feature fusion layer; In the feature fusion layer, the aggregation weight of the node is adaptively determined according to the connection strength between the nodes and the importance of the nodes, and the features of the neighboring nodes are weightedly aggregated according to the aggregation weight to obtain the final multimodal three-dimensional representation of the wound.

4. The system according to claim 1, wherein: Based on the obtained suturing scheme and combined with the position, velocity, and torque sensor information of the suturing robot's joints, the optimal control sequence of the suturing robot is obtained. At the same time, a robust compensation control law is constructed through coordinate transformation and feedback control law. The optimal control sequence and robust compensation control law are integrated into the real-time control system of the suturing robot, including: The mechanical structural parameters of the suturing robot were obtained, and the robot coordinate system, including the base coordinate system, joint coordinate system, manipulator coordinate system, and tissue surface coordinate system, was established. The position of the suturing robot relative to the base coordinate system was solved through coordinate transformation, and the kinematic positive relationship between the robot joint angle and the needle tip position was established. The robot dynamics model describing the relationship between the joint torque and the force on the needle tip was derived by combining the Lagrange equations with the robot joint angle, angular velocity, and torque. The robot's joint angles and angular velocities are used as state variables, joint torques are used as control variables, and needle tip posture tracking error and control input smoothness are used as performance indicators. The optimization problem of suture robot motion control is constructed. The state equations, boundary conditions, and penalty factors corresponding to joint constraints are introduced to transform the optimization problem into an unconstrained optimization problem. A sequential convex optimization method is used to divide the unconstrained optimization problem into multiple convex quadratic subproblems. Through penalty factor adjustment and trust region control, the subproblems are iteratively solved to obtain the optimal control torque sequence of the suture robot's joints. A position sensor is installed at the end of the suturing robot to measure the position deviation of the needle tip relative to the tissue surface in real time. The obtained needle tip position deviation is mapped to the joint space through coordinate transformation. Combined with the pre-built time series prediction model and the robot dynamics model, the joint angle compensation value is generated to compensate for the needle tip position deviation. The optimal control torque sequence is used as the feedforward control input, and the joint angle compensation is used as the feedback control input. The composite control torque of the robot joint is generated by weighted summation, and the final composite control torque is input into the real-time control system of the suture robot.

5. The system according to claim 4, characterized in that Combining the pre-built time series prediction model and the robot dynamics model, the steps before generating the joint angle compensation to compensate for the needle tip posture deviation also include: Constructing a time series dataset of needle tip posture deviation, wherein the time series dataset includes historical observations and corresponding timestamps; Preprocessing the time series data set, and dividing the preprocessed time series data set into a training set, a validation set, and a test set; A multi-scale attention recurrent neural network structure for the needle tip posture deviation prediction model is designed. The network structure includes an input layer, a multi-scale feature extraction layer, an attention fusion layer, a bidirectional recurrent neural network layer, and an output layer. The multi-scale feature extraction layer uses a one-dimensional convolutional neural network to extract local features of the needle tip posture deviation at different time scales with different convolution kernel sizes and step sizes. The attention fusion layer uses a self-attention mechanism to adaptively learn the correlation between different modal features and fuse them into a unified feature representation. The bidirectional recurrent neural network layer uses a bidirectional long short-term memory network to learn the forward and backward dependencies of the needle tip posture deviation in the time dimension and capture its long-term trend characteristics. Construct a loss function for multi-task joint learning, and update the network parameters through the back-propagation algorithm. Repeat the iteration until the model converges to obtain a trained needle tip posture deviation prediction model. The trained needle tip posture deviation prediction model is used to predict the needle tip posture deviation sequence within a certain time range in the future, and the predicted posture deviation sequence is converted into motion compensation instructions in the joint space through kinematic inverse solution and trajectory optimization algorithm.

6. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to call the instructions stored in the memory to execute the method performed by the artificial intelligence-based robot-assisted wound suturing system as described in any one of claims 1 to 5.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method performed by the artificial intelligence-based robot-assisted wound suturing system according to any one of claims 1 to 5 is implemented.

Images