A multimodal marker based in-vivo closed-loop three-dimensional printing system and method, device

Abstract

The application provides a kind of based on multimodal mark in-vivo closed loop three-dimensional printing system and method, equipment, including for obtaining the information of multimodal marker and the information of internal region to be printed, while realizing the conversion of each coordinate system Image acquisition and feedback module;For identifying the position of marker infrared camera imaging and feedback module;For dynamic compensation control according to error model and realizing adaptive adjustment of end effector closed loop control module;For receiving the instruction of closed loop control module to complete the in-vivo three-dimensional printing of biological material printing execution module;Multi-freedom degree mechanical arm is collaborative mechanical arm, and the end of collaborative mechanical arm is provided with printing spray head.The application realizes the unified positioning of preoperative medical image and intraoperative navigation information by constructing multimodal marker system, effectively reduces multimodal registration error and improves spatial mapping precision.

Description

Technical Field

[0001] This invention relates to the fields of medical robotics and biomanufacturing technology, specifically to an in vivo closed-loop 3D printing system, method, and equipment based on multimodal labeling. Background Technology

[0002] With the development of precision medicine and tissue engineering technologies, in vivo 3D bioprinting has gradually become an important technical means to achieve tissue repair and functional reconstruction. Compared with in vitro printing followed by implantation, in vivo printing can directly construct biological structures in target areas inside the human body, resulting in better tissue matching and biofunctional adaptability. However, in the environment of minimally invasive laparoscopic surgery, the target area's position exhibits significant dynamic changes due to physiological movements such as respiration, heartbeat, and tissue peristalsis, making traditional static positioning-based printing methods difficult to meet practical needs. Furthermore, minimally invasive surgery relies on puncture channels as the entry point, making it difficult to obtain and accurately model the location of the area to be printed in real time during the actual operation, further exacerbating system errors. In addition, existing in vivo printing systems mostly employ open-loop control strategies, lacking real-time error feedback and dynamic compensation mechanisms, which can easily lead to cumulative errors in complex environments, affecting printing accuracy and surgical safety. On the other hand, there is usually a lack of a unified reference marker system between intraoperative guided infrared imaging and preoperative planned CT or MRI, resulting in insufficient accuracy of multimodal data fusion.

[0003] Application CN110411308A discloses a method for accuracy testing of customized 3D printed models. This method involves setting markers on CT / MRI images and then specifying these markers using a concrete model. The distances between the markers on the CT / MRI images and the markers on the concrete model are measured and compared to obtain the dimensional deviation between the customized 3D printed human tissue model and the actual human tissue in the CT / MRI images. The method is simple to operate, yields highly accurate results, and has significant reference value. However, this patent's application is biased towards accuracy testing of 3D printed models, specifically post-operative / ex vivo model testing, meaning the entire process occurs after manufacturing. Furthermore, although the markers in this patent can be displayed on CT / MRI, they are only used to extract the distance between two fixed points, limiting their applicability.

[0004] Application CN118849420A discloses a 3D printing intermediate product and a 3D printing method. The intermediate product is placed in a printing cavity of a building unit and generated by 3D printing. It includes a 3D printed part and printing material encasing the 3D printed part. The upper surface of the 3D printed intermediate product is marked for observation and identification. However, this invention is geared towards general industrial 3D printing production lines, mainly used for marking and managing printed semi-finished products (intermediate products), identifying cooling status, and tracing tasks. It cannot be applied to medical robots, biomanufacturing, or in vivo printing. Furthermore, the marking system of this invention is a surface marking of the intermediate product, directly made on the surface of the 3D printed semi-finished product and integrally formed with the printing process. This means that the marking points are rigidly connected to the printed part, making it difficult to adapt to the deformation of human skin. After surgery, this multiphase marking system is difficult to remove directly.

[0005] Therefore, how to achieve multimodal information fusion, precise positioning and closed-loop control in dynamic environments is a key problem that needs to be solved in current in vivo printing technology. At the same time, it can be applied to medical robots, biomanufacturing or in vivo printing, and can adapt to the deformation of human skin. This is the problem that this application needs to solve. Summary of the Invention

[0006] The purpose of this invention is to propose an in vivo closed-loop 3D printing system, method, and device based on multimodal labeling, thereby solving the problems of large positioning errors, insufficient accuracy of multimodal information fusion, difficulty in modeling puncture channel constraints, and lack of stable closed-loop control mechanism in existing intraperitoneal in vivo 3D printing technology in dynamic physiological environments.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The present invention provides an in vivo closed-loop 3D printing system based on multimodal marking, including a multimodal flexible marking module, an image acquisition and feedback module, a closed-loop control module, and an in vivo printing execution module; the multimodal flexible marking module is used to provide biological surface information before and during surgery, including multiphase array marking points arranged on the body surface;

[0008] The image acquisition and feedback module is used to acquire information about multimodal markers and the internal area to be printed, and to realize the transformation of various coordinate systems. This module includes: a CT or MRI imaging and feedback module, used to establish a preoperative and intraoperative model of the internal area to be printed, including CT / MRI equipment.

[0009] Infrared camera imaging and feedback module, used to identify the location of markers, includes a depth camera;

[0010] The closed-loop control module is used for dynamic compensation control based on the error model and to achieve adaptive adjustment of the end effector; the printing execution module includes a multi-degree-of-freedom robotic arm and a biomaterial nozzle, which is used to receive instructions from the closed-loop control module to complete the in vivo three-dimensional printing of biomaterials; wherein, the multi-degree-of-freedom robotic arm is a collaborative robotic arm, and the printing nozzle is installed at the end of the collaborative robotic arm; during system operation, the printing nozzle passes through the puncture channel to print on the target tissue.

[0011] Furthermore, the multiphase array markers include infrared optical marker units, CT imaging material layers, and MRI imaging material layers. The multiphase array markers are arranged irregularly on the body surface around the target tissue and have a unique code, which is used to achieve unified positioning and prevent misidentification between preoperative images and intraoperative optical navigation.

[0012] Furthermore, the depth camera is fixedly mounted to the end of the collaborative robotic arm via a rigid bracket, and the relative position between the two remains unchanged; the depth camera is used to simultaneously output RGB images and depth images to obtain the three-dimensional spatial coordinates of the multiphase array marker points.

[0013] Furthermore, the collaborative robotic arm has three degrees of freedom, including two rotary joints and one kinetic joint, wherein the two rotary joints are used to adjust the X-axis and Y-axis coordinates of the print head, and the kinetic joint is used to adjust the Z-axis coordinate of the print head.

[0014] On the other hand, the present invention also provides an in vivo closed-loop 3D printing method based on multimodal markers implemented by an in vivo closed-loop 3D printing system, comprising the following steps:

[0015] Step S1: Before the operation, acquire CT or MRI image data containing the target tissue and multiphase array markers, construct a three-dimensional model of the target tissue, extract the spatial location, shape boundary and structural features of the area to be printed, and establish an image coordinate system between the target tissue and the markers.

[0016] Step S2: During the operation, the position coordinates of the multiphase array markers on the body surface are obtained by the depth camera, which is used as an infrared optical tracking system. The three-dimensional spatial position information of the biological surface is extracted based on the preset coded markers, and the local coordinate system of the markers is established.

[0017] Step S3: Establish the mapping relationship between the CT or MRI image coordinate system, the optical coordinate system of the marker, and the robot coordinate system of the collaborative robotic arm. Calculate the rotation matrix and translation vector through rigid body transformation to obtain the transformation matrix, thereby realizing the mapping of the target position in the robot coordinate system.

[0018] Step S4: Obtain the real-time pose information of the print head installed at the end of the collaborative robotic arm, calculate the error vector between the target position and the actual position, construct a dynamic error model, and update the error at a fixed frequency.

[0019] Step S5: Generate control input based on the error model to adjust the position of the printing nozzle at the end of the collaborative robotic arm in real time to compensate for disturbances caused by breathing or tissue movement.

[0020] Step S6: According to the instructions issued by the closed-loop control module, the printing nozzle is controlled to move through the puncture channel and deposit the biomaterial into the target tissue area. The printing trajectory is corrected through real-time path tracking to complete the in vivo three-dimensional structure construction.

[0021] As a further improvement, in step S1, a three-dimensional geometric model of the target region is generated using voxel reconstruction or mesh reconstruction methods, and a set of key feature points is extracted, denoted as:

[0022] .

[0023] in, This step represents the spatial point set of the target area. The output of this step is the spatial representation of the target area in the image coordinate system, providing basic data for subsequent coordinate mapping.

[0024] As a further improvement, in step S2, during the procedure, the spatial position of the marker is acquired in real time using an infrared optical tracking system, which is represented as follows:

[0025] ;

[0026] in, Let be the coordinates of point t in the camera coordinate system at time t, and K be the intrinsic parameter matrix of the camera. and Let the focal length be the distance along the u and v directions in the camera coordinate system. The inclination coefficient of the coordinate axis. and Here, T represents the coordinates of the principal point in the image, and T is the extrinsic parameter matrix of the camera. Let be a rotation matrix. It is a translation matrix. Let t be the coordinates of point t in the world coordinate system. World coordinates Homogeneous coordinate notation.

[0027] As a further improvement, step S3 establishes the mapping relationship between the various coordinate systems. The mapping relationship between the CT / MRI scan coordinate system and the world coordinate system is as follows:

[0028] ;

[0029] in, This represents the coordinates of the marker in the world coordinate system W at time t. This represents the transformation matrix from the CT / MRI scan coordinate system to the world coordinate system. This represents the coordinates of the marker point at time t in the CT / MRI scan coordinate system;

[0030] Since the mapping relationship between the camera coordinate system and the world coordinate system has already been established, it is given directly as follows:

[0031] ;

[0032] in, This represents the transformation matrix from the infrared camera coordinate system to the world coordinate system, and it is related to the matrix... They are inverse matrices. This represents the coordinates of the marker point at time t in the infrared camera coordinate system;

[0033] The mapping relationship between the robot coordinate system and the world coordinate system is as follows:

[0034] ;

[0035] in, This represents the transformation matrix from the CT / MRI scan coordinate system to the world coordinate system. This represents the coordinates of the marker point at time t in the CT / MRI scan coordinate system;

[0036] This method enables dynamic mapping updates between other coordinate systems and the actual spatial coordinate system, thus providing a unified reference for subsequent motion compensation.

[0037] As a further improvement, multi-degree-of-freedom robotic arms include, but are not limited to, 3-degree-of-freedom robotic arms and 4-degree-of-freedom robotic arms.

[0038] As a further improvement, the establishment of the mapping relationship in step S3 uses the weighted least squares method to solve for the parameters of the transformation matrix, so as to reduce the impact of the visual measurement error of the depth camera and the recognition error of the multiphase array marker points on the mapping accuracy.

[0039] As a further improvement, the dynamic error model in step S4 includes position error and error change rate, wherein,

[0040] The error between the actual position and the target position of the robot's end effector is: ;

[0041] The rate of change of error is expressed as: ;

[0042] in, For the target location, This refers to the actual position of the robot's end effector.

[0043] As a further improvement, the closed-loop control in step S5 adopts proportional-derivative control, proportional-integral-derivative control, or model predictive control methods, and its control input satisfies: ;

[0044] in and These are the proportional coefficient and the differential coefficient, respectively. For disturbance compensation weights, This represents the disturbance term caused by respiration, tissue peristalsis, or abdominal wall displacement; the update frequency of the closed-loop control is not less than 30Hz. This control model is used to adjust the robot joints in real time and dynamically update the position of the end effector, continuously bringing it closer to the desired printing position.

[0045] As a further improvement, step S6 is in-body printing execution and adaptive path tracking. Based on the control signals output from steps S4 and S5, the print head is driven to move along a preset path, and dynamic correction is performed through a real-time path tracking algorithm to ensure that the error between the target position and the actual position in S5 is satisfied. By minimizing the deviation between the two, the nozzle can accurately reach the preset printing position and complete the precise deposition of biomaterial. In order to make the printing path smooth and avoid collision with tissue, the system combines a path interpolation algorithm to achieve continuous printing. The three-axis robot has two rotary joints and one kinetic joint. The two rotary joints are used to adjust the X-axis and Y-axis coordinates of the end effector, and the kinetic joint is used to adjust the Z-axis coordinate of the end effector.

[0046] As a further improvement, the multi-coordinate system fusion in step S3 uses the weighted least squares method to solve for parameters, so as to reduce the impact of visual measurement error and marker recognition error on the overall mapping accuracy.

[0047] As a further improvement, a disturbance observer is introduced in steps S4 and S5 to estimate breathing disturbances online, thereby improving the system's robustness to periodic and non-periodic disturbances.

[0048] As a further improvement, the path tracking in step S6 adopts a model predictive control method to predict and optimize the trajectory deviation at several future moments, so as to improve the printing stability and trajectory tracking accuracy under complex motion conditions.

[0049] As a further improvement, the update frequency of the closed-loop control system is preferably 50Hz to 100Hz to meet the real-time compensation requirements under dynamic environments.

[0050] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps in the in vivo closed-loop 3D printing method based on multimodal marking.

[0051] Compared with existing technologies, the beneficial effects of this invention are as follows: By constructing a multimodal labeling system, this invention achieves unified positioning of preoperative medical images and intraoperative navigation information, effectively reducing multimodal registration errors and improving spatial mapping accuracy; by constructing a multi-coordinate system fusion model, it achieves unified expression of complex spatial relationships, enhancing the consistency between image space and execution space; by using dynamic error modeling and closed-loop control mechanisms, it achieves real-time compensation for respiratory disturbances and tissue movement, significantly improving printing positioning accuracy; by using an end effector adaptive adjustment strategy, it reduces the lateral additional load at the puncture point, improving the safety of minimally invasive operations; by using path tracking and dynamic correction mechanisms, it enables the nozzle to continuously approach the target position, thereby significantly improving in vivo printing accuracy and stability; by constructing a complete technology chain of image modeling, multimodal fusion, dynamic modeling, closed-loop control, and finally printing execution, it avoids the dependence of traditional methods on human experience and complex analytical models, improving engineering application efficiency and system robustness. Attached Figure Description

[0052] Figure 1 A flowchart of an in vivo closed-loop 3D printing method based on multimodal labeling provided in an embodiment of the present invention;

[0053] Figure 2 This is a block diagram of the overall structure of an in vivo closed-loop 3D printing system based on multimodal marking according to the present invention.

[0054] Figure 3 A schematic diagram of an in vivo printing surgery provided in an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of multiphase array marker points provided in an embodiment of the present invention.

[0056] In the diagram: 1-Collaborative robotic arm; 2-CT / MRI equipment; 3-Depth camera; 4-Printing nozzle; 5-Multiphase array markers; 6-Puncture channel; 7-Target tissue. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0058] like Figure 1 and Figure 3As shown, the in vivo closed-loop 3D printing system and method based on multimodal markers of the present invention consists of a collaborative robotic arm 1, a CT / MRI device 2, a depth camera 3, a printing nozzle 4, multiphase array marker points 5, a puncture channel 6, and target tissue 7.

[0059] The collaborative robotic arm 1 is the actuator of the entire system, possessing three degrees of freedom for adjustment. Two rotary joints, J1 and J2, are used to adjust the X and Y axis coordinates of the end effector, while the translational joint Z1 is used to adjust the Z axis coordinate of the end effector. The collaborative robotic arm 1 is connected to the controller via TCP / IP protocol and can receive motion control commands from the controller.

[0060] The CT / MRI device 2 is positioned above the target tissue 7, at a height parallel to the rotation joint J2 of the collaborative robotic arm 1, with translational offset in the XY plane, while simultaneously aligning with the target tissue 7. This arrangement not only ensures that the CT / MRI device 2 does not interfere with the operation of the collaborative robotic arm 1, but also guarantees a good field of view for tracking markers.

[0061] Depth camera 3 is fixedly mounted on the end of collaborative robotic arm 1 using a bracket; the two can be considered rigidly connected, and their relative positions remain unchanged during movement. Depth camera 3 can simultaneously output RGB video streams and depth streams, and its resolution and frame rate settings can be adjusted as needed. After acquiring the target surface depth information, the CT / MRI device 2 transmits the data to the control system. The control system can then use the transformation relationship between coordinate systems to send new target coordinates, thereby enabling precise control of the robotic arm.

[0062] The printing nozzle 4 is mounted on the end flange of the collaborative robotic arm 1, and the two are concentric. The system adjusts the extrusion of biomaterial by regulating the air pressure. Specifically, the air supply for the printing nozzle 4 is an external air pump, which passes through a set of air pressure valves and relays. The output air pressure of the air pressure valves is adjustable, and the relays are used to control the on / off state of the air supply, thereby achieving precise control of pressure and extrusion speed.

[0063] Multiphase array markers 5 are placed on the skin surface outside the area to be repaired. They can be accurately identified by the CT / MRI equipment 2 and the depth camera 3, and are used for three-dimensional parameter estimation and reconstruction of the multiphase array markers and target tissue 7 (e.g., ...). Figure 4(As shown). Each phase marker at the marked location is irregularly arranged and numbered. This allows for more accurate identification of the dynamic target surface throughout the repair process, distinguishing the specific location of the central lesion and ensuring real-time tracking and precise positioning even when the markers move or rotate. The system controller receives real-time data from the depth camera 3 and the initial scanned point cloud model, performs data fusion, and generates control commands. It comprehensively analyzes the depth information of the target surface and the initial path model, and adjusts the position of the robotic arm and the extrusion speed of the extrusion system through feedback. This enables the system to adapt to dynamic environments, ensuring that the material accurately covers the target area. The printing nozzle 4, based on the multiphase array marker points 5, passes through the puncture channel 6 to repair and print on the target tissue 7.

[0064] In this embodiment, an in vivo closed-loop 3D printing system based on multimodal markers is constructed (e.g., Figure 2 (As shown) This system mainly includes a multimodal marker flexible module, an image acquisition and feedback module, a closed-loop control module, and an in vivo printing execution module. The multimodal marker flexible module is placed on the patient's abdominal surface, made of a flexible substrate material, and integrates an infrared emitting unit, CT imaging material, and MRI imaging material, enabling stable identification in different modalities. The image acquisition and feedback module acquires information about the multimodal markers and the internal area to be printed, while also performing coordinate system transformations. This module can be further divided into two parts: a CT or MRI imaging and feedback module, used to establish a preoperative and intraoperative model of the internal area to be printed; and an infrared camera imaging and feedback module, used to identify the position of the markers. The printing execution module uses a 4-DOF robotic arm with a biomaterial nozzle mounted on its end flange to complete the in vivo printing operation.

[0065] In the specific implementation process, preoperative modeling is first performed. Image data of the target area in the patient's abdominal cavity is acquired using CT or MRI equipment, and medical image processing software is used to segment and reconstruct the images to obtain a three-dimensional model of the target tissue 7. Simultaneously, the spatial coordinate information of the target area is extracted to establish an image coordinate system. In this process, threshold segmentation combined with a region growing algorithm can be used to extract the target tissue 7, and a three-dimensional surface model can be generated using the Marching Cubes algorithm to obtain the geometric boundary of the target area. Subsequently, multimodal flexible markers are placed on the patient's body surface, and the spatial relationship between the marker positions and the target area is established using CT / MRI images.

[0066] During the procedure, the marker is monitored in real time using an infrared optical tracking system to determine its position in the actual space. This enables dynamic mapping and updating between the image coordinate system and the world coordinate system. Simultaneously, coded array markers are placed outside the puncture channel 6. Image data is acquired using an RGB-D camera, and the marker center position information is extracted using a method based on color segmentation and geometric fitting, thereby reducing registration errors.

[0067] During dynamic control, the actual position of the robot's end effector is acquired in real time. And calculate its distance from the target position. Error between:

[0068] ;

[0069] Simultaneously calculate the rate of change of error:

[0070] ;

[0071] Based on the above error model, a closed-loop control system is constructed, and its control input is:

[0072] ;

[0073] in, and These are the proportional and derivative coefficients, respectively. These control inputs are used to adjust the robot's joints, causing the end effector to gradually approach the target position.

[0074] During the printing execution phase, the printhead is driven to move according to the preset printing path, and the trajectory is dynamically corrected through a real-time path tracking algorithm to ensure that the error between the target position and the actual position meets the following requirements:

[0075] ;

[0076] in, The target printing location is defined. To ensure a smooth printing path without colliding with tissue, the system incorporates a path interpolation algorithm for continuous printing. In terms of material selection, hydrogels or bio-inks with good rheological properties are preferred, and stable deposition is achieved through a reasonable extrusion rate.

[0077] In this embodiment, the update frequency of the closed-loop control system is preferably set to 50Hz to ensure that the system has sufficient response capability to breathing disturbances (approximately 0.2–0.5Hz); at the same time, the measurement data is processed by introducing sliding window filtering or Kalman filtering to reduce the impact of noise on system stability.

[0078] Through the above-described embodiments, the present invention realizes a complete technical process from preoperative image modeling, multimodal localization, spatial registration, dynamic error modeling to closed-loop control and printing execution, enabling the system to maintain high precision and high stability in a dynamic abdominal environment, thereby significantly improving the effect and safety of in vivo 3D printing.

[0079] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. An in vivo closed-loop 3D printing system based on multimodal markers, characterized in that, It includes a multimodal flexible labeling module, an image acquisition and feedback module, a closed-loop control module, and an in vivo printing execution module; the multimodal flexible labeling module is used to provide biological surface information before and during surgery, including multiphase array labeling points (5) arranged on the body surface; The image acquisition and feedback module is used to acquire information about multimodal markers and information about the internal area to be printed, and to realize the transformation of each coordinate system. The module includes: a CT or MRI imaging and feedback module, used to establish a model of the internal area to be printed before and during the operation, including a CT / MRI device (2). An infrared camera imaging and feedback module is used to identify the location of markers, including a depth camera (3); The closed-loop control module is used to perform dynamic compensation control based on the error model and realize the adaptive adjustment of the end effector; the printing execution module includes a multi-degree-of-freedom robotic arm and a biomaterial nozzle, which is used to receive instructions from the closed-loop control module to complete the in vivo three-dimensional printing of biomaterials; wherein, the multi-degree-of-freedom robotic arm is a collaborative robotic arm (1), and the end of the collaborative robotic arm (1) is equipped with a printing nozzle (4); during system operation, the printing nozzle (4) passes through the puncture channel (6) to print the target tissue (7); The multiphase array marker (5) includes an infrared optical marker unit, a CT imaging material layer and an MRI imaging material layer. The multiphase array marker (5) is arranged irregularly on the body surface around the target tissue (7) and has a unique code to achieve unified positioning and prevent misidentification between preoperative images and intraoperative optical navigation.

2. The in vivo closed-loop 3D printing system based on multimodal marking according to claim 1, characterized in that, The depth camera (3) is fixedly mounted on the end of the collaborative robotic arm (1) by a rigid bracket, and the relative position between the two remains unchanged; the depth camera (3) is used to output RGB images and depth images simultaneously to obtain the three-dimensional spatial coordinates of the multiphase array marker points (5).

3. The in vivo closed-loop 3D printing system based on multimodal marking according to claim 1, characterized in that, The collaborative robotic arm (1) has three degrees of freedom, including two rotary joints and one locating joint, wherein the two rotary joints are used to adjust the X-axis and Y-axis coordinates of the print head (4), and the locating joint is used to adjust the Z-axis coordinate of the print head (4).

4. A method for in vivo closed-loop 3D printing based on multimodal markers, implemented using the in vivo closed-loop 3D printing system based on multimodal markers as described in any one of claims 1-3, characterized in that, Includes the following steps: Step S1: Before the operation, acquire CT or MRI image data containing the target tissue (7) and multiphase array markers (5), construct a three-dimensional model of the target tissue (7), extract the spatial location, shape boundary and structural features of the area to be printed, and establish the image coordinate system of the target tissue (7) and the markers. Step S2: During the operation, the position coordinates of the multiphase array markers (5) on the body surface are obtained by the depth camera (3) which is used as an infrared optical tracking system, and the three-dimensional spatial position information of the biological surface is extracted based on the preset coded markers to establish the local coordinate system of the markers; Step S3: Establish the mapping relationship between the CT or MRI image coordinate system, the optical coordinate system of the marker, and the robot coordinate system of the collaborative robotic arm (1). Calculate the rotation matrix and translation vector through rigid body transformation to obtain the transformation matrix, thereby realizing the mapping of the target position in the robot coordinate system. Step S4: Obtain the real-time pose information of the printing nozzle (4) installed at the end of the collaborative robotic arm (1), calculate the error vector between the target position and the actual position, construct a dynamic error model, and update the error at a fixed frequency. Step S5, Closed-loop dynamic compensation: Based on the dynamic error model obtained in step S4, control input is generated to adjust the position of the printing nozzle (4) at the end of the collaborative robotic arm (1) in real time in a closed loop. Step S6: According to the instructions issued by the closed-loop control module, the printing nozzle (4) is controlled to move through the puncture channel (6) and deposit the biomaterial into the target tissue (7) area. The printing trajectory is corrected by real-time path tracking to complete the in vivo three-dimensional structure construction.

5. The in vivo closed-loop 3D printing method based on multimodal markers according to claim 4, characterized in that, In step S1, a three-dimensional geometric model of the target region is generated using voxel reconstruction or mesh reconstruction methods, and a set of key feature points is extracted, denoted as: ； in, This step represents the spatial point set of the target area. The output of this step is the spatial representation of the target area in the image coordinate system, providing basic data for subsequent coordinate mapping.

6. The in vivo closed-loop 3D printing method based on multimodal markers according to claim 4, characterized in that, In step S2, during the operation, the spatial position of the marker is acquired in real time using an infrared optical tracking system, which is represented as follows: ； in, Let be the coordinates of point t in the camera coordinate system at time t, and K be the intrinsic parameter matrix of the camera. and Let the focal length be the distance along the u and v directions in the camera coordinate system. The inclination coefficient of the coordinate axis. and Here are the coordinates of the principal point in the image, and T is the extrinsic parameter matrix of the camera. Let be a rotation matrix. It is a translation matrix. Let t be the coordinates of point t in the world coordinate system. World coordinates Homogeneous coordinate notation.

7. The in vivo closed-loop 3D printing method based on multimodal markers according to claim 4, characterized in that, Step S3 establishes the mapping relationship between the various coordinate systems. The mapping relationship between the CT / MRI scan coordinate system and the world coordinate system is as follows: ； in, This represents the coordinates of the marker in the world coordinate system W at time t. This represents the transformation matrix from the CT / MRI scan coordinate system to the world coordinate system. This represents the coordinates of the marker point at time t in the CT / MRI scan coordinate system; Since the mapping relationship between the camera coordinate system and the world coordinate system has already been established, it is given directly as follows: ； in, This represents the transformation matrix from the infrared camera coordinate system to the world coordinate system, and it is related to the matrix... They are inverse matrices. This represents the coordinates of the marker point at time t in the infrared camera coordinate system; The mapping relationship between the robot coordinate system and the world coordinate system is as follows: ； in, This represents the transformation matrix from the CT / MRI scan coordinate system to the world coordinate system. This represents the coordinates of the marker point at time t in the CT / MRI scan coordinate system.

8. The in vivo closed-loop 3D printing method based on multimodal markers according to claim 4, characterized in that, The dynamic error model described in step S4 includes position error and error change rate, wherein, The error between the actual position and the target position of the robot's end effector is: ; The rate of change of error is expressed as: ; in, For the target location, This refers to the actual position of the robot's end effector.

9. The in vivo closed-loop 3D printing method based on multimodal markers according to claim 4, characterized in that, The closed-loop control described in step S6 employs proportional-derivative control, proportional-integral-derivative control, or model predictive control methods, and its control input satisfies: ; in and These are the proportional coefficient and the differential coefficient, respectively. For disturbance compensation weights, This represents a disturbance term caused by respiration, tissue peristalsis, or abdominal wall displacement; the update frequency of the closed-loop control is not less than 30Hz; Step S6 is the in-vivo printing execution and adaptive path tracking. Based on the control signals output from steps S4 and S5, the print head (4) is driven to move along a preset path, and dynamic correction is performed through a real-time path tracking algorithm to ensure that the error between the target position and the actual position in S5 meets the following requirements: |。 10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the in vivo closed-loop 3D printing method based on multimodal marking as described in any one of claims 4-9.

Images