Visual servo-based in-vivo restoration system and restoration control method

Abstract

The invention discloses an in-vivo repair system based on visual servo and a repair control method. The system part comprises a cooperative mechanical arm, a material extrusion system, a depth camera and a three-dimensional scanner. Before repairing, surface three-dimensional information of a to-be-repaired part is obtained by means of a three-dimensional scanner and a positioning mark point and is converted into a corresponding point cloud model to be imported into a system program. And in cooperation with a depth camera carried by a cooperative mechanical arm, a repairing plane can be positioned in real time, an extrusion track is corrected, and biological three-dimensional printing closed-loop control is achieved. According to the system, the problem of real-time in-vivo three-dimensional repair on a complex surface and a non-static surface is solved, and the stability and the dynamic response capability of a printing system can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of biological three-dimensional printing and medical repair equipment, and in particular to an in-body repair system and a repair control method based on visual servoing. Background Art

[0002] Additive manufacturing technology, commonly known as 3D printing technology, originated in the 1980s and is an advanced technology for rapid prototyping. Compared with traditional manufacturing methods, 3D printing can process more precise and complex structures, and the processing time only depends on the size of the model itself, which greatly shortens the time and process from product design to production, and is suitable for highly customized structural manufacturing.

[0003] Visual servo control refers to a method of controlling the movement of a manipulator or robot using image data obtained by a computer vision system. These image data are usually collected in real time by a camera installed on the robot. The control system can accurately perceive the position, shape and posture information of the object through real-time processing and analysis of the input image, and try to feed it back to the system controller to achieve closed-loop control and reduce system errors. Visual servo control methods are mainly divided into two categories: image-based visual servo control (IBVS) and posture / position-based visual servo control (PBVS).

[0004] However, traditional 3D printing technology still has major problems in dealing with the needs of in-vivo repair of complex 3D surfaces and non-static surfaces, mainly due to the following reasons:

[0005] 1. Non-static complex surface: The surface of biological tissue is usually a non-rigid surface with a complex surface shape. It is also affected by biological rhythms and structures. During the repair process of biological tissue, it is inevitable that there will be a certain degree of movement and deformation, which will greatly affect the in vivo repair and printing effects. The printing platform should be able to handle these non-rigid surfaces.

[0006] 2. High-precision positioning requirements: Printing on the surface of biological objects requires the equipment to have high printing accuracy to ensure that the printing effect meets expectations while avoiding mechanical damage.

[0007] 3. Real-time requirements: The printing equipment must be able to quickly respond to changes in the target surface morphology and be able to adjust the printing strategy in real time when the surface changes, including path planning and optimization of printing depth, to ensure that the material can accurately cover the repair area, thereby achieving the best repair effect.

[0008] In the existing technology, open systems are usually used for biomaterial printing. This solution relies on pre-programming and path planning, which cannot meet the needs of real-time in vivo repair. With the development of machine vision and robot visual servo control, a closed-loop control system based on vision can be realized, thereby providing accurate repair of complex surfaces. Summary of the invention

[0009] In view of the above problems, the present invention proposes an in-body repair system and repair control method based on visual servoing, which realizes closed-loop control through machine vision related technologies in conjunction with a robotic arm, and realizes real-time repair functions on complex and non-static surfaces, with the characteristics of dynamic response and high precision.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] An in vivo repair system based on visual servoing includes a collaborative robot arm, a depth camera, a material extrusion system, a three-dimensional scanner, marking points and a checkerboard marking plate. The end of the collaborative robot arm is equipped with a depth camera and a material extrusion system for controlling the printing trajectory during the repair process according to the depth information of the target surface. The depth camera is installed at the end of the collaborative robot arm to collect the depth data of the target surface in real time, generate point cloud information and transmit it to the system controller; the material extrusion system is integrated with the collaborative robot arm and is used to adjust the extrusion amount of the biomaterial according to the instructions and configuration parameters of the system controller to achieve a stable distribution of the material. The three-dimensional scanner is used to scan the target surface before repair, generate a three-dimensional point cloud model and transmit it to the system controller to plan an initial repair path; the marking point is set on the area to be repaired and is used as a reference point for positioning and calibration of the depth camera and the structured light three-dimensional scanner; the checkerboard marking plate is used to calibrate the system before repair.

[0012] As a further improvement of the system of the present invention, the depth camera moves synchronously with the collaborative robot arm and can adjust the viewing angle and collect depth information at any time according to the position changes of the target surface.

[0013] As a further improvement of the system of the present invention, the material extrusion system adjusts the extrusion flow rate of the biomaterial by real-time pressure control to adapt to the dynamic changes of the curvature and depth information of the target surface.

[0014] As a further improvement of the system of the present invention, the three-dimensional scanner performs a complete scan of the target surface before the repair operation, and the generated three-dimensional point cloud model is used to assist the system controller in preliminarily planning the repair path and serves as a positioning reference for subsequent depth camera (2) acquisition.

[0015] As a further improvement of the system of the present invention, the marking points are designed with high-contrast patterns or colors and are evenly distributed around the repaired area, and can be accurately identified by a depth camera and a three-dimensional scanner.

[0016] As a further improvement of the system of the present invention, during the repair process, the printing trajectory of the collaborative robot arm and the material output of the material extrusion system are automatically adjusted according to the target surface change information fed back in real time by the depth camera to achieve closed-loop control.

[0017] As a further improvement of the system of the present invention, the system is suitable for complex and dynamic surface repair scenarios, including in vivo tissue repair, orthopedic implant surface repair, and the production of personalized medical devices.

[0018] The present invention provides a repair control method of an in-body repair system based on visual servoing, and the specific steps are as follows:

[0019] S1: Obtain a 3D point cloud model of the surface of the repaired object, obtain the 3D point cloud of the object surface containing the texture information of the marker points with the help of a 3D scanner, and use the point cloud editing software for preprocessing, select the appropriate sampling density, delete some frames with poor scanning effects, and splice the remaining frame point clouds to obtain more accurate object surface information;

[0020] S2: Generate a repair path suitable for the surface, call the contour and fill path generated by the slicing software, calculate the path suitable for the current repair surface, and process the generated path to make it smooth to meet the acceleration and deceleration requirements of the robot arm during movement. The new path information will be uploaded to the storage medium for use in body repair;

[0021] S3: Obtain the depth camera video stream and depth stream. After aligning the depth camera video frame with the depth frame, output the depth information of the specified pixel, and further obtain accurate point cloud information;

[0022] S4: Run the marker point extraction and matching algorithm, and use multi-indicator comprehensive evaluation to deal with occlusion and false positive situations. All detected marker points will be screened and matched to obtain the optimal set of parameters for reconstruction;

[0023] S5: Perform real-time 3D reconstruction of the object according to the optimal parameters to obtain the current position and posture of the object, and obtain the motion information of the object in combination with the motion information of the robot arm;

[0024] S6: Input the object posture into the motion control and prediction algorithm for trajectory correction to obtain the motion trajectory of the extrusion nozzle.

[0025] As a further improvement of the method of the present invention, the step S1 arranges visual marking points near the repair area, specifically as follows:

[0026] The checkerboard marker plate 6 is used to obtain and calibrate the camera's internal parameters, and the calibration plate is used to complete the hand-eye calibration. In addition, four parameters need to be obtained: Represents the transformation matrix between the robot coordinate system and the base coordinate system, which is a known quantity that can be obtained from the robot arm system; Represents the transformation matrix from the camera coordinate system to the robot coordinate system. This needs to be calculated, but this transformation relationship remains unchanged during the movement of the robot. The transformation matrix from the camera coordinate system to the calibration plate coordinate system is also a known quantity that can be obtained by camera calibration; The matrix representing the calibration plate coordinate system to the base coordinate system is the final desired result. As long as the relative position of the robot arm and the calibration plate remains unchanged, this transformation matrix does not change. At this time, the coordinate system transformation relationship between the depth camera (2) and the collaborative robot arm (1) is obtained. The specific calculation process is as follows;

[0027] First, control the robot arm to move from the initial position to position 1 and position 2, and establish the position formulas of position 1 and position 2. Since the robot arm coordinate system and the calibration plate coordinate system are relatively fixed, the transformation matrix between the two remains unchanged. Solve the transformation relationship between the camera and tool coordinate systems:

[0028]

[0029]

[0030] Combining equation (1) and equation (2), we can get:

[0031]

[0032] Eliminate P cal Arrange to the same side and simplify:

[0033]

[0034] in Known, It can be obtained by camera calibration. is the transformation matrix from the end tool coordinate system to the camera. Since we know the structure of the robot arm and calibrate the tool coordinate system, the transformation between the robot arm coordinate system and the tool coordinate system is known, so we can calculate The above algorithm is implemented using code to complete the calibration work.

[0035] As a further improvement of the method of the present invention, the step S3: obtaining the video stream and the depth stream of the depth camera, aligning the video frame of the depth camera with the depth frame, outputting the depth information of the specified pixel, and further obtaining accurate point cloud information;

[0036] First, we need to find the elliptical marker points on the arm in the camera image. Next, we need to calculate the transformation matrix T. bThe ellipse is restored to a circle for subsequent positioning and attitude estimation. Here we find T b Using the following method, assume a matrix T y , the eigenvalues ​​are λ1, λ2, λ3, then their eigenvectors are η1, η2, η3 respectively, and we get:

[0037] T y =VΛV T (6)

[0038]

[0039] Assuming two variables m and n, the rotation matrix T b It is expressed as:

[0040]

[0041] Here θ represents any rotation angle around the normal of the marking plane. Theoretically, the rotation angle θ represents the transformation matrix that transforms the ellipse into a circle in any rotation state. Simplifying it to 0 reduces the amount of calculation and reduces the calculation error. At this time, T b for:

[0042]

[0043] Here, m and n can only be 1 and -1, and there will be 4 different rotation matrices corresponding to the four different directions when transforming the ellipse into a circle. In actual scenarios, since the marker is planar, each detected ellipse needs to be evaluated to determine how the ellipse can be better transformed into a circle under these two possible directions, so as to further determine the position and posture information of the marker.

[0044] Beneficial effects: The visual servo closed-loop in-body repair system of the present invention has the following beneficial effects:

[0045] 1. The system has remarkable precision repair capabilities. Through the integrated control of collaborative robotic arms, depth cameras, 3D scanners, material extrusion devices and visual markers, the system can achieve precise positioning and operation of the area to be repaired. The system controller can dynamically adjust the trajectory of the robotic arm and the amount of air pressure extrusion based on feedback data to achieve precise coverage of biomaterials, avoid material waste, and significantly improve repair accuracy.

[0046] 2. In terms of dynamic response and closed-loop control, this system demonstrates excellent performance. The closed-loop control scheme based on visual servoing enables this system to have real-time dynamic response capabilities on non-static or complex surfaces. The depth camera outputs RGB and depth video streams at a fixed frame rate to calculate the depth information of the target surface. The controller instantly adjusts the operating parameters of the robot arm and extrusion system based on the real-time feedback of the printing plane posture to ensure that the preset trajectory can be correctly printed on the changing curved surface. This closed-loop control gives the system strong adaptability and effectively overcomes the operational instability limitations of traditional bioprinting technology in dynamic environments.

[0047] 3. The system has higher stability and reliability. The marking points provide precise positioning benchmarks and constraints to ensure the positioning accuracy during the repair process, and with the help of redundant markings, it can effectively deal with problems such as occlusion, missing markings, and recognition errors caused by the three-dimensional movement of objects. The system has good stability in both data acquisition and execution stages, making the repair effect more uniform and avoiding errors caused by external interference or surface movement to a certain extent. This high-stability design is particularly suitable for biological three-dimensional printing and in-body repair applications of medical repair equipment, which improves the overall reliability of the system.

[0048] 4. The system has obvious advantages in improving repair efficiency and reducing material waste. The material system extrusion can provide a constant extrusion flow rate during the printing process to ensure the consistency of the extrusion line width, and can be adjusted according to the stored material printing parameters, so that the system can adapt to the printing needs of various materials while reducing material waste.

[0049] 5. The system has wide applicability. The visual servo printing system does not require implanting markers or performing harmful operations inside the organism. It can be applied to a variety of repair scenarios, such as epidermal tissue defect repair, open wound repair, in vivo repair of bone defects, etc.

[0050] 6. The system has a high degree of automation. The system can access mature slicing engines such as 3DSlicer, Simplify3D, etc. to assist in the planning and generation of structures and paths. After completing this step, the system will use the surface information provided by the 3D scanner to automatically calculate and generate a conformal printing trajectory suitable for the surface. During the repair process, the depth camera automatically implements trajectory waypoint correction and closed-loop control without manual intervention. In the event of serious errors such as target loss or system failure, it will automatically stop printing and alarm. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the calibration working principle required before the system of the present invention is operated;

[0052] Figure 2 This is a structural diagram of the visual servo closed-loop in-body repair system of the present invention;

[0053] Figure 3 A flowchart of the steps of performing real-time in vivo repair according to the present invention;

[0054] The reference numerals are as follows:

[0055] 1. Collaborative robotic arm; 2. Depth camera; 3. Material extrusion system; 4. 3D scanner; 5. Marking points; 6. Checkerboard marking board. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0057] The present invention discloses an in vivo repair system based on visual servoing, and the system comprises: a collaborative robot arm 1, at the end of which a depth camera and a material extrusion system are installed, and the system is used to control the printing trajectory in the repair process according to the depth information of the target surface; a depth camera 2, which is installed at the end of the collaborative robot arm, collects the depth data of the target surface in real time, generates point cloud information and transmits it to the system controller; a material extrusion system 3, which is integrated with the collaborative robot arm, and is used to adjust the extrusion amount of the biomaterial according to the instructions and configuration parameters of the system controller to achieve a stable distribution of the material; a three-dimensional scanner 4, which is used to scan the target surface before repair, generate a three-dimensional point cloud model and transmit it to the system controller to plan an initial repair path; a marking point 5, which is set on the area to be repaired and is used as a reference point for positioning and calibration of the depth camera and the structured light three-dimensional scanner; a checkerboard marking plate 6, which is used to calibrate the system before repair.

[0058] The depth camera 2 described in the present invention operates synchronously with the collaborative robot arm 1, and can adjust the viewing angle and collect depth information at any time according to the position change of the target surface.

[0059] The material extrusion system 3 of the present invention adjusts the extrusion flow rate of the biomaterial by real-time pressure control to adapt to the dynamic changes of the curvature and depth information of the target surface.

[0060] The three-dimensional scanner 4 of the present invention performs a complete scan of the target surface before the repair operation, and the generated three-dimensional point cloud model is used to assist the system controller in preliminarily planning the repair path and serves as a positioning reference for subsequent depth camera 2 acquisition.

[0061] The marking points 5 of the present invention are designed with high contrast patterns or colors and are evenly distributed around the repaired part, and can be accurately identified by a depth camera and a three-dimensional scanner.

[0062] During the repair process of the present invention, the printing trajectory of the collaborative robot arm (1) and the material output of the material extrusion system 3 are automatically adjusted according to the target surface change information fed back in real time by the depth camera, thereby realizing closed-loop control.

[0063] The system of the present invention is suitable for complex and dynamic surface repair scenarios, including in vivo tissue repair, orthopedic implant surface repair, and the production of personalized medical devices.

[0064] like Figure 1 As shown in the figure, first of all, the system needs to be calibrated before it runs, including the calibration of the depth camera coordinate system, the robot arm coordinate system, and the scanner coordinate system. Specifically, 6 (a checkerboard calibration board consisting of multiple rows of black and white squares with known side lengths) is used to obtain and calibrate the camera's internal parameters, and the calibration board is used to complete the hand-eye calibration. In addition, four parameters need to be obtained. Represents the transformation matrix between the robot coordinate system and the base coordinate system, which is a known quantity that can be obtained from the robot arm system; Represents the transformation matrix from the camera coordinate system to the robot coordinate system. This needs to be calculated, but this transformation relationship remains unchanged during the movement of the robot. The transformation matrix (camera extrinsic parameter) from the camera coordinate system to the calibration plate coordinate system is also a known quantity that can be obtained by camera calibration. The matrix representing the calibration plate coordinate system to the base coordinate system is the final desired result. As long as the relative position of the robot arm and the calibration plate remains unchanged, this transformation matrix does not change. At this point, the coordinate system transformation relationship between the depth camera 2 and the collaborative robot arm 1 can be obtained. Combination Figure 1 Its specific calculation process is as follows.

[0065] First, control the robot arm to move from the initial position to position 1 and 2. Combine the two equations. Since the robot arm coordinate system and the calibration plate coordinate system are relatively fixed, the transformation matrix between the two remains unchanged. The transformation relationship between the camera and tool coordinate systems can be solved:

[0066]

[0067] Combining equations 1 and 2, we can get:

[0068]

[0069] Eliminate P cal Arrange to the same side and simplify:

[0070]

[0071] in Known, It can be obtained by camera calibration. is the transformation matrix from the end tool coordinate system to the camera. Since we know the structure of the robot arm and calibrate the tool coordinate system, the transformation between the robot arm coordinate system and the tool coordinate system is known (TCP calibration), so we can calculate The above algorithm is implemented using code to complete the calibration work.

[0072] like Figure 2 As shown, the visual servo closed-loop on-body repair system of the present invention is composed of a collaborative robot arm 1, a depth camera 2, a material extrusion system 3, a three-dimensional scanner 4 and a marking point 5.

[0073] Among them, the collaborative robot arm 1 is the actuator of the entire system, which has the ability to adjust three degrees of freedom. It is connected to the controller through the network port and can accept motion control instructions from the controller (based on TCP byte stream).

[0074] The depth camera 2 is fixedly mounted on the end of the collaborative robot arm 1 with the help of a bracket. The two can be regarded as a rigid connection, and the relative position relationship between the two can be considered unchanged during the movement. The depth camera can output RGB video stream and depth stream at the same time, and the resolution and frame rate settings can be adjusted as needed. After the depth camera 2 obtains the depth information of the target surface, it transmits the data to the control system, which can use this transformation relationship to accurately control the robot arm.

[0075] The material extrusion system 3 is also installed at the end of the robot arm and cooperates with the depth camera 2 to achieve dynamic printing control. The system adjusts the output of the biomaterial through the end effector, and the extrusion amount is adjusted according to the depth information of the target surface and real-time feedback to ensure that the material covers the repair area well. The material extrusion system 3 is electrically connected to the system controller and is connected to the system reference air pressure source air circuit to achieve precise control of pressure and extrusion speed.

[0076] The marker points 5 are placed in the area to be repaired, which can be accurately identified by the depth camera 2 and the 3D scanner 4, and used to estimate and reconstruct the 3D parameters of the current object to be repaired. The marker points 5 can ensure the real-time tracking and positioning of the dynamic target surface by the system during the entire repair process, and can cope with the false positive interference caused by factors such as occlusion and illumination changes caused by the movement of objects or robotic arms. The following implementation method:

[0077] We know that the projection transformation that commonly occurs during visual inspection will cause a circle to appear in the form of an ellipse. First, we need to find the elliptical marker points on the arm in the camera image (that is, the circular points before the projection transformation). The next step is to calculate the transformation matrix T b The ellipse is restored to a circle for subsequent positioning and attitude estimation. Here we find T b We adopt a more common method, assuming a matrix Ty , the eigenvalues ​​are λ1, λ2, λ3, then their eigenvectors are η1, η2, η3 respectively, we can get:

[0078] T y =VΛV T (6)

[0079]

[0080] Assuming two variables m and n, the rotation matrix T b It can be expressed as:

[0081]

[0082] Here θ represents any rotation angle around the normal of the marking plane. Theoretically, the rotation angle θ can represent the transformation matrix that transforms the ellipse into a circle in any rotation state. However, since this paper focuses on marker detection, positioning, and transforming the ellipse into a circle and performing subsequent operations, there is no need to consider the complete rotation information represented by θ. Simplifying it to 0 reduces the amount of calculation and reduces the calculation error, which helps to improve the efficiency and stability of the entire algorithm in practical applications. At this time, T b for:

[0083]

[0084] Where m and n can only be 1 and -1, so there are four different combinations, and there will be 4 different rotation matrices corresponding to the four different directions of transforming the ellipse into a circle. In the actual scene, since the marker is flat, two of the directions will cause the marker's direction to be inconsistent with the camera's line of sight (for example, it will make the marker look upside down or tilted too much, which does not conform to the reasonable posture of the marker relative to the camera in the actual physical scene). These can be directly excluded, and the rotation matrices corresponding to the remaining two directions (T b1 With T b2 ) is reasonable, and each detected ellipse needs to be evaluated to determine how the ellipse can be better transformed into a circle under these two possible directions, so as to further determine the position, posture and other information of the marker.

[0085] For all detected ellipse pairs, their rotation (i.e., T b1 With T b2 ) to form four feasible rotation pairs, and then filter out the rotation pairs whose inner product is higher than a fixed threshold by calculating the inner product of the two rotation matrices in the rotation pair. This can avoid selecting ellipse pairs with inconsistent directions, such as Figure 2 As shown on the right, because the marker is flat, the directions of its points should be consistent to a certain extent, so as to select the correct transformation matrix T bThen the desired posture is obtained through the verification code.

[0086] The system controller receives the real-time data from the depth camera 2 and the point cloud model of the initial scan, performs data fusion and generates control instructions, comprehensively analyzes the depth information and initial path model of the target surface, and adjusts the position of the robotic arm, the extrusion speed of the extrusion system and the material distribution through feedback, so that the system has the ability to adapt in a dynamic environment and ensure that the material accurately covers the target area.

[0087] Figure 3 The control logic of the real-time in vivo repair of the present invention includes the following steps:

[0088] S1: Obtain a 3D point cloud model of the surface of the repaired object. Use a 3D scanner to obtain a 3D point cloud of the object surface containing the texture information of the marker points, and use point cloud editing software for preprocessing, select a suitable sampling density, delete some frames with poor scanning effects, and splice the remaining frame point clouds to obtain more accurate object surface information.

[0089] S2: Generate a repair path suitable for the surface, call the contour and fill path generated by the slicing software, calculate the path suitable for the current repair surface, and process the generated path to make it smooth to meet the acceleration and deceleration requirements of the robot arm during movement. The new path information will be uploaded to the storage medium for use in body repair.

[0090] S3: Get the depth camera video stream and depth stream. After aligning the depth camera video frame with the depth frame, the depth information of the specified pixel can be output, and more accurate point cloud information can be obtained.

[0091] S4: Run the marker point extraction and matching algorithm, and use multi-indicator comprehensive evaluation to deal with occlusion and false positive situations. All detected marker points will be screened and matched to obtain the optimal set of parameters for reconstruction.

[0092] S5: Perform real-time 3D reconstruction of the object according to the optimal parameters to obtain the current position and posture of the object, and obtain the motion information of the object in combination with the motion information of the robotic arm.

[0093] S6: Input the object posture into the motion control and prediction algorithm for trajectory correction to obtain the motion trajectory of the extrusion nozzle.

[0094] Of course, the system components may also include fewer or more components, which is not limited in this embodiment.

[0095] The optional application of the present invention also provides a computer-readable storage medium, in which a computer and a controller can read the program stored, and the program is loaded by a processor to execute the real-time in vivo repair method disclosed in the present invention.

[0096] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. An in vivo repair system based on visual servoing, comprising a collaborative robot arm (1), a depth camera (2), a material extrusion system (3), a three-dimensional scanner (4), a marking point (5) and a checkerboard marking plate (6), characterized in that: A depth camera (2) and a material extrusion system (3) are installed at the end of the collaborative robot arm (1) for controlling the printing trajectory during the repair process according to the depth information of the target surface. The depth camera (2) is installed at the end of the collaborative robot arm (1) to collect the depth data of the target surface in real time, generate point cloud information and transmit it to the system controller; the material extrusion system (3) is integrated with the collaborative robot arm (1) and is used to adjust the extrusion amount of the biomaterial according to the instructions and configuration parameters of the system controller to achieve a stable distribution of the material; the three-dimensional scanner (4) is used to scan the target surface before repair, generate a three-dimensional point cloud model and transmit it to the system controller to plan an initial repair path; the marking point (5) is set on the area to be repaired and used as a reference point for positioning and calibration of the depth camera and the structured light three-dimensional scanner; the checkerboard marking plate (6) is used to calibrate the system before repair.

2. The in vivo repair system based on visual servoing according to claim 1, characterized in that: The depth camera (2) moves synchronously with the collaborative robot arm (1).

3. The in vivo repair system based on visual servoing according to claim 1, characterized in that: The material extrusion system (3) adjusts the extrusion flow rate of the biomaterial by controlling the pressure in real time.

4. The in-vivo repair system and repair control method based on visual servoing according to claim 1, characterized in that: The three-dimensional scanner (4) performs a complete scan on the target surface before the repair operation, and the generated three-dimensional point cloud model is used to assist the system controller in preliminarily planning the repair path and serves as a positioning reference for subsequent depth camera (2) acquisition.

5. The in-vivo repair system and repair control method based on visual servoing according to claim 1, characterized in that: The marking points (5) are designed with high contrast patterns or colors.

6. The repair control method of the in-body repair system based on visual servoing according to any one of claims 1 to 5, characterized in that: The specific steps are as follows; S1: Obtain a 3D point cloud model of the surface of the repaired object, obtain the 3D point cloud of the object surface containing the texture information of the marker points with the help of a 3D scanner, and use the point cloud editing software for preprocessing, select the appropriate sampling density, delete some frames with poor scanning effects, and splice the remaining frame point clouds to obtain more accurate object surface information; S2: Generate a repair path suitable for the surface, call the contour and fill path generated by the slicing software, calculate the path suitable for the current repair surface, and process the generated path to make it smooth to meet the acceleration and deceleration requirements of the robot arm during movement. The new path information will be uploaded to the storage medium for use in body repair; S3: Obtain the depth camera video stream and depth stream. After aligning the depth camera video frame with the depth frame, output the depth information of the specified pixel, and further obtain accurate point cloud information; S4: Run the marker point extraction and matching algorithm, and use multi-indicator comprehensive evaluation to deal with occlusion and false positive situations. All detected marker points will be screened and matched to obtain the optimal set of parameters for reconstruction; S5: Perform real-time 3D reconstruction of the object according to the optimal parameters to obtain the current position and posture of the object, and obtain the motion information of the object in combination with the motion information of the robot arm; S6: Input the object posture into the motion control and prediction algorithm for trajectory correction to obtain the motion trajectory of the extrusion nozzle.

7. The repair control method of the in-vivo repair system based on visual servoing according to claim 6 is characterized in that The step S1 arranges visual marking points near the repair area, specifically as follows: The checkerboard marker plate 6 is used to obtain and calibrate the camera's internal parameters, and the calibration plate is used to complete the hand-eye calibration. In addition, four parameters need to be obtained: Represents the transformation matrix between the robot coordinate system and the base coordinate system, which is a known quantity that can be obtained from the robot arm system; Represents the transformation matrix from the camera coordinate system to the robot coordinate system. This needs to be calculated, but this transformation relationship remains unchanged during the movement of the robot. The transformation matrix from the camera coordinate system to the calibration plate coordinate system is also a known quantity that can be obtained by camera calibration; The matrix representing the calibration plate coordinate system to the base coordinate system is the final desired result. As long as the relative position of the robot arm and the calibration plate remains unchanged, this transformation matrix does not change. At this time, the coordinate system transformation relationship between the depth camera (2) and the collaborative robot arm (1) is obtained. The specific calculation process is as follows; First, control the robot arm to move from the initial position to position 1 and position 2, and establish the position formulas of position 1 and position 2. Since the robot arm coordinate system and the calibration plate coordinate system are relatively fixed, the transformation matrix between the two remains unchanged. Solve the transformation relationship between the camera and tool coordinate systems: Combining equation (1) and equation (2), we can get: Eliminate P cal Arrange to the same side and simplify: in Known, It can be obtained by camera calibration. is the transformation matrix from the end tool coordinate system to the camera. Since we know the structure of the robot arm and calibrate the tool coordinate system, the transformation between the robot arm coordinate system and the tool coordinate system is known, so we can calculate The above algorithm is implemented using code to complete the calibration work.

8. The repair control method of the in-vivo repair system based on visual servoing according to claim 6 is characterized in that The step S3: obtaining the video stream and depth stream of the depth camera, aligning the video frame of the depth camera with the depth frame, outputting the depth information of the specified pixel, and further obtaining accurate point cloud information; First, we need to find the elliptical marker points on the arm in the camera image. Next, we need to calculate the transformation matrix T. b The ellipse is restored to a circle for subsequent positioning and attitude estimation. Here we find T b Using the following method, assume a matrix T y , the eigenvalues ​​are λ1, λ2, λ3, then their eigenvectors are η1, η2, η3 respectively, and we get: T y =VΛV T (6) Assuming two variables m and n, the rotation matrix T b It is expressed as: Here θ represents any rotation angle around the normal of the marking plane. Theoretically, the rotation angle θ represents the transformation matrix that transforms the ellipse into a circle in any rotation state. Simplifying it to 0 reduces the amount of calculation and reduces the calculation error. At this time, T b for: Here, m and n can only be 1 and -1, and there will be 4 different rotation matrices corresponding to the four different directions when transforming the ellipse into a circle. In actual scenarios, since the marker is planar, each detected ellipse needs to be evaluated to determine how the ellipse can be better transformed into a circle under these two possible directions, so as to further determine the position and posture information of the marker.

Images