In-vivo repair system and repair control method based on visual servoing

Abstract

The system part includes a collaborative robot, a material extrusion system, a depth camera, and a three-dimensional scanner. Before repair, the three-dimensional information of the surface of the part to be repaired is obtained by means of the three-dimensional scanner and the positioning mark point, and is converted into a corresponding point cloud model and imported into the system program. The depth camera carried by the collaborative robot can locate the repair plane in real time and correct the extrusion trajectory, realizing closed-loop control of biological three-dimensional printing. The system solves the problem of real-time in-vivo three-dimensional repair on complex surfaces and non-stationary surfaces, and can improve the stability and dynamic response capability of the printing system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of biological three-dimensional printing and medical repair equipment, and particularly relates to an in-vivo repair system based on visual servoing and a repair control method. BACKGROUND

[0002] Additive manufacturing technology, commonly known as 3D printing technology, originated in the 1980s and is an advanced technology of rapid prototyping. Compared with traditional manufacturing methods, 3D printing can process more precise and complex structures, and the processing time only depends on the size of the model itself, greatly shortening the time from product design to production and the process, and is suitable for highly customized structure manufacturing.

[0003] Visual servoing control is a method of using image data obtained by a computer vision system to control the movement of a robotic arm or robot. These image data are usually collected in real time by a camera installed on the robot. Through real-time processing and analysis of the input image, the control system can accurately perceive the position, shape and attitude information of the object, and feedback to the system controller to achieve closed-loop control and reduce system error. Visual servoing control methods are mainly divided into two categories: image-based visual servoing control (IBVS) and pose / position-based visual servoing control (PBVS).

[0004] However, traditional three-dimensional printing technology still has great problems in dealing with the demand for in-vivo repair of complex three-dimensional surfaces and non-stationary surfaces, mainly due to the following reasons:

[0005] 1. Non-stationary complex surface: The surface of a biological tissue is usually a non-rigid surface, and the shape of the surface is usually complex, and is affected by biological rhythms and structures. During the repair process of the biological tissue, there will inevitably be a certain degree of movement and deformation, which will greatly affect the in-vivo repair and printing effect. The printing platform should have the ability to handle these non-rigid surfaces.

[0006] 2. High precision positioning requirement: Printing on the surface of a biological body requires the equipment to have high printing precision to ensure that the printing effect meets the expectations, while avoiding mechanical damage.

[0007] 3. Real-time requirement: The printing equipment must have the ability to quickly respond to changes in the target surface morphology and adjust the printing strategy in real time when the surface changes, including path planning and optimization of printing depth, to ensure that the material can accurately cover the repair area, thereby achieving the best repair effect.

[0008] In the prior art, open systems are usually used for biological material printing, which relies on pre-off-line programming and path planning and cannot meet the needs of real-time in-vivo repair. With the development of machine vision and robot vision servo control, a closed-loop control system based on vision can be realized, thereby providing accurate repair of complex surfaces. SUMMARY

[0009] To solve the above problems, the application provides an in-vivo repair system based on visual servoing and a repair control method. The system realizes real-time repair on complex and non-stationary surfaces through closed-loop control of machine vision-related technologies and a robot arm, and has dynamic response and high precision.

[0010] To achieve the above object, the technical solution adopted by the application is as follows:

[0011] The in-vivo repair system based on visual servoing comprises a collaborative robot arm, a depth camera, a material extrusion system, a three-dimensional scanner, a marker point, and a checkerboard marker plate. The depth camera and the material extrusion system are installed at the end of the collaborative robot arm, and are used to control the printing trajectory during the repair process according to the target surface depth information. The depth camera is installed at the end of the collaborative robot arm, and is used to collect depth data of the target surface in real time, generate point cloud information, and transmit the point cloud information to a system controller. The material extrusion system is integrated with the collaborative robot arm, and is used to adjust the extrusion amount of biological materials according to the instructions and configuration parameters of the system controller, so as to realize stable distribution of the materials. The three-dimensional scanner is used to scan the target surface before repair, generate a three-dimensional point cloud model, and transmit the three-dimensional point cloud model to the system controller to plan an initial repair path. The marker point is arranged on the to-be-repaired area, and is used as a reference point for positioning and calibration of the depth camera and the structured light three-dimensional scanner. The checkerboard marker plate is used to calibrate the system before repair.

[0012] As a further improvement of the system of the application, the depth camera and the collaborative robot arm move synchronously, and can adjust the viewing angle and collect depth information at any time according to the position change of the target surface.

[0013] As a further improvement of the system of the application, the material extrusion system adjusts the extrusion flow of biological materials by real-time control of pressure, so as to adapt to the dynamic changes of the target surface curvature and depth information.

[0014] As a further improvement of the system of the application, the three-dimensional scanner scans the target surface completely before repair, and the generated three-dimensional point cloud model is used to assist the system controller to preliminarily plan a repair path, and serves as a positioning reference for subsequent collection of the depth camera (2).

[0015] As a further improvement of the system of the application, the marker point is designed as a high-contrast pattern or color, and is uniformly distributed around the repair site, and can be accurately recognized by the depth camera and the three-dimensional scanner.

[0016] As a further improvement of the system of the application, the target surface change information is automatically adjusted according to the real-time feedback of the depth camera during the repair process, the printing trajectory of the collaborative robot arm and the material output of the material extrusion system are automatically adjusted, and closed-loop control is realized.

[0017] As a further improvement of the system of the application, the system is suitable for complex and dynamic surface repair scenarios, including in-vivo tissue repair, orthopedic implant surface repair, and personalized medical device manufacturing.

[0018] The application provides a repair control method for an in-vivo repair system based on visual servoing, and the specific steps are as follows:

[0019] S1: Obtain the three-dimensional point cloud model of the repaired object surface, obtain the three-dimensional point cloud of the object surface containing the marker point texture information by means of a three-dimensional scanner, and pre-process it by using a point cloud editing software, select a suitable sampling density, delete some frames with poor scanning effect, and splice the remaining frame point clouds to obtain more accurate object surface information;

[0020] S2: Generate a repair path suitable for the surface, call the contour and filling path generated by the slicing software, calculate the path suitable for the current repair surface, and process the generated path to make it smooth, meeting the acceleration and deceleration requirements of the robot arm movement. The new path information will be uploaded to the storage medium for use in in-vivo repair;

[0021] S3: Obtain the depth camera video stream and depth stream, align the video frame and depth frame of the depth camera, output the depth information of the specified pixel, and further obtain accurate point cloud information;

[0022] S4: Run the marker point extraction and matching algorithm, adopt multi-index comprehensive judgment to deal with the occlusion and false positive situations, and all detected marker points will be screened and matched to obtain the optimal set of parameters for reconstruction;

[0023] S5: Perform real-time three-dimensional reconstruction of the object according to the optimal parameters, obtain the pose of the current object, and obtain the motion information of the object combined with the motion information of the robot arm;

[0024] S6: Input the object pose into the motion control and prediction algorithm for trajectory correction to obtain the motion trajectory of the extrusion nozzle.

[0025] As a further improvement of the method of the application, the step S1 arranges visual markers near the repair area, and the specific steps are as follows:

[0026] The checkerboard marker board 6 is used to obtain and correct the camera's intrinsic parameters, and the hand-eye calibration is completed by using the calibration board. In addition, four parameters need to be obtained, R represents the transformation matrix of the robot coordinate system to the base coordinate system, which is a known quantity and can be derived from the robot system; R represents the transformation matrix of the camera coordinate system to the robot coordinate system, which needs to be calculated but the transformation relationship does not change during the movement of the robot; R represents the transformation matrix of the camera coordinate system to the calibration board coordinate system, which is a known quantity and can be derived from camera calibration; R represents the transformation matrix of the calibration board coordinate system to the base coordinate system, which is the final result we want to obtain; as long as the relative position of the robot and the calibration board does not change, the transformation matrix does not change; at this time, the coordinate transformation relationship between the depth camera (2) and the collaborative robot (1) is obtained The specific calculation process is as follows:

[0027] First, control the robot to move from the initial position to position 1, position 2, and position 1 and position 2 are solved by position formula, since the robot coordinate system and the calibration board coordinate system are relatively fixed, the transformation matrix between them does not change, and the transformation relationship between the camera and the tool coordinate system is solved:

[0028]

[0029]

[0030] Solving equation (1) and equation (2) gives:

[0031]

[0032] Eliminate P cal Arrange to the same side and simplify:

[0033]

[0034] Wherein Given, can be derived from camera calibration, is the transformation matrix of the end tool coordinate system to the camera, since we know the structure of the robot, the calibration of the tool coordinate system is performed, the transformation of the robot coordinate system and the tool coordinate system is known, and then can be solved The above algorithm is realized by code to complete the calibration work.

[0035] As a further improvement of the method of the application, in step S3: obtaining the depth camera video stream and the depth stream, the video frame and the depth frame of the depth camera are aligned, and the depth information of the specified pixel is output, and accurate point cloud information can be further obtained;

[0036] First, the elliptical marker point on the arm needs to be found in the camera image, and then the transformation matrix T is calculated bEllipse is reduced to circle for subsequent positioning and attitude estimation problem, here T b The following method is adopted, assuming a matrix T y , eigenvalues λ1, λ2, λ3, then their eigenvectors are η1, η2, η3, obtain:

[0037] T y = VΛV T (6)

[0038]

[0039] Assuming two variables m, n, the rotation matrix T b Is expressed as:

[0040]

[0041] Here θ represents an arbitrary rotation angle around the normal of the marker plane, theoretically, the rotation angle θ represents the transformation matrix that transforms the ellipse into a circle in any rotation state, and simplifying it to 0 reduces the calculation amount and reduces the calculation error, then T b For:

[0042]

[0043] Where m, n can only take 1 and -1, there will be four different rotation matrices corresponding to the four different directions when transforming the ellipse into a circle, in the actual scene, since the marker is planar, each detected ellipse needs to be evaluated to determine how the ellipse is better transformed into a circle in the two possible directions, so as to further determine the position and attitude information of the marker.

[0044] Beneficial effects: the visual servo closed loop of the in-vivo repair system has the following beneficial effects:

[0045] 1、The system has significant accurate repair ability. Through the comprehensive control of the collaborative manipulator, the depth camera, the three-dimensional scanner, the material extrusion device and the visual marker point, the system can realize the accurate positioning and operation of the repair area, the system controller can dynamically adjust the manipulator trajectory and the air pressure extrusion amount according to the feedback data, realize the accurate coverage of biological materials, avoid material waste, and significantly improve the repair precision.

[0046] 2. In terms of dynamic response and closed-loop control, the system demonstrates excellent performance. The closed-loop control scheme based on visual servoing enables the system to have real-time dynamic response capability on non-stationary or complex surfaces. The depth camera outputs RGB and depth video streams at a fixed frame rate to calculate the depth information of the target surface. The controller adjusts the relevant operating parameters of the robotic arm and extrusion system based on real-time feedback of the printing plane pose to ensure correct printing of the preset trajectory on the changing curved surface. This closed-loop control gives the system strong adaptability and effectively overcomes the instability of traditional bio-printing technology in dynamic environments.

[0047] 3. The system has higher stability and reliability. The system provides accurate positioning reference and constraints through the marker points to ensure the accuracy of positioning during the repair process, and effectively deals with problems such as occlusion, marker loss, and recognition errors caused by three-dimensional motion of the object with the help of redundant markers. The system has good stability in both data acquisition and execution stages, making the repair effect more uniform and avoiding errors caused by external interference or surface movement to some extent. This high stability design is particularly suitable for bio-3D printing and in-vivo repair applications of medical repair equipment, improving the overall reliability of the system.

[0048] 4. The system has obvious advantages in improving repair efficiency and reducing material waste. The material system extrusion can provide a constant extrusion flow during printing to ensure the consistency of the extrusion line width, and can be adjusted according to the stored material printing parameters, so that the system can adapt to the printing needs of various materials while reducing material waste.

[0049] 5. The system has wide applicability. The visual servo printing system does not require the implantation of markers or harmful operations in the body, and can be applied to various repair scenarios such as epidermal tissue defect repair, open wound repair, and in-vivo repair of bone defect sites.

[0050] 6. The system has a high degree of automation. The system can access mature slicing engines such as 3DSlicer, Simplify3D, etc. to assist in structure and path planning and generation. After completing this step, the system will automatically calculate and generate conformal printing trajectories suitable for the surface based on the surface information provided by the three-dimensional scanner. During the repair process, the depth camera automatically corrects the trajectory waypoints and implements closed-loop control without human intervention. In the event of a serious error such as target loss or system failure, the system will automatically stop printing and alarm. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 Schematic diagram of the calibration principle required by the system of the present application before operation;

[0052] Figure 2 Structure diagram of the visual servo closed-loop in-vivo repair system of the present application;

[0053] Figure 3 Flow chart of steps for real-time in-vivo repair of the present application;

[0054] Reference signs are as follows:

[0055] 1, collaborative robot; 2, depth camera; 3, material extrusion system; 4, three-dimensional scanner; 5, marker point; 6, checkerboard marker plate. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0057] The in-vivo repair system based on visual servoing of the present application comprises: a collaborative robot 1, the end of which is provided with a depth camera and a material extrusion system, which is used to control the printing track in the repair process according to the depth information of the target surface; a depth camera 2, which is installed at the end of the collaborative robot, is used to collect the depth data of the target surface in real time, generate point cloud information and transmit it to the system controller; a material extrusion system 3, which is integrated with the collaborative robot, is used to adjust the extrusion amount of the biological material according to the instructions and configuration parameters of the system controller, so as to realize the stable distribution of the material; a three-dimensional scanner 4, which is used to scan the target surface before repair, generate a three-dimensional point cloud model and transmit it to the system controller to plan the initial repair path; a marker point 5, which is arranged on the to-be-repaired area, is used as a reference point for positioning and calibration of the depth camera and the structured light three-dimensional scanner; and a checkerboard marker plate 6, which is used to calibrate the system before repair.

[0058] The depth camera 2 and the collaborative robot 1 of the present application move synchronously, can adjust the viewing angle and collect the depth information at any time according to the position change of the target surface.

[0059] The material extrusion system 3 of the present application adjusts the extrusion flow of the biological material by real-time control of the pressure, so as to adapt to the dynamic change of the curvature and depth information of the target surface.

[0060] The three-dimensional scanner 4 of the present application scans the target surface completely before the repair operation, and the generated three-dimensional point cloud model is used to assist the system controller to preliminarily plan the repair path, and is used as the positioning reference of the subsequent depth camera 2.

[0061] The marker point 5 of the present application is designed as a high-contrast pattern or color, and is uniformly distributed around the repair site, which can be accurately recognized by the depth camera and the three-dimensional scanner.

[0062] The present application adjusts the printing track of the collaborative robot (1) and the material output of the material extrusion system 3 according to the real-time feedback of the target surface change information of the depth camera in the repairing process, and realizes closed-loop control.

[0063] The system is suitable for complex and dynamic surface repairing scenes, including in-vivo tissue repairing, orthopedic implant surface repairing and personalized medical instrument manufacturing.

[0064] As shown in Figure 1 , first, before the system runs, corresponding calibration work is needed, including calibration of the depth camera coordinate system, the robot coordinate system and the scanner coordinate system. Specifically, 6 (a chessboard calibration plate formed by alternately arranging multiple rows of black and white squares with known side length) is used to obtain and correct the camera's internal parameters, and the hand-eye calibration is completed by using the calibration plate. In addition, 4 parameters, represent the conversion matrix of the robot coordinate system and the base coordinate system, which is a known quantity and can be obtained from the robot system; represent the conversion matrix of the camera coordinate system to the robot coordinate system, which needs to be calculated, but this conversion relationship does not change during the movement of the robot; represent the conversion matrix of the camera coordinate system to the calibration plate coordinate system (camera external parameters), which is also a known quantity and can be obtained by camera calibration; represent the matrix of the calibration plate coordinate system to the base coordinate system, which is the final result we want to get; as long as the relative position of the robot and the calibration plate does not change, this conversion matrix does not change; at this time, the coordinate system conversion relationship between the depth camera 2 and the collaborative robot 1 can be obtained Combined Figure 1 its specific calculation process is as follows.

[0065] First, control the robot to move from the initial position to position 1, 2, and solve the two equations simultaneously. Since the robot coordinate system and the calibration plate coordinate system are relatively fixed, the conversion matrix between them does not change, and the conversion relationship between the camera and the tool coordinate system can be solved:

[0066]

[0067] Solve 1, 2 simultaneously:

[0068]

[0069] Eliminate P cal Arrange to the same side and simplify:

[0070]

[0071] Among them known, can be obtained by camera calibration, is the transformation matrix from the end tool coordinate system to the camera, since we know the structure of the robot arm, calibrate the tool coordinate system, the transformation of the robot coordinate system and the tool coordinate system is known (TCP calibration), then the transformation matrix T can be calculated The above algorithm is implemented by using code to complete the calibration work.

[0072] As shown in Figure 2 The visual servo closed loop in the body repair system of the application is composed of a collaborative robot arm 1, a depth camera 2, a material extrusion system 3, a three-dimensional scanner 4 and a marker point 5.

[0073] Among them, the collaborative robot arm 1 is the actuator of the whole system, which has three degrees of freedom of adjustment, and is connected to the controller through the network port, and can accept the motion control instruction (based on TCP byte stream) from the controller.

[0074] The depth camera 2 is fixedly installed at the end of the collaborative robot arm 1 by means of a support, and the relative position relationship between the two is considered unchanged during the movement. The depth camera can output RGB video stream and depth stream at the same time, and the resolution and frame rate can be adjusted according to the need. After the depth camera 2 obtains the depth information of the target surface, the data is transmitted to the control system, and the control system can implement accurate control of the robot arm by means of the transformation relationship.

[0075] The material extrusion system 3 is also installed at the end of the robot arm, and cooperates with the depth camera 2 to realize dynamic printing control. The system adjusts the output of the biological material through the end effector, and the extrusion amount is adjusted according to the depth information of the target surface and the real-time feedback, so as to ensure that the material covers the repair area well. The material extrusion system 3 is electrically connected to the system controller, and is also connected to the system reference air pressure source air path, so as to realize accurate control of the pressure and the extrusion speed.

[0076] The marker point 5 is arranged in the area to be repaired, and can be accurately recognized by the depth camera 2 and the three-dimensional scanner 4, and is used for three-dimensional parameter estimation and reconstruction of the current object to be repaired. The marker point 5 can ensure that the system can track and locate the dynamic target surface in real time during the whole repair process, and can deal with the detection false positive interference caused by factors such as occlusion and illumination change caused by object or robot arm movement, and the following implementation method:

[0077] We know that the projective transformation commonly occurring in visual detection will cause the circle to appear in the form of an ellipse. First, find the marker point on the arm in the form of an ellipse (i.e. the point in the form of a circle before projection transformation) in the camera image, and then calculate the transformation matrix T b The ellipse is restored to a circle for subsequent positioning and attitude estimation, etc. Here T b We adopt a common method, and assume a matrix Ty , eigenvalues are λ1, λ2, λ3, and their eigenvectors are η1, η2, η3, respectively, then we have:

[0078] T y = VΛV T (6)

[0079]

[0080] Assuming two variables m, n, then the rotation matrix T b can be expressed as:

[0081]

[0082] Here θ represents an arbitrary rotation angle around the normal of the marker plane. In theory, the rotation angle θ can represent the transformation matrix that transforms the ellipse into a circle in any rotation state. However, since this paper focuses on marker detection, positioning, and transforming the ellipse into a circle for subsequent operations, it is not necessary to consider the complete rotation information represented by θ. Simplifying it to 0 reduces the amount of calculation and reduces calculation errors, which helps to improve the efficiency and stability of the entire algorithm in practical applications. Therefore, at this time, T b is:

[0083]

[0084] where m, n can only take 1 and -1, so there are four different combinations, which will appear 4 different rotation matrices corresponding to the four different directions when transforming the ellipse into a circle. In actual scenarios, since the marker is planar, two of the directions will cause the direction of the marker to be inconsistent with the line of sight of the camera (for example, it will make the marker appear upside down or tilted too much, which does not conform to the reasonable pose of the marker relative to the camera in the actual physical scene) can be directly excluded, leaving the remaining two directions corresponding to the rotation matrices (T b1 and T b2 ) are reasonable, and each detected ellipse needs to be evaluated to determine how the ellipse is better transformed into a circle in these two possible directions, thereby further determining the position, pose, and other information of the marker.

[0085] For all detected ellipses, combine their rotations (i.e., T b1 and T b2 ) to form four feasible rotation pairs, and then filter out rotation pairs with an inner product higher than a fixed threshold by calculating the inner product of the two rotation matrices in the rotation pair. This can avoid selecting ellipse pairs with inconsistent directions, such as Figure 2 as shown on the right, because the marker is planar, the direction of its points should have a certain consistency, so as to filter out the correct transformation matrix T bThe desired pose is then obtained by checking the code.

[0086] The system controller receives real-time data of the depth camera 2 and the point cloud model of the initial scan, performs data fusion and generates control instructions, comprehensively analyzes the depth information of the target surface and the initial path model, adjusts the position of the mechanical arm, the extrusion speed of the extrusion system and the material distribution through feedback, so that the system has self-adaptive ability in dynamic environment, and ensures that the material accurately covers the target area.

[0087] Figure 3 The control logic of the real-time in-situ repair of the application comprises the following steps:

[0088] S1: Obtain the three-dimensional point cloud model of the repaired object surface. Obtain the three-dimensional point cloud of the object surface containing the texture information of the marker points by means of a three-dimensional scanner, and pre-process it by using a point cloud editing software, select appropriate sampling density, delete some frames with poor scanning effect, and splice the remaining frame point clouds to obtain more accurate object surface information.

[0089] S2: Generate a repair path suitable for the surface, call the contour and filling path generated by the slicing software, calculate a path suitable for the current repair surface, and process the generated path to make it smooth, meeting the acceleration and deceleration requirements when the mechanical arm moves. The new path information will be uploaded to the storage medium for use in in-situ repair.

[0090] S3: Obtain the depth camera video stream and depth stream. After aligning the video frames and depth frames of the depth camera, the depth information of the specified pixels can be output, and further more accurate point cloud information can be obtained.

[0091] S4: Run the marker point extraction and matching algorithm, and use multi-index comprehensive judgment to deal with the occlusion and false positive situations. All detected marker points will be screened and matched to obtain the optimal set of parameters for reconstruction.

[0092] S5: According to the optimal parameters, the object is reconstructed in real time, the pose of the current object is obtained, and the motion information of the object is obtained in combination with the motion information of the mechanical arm.

[0093] S6: Input the object pose into the motion control and prediction algorithm to correct the trajectory, and obtain the motion trajectory of the extrusion nozzle.

[0094] Of course, the system components can also include fewer or more components, and the present embodiment does not limit this.

[0095] Optionally, the application also provides a computer-readable storage medium, which can be read by a computer and a controller, and the program stored therein is loaded by a processor and executes the real-time in-situ repair method disclosed in the application.

[0096] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in any other form, and any modification or equivalent variation made according to the technical essence of the present application still falls within the scope of the present application.

Claims

1. A vision servoing based in-vivo repair system comprising a collaborative robot (1), a depth camera (2), a material extrusion system (3), a three-dimensional scanner (4), a marker point (5) and a checkerboard marker plate (6), characterized in that: The end of the collaborative robot arm (1) is provided with a depth camera (2) and a material extrusion system (3) for controlling the printing track during the repair process according to the target surface depth information, the depth camera (2) is installed at the end of the collaborative robot arm (1), which can collect the depth data of the target surface in real time, generate point cloud information and transmit it to the system controller; the material extrusion system (3) is integrated with the collaborative robot arm (1), which is used to adjust the extrusion amount of biological materials according to the instructions and configuration parameters of the system controller, so as to realize the stable distribution of materials, the three-dimensional scanner (4) is used to scan the target surface before repair, generate a three-dimensional point cloud model and transmit it to the system controller to plan the initial repair path; the marker point (5) is arranged on the area to be repaired, which is used as a reference point for positioning and calibration of the depth camera and the structured light three-dimensional scanner; the checkerboard marker plate (6) is used for calibration before repair.

2. The visual servoing based in-vivo repair system of claim 1, wherein: The depth camera (2) and the collaborative robot arm (1) move synchronously.

3. The visual servo based in-vivo repair system of claim 1, wherein: The material extrusion system (3) adjusts the extrusion flow of biological materials by real-time control of pressure.

4. The visual servoing based in-vivo repair system and repair control method of claim 1, wherein: The three-dimensional scanner (4) scans the target surface completely before the repair operation, and the generated three-dimensional point cloud model is used to assist the system controller to preliminarily plan the repair path, and is used as the positioning reference of the subsequent depth camera (2). 5.The in-vivo vision servo-based repair system and repair control method according to claim 1, wherein: The marker point (5) is designed as a high-contrast pattern or color.

6. The visual servo based in-vivo repair system of claim 1-5, wherein: The specific steps are as follows: S1: Obtain the three-dimensional point cloud model of the repaired object surface, obtain the three-dimensional point cloud of the object surface containing the marker point texture information by means of the three-dimensional scanner, and pre-process it by using the point cloud editing software, select the appropriate sampling density, delete some frames with poor scanning effect, and splice the remaining frame point clouds to obtain more accurate object surface information; S2: Generate a repair path suitable for the surface, call the contour and filling path generated by the slicing software, calculate a path suitable for the current repair surface, and process the generated path to make it smooth, meet the acceleration and deceleration requirements when the robot arm moves, and upload the new path information to the storage medium for use during in-vivo repair; S3: Obtain the depth camera video stream and depth stream, align the video frame and depth frame of the depth camera, output the depth information of the specified pixel, and further obtain accurate point cloud information; S4: Run the marker point extraction and matching algorithm, use multi-index comprehensive judgment to deal with the shielding and false positive situations, and select and match all detected marker points to obtain the optimal parameter set for reconstruction; S5: According to the optimal parameters, the object is reconstructed in real time, the pose of the current object is obtained, and the motion information of the object is obtained by combining the motion information of the robot arm; S6: Input the object pose into the motion control and prediction algorithm to correct the trajectory of the extrusion nozzle.

7. The repair control method of the in-vivo repair system based on visual servoing according to claim 6, wherein The step S1 arranges visual markers near the repair area, which is specifically as follows: The checkerboard marker board (6) is used to obtain and correct the camera's internal parameters, and the hand-eye calibration is completed by using the checkerboard marker board (6). In addition, four parameters need to be obtained. represents the conversion matrix of the mechanical arm coordinate system and the base coordinate system, which is a known quantity and can be obtained from the machine arm system; represents the conversion matrix of the camera coordinate system to the mechanical arm coordinate system, which needs to be calculated, but this conversion relationship does not change during the movement of the mechanical arm; represents the conversion matrix of the camera coordinate system to the checkerboard marker board coordinate system, which is a known quantity and can be obtained by camera calibration; represents the conversion matrix of the checkerboard marker board coordinate system to the base coordinate system, which is the final result to be obtained; as long as the relative position of the mechanical arm and the checkerboard marker board does not change, this conversion matrix does not change; at this time, the coordinate system conversion relationship between the depth camera (2) and the collaborative mechanical arm (1) is obtained The specific calculation process is as follows: Firstly, the robot arm is controlled to move from the initial position to position 1, position 2 and position 1 and position 2 are solved simultaneously. Since the robot arm coordinate system and the checkerboard marker plate coordinate system are relatively fixed, the transformation matrix between the two is unchanged, and the transformation relationship between the camera and the tool coordinate system is solved: （1） （2） The simultaneous equations (1) and (2) can be obtained: （3） Elimination Collapsing to the same side and simplifying gives: （4） （5） wherein It is known that, The transformation matrix from the end tool coordinate system to the camera can be obtained by camera calibration, is the transformation matrix from the end tool coordinate system to the camera, since the structure of the robot arm is known, the robot arm coordinate system and the tool coordinate system are transformed, and then the transformation matrix from the end tool coordinate system to the camera can be obtained. The above algorithm is implemented by code to complete the calibration work.

8. The repair control method of the visual servo based in-vivo repair system according to claim 6, characterized in that In the step S3, the depth camera video stream and the depth stream are acquired, the video frame and the depth frame of the depth camera are aligned, the depth information of the specified pixel is output, and accurate point cloud information can be further obtained. First, the elliptic marker points on the arm need to be found in the camera image, the next step is to calculate the transformation matrix The ellipse is reduced to a circle for the subsequent localization and pose estimation problem, here we solve The following method is used, assuming a matrix , the eigenvalues are , , , their eigenvectors are , , , we get: （6） （7） Assume two variables , , then the rotation matrix is expressed as: （8） Here a representative arbitrary rotation angle around the normal of the marking plane, the theoretical rotation angle denotes the transformation matrix that transforms the ellipse into a circle in any rotated state, reducing the calculation amount and lowering the calculation error by simplifying it to 0, then is: （9） wherein , Only 1 and -1 can be taken, and four different rotation matrices will appear, respectively corresponding to four different directions when transforming the ellipse into a circle. In actual scenarios, since the marker is planar, each detected ellipse needs to be evaluated to determine how the ellipse is better transformed into a circle in the two possible directions, thereby further determining the position and attitude information of the marker.

Images