Puncture robot calibration method and system based on binocular vision

Abstract

The invention discloses a calibration method and system of a puncture robot based on binocular vision. The method comprises the following steps: designing a multi-modal calibration plate, constructing a polynomial distortion model for the puncture robot based on multi-view geometric calibration, and optimizing a radial distortion coefficient; performing global search on the initial calibration parameters by using a particle swarm optimization algorithm, and screening to obtain an optimal parameter set; an improved differential evolution algorithm is adopted to optimize the needle tip track fitting error, and optimized kinematics parameters are obtained; constructing a lightweight convolutional neural network, and taking the binocular image as input to obtain a high-precision needle point coordinate; and performing three-dimensional reconstruction by adopting structured light assistance to obtain a three-dimensional motion trail of the needle tip. The system comprises a multi-view geometric calibration module, an initial calibration parameter determination module, an optimal parameter set screening module, a kinematic parameter optimization module, a convolutional neural network module and a three-dimensional reconstruction module. The positioning precision of the puncture robot is improved, and the stability and the real-time performance of the puncture robot in a complex operation environment are enhanced.

Description

Technical Field

[0001] The present invention relates to the cross - technical field of medical image processing and precise control of robots, and particularly relates to a calibration method and system for a puncture robot based on binocular vision. Background Art

[0002] With the continuous development of medical imaging technology and robotics, minimally invasive surgeries, such as percutaneous biopsy, tumor ablation, etc., are increasingly widely used in the medical field. To ensure the high precision and safety of these surgeries, the positioning and control accuracy of the puncture robot becomes a key technology. The robot - assisted surgical system usually uses visual feedback to assist in precise positioning and control, and the calibration accuracy of the visual system directly determines the accuracy and safety of the surgical operation. In recent years, with the rapid development of deep learning, computer vision, and robotics, the vision - based surgical robot positioning system has gradually become a research hotspot. In particular, the binocular vision technology, which uses the images obtained by two cameras, the left and the right, to reconstruct three - dimensional space information, has been widely applied in puncture robots.

[0003] With the popularization of minimally invasive surgery technology, especially the continuous development of robot - assisted puncture surgeries (such as tumor ablation, biopsy, etc.), the market demand for precise positioning, motion control, and real - time feedback is increasing day by day. The precise positioning and control of the puncture robot will directly affect the success rate of the surgery and the patient's recovery. Therefore, how to improve the calibration accuracy of the puncture robot and ensure the stable operation of the robot in a complex surgical environment has become a hot topic in current technical research. The current market demand for vision - based puncture robot systems is growing rapidly, especially in the following aspects:

[0004] (1) High - precision positioning. To reduce accidental injury and ensure surgical accuracy, precise needle tip positioning and motion control are key requirements in minimally invasive surgery; (2) Real - time feedback and intelligent adjustment. During the surgical process, the robot system needs to adjust its motion state and position in real time. Therefore, the calibration method needs to have real - time performance and adaptive ability; (3) Anti - interference ability. During the surgical process, factors such as blood, tissue deformation, and instrument movement may cause image interference. Therefore, the system needs to have strong robustness to cope with changes in a complex environment.

[0005] In the prior art, the calibration methods for puncture robots based on binocular vision usually have the following technical defects and problems:

[0006] (1) The accuracy of traditional calibration methods is limited. Traditional calibration methods usually rely on checkerboard calibration plates for geometric calibration. Although this method is widely used in control systems, its accuracy is often not ideal in complex surgical environments. Especially in the face of a dynamically changing surgical environment, such as blood and tissue occlusion, existing calibration methods are difficult to provide sufficient robustness, resulting in low needle tip positioning accuracy and large surgical errors.

[0007] (2) Insufficient distortion correction. The optical distortion of the camera lens is an important factor affecting visual accuracy. Although there are some methods in the prior art that use the radial distortion model for correction, in practical applications, due to the non-ideal optical characteristics of the lens and the complex surgical environment, it often leads to insufficient distortion correction, thus affecting the calibration accuracy. Especially in minimally invasive surgery, the requirement for precise needle tip positioning is very high, and the impact of uncorrected distortion is particularly prominent.

[0008] (3) Limitations of optimization algorithms. Existing calibration methods mostly rely on traditional non-linear optimization algorithms in parameter optimization, such as the least squares method, Newton's method, etc. These methods are prone to falling into local optimal solutions and are difficult to ensure the accuracy of the global optimal solution. Especially for calibration problems with multiple unknown parameters, these traditional optimization algorithms often have a slow convergence speed and are sensitive to the selection of initial parameters, resulting in an unstable calibration process.

[0009] (4) Real-time and robustness issues. Currently, many calibration methods have a high computational complexity and are difficult to meet the requirements in real-time surgical scenarios. For example, deep learning methods have a large amount of computation, especially when running on an embedded platform, it may not be able to ensure sufficient real-time performance. In addition, dynamic interferences during surgery, such as tissue occlusion and the rapid movement of the needle tip, will also affect the robustness of existing systems, resulting in inaccurate positioning or failure. Summary of the Invention

[0010] In view of this, the purpose of the embodiments of the present invention is to provide a calibration method and system for a puncture robot based on binocular vision. Through a multi-modal calibration plate, particle swarm optimization, differential evolution algorithm, lightweight convolutional neural network, and structured light-assisted three-dimensional reconstruction, etc., it has the advantages of high accuracy, low cost, easy operation, strong anti-interference, and high system robustness, improving the positioning accuracy of the puncture robot and enhancing its stability and real-time performance in complex surgical environments.

[0011] The embodiments of the present invention are implemented as follows:

[0012] A calibration method for a puncture robot based on binocular vision, which includes:

[0013] Design a multi-modal calibration plate and perform multi-viewpoint geometric calibration on the puncture robot.

[0014] Based on multi-view geometry calibration, a polynomial distortion model is constructed for the puncture robot, the radial distortion coefficient is optimized, and the initial calibration parameters are obtained.

[0015] The particle swarm optimization algorithm is used to globally search the initial calibration parameters, and the optimal parameter set is screened out.

[0016] Taking the optimal parameter set as the input, the improved differential evolution algorithm is used to optimize the fitting error of the needle tip trajectory, and the optimized kinematic parameters are obtained.

[0017] A lightweight convolutional neural network is constructed, and binocular images are used as the input to obtain high-precision needle tip coordinates.

[0018] Structured light assistance is used for three-dimensional reconstruction, and the optimized kinematic parameters and the high-precision needle tip coordinates are incorporated into the needle tip kinematic model to reconstruct the three-dimensional motion trajectory of the needle tip.

[0019] In a preferred embodiment of the present invention, in the above-mentioned calibration method of the puncture robot based on binocular vision, the design of the multi-modal calibration board and the multi-view geometry calibration of the puncture robot include:

[0020] Select a black matte carbon fiber board as the calibration board, and a periodic fluorescent marker array is embedded on the surface of the calibration board.

[0021] Control the end of the robot to fix the binocular camera, and take pictures of the calibration board from 2N non-coplanar viewpoints, where N is a natural number greater than 0, to obtain a set of consecutive frame images. Extract the fluorescent marker points from the set of consecutive frame images, and locate the central pixel coordinates of the fluorescent marker points.

[0022] Let the world coordinates of the k-th marker point on the calibration board be P k =[X k ,Y k ,0] T , and the pixel coordinates projected onto the image of the i-th viewpoint are p ik =[u ik ,v ik T , and a projection equation is constructed where K i is the camera internal parameter matrix, R i is the external parameter rotation matrix, t i is the translation vector, and λ ik is the scale factor.

[0023] Through the singular value decomposition method, the initial value of the camera internal parameter matrix K i is solved.

[0024] The external parameter rotation matrix R i and the translation vector t are solved​i Value

[0025] Its technical effects are as follows: The calibration board adopts a periodic fluorescence marking array, providing more accurate calibration points, enabling more accurate geometric information to be obtained when shooting from different perspectives; the design of the calibration board includes three dynamic modes, improving the robustness of calibration through the periodic fluorescence marking array, helping to suppress environmental light interference, adapting to traditional calibration algorithms, and increasing the adaptability to environmental changes and uneven illumination; the calibration board integrates an FPGA chip, and realizes multi-frequency flashing of 1Hz / 5Hz / 10Hz by modulating the LED emission timing through PWM to synchronize the exposure signals of the binocular cameras, effectively suppressing the interference of environmental light noise.

[0026] In a preferred embodiment of the present invention, in the above-mentioned calibration method of the puncture robot based on binocular vision, in the multi-viewpoint geometry calibration, a polynomial distortion model is constructed for the puncture robot, and the radial distortion coefficient is optimized. The initial calibration parameters obtained include:

[0027] Use a second-order radial distortion model to describe lens distortion where (u, v) are the distorted pixel coordinates, (u corrected , v corrected ) are the corrected coordinates, (u 0 , v 0 ) are the principal point coordinates, obtained from the camera internal parameter matrix K i , represents the radial distance from the pixel point to the principal point, and k 1 and k 2 are the radial distortion coefficients to be optimized.

[0028] Convert the world coordinate P k = [X k , Y k , 0] T to the camera coordinate system through the external parameter matrix

[0029] Project the three-dimensional point in the camera coordinate system onto the normalized plane and calculate the normalized coordinates

[0030] Calculate the square of the radial distance from the normalized coordinates to the principal point Obtain the corrected normalized coordinates (x d = x n (1 + k 1 r 2 + k 2 r 4 ), y d = y n (1 + k 1 r2 +k 2 r 4 ))。

[0031] Map the corrected normalized coordinates to the actual pixel coordinate system where (f x , f y ) is the focal length of the camera.

[0032] Construct the reprojection error function, defined as the sum of the squared errors of all marked points at all viewpoints, The goal is to adjust the radial distortion coefficient k to be optimized through an optimization algorithm 1 and k 2 , such that the reprojection error E is minimized, where p ik =(u ik , v ik ) is the actual observed coordinate, is the theoretical projection coordinate.

[0033] For each marked point, calculate the partial derivatives of the radial distortion coefficients k 1 and k 2 through the Jacobian matrix Take the partial derivative with respect to Take the partial derivative For Take the partial derivative

[0034] Perform iterative initialization, set the initial distortion coefficients k 1 =0, k 2 =0, the initial damping factor λ = 0.001, and the maximum number of iterations T = 100.

[0035] Perform parameter update Δθ = -(J T J + λI) -1 J T E, where Δθ = [Δk 1 , Δk 2 T 。

[0036] After the update, perform convergence judgment. If the reprojection error E decreases after the update, accept the update, k 1 ← k 1 + Δk 1 , k 2 ← k 2 + Δk 2 , decrease the damping factor λ ← 0.1λ. If the reprojection error increases after the update, reject the update and increase the damping factor λ ← 10λ.

[0037] The convergence condition is that the parameter change amount ||Δθ|| < 10 -6 ​, the error decrease amount |E new - E old | < 10 -6 , and the maximum iteration number T is reached.

[0038] Its technical effect is as follows: By adopting a second-order radial distortion model, the optical distortion of the lens is accurately described. Through the optimization of the radial distortion coefficients, the image deviation caused by the lens distortion is effectively reduced; a reprojection error function is constructed and its value is minimized to ensure that the error of each marker point during the calibration process is minimized at all viewing angles, optimizing the internal and external parameters of the camera, ensuring the accuracy of the vision system, and improving the accuracy of the needle tip positioning of the puncture robot; the partial derivatives of the radial distortion coefficients are calculated using the Jacobian matrix. The introduction of the Jacobian matrix can accelerate convergence during the optimization process, avoiding the slow convergence problem caused by inappropriate initial values or local optimal solutions in traditional methods; through the adjustment of the damping factor, over-updating or premature convergence can be effectively avoided. If the reprojection error does not decrease significantly after the update, the system will appropriately increase the damping factor to avoid ineffective updates.

[0039] In a preferred embodiment of the present invention, in the above-mentioned calibration method of the puncture robot based on binocular vision, the global search for the initial calibration parameters using the particle swarm optimization algorithm and the screening to obtain the optimal parameter set include:

[0040] Encoding the initial calibration parameters into a particle position vector where (f x , f y , u 0 , v 0 ) are the parameters of the camera internal parameter matrix K i , (k 1 , k 2 ) are the radial distortion coefficients, is the Euler angle parameterization of the external parameter rotation matrix R i , (t x , t y , t z ) are the components of the translation vector t i .

[0041] According to the physical meaning, set the search space to perform range constraints on the initial calibration parameters.

[0042] Set the particle swarm size, the number of particles N particles = 200, set the initial position of the central particle, and other particles are uniformly randomly distributed within the parameter range. The initial velocity of the particles is set at 10% of the parameter range.

[0043] For each particle j, update its velocity v j and position x j, the velocity at the (t + 1)-th iteration is updated as where w is the inertia weight, with an initial value of w max = 0.9, linearly decreasing to w min = 0.4, c 1 and c 2 are learning factors, c 1 = c 2 = 2, r 1 and r 2 are random numbers, uniformly distributed between [0, 1], and the position is updated as

[0044] If the updated position exceeds the parameter range, it is set to the nearest boundary value for boundary truncation. The boundary truncation formula is

[0045] Record the historical optimal position of each particle as the optimal individual for update.

[0046] Record the individual optimal position among all particles as the global optimal g best for update.

[0047] When the set maximum number of iterations is reached, or when the parameter change amount is less than the preset threshold, terminate the iteration and output the global optimal parameter set g best .

[0048] Its technical effect is that: the Particle Swarm Optimization (PSO) algorithm can explore the entire parameter space through global search of the initial calibration parameters, avoiding the common local optimal solution problem in traditional optimization methods. PSO does not depend on the choice of the initial solution and can effectively find the global optimal solution, ensuring the best calibration results are found under various parameter combinations; during the particle swarm optimization process, by reasonably setting the search space and range constraints, it can reduce ineffective searches and ensure that particles search within a physically meaningful parameter space; the inertia weight and learning factors in the particle swarm optimization algorithm can be dynamically adjusted according to the feedback during the search process to balance global exploration and local exploitation, ensuring that the algorithm can conduct extensive exploration in the initial stage; through boundary truncation, the stability of the algorithm is ensured, avoiding ineffective calculations during the search process.

[0049] In a preferred embodiment of the present invention, in the above calibration method of the binocular vision-based puncture robot, using the optimal parameter set as input, the improved differential evolution algorithm is used to optimize the fitting error of the needle tip trajectory, and the optimized kinematic parameters obtained include:

[0050] The optimal parameter set is p j= [K, d, R, t], where K is the camera intrinsic matrix, d is the radial distortion coefficient, R is the extrinsic rotation matrix, and t is the translation vector.

[0051] Let the three-dimensional trajectory of the needle tip include the point set P = {p 1 , p 2 ,..., p i ,..., p N}, where p i = [x i , y i , z i is the position of the needle tip at each moment.

[0052] Define the calculation formula of the optimal fitness P real (t) is the true three-dimensional coordinate of the needle tip in the t-th frame image, and P pred (t) is the predicted position of the needle tip calculated according to the kinematic model and calibration parameters. N is the total number of time steps.

[0053] Minimize the objective function to minimize the fitting error and obtain the optimized kinematic parameters.

[0054] Its technical effect is as follows: For the position of the needle tip at each moment, through the optimal fitness calculation formula, the error between the true three-dimensional coordinate of the needle tip in each frame image and the predicted position of the needle tip by the kinematic model is defined, so that at different time steps, the predicted position of the needle tip is as close as possible to the true position; The improved differential evolution algorithm MDE has a powerful global optimization ability, can avoid falling into local optimal solutions, and can accurately adjust the kinematic parameters through iterative optimization, so that the fitting error of the needle tip trajectory is minimized. In complex and dynamic environments, such as tissue deformation, blood interference, etc., it continuously provides high-precision needle tip positioning; Minimize the objective function (fitting error function) to minimize the error generated during the movement of the needle tip, significantly reduce the deviation and instability in the robot movement, and ensure the accurate tracking and positioning of the needle tip throughout the surgical process.

[0055] In a preferred embodiment of the present invention, in the above calibration method of the binocular vision-based puncture robot, the construction of the lightweight convolutional neural network, with binocular images as input, to obtain high-precision needle tip coordinates includes:

[0056] Take the left and right images of the binocular camera as input, normalize and align the sizes of the input images to ensure the spatial consistency of the binocular images.

[0057] Design a feature extraction backbone network, including an early feature fusion module and depthwise separable convolution blocks. The early feature fusion module adopts a two-stream input branch. The left and right images respectively extract low-level features through independent shallow convolutional layers, and the feature maps of the left and right images are concatenated in the channel dimension to form a multi-modal fusion feature. Each depthwise separable convolution block contains a depth convolution for channel-wise feature extraction and a pointwise convolution for inter-channel feature fusion.

[0058] Design an attention enhancement network, including a squeeze-and-excitation module and a spatial attention mask. The squeeze-and-excitation module compresses the spatial information of the feature map through global average pooling, learns channel weights, and strengthens important feature channels. The spatial attention mask generates a spatial weight map through a convolutional layer to highlight the significant features of the tip region.

[0059] Design a multi-scale feature fusion network, including a pyramid pooling module, which uses pooling kernels of different scales to extract multi-scale features, enhances the network's sensitivity to small displacements of the tip, and the pooled features are upsampled to restore the resolution and fused with the original feature map.

[0060] Design an output layer, including a fully connected regression head, which flattens the finally fused feature map into a vector, gradually reduces the dimension through two fully connected layers, and outputs the three-dimensional coordinates of the high-precision tip. The number of output layer nodes is 3.

[0061] Its technical effects are as follows: By normalizing and aligning the sizes of the input left and right binocular images, the spatial consistency of the images is ensured, the scale differences and offsets between the images can be eliminated, enabling subsequent feature extraction and coordinate calculation to be carried out based on a unified spatial coordinate system; The early feature fusion module extracts low-level features from the left and right images through independent shallow convolutional layers respectively, and splices the feature maps of the left and right images in the channel dimension to form multi-modal fusion features, making full use of the different perspective information of the left and right images, enhancing the network's ability to capture spatial information, thereby improving the positioning accuracy of the needle tip coordinates; Depthwise separable convolution is used instead of traditional convolution. Depthwise separable convolution decomposes the standard convolution operation into two steps: depth convolution and pointwise convolution. While ensuring the ability to extract features, it significantly reduces the computational cost and memory consumption, and greatly reduces the number of parameters and the amount of computation; In the squeeze-and-excitation module, the spatial information of the feature map is compressed through global average pooling to learn channel weights, thereby strengthening important feature channels. The spatial attention mask generates a spatial weight map through a convolutional layer to highlight the significant features in the needle tip area, which can effectively reduce the influence of background noise on the needle tip positioning accuracy; The pyramid pooling module extracts multi-scale features through pooling kernels of different scales, enhancing the network's sensitivity to the tiny displacement of the needle tip. Multi-scale feature fusion can help the network capture different levels of features in spaces of different scales. Especially when positioning the needle tip and detecting tiny displacements, it can better identify and locate the position of the needle tip. Through the upsampling of the pooled features and the fusion with the original feature map, the network can maintain high-resolution features and improve the positioning accuracy; The output layer of the network is designed as a fully connected regression head, which flattens the finally fused feature map into a vector in a step-by-step dimensionality reduction manner and outputs the three-dimensional coordinates of the high-precision needle tip, enabling the network to accurately map the image features to the three-dimensional position space of the needle tip.

[0062] In a preferred embodiment of the present invention, in the above calibration method of the puncture robot based on binocular vision, the three-dimensional reconstruction assisted by structured light and the integration of the optimized kinematic parameters and the high-precision needle tip coordinates into the needle tip kinematic model to reconstruct the three-dimensional motion trajectory of the needle tip include:

[0063] Establish a structured light assisted system, including a projection module and a binocular vision module. The projection module uses a high-precision blue light or infrared structured light projector to project a preset optical pattern onto the puncture needle and the surrounding area. The binocular vision module uses a synchronously triggered binocular camera to capture the scene images under the structured light irradiation from different perspectives respectively, perform image analysis, and output the three-dimensional point cloud motion trajectory of the needle tip.

[0064] Design a kinematic model for a multi-joint chain needle tip. According to the mechanical structure of the puncture robot, define the joint coordinate system and link parameters. The pose of the needle tip is calculated through forward kinematics. The input includes the optimized kinematic parameters and the high-precision needle tip coordinates, and the output of the model is the theoretical three-dimensional coordinates of the needle tip.

[0065] Introduce a joint clearance compensation error term and a thermal deformation compensation error term into the multi-joint chain needle tip kinematic model to optimize the output accuracy and obtain the high-precision three-dimensional coordinates of the needle tip.

[0066] Design an extended Kalman filter model. Take the high-precision three-dimensional coordinates of the needle tip of the needle tip kinematic model and the three-dimensional point cloud motion trajectory reconstructed by structured light as the model input. Perform spline interpolation fitting on the fused needle tip trajectory to eliminate high-frequency noise and optimize the trajectory smoothness, and obtain the optimized three-dimensional motion trajectory of the needle tip.

[0067] Its technical effects are as follows: Structured light assistance projects an optical pattern through a high-precision blue light or infrared structured light projector, and uses a synchronously triggered binocular camera to capture scene images from different perspectives to obtain high-precision three-dimensional point cloud data. Through the generation of depth maps and point cloud reconstruction, the three-dimensional motion trajectory of the needle tip is accurately captured, enhancing the needle tip positioning accuracy, especially in complex and dynamic surgical environments; Through the multi-joint chain needle tip kinematic model, considering the mechanical structure of the puncture robot and the joint coordinate system (such as DH parameters), accurately calculate the pose of the needle tip. The combined optimized kinematic parameters and high-precision needle tip coordinates greatly improve the accuracy of the kinematic model, ensuring that the theoretical three-dimensional coordinates of the needle tip output by the model are more consistent with the actual position, and precisely controlling the movement of the robot; Introduce a joint clearance compensation error term and a thermal deformation compensation error term to further optimize the output accuracy of the kinematic model, compensating for errors caused by factors such as mechanical structure, joint clearance, or thermal deformation, and ensuring that the robot can maintain high-precision motion control in complex environments, especially in high-temperature operating environments; The extended Kalman filter model combines the needle tip kinematic model and the three-dimensional point cloud data reconstructed by structured light for fusion, eliminates high-frequency noise in the reconstruction process, optimizes the smoothness of the trajectory, filters out noise through spline interpolation fitting, and ensures the smoothness of the needle tip motion trajectory, thereby improving the accuracy of needle tip positioning, especially during rapid movement or surgical procedures, and being able to maintain stable and precise operations.

[0068] In a preferred embodiment of the present invention, in the above-mentioned calibration method for a puncture robot based on binocular vision, the steps of image analysis of the structured light assistance system include:

[0069] Perform optical pattern design and project multiple groups of stripe patterns with different frequencies in sequence by the projector.

[0070] Analyze the stripe images captured by the binocular camera and extract the phase value of each point.

[0071] Combining phase values ​​at different frequencies eliminates ambiguity caused by phase jumps and generates a continuous absolute phase map.

[0072] The absolute phase map is converted into accurate three-dimensional depth data by combining the parallax information and phase difference of the binocular camera to generate a high-resolution depth map.

[0073] Each pixel in the high-resolution depth map is converted into a three-dimensional spatial point to form a dense point cloud of the needle tip and surrounding tissue.

[0074] The dense point cloud is converted into a smooth three-dimensional surface model using a mesh generation algorithm, the geometric features of the needle tip are highlighted, and the output is a three-dimensional point cloud motion trajectory of the needle tip.

[0075] Its technical effects are as follows: through the projection of multi-frequency fringe patterns, a wide range of depth information and high-resolution data of local details can be obtained simultaneously. Low-frequency thick stripes help to quickly obtain rough depth information of the scene, while high-frequency thin stripes can improve the resolution of local details; the accurate extraction of phase values ​​enables each point in the image to correspond to a specific three-dimensional spatial position, thereby improving the accuracy of three-dimensional reconstruction; by combining phase values ​​of different frequencies, the blurring caused by phase jumps can be eliminated, and a continuous absolute phase map is generated, which effectively avoids the accumulation of errors during phase unwrapping and ensures the continuity and accuracy of the three-dimensional reconstruction data; combining parallax information and phase difference, the phase map can be converted into accurate three-dimensional depth data through the parallax information of the binocular camera, and the generated high-resolution depth map is the needle tip and surrounding tissue It provides detailed spatial information and effectively captures tiny displacements and detail features; it effectively suppresses random noise caused by environmental interference (through multi-frame fusion optimization, ensures the stability of point cloud data, and aligns point cloud data within multiple time steps. Through spatial registration, the continuity and accuracy of the needle tip trajectory are guaranteed. Based on the rigid body motion assumption of the needle tip, the point cloud data can accurately reflect the motion trajectory of the needle tip at different time points, ensuring the accuracy and reliability of motion analysis; using a mesh generation algorithm (such as Poisson reconstruction) to process dense point clouds can generate a smooth three-dimensional surface model and highlight the geometric features of the needle tip. Through this reconstruction method, the point cloud data is converted into a three-dimensional model with precise geometric features, which greatly improves the reconstruction accuracy of the needle tip position and provides higher quality visual support for the three-dimensional motion trajectory of the needle tip.

[0076] A calibration system for a puncture robot based on binocular vision, comprising:

[0077] The multi-viewpoint geometric calibration module is used to design a multi-modal calibration plate and perform multi-viewpoint geometric calibration on the puncture robot.

[0078] An initial calibration parameter determination module, configured to construct a polynomial distortion model for the puncture robot based on multi-viewpoint geometry calibration, optimize the radial distortion coefficient, and obtain initial calibration parameters.

[0079] An optimal parameter set screening module, configured to perform a global search on the initial calibration parameters using a particle swarm optimization algorithm, and screen out an optimal parameter set.

[0080] A kinematic parameter optimization module, configured to use the optimal parameter set as input, and optimize the fitting error of the needle tip trajectory using an improved differential evolution algorithm to obtain optimized kinematic parameters.

[0081] A convolutional neural network module, configured to construct a lightweight convolutional neural network, use binocular images as input, and obtain high-precision needle tip coordinates.

[0082] A three-dimensional reconstruction module, configured to perform three-dimensional reconstruction assisted by structured light, integrate the optimized kinematic parameters and the high-precision needle tip coordinates into a needle tip kinematic model, and reconstruct a three-dimensional motion trajectory of the needle tip.

[0083] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the calibration method of the puncture robot based on binocular vision as described above.

[0084] The beneficial effects of the embodiments of the present invention are:

[0085] Through a multi-modal calibration board and multi-viewpoint geometry calibration, the present invention overcomes the error problems in traditional calibration methods. In particular, through the design of a fluorescence marker array and the selection of non-coplanar viewpoints, geometric information of the puncture robot at different angles can be accurately obtained, greatly improving the calibration accuracy. The introduction of multiple calibration board modes and adaptive structured light frequency adjustment during the calibration process can ensure the stable operation of the system in complex environments such as tissue deformation and blood occlusion, enhancing the robustness of the calibration process.

[0086] During the structured light-assisted three-dimensional reconstruction process of the present invention, by combining stripe patterns of different frequencies, the projection frequency and optical pattern can be flexibly adjusted to cope with changes in the dynamic surgical environment, ensuring that the structured light system can effectively handle blood interference and tissue deformation, and improving the adaptability of the system in the intraoperative dynamic environment. Through accurate phase value extraction, phase difference combination, and fusion of parallax information, it is ensured that high-resolution depth maps and accurate three-dimensional point cloud data can still be obtained in complex environments, providing detailed three-dimensional spatial information for the needle tip position.

[0087] The present invention conducts a global search for the initial calibration parameters through the particle swarm optimization algorithm, avoiding the local optimum problem, and further optimizes the fitting error of the needle tip trajectory through an improved differential evolution algorithm. Through the combination of global optimization and local optimization, the kinematic parameters can be accurately adjusted to ensure that the robot maintains precise movement in various complex surgical environments. During the optimization process, by reasonably setting the number of iterations and the convergence criterion, fast convergence can be achieved, improving the accuracy and efficiency of the system in real-time surgery and meeting the high real-time requirements.

[0088] The present invention designs a lightweight convolutional neural network, LightCNN, which not only ensures the output of high-precision needle tip coordinates but also effectively reduces the computational complexity. It is suitable for embedded platforms, greatly reducing the system's demand for hardware resources, reducing the system cost, while improving the computational efficiency and reducing the energy consumption. The use of the lightweight network simplifies the hardware configuration of the system, making the present invention applicable to a wider range of medical device scenarios, reducing the hardware cost, and simplifying the manufacturing process.

[0089] The present invention reduces noise interference through the fusion and optimization of multi-frame point cloud data, ensures the continuity and accuracy of the needle tip trajectory, enhances the system's adaptability to environmental changes, enables the system to operate stably under different environmental conditions, and ensures accurate three-dimensional trajectory reconstruction. The high robustness, precise optimization process, and real-time feedback mechanism of the present invention contribute to reducing the failure rate and enhancing the overall stability of the system, especially maintaining high reliability under complex surgical conditions. Brief Description of the Drawings

[0090] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can be obtained based on these drawings.

[0091] Figure 1 It is a flowchart of the calibration method for the puncture robot based on binocular vision of the present invention. Detailed Embodiments

[0092] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0093] Please refer to Figure 1, the first embodiment of the present invention provides a calibration method for a puncture robot based on binocular vision, which includes: designing a multi-modal calibration board to perform multi-viewpoint geometric calibration on the puncture robot; based on the multi-viewpoint geometric calibration, constructing a polynomial distortion model for the puncture robot, optimizing the radial distortion coefficient, correcting the optical distortion of the lens of the puncture robot, and obtaining initial calibration parameters; using the particle swarm optimization algorithm to perform a global search on the initial calibration parameters, and screening to obtain an optimal parameter set; using the optimal parameter set as input, adopting an improved differential evolution algorithm to optimize the fitting error of the needle tip trajectory, and obtaining optimized kinematic parameters; constructing a lightweight convolutional neural network, using binocular images as input, and obtaining high-precision needle tip coordinates; using structured light assistance for three-dimensional reconstruction, integrating the optimized kinematic parameters and the high-precision needle tip coordinates into the needle tip kinematic model, and reconstructing to obtain the three-dimensional motion trajectory of the needle tip.

[0094] In a preferred embodiment of the present invention, in the above calibration method for a puncture robot based on binocular vision, the designing of the multi-modal calibration board and performing multi-viewpoint geometric calibration on the puncture robot includes: selecting a black matte carbon fiber board as the calibration board, and embedding a periodic fluorescent marker array on the surface of the calibration board; specifically, the periodic fluorescent marker array can adopt three dynamic modes: hexagonal honeycomb arrangement, with a spacing of 15 mm, used to avoid symmetry matching ambiguity; random point cloud distribution, with a density of 10 points / cm 2 , used for anti-environmental light interference verification; rectangular grid arrangement, with a spacing of 20 mm, adapted to traditional calibration algorithms. The control module of the calibration board integrates an FPGA chip, and realizes multi-frequency flashing of 1 Hz / 5 Hz / 10 Hz by modulating the LED emission timing through PWM, synchronizes the exposure signals of the binocular cameras, and suppresses environmental light noise. The calibration board communicates with the binocular cameras through the RS-485 protocol, and the LED flashes with a pulse width of 10 ms during the camera exposure period and is turned off at other times.

[0095] Control the end of the robot to fix the binocular cameras, and take pictures of the calibration board from 2N non-coplanar viewpoints, where N is a natural number greater than 0, to obtain a set of continuous frame images, extract the fluorescent marker points from the set of continuous frame images, and locate the central pixel coordinates of the fluorescent marker points; specifically, the shooting viewpoints cover a pitch angle of ±45°, a yaw angle of 0° to 360° at intervals of 45°, a distance of 50 cm ± 5 cm from the calibration board, and 10 frames of images are continuously taken at each viewpoint. The exposure time can adopt 20 ms / 40 ms / 60 ms, and the dynamic range is improved through HDR fusion to obtain a set of continuous frame images. The set of continuous frame images is processed by non-local mean filtering NL-Means to remove sensor noise, retain the edge details of the marker points, and perform sub-pixel-level positioning on the central coordinates of the marker points based on the Gaussian fitting algorithm to obtain the central pixel coordinates of the fluorescent marker points.

[0096] Let the world coordinates of the k-th fiducial point on the calibration board be P k =[X k ,Y k ,0] T , and its pixel coordinates projected onto the i-th perspective image be p ik =[u ik ,v ik T , and construct the projection equation where K i is the camera intrinsic matrix, R i is the extrinsic rotation matrix, t i is the translation vector, and λ ik is the scale factor; through singular value decomposition, the initial value of the camera intrinsic matrix K i is obtained; specifically, in the singular value decomposition method, for each perspective i, the homography matrix H i is solved by the direct linear transformation algorithm, such that After expansion, a linear equation system Two equations are constructed for each fiducial point to form an overdetermined equation system A i h = 0, and through SVD decomposition, A i = UΣV T , and the last column of V is taken as the least squares solution of H i .

[0097] The relationship between the homography matrix and the intrinsic matrix K i is H i = K i [r 1 r 2 t], and using the orthogonality of the rotation matrix the constraint equation is obtained, where v ij =[h i1 h j1 ,h i1 h j2 +h i2 h j1 ,h i2 h j2 ,h i3 h j1 +h i1 h j3 ,h i3 h j2 +h i2 h j3 ,h i3 h j3 T , and b is the vector form of B

[0098] ​​Stack the constraint equations from multiple perspectives to form an overdetermined system of equations \(Ab = 0\); perform SVD decomposition on matrix \(A\), \(A=U\Sigma V\) T , and take the last column of \(V\) as the solution of \(b\); reconstruct \(b\) into a symmetric matrix \(B\) that satisfies Perform Cholesky decomposition on \(B\), \(B = LL\) T , to obtain \(K\) i -1 = \(L\), and find the inverse to obtain the intrinsic matrix \(K\) i .

[0099] Solve to obtain the extrinsic rotation matrix \(R\) i and translation vector \(t\) i .

[0100] Specifically, according to \(H\) i = \(K\) i [r 1 r 2 t], solve to obtain where \(\lambda = 1 / ||K\) i -1 h 1 || is used to normalize \(r\) 1 and \(r\); construct an approximate rotation matrix \(R\) raw = [r 1 r 2 r 1 ×r 2 , perform SVD decomposition on the approximate rotation matrix \(R\) raw to obtain \(R\) raw = \(U\Sigma V\) T , ensure that is orthonormalized to obtain \(R\) i = \(UV\) T ; adjust the translation vector by minimizing the reprojection error to obtain \(t\) i = \(\lambda K\) i -1 h 3 .

[0101] Its technical effects are as follows: The calibration board uses a periodic fluorescent marker array to provide more accurate calibration points, enabling more accurate geometric information to be obtained when shooting from different perspectives; the design of the calibration board includes three dynamic modes, improving the robustness of calibration through the periodic fluorescent marker array, helping to suppress environmental light interference, adapting to traditional calibration algorithms, and increasing the adaptability to environmental changes and uneven lighting; the calibration board integrates an FPGA chip, and realizes multi-frequency flashing of 1Hz / 5Hz / 10Hz by modulating the LED emission timing through PWM to synchronize the binocular camera exposure signal, effectively suppressing the interference of environmental light noise.

[0102] In a preferred embodiment of the present invention, in the above calibration method of the puncture robot based on binocular vision, in the multi-viewpoint geometry calibration, a polynomial distortion model is constructed for the puncture robot, and the radial distortion coefficients are optimized to obtain the initial calibration parameters, including: using a second-order radial distortion model to describe the lens distortion. where (u, v) are the distorted pixel coordinates, (u corrected , v corrected ) are the corrected coordinates, (u 0 , v 0 ) are the principal point coordinates, obtained from the camera internal parameter matrix K i ; represents the radial distance from the pixel point to the principal point, and k 1 and k 2 are the radial distortion coefficients to be optimized; the world coordinate P k = [X k , Y k , 0] T is transformed to the camera coordinate system through the external parameter matrix The three-dimensional points in the camera coordinate system are projected onto the normalized plane to calculate the normalized coordinates Calculate the square of the radial distance from the normalized coordinates to the principal point Obtain the corrected normalized coordinates (x d = x n (1 + k 1 r 2 + k 2 r 4 ), y d = y n (1 + k 1 r 2 + k 2 r 4 )); Map the corrected normalized coordinates to the actual pixel coordinate system where (f x , f y ) are the focal lengths of the camera; construct a reprojection error function, defined as the sum of the squared errors of all marked points at all viewpoints, The goal is to adjust the radial distortion coefficients k 1 and k 2 to be optimized through an optimization algorithm so that the reprojection error E is minimized, where p ik = (u ik , v ik ) are the actual observed coordinates, is the theoretical projection coordinate; for each marked point, calculate the partial derivatives of the radial distortion coefficients k 1 and k 2 through the Jacobian matrix For Partial derivative For Take the partial derivative Perform iterative initialization, set the initial distortion coefficient k 1 = 0, k 2 = 0, the initial damping factor λ = 0.001, the maximum number of iterations T = 100; perform parameter update Δθ = -(J T J + λI) -1 J T E, where Δθ = [Δk 1 , Δk 2 T ; after the update, perform convergence judgment. If the reprojection error decreases after the update E, accept the update, k 1 ← k 1 + Δk 1 , k 2 ← k 2 + Δk 2 , reduce the damping factor λ ← 0.1λ. If the reprojection error increases after the update, reject the update and increase the damping factor λ ← 10λ; the convergence condition is that the parameter change amount ||Δθ|| < 10 -6 , the error decrease amount ∣E new - E old ∣ < 10 -6 , reach the maximum number of iterations T.

[0103] Its technical effect is as follows: adopting a second-order radial distortion model to accurately describe the optical distortion of the lens, effectively reducing the image deviation caused by lens distortion through the optimization of the radial distortion coefficient; constructing a reprojection error function and minimizing its value to ensure that the error of each marker point in the calibration process is minimized at all viewing angles, optimizing the internal and external parameters of the camera, ensuring the accuracy of the vision system, and improving the accuracy of the needle tip positioning of the puncture robot; using the Jacobian matrix to calculate the partial derivative of the radial distortion coefficient, the introduction of the Jacobian matrix can accelerate convergence in the optimization process, avoiding the slow convergence problem caused by inappropriate initial values or local optimal solutions in traditional methods; through the adjustment of the damping factor, it can effectively avoid over-updating or premature convergence. If the reprojection error does not decrease significantly after the update, the system will appropriately increase the damping factor to avoid ineffective updates.

[0104] In a preferred embodiment of the present invention, in the above calibration method of the puncture robot based on binocular vision, the use of the particle swarm optimization algorithm to perform a global search on the initial calibration parameters and screening to obtain the optimal parameter set includes: encoding the initial calibration parameters as the particle position vector x = [f x , f y , u 0 , v 0 , k 1 , k 2 , θ​R1 , θ R2 , θ R3 , t x , t y , t z , where, (f x , f y , u 0 , v 0 ) are the parameters of the camera intrinsic matrix K i 's parameters, (k 1 , k 2 ) are the radial distortion coefficients, is the extrinsic rotation matrix R i 's Euler angle parameterization, (t x , t y , t z ) are the components of the translation vector t i ; According to the physical meaning, set the search space to perform range constraints on the initial calibration parameters, where, f x and f y are set near the initial focal length and are set according to the empirical range of the radial distortion coefficient; Set the particle swarm size, the number of particles N particles = 200, set the initial position of the central particle, and other particles are uniformly randomly distributed within the parameter range, and the initial velocity of the particles is set to 10% of the parameter range; For each particle j, update its velocity v j and position x j , the velocity update at the (t + 1)-th iteration is where, w is the inertia weight, used to balance global exploration and local exploitation, with the initial value w max = 0.9, linearly decreasing to w min = 0.4, c 1 and c 2 are the learning factors, which are the weights for controlling individual experience and social experience, c 1 = c 2 = 2, r 1 and r 2 are random numbers, used to increase the search randomness, uniformly distributed between [0, 1], and the position update is If the updated position exceeds the parameter range, then set it to the nearest boundary value for boundary truncation, and the boundary truncation formula is Record the historical optimal position of each particle as the optimal individual for update; Record the individual optimal position among all particles as the global optimal g best for update; When the set maximum number of iterations is reached, or the parameter change amount is less than the preset threshold, such as the focal length f x and fy When the change amount is less than 0.01%, terminate the iteration and output the global optimal parameter set g best .

[0105] Its technical effects are as follows: The Particle Swarm Optimization (PSO) algorithm can explore the entire parameter space through global search of the initial calibration parameters, avoiding the common local optimal solution problem in traditional optimization methods. PSO does not depend on the choice of the initial solution and can effectively find the global optimal solution, ensuring the best calibration results are found under various parameter combinations; during the particle swarm optimization process, by reasonably setting the search space and range constraints, invalid searches can be reduced, ensuring that particles search within a physically meaningful parameter space; the inertia weight and learning factor in the particle swarm optimization algorithm can be dynamically adjusted according to the feedback during the search process to balance global exploration and local exploitation, ensuring that the algorithm can conduct extensive exploration in the initial stage; through boundary truncation, the stability of the algorithm is ensured, avoiding invalid calculations during the search process.

[0106] In a preferred embodiment of the present invention, in the above-mentioned calibration method of the puncture robot based on binocular vision, using the optimal parameter set as input, an improved differential evolution algorithm is used to optimize the fitting error of the needle tip trajectory, and the optimized kinematic parameters obtained include: the optimal parameter set is p j =[K, d, R, t], where K is the camera internal parameter matrix, d is the radial distortion coefficient, R is the external parameter rotation matrix, and t is the translation vector; let the three-dimensional trajectory of the needle tip include the point set P = {p 1 , p 2 ,..., p i ,..., p N}, where p i =[x i , y i , z i is the needle tip position at each moment; define the calculation formula of the optimal fitness P real (t) is the true three-dimensional coordinate of the needle tip in the t-th frame image, P pred (t) is the predicted position of the needle tip calculated according to the kinematic model and calibration parameters, and N is the total number of time steps; minimize the objective function to minimize the fitting error and obtain the optimized kinematic parameters.

[0107] Its technical effects are as follows: For the needle tip position at each moment, by means of the optimal fitness calculation formula, the error between the true three-dimensional coordinates of the needle tip in each frame of image and the needle tip position predicted by the kinematic model is defined, so that at different time steps, the predicted position of the needle tip is as close as possible to the true position; The improved differential evolution algorithm MDE has a powerful global optimization ability, can avoid falling into local optimal solutions, and precisely adjusts the kinematic parameters through iterative optimization, minimizing the needle tip trajectory fitting error, and continuously providing high-precision needle tip positioning in complex and dynamic environments such as tissue deformation and blood interference; Minimize the objective function (fitting error function), minimize the error generated by the needle tip during movement, significantly reduce the deviation and instability in the robot movement, and ensure the precise tracking and positioning of the needle tip throughout the surgical process.

[0108] In a preferred embodiment of the present invention, in the above calibration method of the puncture robot based on binocular vision, the construction of the lightweight convolutional neural network, with binocular images as input to obtain high-precision needle tip coordinates includes: Taking the left and right images of the binocular camera as input, normalizing and aligning the sizes of the input images to ensure the spatial consistency of the binocular images; Designing a feature extraction backbone network, including an early feature fusion module and depthwise separable convolution blocks. The early feature fusion module adopts a two-stream input branch, and the left and right images respectively extract low-level features through independent shallow convolutional layers, and the feature maps of the left and right images are concatenated in the channel dimension to form a multi-modal fusion feature. Each depthwise separable convolution block includes a depth convolution for extracting features channel by channel and a pointwise convolution for inter-channel feature fusion. Using depthwise separable convolution to replace the standard convolution can significantly reduce the number of parameters and the amount of calculation; Designing an attention enhancement network, including a squeeze-and-excitation module and a spatial attention mask. The squeeze-and-excitation module compresses the spatial information of the feature map through global average pooling, learns channel weights, and strengthens important feature channels. The spatial attention mask generates a spatial weight map through a convolutional layer to highlight the significant features of the needle tip area; Designing a multi-scale feature fusion network, including a pyramid pooling module, using pooling kernels of different scales to extract multi-scale features, enhancing the network's sensitivity to the tiny displacement of the needle tip. After pooling, the features are upsampled to restore the resolution and fused with the original feature map; Designing an output layer, including a fully connected regression head, flattening the finally fused feature map into a vector, gradually reducing the dimension through two fully connected layers, and outputting the three-dimensional coordinates of the high-precision needle tip. The number of nodes in the output layer is 3.

[0109] Its technical effects are as follows: By normalizing and aligning the sizes of the input left and right binocular images, the spatial consistency of the images is ensured, the scale differences and offsets between the images can be eliminated, enabling subsequent feature extraction and coordinate calculation to be carried out based on a unified spatial coordinate system; The early feature fusion module extracts low-level features from the left and right images through independent shallow convolutional layers respectively, and splices the feature maps of the left and right images in the channel dimension to form multi-modal fusion features, making full use of the different perspective information of the left and right images, enhancing the network's ability to capture spatial information, and thus improving the positioning accuracy of the needle tip coordinates; Depthwise separable convolution is used instead of traditional convolution. Depthwise separable convolution decomposes the standard convolution operation into two steps: depth convolution and pointwise convolution. While ensuring the ability to extract features, it significantly reduces the computational cost and memory consumption, and greatly reduces the number of parameters and the amount of computation; In the squeeze-and-excitation module, the spatial information of the feature map is compressed through global average pooling to learn channel weights, thereby strengthening important feature channels. The spatial attention mask generates a spatial weight map through a convolutional layer to highlight the significant features in the needle tip area, which can effectively reduce the influence of background noise on the needle tip positioning accuracy; The pyramid pooling module extracts multi-scale features through pooling kernels of different scales, enhancing the network's sensitivity to the tiny displacement of the needle tip. Multi-scale feature fusion can help the network capture different levels of features in spaces of different scales. Especially when positioning the needle tip and detecting tiny displacements, it can better identify and locate the position of the needle tip. Through the upsampling of the pooled features and the fusion with the original feature map, the network can maintain high-resolution features and improve the positioning accuracy; The output layer of the network is designed as a fully connected regression head, which flattens the finally fused feature map into a vector in a step-by-step dimensionality reduction manner and outputs the three-dimensional coordinates of the high-precision needle tip, enabling the network to accurately map the image features to the three-dimensional position space of the needle tip.

[0110] In a preferred embodiment of the present invention, in the above calibration method of the puncture robot based on binocular vision, the three-dimensional reconstruction assisted by structured light is adopted, and the optimized kinematic parameters and the high-precision needle tip coordinates are incorporated into the needle tip kinematic model, and the three-dimensional motion trajectory of the needle tip is reconstructed as follows: A structured light assisted system is established, including a projection module and a binocular vision module. The projection module uses a high-precision blue light or infrared structured light projector to project a preset optical pattern onto the puncture needle and the surrounding area. The binocular vision module uses a synchronously triggered binocular camera to capture the scene images under the structured light irradiation from different perspectives respectively, perform image analysis, and output the three-dimensional point cloud motion trajectory of the needle tip; A multi-joint chain needle tip kinematic model is designed. According to the mechanical structure of the puncture robot (such as a serial manipulator), the joint coordinate system and link parameters (such as DH parameters) are defined. The pose of the needle tip (end effector) is calculated by forward kinematics. The input includes the optimized kinematic parameters and the high-precision needle tip coordinates, and the output of the model is the theoretical three-dimensional coordinates of the needle tip; In the multi-joint chain needle tip kinematic model, a joint clearance compensation error term and a thermal deformation compensation error term are introduced to optimize the output accuracy and obtain the high-precision three-dimensional coordinates of the needle tip; An extended Kalman filter model is designed. The high-precision three-dimensional coordinates of the needle tip kinematic model and the three-dimensional point cloud motion trajectory reconstructed by structured light are used as the model input, and the spline interpolation fitting is performed on the fused needle tip trajectory to eliminate high-frequency noise and optimize the trajectory smoothness, and the optimized three-dimensional motion trajectory of the needle tip is obtained.

[0111] The technical effects are as follows: The structured light assistance projects an optical pattern through a high-precision blue light or infrared structured light projector, and uses a synchronously triggered binocular camera to capture the scene images from different perspectives to obtain high-precision three-dimensional point cloud data. Through the generation of depth maps and point cloud reconstruction, the three-dimensional motion trajectory of the needle tip is accurately captured, enhancing the needle tip positioning accuracy, especially in complex and dynamic surgical environments; Through the multi-joint chain needle tip kinematic model, considering the mechanical structure and joint coordinate system of the puncture robot (such as DH parameters), the pose of the needle tip is accurately calculated. The combined optimized kinematic parameters and high-precision needle tip coordinates greatly improve the accuracy of the kinematic model, ensuring that the theoretical three-dimensional coordinates of the needle tip output by the model are more consistent with the actual position and accurately controlling the robot motion; The introduction of the joint clearance compensation error term and the thermal deformation compensation error term further optimizes the output accuracy of the kinematic model, compensating for the errors caused by factors such as mechanical structure, joint clearance or thermal deformation, and ensuring that the robot can maintain high-precision motion control in complex environments, especially in high-temperature operating environments; The extended Kalman filter model combines the needle tip kinematic model and the three-dimensional point cloud data reconstructed by structured light for fusion, eliminates the high-frequency noise in the reconstruction process, optimizes the smoothness of the trajectory, filters the noise through spline interpolation fitting, and ensures the smoothness of the needle tip motion trajectory, thereby improving the accuracy of needle tip positioning, especially during fast motion or surgical procedures, and can maintain stable and precise operation.

[0112] In a preferred embodiment of the present invention, in the above calibration method of the puncture robot based on binocular vision, the steps of image analysis of the structured light assist system include: performing optical pattern design, and sequentially projecting multiple sets of stripe patterns with different frequencies by the projector. Specifically, low-frequency thick stripes are used to quickly obtain depth information in a large range, and high-frequency thin stripes are used to improve the resolution of local details. According to the complexity of the intraoperative scene, such as tissue deformation and blood occlusion, the projection frequency and pattern type are adaptively adjusted to ensure stable imaging in a dynamic environment; analyzing the stripe images captured by the binocular cameras, and extracting the phase value of each point. Specifically, the phase value reflects the deformation degree of the stripe on the object surface and is directly related to the three-dimensional shape of the object surface; combining the phase values of different frequencies to eliminate the ambiguity caused by phase jumps and generating a continuous absolute phase map; combining the disparity information and phase difference of the binocular cameras, converting the absolute phase map into accurate three-dimensional depth data, and generating a high-resolution depth map; converting each pixel in the high-resolution depth map into a three-dimensional spatial point to form a dense point cloud of the needle tip and the surrounding tissues. Specifically, it further includes performing multi-frame fusion optimization, performing moving average or median filtering on consecutive multi-frame point clouds to suppress random noise, performing spatial registration, and based on the rigid body motion assumption of the needle tip, aligning the point cloud data at different times to ensure trajectory continuity; using a mesh generation algorithm (such as Poisson reconstruction) to convert the dense point cloud into a smooth three-dimensional surface model, highlighting the geometric features of the needle tip, and outputting the three-dimensional point cloud motion trajectory of the needle tip.

[0113] Its technical effects are as follows: By projecting multi-frequency stripe patterns, a wide range of depth information and high-resolution data of local details can be obtained simultaneously. The low-frequency thick stripes help to quickly acquire the rough depth information of the scene, while the high-frequency thin stripes can improve the resolution of local details; The accurate extraction of phase values enables each point in the image to correspond to a specific three-dimensional spatial position, thereby improving the accuracy of 3D reconstruction; By combining phase values of different frequencies, the ambiguity phenomenon caused by phase jumps can be eliminated, generating a continuous absolute phase map, effectively avoiding the error accumulation during phase unwrapping, and ensuring the continuity and accuracy of 3D reconstruction data; By combining parallax information and phase difference, through the parallax information of the binocular camera, the phase map can be converted into accurate three-dimensional depth data, and the generated high-resolution depth map provides detailed spatial information for the needle tip and surrounding tissues, effectively capturing minute displacements and detailed features; Through multi-frame fusion optimization, the random noise caused by environmental interference is effectively suppressed, ensuring the stability of the point cloud data, and aligning the point cloud data within multiple time steps. Through spatial registration, the continuity and accuracy of the needle tip trajectory are ensured. Based on the rigid body motion assumption of the needle tip, the point cloud data can accurately reflect the motion trajectory of the needle tip at different time points, ensuring the accuracy and reliability of motion analysis; Using a mesh generation algorithm (such as Poisson reconstruction) to process the dense point cloud can generate a smooth three-dimensional surface model, highlighting the geometric features of the needle tip. Through this reconstruction method, the point cloud data is converted into a three-dimensional model with accurate geometric features, greatly improving the reconstruction accuracy of the needle tip position and providing higher-quality visual support for the three-dimensional motion trajectory of the needle tip.

[0114] The second embodiment of the present invention provides a calibration system for a puncture robot based on binocular vision, which includes: a multi-viewpoint geometry calibration module for designing a multi-modal calibration board to perform multi-viewpoint geometry calibration on the puncture robot; an initial calibration parameter determination module for constructing a polynomial distortion model for the puncture robot based on multi-viewpoint geometry calibration, optimizing the radial distortion coefficient, and obtaining initial calibration parameters; an optimal parameter set screening module for globally searching the initial calibration parameters using a particle swarm optimization algorithm and screening out the optimal parameter set; a kinematic parameter optimization module for using the optimal parameter set as input and adopting an improved differential evolution algorithm to optimize the fitting error of the needle tip trajectory to obtain optimized kinematic parameters; a convolutional neural network module for constructing a lightweight convolutional neural network, taking binocular images as input, and obtaining high-precision needle tip coordinates; a three-dimensional reconstruction module for performing three-dimensional reconstruction assisted by structured light, integrating the optimized kinematic parameters and the high-precision needle tip coordinates into the needle tip kinematic model, and reconstructing the three-dimensional motion trajectory of the needle tip.

[0115] The third embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the calibration method of the binocular vision-based puncture robot as described above.

[0116] The computer program product of the calibration method and device of the binocular vision-based puncture robot provided by the embodiments of the present invention includes a computer-readable storage medium storing program codes. The instructions included in the program codes can be used to execute the methods in the foregoing method embodiments. For specific implementation, reference can be made to the method embodiments, which will not be elaborated herein.

[0117] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium runs, it can execute the above-mentioned calibration method of the binocular vision-based puncture robot, thereby improving the positioning accuracy of the puncture robot and enhancing its stability and real-time performance in a complex surgical environment.

[0118] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0119] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A calibration method for a puncture robot based on binocular vision, characterized in that: include: Design a multi-modal calibration board to perform multi-viewpoint geometric calibration on the puncture robot; Based on multi-viewpoint geometric calibration, a polynomial distortion model is constructed for the puncture robot, the radial distortion coefficient is optimized, and the initial calibration parameters are obtained; The particle swarm optimization algorithm is used to perform a global search on the initial calibration parameters and screen out the optimal parameter set; Taking the optimal parameter set as input, an improved differential evolution algorithm is used to optimize the needle tip trajectory fitting error to obtain optimized kinematic parameters; Construct a lightweight convolutional neural network, use binocular images as input, and obtain high-precision needle tip coordinates; The three-dimensional reconstruction is performed with the assistance of structured light, and the optimized kinematic parameters and the high-precision needle tip coordinates are integrated into the needle tip kinematic model to reconstruct the three-dimensional motion trajectory of the needle tip.

2. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The design of the multi-modal calibration board to perform multi-viewpoint geometric calibration on the puncture robot includes: A matte black carbon fiber plate is selected as a calibration plate, and a periodic fluorescent marker array is embedded on the surface of the calibration plate; Control the binocular camera fixed at the end of the robot to shoot the calibration plate from 2N non-coplanar viewing angles, where N is a natural number greater than 0, to obtain a set of continuous frame images, extract the fluorescent marking point from the set of continuous frame images, and locate the central pixel coordinate of the fluorescent marking point; Assume that the world coordinates of the kth marking point on the calibration plate are P k =[X k ,Y k ,0] T , the pixel coordinate projected to the i-th viewing angle image is p ik =[u ik ,v ik ] T , construct the projection equation Among them, K i is the camera intrinsic parameter matrix, R i Extrinsic rotation matrix, t i is the translation vector, λ ik is the scale factor; Through the singular value decomposition method, the camera intrinsic parameter matrix K is obtained i The initial value of Solve to get the external parameter rotation matrix R i and the translation vector t i The value of .

3. The calibration method of the puncture robot based on binocular vision according to claim 2, characterized in that: Based on the multi-viewpoint geometric calibration, a polynomial distortion model is constructed for the puncture robot, and the radial distortion coefficient is optimized to obtain the initial calibration parameters including: Using the second-order radial distortion model to describe lens distortion Among them, (u,v) is the distorted pixel coordinate, (u corrected ,v corrected ) is the corrected coordinate, (u0, v0) is the principal point coordinate, and from the camera intrinsic parameter matrix K i Get represents the radial distance from the pixel to the principal point, k1 and k2 are the radial distortion coefficients to be optimized; The world coordinate P k =[X k ,Y k ,0] T Transform to the camera coordinate system through the external parameter matrix Project the 3D point in the camera coordinate system onto the normalized plane and calculate the normalized coordinates Calculate the square of the radial distance r from the normalized coordinate to the principal point 2 =x n 2 +y n 2 , and obtain the corrected normalized coordinates (x d =x n (1+k1r 2 +k2r 4 ),y d =y n (1+k1r 2 +k2r 4 )); Map the corrected normalized coordinates to the actual pixel coordinate system Among them, (f x ,f y ) is the focal length of the camera; Construct the reprojection error function, which is defined as the sum of squared errors of all markers at all viewing angles, The goal is to adjust the radial distortion coefficients k1 and k2 to be optimized through the optimization algorithm to minimize the reprojection error E, where p ik =(u ik ,v ik ) is the actual observation coordinate, is the theoretical projection coordinate; For each marker point, the partial derivatives of the radial distortion coefficients k1 and k2 are calculated through the Jacobian matrix right Find partial derivatives right Find partial derivatives Perform iterative initialization, set the initial distortion coefficients k1=0, k2=0, the initial damping factor λ=0.001, and the maximum number of iterations T=100; Perform parameter update Δθ=-(J T J+λI) -1 J T E, where Δθ=[Δk1,Δk2] T ; After the update, a convergence judgment is performed. If the reprojection error is reduced by E after the update, the update is accepted, k1←k1+Δk1, k2←k2+Δk2, and the damping factor λ←0.1λ is reduced. If the reprojection error increases after the update, the update is rejected and the damping factor λ←10λ is increased. The convergence condition is that the parameter change ||Δθ||<10 -6 , error reduction |E new -E old ∣<10 -6 , reaching the maximum number of iterations T.

4. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The particle swarm optimization algorithm is used to perform a global search on the initial calibration parameters, and the optimal parameter set obtained by screening includes: Encode the initial calibration parameters as particle position vectors Among them, (f x ,f y ,u0,v0) is the camera internal parameter matrix K i , (k1, k2) are radial distortion coefficients, is the external parameter rotation matrix R i Euler angle parameterization, (t x ,t y ,t z ) is the translation vector t i The amount of According to the physical meaning, set the search space Performing range constraints on the initial calibration parameters; Set the particle swarm size, the number of particles N particles =200, set the initial position of the central particle, and the other particles are evenly and randomly distributed within the parameter range. The initial velocity of the particles is set to 10% of the parameter range; For each particle j, update its velocity v j and position x j , the speed of the t+1th iteration is updated to Among them, w is the inertia weight, the initial value w max =0.9, linearly decreasing to w min =0.4, c1 and c2 are learning factors, c1=c2=2, r1 and r2 are random numbers, evenly distributed between [0,1], and the position update is If the updated position If the parameter value exceeds the range, it is set to the nearest boundary value for boundary truncation. The boundary truncation formula is: Record the historical optimal position of each particle as the optimal individual p bestj Make updates; Record the individual optimal positions of all particles as the global optimal g best Make updates; When the maximum number of iterations is reached or the parameter change is less than the preset threshold, the iteration is terminated and the global optimal parameter set g is output. best .

5. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The optimal parameter set is used as input, and the improved differential evolution algorithm is used to optimize the needle tip trajectory fitting error, and the optimized kinematic parameters obtained include: The optimal parameter set is p j =[K, d, R, t], where K is the camera intrinsic matrix, d is the radial distortion coefficient, R is the extrinsic rotation matrix, and t is the translation vector; Assume that the three-dimensional trajectory of the needle tip includes the point set P = {p1, p2, ..., p i ,...,p N }, where p i =[x i ,y i ,z i ] is the needle tip position at each moment; Define the calculation formula for optimal fitness P real (t) is the true three-dimensional coordinate of the needle tip in the t-th frame image, P pred (t) is the predicted position of the needle tip calculated based on the kinematic model and calibration parameters, and N is the total number of time steps; Minimize the objective function The fitting error is minimized and the optimized kinematic parameters are obtained.

6. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The method of constructing a lightweight convolutional neural network and taking a binocular image as input to obtain high-precision needle tip coordinates includes: The left and right images of the stereo camera are taken as input, and the input images are normalized and size-aligned to ensure the spatial consistency of the stereo images; Design a feature extraction backbone network, including an early feature fusion module and a deep separable convolution block. The early feature fusion module adopts a dual-stream input branch. The left and right images are respectively subjected to independent shallow convolution layers to extract low-level features. The feature maps of the left and right images are spliced ​​in the channel dimension to form multimodal fusion features. Each of the deep separable convolution blocks contains a deep convolution for extracting features channel by channel and a point-by-point convolution for inter-channel feature fusion. Design an attention enhancement network, including a compression excitation module and a spatial attention mask. The compression excitation module compresses the spatial information of the feature map through global average pooling, learns channel weights, and strengthens important feature channels. The spatial attention mask generates a spatial weight map through a convolutional layer to highlight the salient features of the needle tip area. Design a multi-scale feature fusion network, including a pyramid pooling module, which uses pooling kernels of different scales to extract multi-scale features and enhance the network's sensitivity to tiny displacements of the needle tip. After pooling, the features are upsampled to restore the resolution and fused with the original feature map. The output layer is designed, including a fully connected regression head, which flattens the final fused feature map into a vector, gradually reduces the dimension through two fully connected layers, and outputs the three-dimensional coordinates of the high-precision needle tip. The number of nodes in the output layer is 3.

7. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The three-dimensional reconstruction is performed with the aid of structured light, and the optimized kinematic parameters and the high-precision needle tip coordinates are integrated into the needle tip kinematic model to reconstruct the three-dimensional motion trajectory of the needle tip, including: Establish a structured light auxiliary system, including a projection module and a binocular vision module. The projection module uses a high-precision blue light or infrared structured light projector to project a preset optical pattern onto the puncture needle and the surrounding area. The binocular vision module uses a synchronously triggered binocular camera to capture scene images under structured light illumination from different perspectives, perform image analysis, and output a three-dimensional point cloud motion trajectory of the needle tip. A multi-joint chain needle tip kinematic model is designed. According to the mechanical structure of the puncture robot, the joint coordinate system and the connecting rod parameters are defined. The position and posture of the needle tip are calculated by forward kinematics. The input includes the optimized kinematic parameters and the high-precision needle tip coordinates. The model output is the theoretical three-dimensional coordinates of the needle tip. Joint clearance compensation error terms and thermal deformation compensation error terms are introduced into the multi-joint chain needle tip kinematic model to optimize output accuracy and obtain high-precision three-dimensional coordinates of the needle tip; An extended Kalman filter model is designed, and the high-precision three-dimensional coordinates of the needle tip kinematic model and the three-dimensional point cloud motion trajectory reconstructed by structured light are used as model input. Spline interpolation fitting is performed on the fused needle tip trajectory to eliminate high-frequency noise, optimize trajectory smoothness, and obtain the optimized three-dimensional motion trajectory of the needle tip.

8. The binocular vision-based puncture robot calibration method according to claim 7, characterized in that: The steps of image analysis of the structured light auxiliary system include: Design the light pattern, and use the projector to sequentially project multiple sets of stripe patterns with different frequencies; Analyze the fringe image captured by the binocular camera and extract the phase value of each point; Combine phase values ​​of different frequencies to eliminate ambiguity caused by phase jumps and generate a continuous absolute phase map; Combining the parallax information and phase difference of the binocular camera, the absolute phase map is converted into accurate three-dimensional depth data to generate a high-resolution depth map; Convert each pixel in the high-resolution depth map into a three-dimensional spatial point to form a dense point cloud of the needle tip and surrounding tissue; The dense point cloud is converted into a smooth three-dimensional surface model using a mesh generation algorithm, the geometric features of the needle tip are highlighted, and the output is a three-dimensional point cloud motion trajectory of the needle tip.

9. A calibration system for a puncture robot based on binocular vision, characterized in that: include: Multi-viewpoint geometric calibration module, used to design multi-modal calibration plates and perform multi-viewpoint geometric calibration on the puncture robot; An initial calibration parameter determination module is used to construct a polynomial distortion model for the puncture robot based on multi-viewpoint geometric calibration, optimize the radial distortion coefficient, and obtain initial calibration parameters; The optimal parameter set screening module is used to perform a global search on the initial calibration parameters using the particle swarm optimization algorithm to screen out the optimal parameter set; A kinematic parameter optimization module, which is used to optimize the needle tip trajectory fitting error using the optimal parameter set as input, and obtain optimized kinematic parameters by using an improved differential evolution algorithm; Convolutional neural network module, used to build a lightweight convolutional neural network, using binocular images as input to obtain high-precision needle tip coordinates; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction with the aid of structured light, integrate the optimized kinematic parameters and the high-precision needle tip coordinates into the needle tip kinematic model, and reconstruct the three-dimensional motion trajectory of the needle tip.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the calibration method of the puncture robot based on binocular vision as described in any one of claims 1 to 8 is implemented.

Images