A calibration method and system for a puncture robot based on binocular vision

Abstract

The present invention discloses a calibration method and system for a puncture robot based on binocular vision. The method includes designing a multimodal calibration plate, constructing a polynomial distortion model for the puncture robot based on multi-viewpoint geometric calibration, and optimizing the radial distortion coefficient; using a particle swarm optimization algorithm to perform a global search on the initial calibration parameters and screen out the optimal parameter set; using an improved differential evolution algorithm to optimize the needle tip trajectory fitting error and obtain optimized kinematic parameters; constructing a lightweight convolutional neural network, using binocular images as input, to obtain high-precision needle tip coordinates; and using structured light to assist in three-dimensional reconstruction to obtain the three-dimensional motion trajectory of the needle tip. The system includes a multi-viewpoint geometric calibration module, an initial calibration parameter determination module, an optimal parameter set screening module, a kinematic parameter optimization module, a convolutional neural network module, and a three-dimensional reconstruction module. The present invention improves the positioning accuracy of the puncture robot and enhances its stability and real-time performance in complex surgical environments.

Description

Technical Field

[0001] The present invention relates to the technical field of intersection of medical image processing and robot precision control, and in particular to a calibration method and system for a puncture robot based on binocular vision. Background Art

[0002] With the continuous advancement of medical imaging and robotics technologies, minimally invasive surgeries such as percutaneous biopsy and tumor ablation are becoming increasingly widespread in the medical field. To ensure the high precision and safety of these surgeries, the positioning and control accuracy of puncture robots have become key technologies. Robot-assisted surgical systems typically use visual feedback to assist with precise positioning and control, and the calibration accuracy of the visual system directly determines the accuracy and safety of surgical operations. In recent years, with the rapid development of deep learning, computer vision, and robotics, vision-based surgical robot positioning systems have gradually become a research hotspot. In particular, binocular vision technology, which uses images acquired by two cameras to reconstruct three-dimensional spatial information, has been widely used in puncture robots.

[0003] With the popularization of minimally invasive surgical techniques, especially the continuous development of robot-assisted puncture surgery (such as tumor ablation and biopsy), the market demand for precise positioning, motion control, and real-time feedback is increasing. Accurate positioning and control of puncture robots will directly affect the success rate of surgery and the patient's recovery. Therefore, how to improve the calibration accuracy of puncture robots and ensure the stable operation of robots in complex surgical environments has become a hot topic in current technical research. The current market demand for vision-based puncture robot systems is growing rapidly, especially in the following aspects:

[0004] (1) High-precision positioning. In order to reduce accidental injuries and ensure surgical accuracy, accurate needle tip positioning and motion control are key requirements in minimally invasive surgery. (2) Real-time feedback and intelligent adjustment. During the operation, the robot system needs to adjust its motion state and position in real time. Therefore, the calibration method needs to have real-time and adaptive capabilities. (3) Anti-interference ability. During the operation, factors such as blood, tissue deformation and instrument movement may cause image interference. Therefore, the system needs to have strong robustness to cope with changes in complex environments.

[0005] In the existing technology, the puncture robot calibration method based on binocular vision usually has the following technical defects and problems:

[0006] (1) Traditional calibration methods have limited accuracy. Traditional calibration methods usually rely on a checkerboard calibration plate for geometric calibration. Although this method is widely used in control systems, its accuracy is often not ideal in complex surgical environments. Especially in the face of dynamically changing surgical environments, such as blood and tissue occlusion, existing calibration methods are difficult to provide sufficient robustness, resulting in low needle tip positioning accuracy and large surgical errors.

[0007] (2) Insufficient distortion correction. The optical distortion of the camera lens is an important factor affecting visual accuracy. Although some existing methods use radial distortion models for correction, in actual applications, due to the non-ideal optical properties of the lens and the complex surgical environment, the distortion correction is often insufficient, thus affecting the calibration accuracy. Especially in minimally invasive surgery, the precise positioning of the needle tip is very demanding, and the impact of uncorrected distortion is particularly prominent.

[0008] (3) Limitations of optimization algorithms. Existing calibration methods mostly rely on traditional nonlinear optimization algorithms for parameter optimization, such as the least squares method and Newton's method. These methods are prone to falling into local optimal solutions and are difficult to guarantee the accuracy of the global optimal solution. Especially for calibration problems with multiple unknown parameters, these traditional optimization algorithms often converge slowly and are sensitive to the choice of initial parameters, resulting in an unstable calibration process.

[0009] (4) Real-time and robustness issues. Currently, many calibration methods have high computational complexity and are difficult to meet the needs of real-time surgical scenarios. For example, deep learning methods have a large computational workload, especially when running on embedded platforms, and may not be able to guarantee sufficient real-time performance. In addition, dynamic interference during surgery, such as tissue occlusion and rapid movement of the needle tip, can also affect the robustness of existing systems, resulting in inaccurate positioning or failure. Summary of the Invention

[0010] In view of this, the purpose of an embodiment of the present invention is to provide a calibration method and system for a puncture robot based on binocular vision. Through a multimodal calibration plate, particle swarm optimization, differential evolution algorithm, lightweight convolutional neural network and structured light-assisted three-dimensional reconstruction, it has the advantages of high precision, low cost, easy operation, strong anti-interference, and high system robustness, thereby improving the positioning accuracy of the puncture robot and enhancing its stability and real-time performance in complex surgical environments.

[0011] The embodiment of the present invention is achieved as follows:

[0012] A calibration method for a puncture robot based on binocular vision, comprising:

[0013] A multimodal calibration plate is designed to perform multi-viewpoint geometric calibration on the puncture robot.

[0014] Based on multi-viewpoint geometric calibration, a polynomial distortion model is constructed for the puncture robot, and the radial distortion coefficient is optimized to obtain initial calibration parameters.

[0015] The particle swarm optimization algorithm is used to perform a global search on the initial calibration parameters and screen out the optimal parameter set.

[0016] Taking the optimal parameter set as input, an improved differential evolution algorithm is used to optimize the needle tip trajectory fitting error to obtain optimized kinematic parameters.

[0017] A lightweight convolutional neural network is constructed, which takes binocular images as input to obtain high-precision needle tip coordinates.

[0018] Structured light is used to assist in three-dimensional reconstruction, and the optimized kinematic parameters and the high-precision needle tip coordinates are integrated into the needle tip kinematic model to reconstruct the three-dimensional motion trajectory of the needle tip.

[0019] In a preferred embodiment of the present invention, in the above-mentioned binocular vision-based puncture robot calibration method, the design of a multimodal calibration plate to perform multi-viewpoint geometric calibration on the puncture robot includes:

[0020] A black matte carbon fiber plate is selected as the calibration plate, and a periodic fluorescent marker array is embedded on the surface of the calibration plate.

[0021] Control the binocular camera fixed at the end of the robot to shoot the calibration plate from 2N non-coplanar perspectives, where N is a natural number greater than 0, to obtain a set of continuous frame images, extract the fluorescent marker from the set of continuous frame images, and locate the central pixel coordinates of the fluorescent marker.

[0022] Assume that the world coordinates of the kth marking point on the calibration plate are P k =[X k ,Y k ,0] T , the pixel coordinate projected to the i-th perspective image is p ik =[u ik ,v ik ] T , construct the projection equation Among them, K i is the camera intrinsic parameter matrix, R i Extrinsic rotation matrix, t i is the translation vector, λ ik is the scale factor.

[0023] By using the singular value decomposition method, the camera intrinsic parameter matrix K is obtained. i The initial value of .

[0024] Solve to get the external parameter rotation matrix R i and the translation vector ti The value of .

[0025] Its technical effects are as follows: the calibration plate uses a periodic fluorescent marker array to provide more precise calibration points, enabling more accurate geometric information to be obtained when shooting from different perspectives; the design of the calibration plate includes three dynamic modes, which improves the robustness of the calibration through the periodic fluorescent marker array, helps to suppress ambient light interference, adapts to traditional calibration algorithms, and increases the ability to adapt to environmental changes and uneven lighting; the calibration plate integrates an FPGA chip, and uses PWM to modulate the LED light emission timing to achieve 1Hz / 5Hz / 10Hz multi-frequency flashing to synchronize the binocular camera exposure signal, effectively suppressing the interference of ambient light noise.

[0026] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the multi-viewpoint geometric calibration is based on constructing a polynomial distortion model for the puncture robot, optimizing the radial distortion coefficient, and obtaining the initial calibration parameters including:

[0027] Use the second-order radial distortion model to describe lens distortion Among them, (u,v) is the distorted pixel coordinate, (u corrected ,v corrected ) is the corrected coordinate, (u0, v0) is the principal point coordinate, and the camera intrinsic parameter matrix K i Get represents the radial distance from the pixel to the principal point, k1 and k2 are the radial distortion coefficients to be optimized.

[0028] The world coordinate P k =[X k ,Y k ,0] T Transform to the camera coordinate system through the external parameter matrix

[0029] Project the 3D point in the camera coordinate system onto the normalized plane and calculate the normalized coordinates

[0030] Calculate the square of the radial distance from the normalized coordinate to the principal point Get the normalized coordinates after correction (x d =x n (1+k1r 2 +k2r 4 ),y d =y n (1+k1r 2 +k2r 4 )).

[0031] Map the corrected normalized coordinates to the actual pixel coordinate system Among them, (f x ,f y ) is the focal length of the camera.

[0032] Construct the reprojection error function, which is defined as the sum of squared errors of all markers under all viewing angles, The goal is to adjust the radial distortion coefficients k1 and k2 to be optimized through the optimization algorithm to minimize the reprojection error E, where p ik =(u ik ,v ik ) is the actual observation coordinate, are the theoretical projection coordinates.

[0033] For each marker point, the partial derivatives of the radial distortion coefficients k1 and k2 are calculated through the Jacobian matrix right Find partial derivatives right Find partial derivatives

[0034] Perform iterative initialization, set the initial distortion coefficients k1=0, k2=0, the initial damping factor λ=0.001, and the maximum number of iterations T=100.

[0035] Perform parameter update Δθ=-(J T J+λI) -1 J T E, where Δθ=[Δk1,Δk2] T .

[0036] After the update, a convergence check is performed. If the reprojection error decreases by E after the update, the update is accepted, k1←k1+Δk1, k2←k2+Δk2, and the damping factor λ←0.1λ is reduced. If the reprojection error increases after the update, the update is rejected and the damping factor λ←10λ is increased.

[0037] The convergence condition is that the parameter change ||Δθ||<10 -6 , error reduction |E new -E old ∣<10 -6 , reaching the maximum number of iterations T.

[0038] Its technical effects are: using a second-order radial distortion model to accurately describe the optical distortion of the lens, and effectively reducing the image deviation caused by lens distortion by optimizing the radial distortion coefficient; constructing a reprojection error function and minimizing its value to ensure that the error of each marking point in the calibration process is minimized under all viewing angles, optimizing the internal and external parameters of the camera, ensuring the accuracy of the visual system, and improving the accuracy of the needle tip positioning of the puncture robot; using the Jacobian matrix to calculate the partial derivatives of the radial distortion coefficient, the introduction of the Jacobian matrix can accelerate convergence in the optimization process, avoiding the slow convergence problem caused by inappropriate initial values ​​or local optimal solutions in traditional methods; by adjusting the damping factor, it can effectively avoid excessive updates or premature convergence. If the reprojection error does not decrease significantly after the update, the system will appropriately increase the damping factor to avoid invalid updates.

[0039] In a preferred embodiment of the present invention, in the above-mentioned binocular vision-based puncture robot calibration method, the particle swarm optimization algorithm is used to perform a global search on the initial calibration parameters to screen out the optimal parameter set, which includes:

[0040] Encode the initial calibration parameters as particle position vectors Among them, (f x ,f y ,u0,v0) is the camera intrinsic parameter matrix K i Parameters, (k1, k2) are radial distortion coefficients, is the external parameter rotation matrix R i Euler angle parameterization, (t x ,t y ,t z ) is the translation vector t i The weight.

[0041] According to the physical meaning, set the search space The initial calibration parameters are subject to range constraints.

[0042] Set the particle swarm size, the number of particles N particles =200, set the initial position of the central particle, and the other particles are evenly and randomly distributed within the parameter range. The initial velocity of the particles is set to 10% of the parameter range.

[0043] For each particle j, update its velocity v j and position x j , the speed update of the t+1th iteration is Among them, w is the inertia weight, the initial value w max =0.9, linearly decreasing to w min =0.4, c1 and c2 are learning factors, c1=c2=2, r1 and r2 are random numbers, evenly distributed between [0,1], and the position update is

[0044] If the updated position If the parameter range is exceeded, it is set to the nearest boundary value and the boundary is truncate. The boundary truncation formula is:

[0045] Record the historical optimal position of each particle as the optimal individual to update.

[0046] Record the individual optimal positions of all particles as the global optimal g best to update.

[0047] When the maximum number of iterations is reached or the parameter change is less than the preset threshold, the iteration is terminated and the global optimal parameter set g is output. best .

[0048] Its technical effects are as follows: the particle swarm optimization algorithm (PSO) can explore the entire parameter space by performing a global search on the initial calibration parameters, avoiding the local optimal solution problem common in traditional optimization methods. PSO does not rely on the selection of the initial solution and can effectively find the global optimal solution, ensuring that the best calibration result is found under multiple parameter combinations; in the particle swarm optimization process, by reasonably setting the search space and range constraints, invalid searches can be reduced, ensuring that particles search within a physically meaningful parameter space; the inertia weight and learning factor in the particle swarm optimization algorithm can be dynamically adjusted according to feedback during the search process to balance global exploration and local development, ensuring that the algorithm can conduct extensive exploration in the initial stage; through boundary truncation, the stability of the algorithm is ensured, and invalid calculations during the search process are avoided.

[0049] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the optimal parameter set is used as input, and the improved differential evolution algorithm is used to optimize the needle tip trajectory fitting error to obtain the optimized kinematic parameters, which include:

[0050] The optimal parameter set is p j =[K,d,R,t], where K is the camera intrinsic matrix, d is the radial distortion coefficient, R is the extrinsic rotation matrix, and t is the translation vector.

[0051] Assume that the three-dimensional trajectory of the needle tip includes the point set P = {p1, p2, ..., p i ,...,p N}, where p i =[x i ,y i ,z i ] is the needle tip position at each moment.

[0052] Define the calculation formula for optimal fitness P real (t) is the true three-dimensional coordinate of the needle tip in the t-th frame image, P pred (t) is the predicted position of the needle tip calculated based on the kinematic model and calibration parameters, and N is the total number of time steps.

[0053] Minimize the objective function The fitting error is minimized and the optimized kinematic parameters are obtained.

[0054] Its technical effects are: for the needle tip position at each moment, the optimal fitness calculation formula is used to define the error between the real three-dimensional coordinates of the needle tip in each frame image and the needle tip position predicted by the kinematic model, so that the predicted position of the needle tip is as close to the real position as possible at different time steps; the improved differential evolution algorithm MDE has a strong global optimization capability, which can avoid falling into the local optimal solution, and accurately adjust the kinematic parameters through iterative optimization to minimize the needle tip trajectory fitting error, and continuously provide high-precision needle tip positioning in complex and dynamic environments, such as tissue deformation, blood interference, etc.; minimize the objective function (fitting error function) to minimize the error generated by the needle tip during movement, significantly reduce the deviation and instability in the robot movement, and ensure the accurate tracking and positioning of the needle tip throughout the operation.

[0055] In a preferred embodiment of the present invention, in the above-mentioned binocular vision-based puncture robot calibration method, the construction of a lightweight convolutional neural network, using binocular images as input to obtain high-precision needle tip coordinates includes:

[0056] The left and right images of the binocular camera are taken as input, and the input images are normalized and size aligned to ensure the spatial consistency of the binocular images.

[0057] A feature extraction backbone network is designed, including an early feature fusion module and a depthwise separable convolution block. The early feature fusion module adopts a dual-stream input branch. The left and right images are respectively subjected to independent shallow convolution layers to extract low-level features. The feature maps of the left and right images are spliced ​​in the channel dimension to form multimodal fusion features. Each of the depthwise separable convolution blocks contains depthwise convolution for channel-by-channel feature extraction and point-by-point convolution for inter-channel feature fusion.

[0058] An attention enhancement network is designed, including a compression excitation module and a spatial attention mask. The compression excitation module compresses the spatial information of the feature map through global average pooling, learns channel weights, and strengthens important feature channels. The spatial attention mask generates a spatial weight map through a convolutional layer to highlight the salient features of the needle tip area.

[0059] A multi-scale feature fusion network is designed, including a pyramid pooling module. Pooling kernels of different scales are used to extract multi-scale features to enhance the network's sensitivity to tiny displacements of the needle tip. After pooling, the features are upsampled to restore the resolution and fused with the original feature map.

[0060] The output layer is designed, including a fully connected regression head, which flattens the final fused feature map into a vector. The dimensionality is gradually reduced through two fully connected layers to output the high-precision three-dimensional coordinates of the needle tip. The number of nodes in the output layer is 3.

[0061] Its technical effects are: by normalizing and aligning the size of the input left and right binocular images, the spatial consistency of the images is ensured, the scale differences and offsets between images can be eliminated, and the subsequent feature extraction and coordinate calculation can be based on a unified spatial coordinate system; the early feature fusion module extracts low-level features from the left and right images through independent shallow convolution layers, and splices the feature maps of the left and right images in the channel dimension to form multimodal fusion features, making full use of the different perspective information of the left and right images, enhancing the network's ability to capture spatial information, and thus improving the positioning accuracy of the needle tip coordinates; using depthwise separable convolution instead of traditional convolution, depthwise separable convolution decomposes the standard convolution operation into two steps, depthwise convolution and pointwise convolution, while ensuring the ability to extract features, significantly reducing the computational cost and memory consumption, and greatly reducing the number of parameters and calculations; in the compression excitation module, the spatial information of the feature map is compressed by global average pooling. The spatial attention mask generates a spatial weight map through the convolution layer, highlighting the significant features of the needle tip area, which can effectively reduce the impact of background noise on the needle tip positioning accuracy; the pyramid pooling module extracts multi-scale features through pooling kernels of different scales, enhancing the network's sensitivity to tiny displacements of the needle tip. Multi-scale feature fusion can help the network capture features at different levels in spaces of different scales, especially in needle tip positioning and tiny displacement detection, which can better identify and locate the needle tip position. By upsampling the pooled features and fusing them with the original feature map, the network can maintain high-resolution features and improve positioning accuracy. The output layer of the network is designed as a fully connected regression head, which flattens the final fused feature map into a vector through gradual dimensionality reduction and outputs the high-precision three-dimensional coordinates of the needle tip, so that the network can accurately map image features to the three-dimensional position space of the needle tip.

[0062] In a preferred embodiment of the present invention, in the above-mentioned binocular vision-based puncture robot calibration method, the structured light-assisted three-dimensional reconstruction is performed, and the optimized kinematic parameters and the high-precision needle tip coordinates are integrated into the needle tip kinematic model to reconstruct the three-dimensional motion trajectory of the needle tip, including:

[0063] A structured light assistance system is established, including a projection module and a binocular vision module. The projection module uses a high-precision blue light or infrared structured light projector to project a preset optical pattern onto the puncture needle and the surrounding area. The binocular vision module uses a synchronously triggered binocular camera to capture scene images under structured light illumination from different perspectives, perform image analysis, and output a three-dimensional point cloud motion trajectory of the needle tip.

[0064] A multi-joint chain needle tip kinematic model is designed. According to the mechanical structure of the puncture robot, the joint coordinate system and link parameters are defined. The position of the needle tip is calculated by forward kinematics. The input includes the optimized kinematic parameters and the high-precision needle tip coordinates. The model output is the theoretical three-dimensional coordinates of the needle tip.

[0065] Joint clearance compensation error terms and thermal deformation compensation error terms are introduced into the multi-joint chain needle tip kinematic model to optimize output accuracy and obtain high-precision three-dimensional coordinates of the needle tip.

[0066] An extended Kalman filter model is designed, and the high-precision three-dimensional coordinates of the needle tip kinematic model and the three-dimensional point cloud motion trajectory reconstructed by structured light are used as model input. Spline interpolation fitting is performed on the fused needle tip trajectory to eliminate high-frequency noise and optimize the trajectory smoothness to obtain the optimized three-dimensional motion trajectory of the needle tip.

[0067] Its technical effects are: structured light assists in projecting optical patterns through high-precision blue light or infrared structured light projectors, and uses synchronously triggered binocular cameras to capture scene images from different perspectives to obtain high-precision three-dimensional point cloud data. Through the generation of depth maps and point cloud reconstruction, the three-dimensional motion trajectory of the needle tip is accurately captured, and the needle tip positioning accuracy is enhanced, especially in complex and dynamic surgical environments; through the multi-joint chain needle tip kinematic model, the mechanical structure and joint coordinate system (such as DH parameters) of the puncture robot are considered to accurately calculate the position and posture of the needle tip. The optimized kinematic parameters are combined with high-precision needle tip coordinates to greatly improve the accuracy of the kinematic model, ensure that the theoretical three-dimensional coordinates of the needle tip output by the model are more consistent with the actual position, and accurately control the robot movement; the introduction of joint gap compensation error terms and thermal deformation compensation error terms further optimizes the output accuracy of the kinematic model. , to compensate for errors caused by factors such as mechanical structure, joint clearance or thermal deformation, and ensure that the robot can maintain high-precision motion control in complex environments, especially high-temperature operating environments; the extended Kalman filter model is combined with the needle tip kinematic model and the three-dimensional point cloud data of structured light reconstruction to eliminate high-frequency noise in the reconstruction process, optimize the smoothness of the trajectory, and filter out noise through spline interpolation fitting to ensure the smoothness of the needle tip motion trajectory, thereby improving the accuracy of needle tip positioning, especially during rapid movement or surgery, and being able to maintain stable and precise operation.

[0068] In a preferred embodiment of the present invention, in the above-mentioned binocular vision-based puncture robot calibration method, the step of image analysis of the structured light-assisted system includes:

[0069] The light pattern is designed, and the projector projects multiple sets of stripe patterns of different frequencies in sequence.

[0070] The fringe image captured by the binocular camera is analyzed and the phase value of each point is extracted.

[0071] Combining phase values ​​at different frequencies eliminates ambiguity caused by phase jumps and generates a continuous absolute phase map.

[0072] The absolute phase map is converted into accurate three-dimensional depth data by combining the parallax information and phase difference of the binocular camera to generate a high-resolution depth map.

[0073] Each pixel in the high-resolution depth map is converted into a three-dimensional spatial point to form a dense point cloud of the needle tip and surrounding tissue.

[0074] The dense point cloud is converted into a smooth three-dimensional surface model using a mesh generation algorithm, the geometric features of the needle tip are highlighted, and the output is a three-dimensional point cloud motion trajectory of the needle tip.

[0075] Its technical effect is: through the projection of multi-frequency fringe patterns, a wide range of depth information and high-resolution data of local details can be obtained simultaneously. Low-frequency coarse stripes help to quickly obtain rough depth information of the scene, while high-frequency fine stripes can improve the resolution of local details; the accurate extraction of phase values ​​enables each point in the image to correspond to a specific three-dimensional spatial position, thereby improving the accuracy of three-dimensional reconstruction; by combining phase values ​​of different frequencies, the blurring caused by phase jumps can be eliminated, and a continuous absolute phase map is generated, which effectively avoids the error accumulation during phase unwrapping and ensures the continuity and accuracy of the three-dimensional reconstruction data; combining parallax information and phase difference, the phase map can be converted into accurate three-dimensional depth data through the parallax information of the binocular camera, and the generated high-resolution depth map is the needle tip and surrounding tissue It provides detailed spatial information and effectively captures tiny displacements and detailed features. It effectively suppresses random noise caused by environmental interference through multi-frame fusion optimization, ensures the stability of point cloud data, and aligns point cloud data within multiple time steps. The continuity and accuracy of the needle tip trajectory are guaranteed through spatial registration. Based on the rigid body motion assumption of the needle tip, the point cloud data can accurately reflect the motion trajectory of the needle tip at different time points, ensuring the accuracy and reliability of motion analysis. The dense point cloud is processed using a mesh generation algorithm (such as Poisson reconstruction) to generate a smooth three-dimensional surface model that highlights the geometric features of the needle tip. Through this reconstruction method, the point cloud data is converted into a three-dimensional model with precise geometric features, which greatly improves the reconstruction accuracy of the needle tip position and provides higher-quality visual support for the three-dimensional motion trajectory of the needle tip.

[0076] A calibration system for a puncture robot based on binocular vision, comprising:

[0077] The multi-viewpoint geometric calibration module is used to design a multi-modal calibration plate and perform multi-viewpoint geometric calibration on the puncture robot.

[0078] The initial calibration parameter determination module is used to construct a polynomial distortion model for the puncture robot based on multi-viewpoint geometric calibration, optimize the radial distortion coefficient, and obtain initial calibration parameters.

[0079] The optimal parameter set screening module is used to perform a global search on the initial calibration parameters using the particle swarm optimization algorithm to screen out the optimal parameter set.

[0080] The kinematic parameter optimization module is used to optimize the needle tip trajectory fitting error using the optimal parameter set as input and obtain optimized kinematic parameters using an improved differential evolution algorithm.

[0081] The convolutional neural network module is used to build a lightweight convolutional neural network that uses binocular images as input to obtain high-precision needle tip coordinates.

[0082] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction with the assistance of structured light, integrate the optimized kinematic parameters and the high-precision needle tip coordinates into the needle tip kinematic model, and reconstruct the three-dimensional motion trajectory of the needle tip.

[0083] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned binocular vision-based puncture robot calibration method.

[0084] The beneficial effects of the embodiments of the present invention are:

[0085] By utilizing a multimodal calibration plate and multi-viewpoint geometric calibration, this method overcomes the errors inherent in traditional calibration methods. In particular, through the design of a fluorescent marker array and the selection of non-coplanar viewing angles, the system accurately captures the geometric information of the puncture robot at different angles, significantly improving calibration accuracy. The introduction of multiple calibration plate patterns and adaptive structured light frequency adjustment during the calibration process ensures stable system operation in complex environments, such as those with tissue deformation and blood occlusion, enhancing the robustness of the calibration process.

[0086] During structured light-assisted 3D reconstruction, this invention combines fringe patterns of varying frequencies to flexibly adjust the projection frequency and optical pattern to accommodate changes in the dynamic surgical environment. This ensures that the structured light system can effectively address blood interference and tissue deformation, improving the system's adaptability in dynamic intraoperative environments. Through precise phase value extraction, phase difference integration, and parallax information fusion, high-resolution depth maps and accurate 3D point cloud data can be acquired even in complex environments, providing detailed 3D spatial information on the needle tip's position.

[0087] This invention uses a particle swarm optimization algorithm to perform a global search for initial calibration parameters, avoiding local optimality. It further optimizes the needle tip trajectory fitting error through an improved differential evolution algorithm. By combining global and local optimization, the system can precisely adjust kinematic parameters, ensuring precise robot motion in a variety of complex surgical environments. By properly setting the number of iterations and convergence criteria during the optimization process, rapid convergence is achieved, improving the system's accuracy and efficiency during real-time surgery and meeting high real-time requirements.

[0088] By designing a lightweight convolutional neural network, LightCNN, this paper not only ensures the output of high-precision needle tip coordinates but also effectively reduces computational complexity. This makes it suitable for embedded platforms, significantly reducing the system's hardware resource requirements and system costs while also improving computational efficiency and lowering energy consumption. The use of a lightweight network simplifies the system's hardware configuration, making this invention applicable to a wider range of medical device scenarios, reducing hardware costs, and simplifying the manufacturing process.

[0089] By fusing and optimizing multi-frame point cloud data, the present invention reduces noise interference, ensures the continuity and accuracy of the needle tip trajectory, and enhances the system's adaptability to environmental changes, enabling stable operation under diverse environmental conditions and ensuring accurate three-dimensional trajectory reconstruction. The present invention's high robustness, precise optimization process, and real-time feedback mechanism help reduce the incidence of failures and enhance overall system stability, maintaining high reliability, especially under complex surgical conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0091] Figure 1 The figure is a flow chart of the calibration method of the puncture robot based on binocular vision of the present invention. DETAILED DESCRIPTION

[0092] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0093] Please refer to Figure 1 The first embodiment of the present invention provides a calibration method for a puncture robot based on binocular vision, which includes: designing a multimodal calibration plate to perform multi-viewpoint geometric calibration on the puncture robot; based on the multi-viewpoint geometric calibration, constructing a polynomial distortion model for the puncture robot, optimizing the radial distortion coefficient, correcting the optical distortion of the puncture robot lens, and obtaining initial calibration parameters; using a particle swarm optimization algorithm to perform a global search on the initial calibration parameters and screen out an optimal parameter set; using the optimal parameter set as input, using an improved differential evolution algorithm to optimize the needle tip trajectory fitting error and obtain optimized kinematic parameters; constructing a lightweight convolutional neural network, using binocular images as input, and obtaining high-precision needle tip coordinates; using structured light to assist in three-dimensional reconstruction, integrating the optimized kinematic parameters and the high-precision needle tip coordinates into the needle tip kinematic model, and reconstructing the three-dimensional motion trajectory of the needle tip.

[0094] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the design of a multimodal calibration plate and the multi-viewpoint geometric calibration of the puncture robot include: selecting a black matte carbon fiber plate as the calibration plate, and embedding a periodic fluorescent marker array on the surface of the calibration plate; specifically, the periodic fluorescent marker array can adopt three dynamic modes: hexagonal honeycomb arrangement with a spacing of 15mm to avoid symmetry matching ambiguity; random point cloud distribution with a density of 10 points / cm 2 , used for verifying resistance to ambient light interference; arranged in a rectangular grid with 20mm spacing, it is compatible with traditional calibration algorithms. The calibration board's control module integrates an FPGA chip, which uses PWM to modulate the LED light timing, achieving multiple flashes of 1Hz / 5Hz / 10Hz, synchronizing the binocular camera exposure signal and suppressing ambient light noise. The calibration board communicates with the binocular camera via the RS-485 protocol. The LED flashes with a 10ms pulse width during camera exposure and is off the rest of the time.

[0095] Control the fixed binocular camera at the end of the robot to shoot the calibration plate from 2N non-coplanar perspectives, where N is a natural number greater than 0, to obtain a set of continuous frame images, extract the fluorescent marker from the continuous frame image set, and locate the center pixel coordinates of the fluorescent marker; specifically, the shooting perspective covers a pitch angle of ±45° and a yaw angle of 0° to 360° with an interval of 45°, and the distance from the calibration plate is 50cm±5cm. Ten frames of images are continuously shot at each perspective, and the exposure time can be 20ms / 40ms / 60ms. The dynamic range is improved through HDR fusion to obtain a set of continuous frame images, and the continuous frame image set is processed by non-local mean filtering NL-Means to remove sensor noise and retain the edge details of the marker. Based on the Gaussian fitting algorithm, the center coordinates of the marker are located at the sub-pixel level to obtain the center pixel coordinates of the fluorescent marker.

[0096] Assume that the world coordinates of the kth marking point on the calibration plate are P k =[X k ,Y k ,0] T , the pixel coordinate projected to the i-th perspective image is p ik =[u ik ,v ik ] T , construct the projection equation Among them, K i is the camera intrinsic parameter matrix, R i Extrinsic rotation matrix, t i is the translation vector, λ ik is the scale factor; the camera intrinsic parameter matrix K is obtained by singular value decomposition method iSpecifically, in the singular value decomposition method, for each view i, the homography matrix K is solved by the direct linear transformation algorithm i , making After expansion, we get the linear equation system Construct two equations for each marked point to form an overdetermined system of equations A i h=0, and A is obtained by SVD decomposition. i =UΣV T , take the last column of V as H i The least squares solution of .

[0097] Homography matrix and intrinsic parameter matrix K i The relationship is H i =K i [r1 r2 t], using the orthogonality of the rotation matrix Obtain the constraint equation Among them, v ij =[h i1 h j1 ,h i1 h j2 +h i2 h j1 ,h i2 h j2 ,h i3 h j1 +h i1 h j3 ,h i3 h j2 +h i2 h j3 ,h i3 h j3 ] T , b is the vector form of B.

[0098] Stack the constraint equations from multiple perspectives to form an overdetermined equation system Ab=0; perform SVD decomposition on the matrix A=UΣV T , take the last column of V as the solution of b; reconstruct b into a symmetric matrix B, satisfying Perform Cholesky decomposition on B = LL T , get K i -1 =L, inverse to get the internal parameter matrix K i .

[0099] Solve to get the external parameter rotation matrix R i and the translation vector t i The value of .

[0100] Specifically, according to H i =K i [r1 r2 t], the solution is where λ = 1 / ||K i -1 h1|| is used to normalize r1 and r; construct an approximate rotation matrix R raw =[r1 r2 r1×r2], for the approximate rotation matrix R raw Perform SVD decomposition to get R raw =UΣV T ,make sure Orthogonalize and get R i =UV T ; Adjust the translation vector by minimizing the reprojection error to obtain t i =λK i -1 h3.

[0101] Its technical effects are as follows: the calibration plate uses a periodic fluorescent marker array to provide more precise calibration points, enabling more accurate geometric information to be obtained when shooting from different perspectives; the design of the calibration plate includes three dynamic modes, which improves the robustness of the calibration through the periodic fluorescent marker array, helps to suppress ambient light interference, adapts to traditional calibration algorithms, and increases the ability to adapt to environmental changes and uneven lighting; the calibration plate integrates an FPGA chip, and uses PWM to modulate the LED light emission timing to achieve 1Hz / 5Hz / 10Hz multi-frequency flashing to synchronize the binocular camera exposure signal, effectively suppressing the interference of ambient light noise.

[0102] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the multi-viewpoint geometric calibration is based on constructing a polynomial distortion model for the puncture robot, optimizing the radial distortion coefficient, and obtaining the initial calibration parameters, which includes: using a second-order radial distortion model to describe the lens distortion Among them, (u,v) is the distorted pixel coordinate, (u corrected ,v corrected ) is the corrected coordinate, (u0, v0) is the principal point coordinate, and the camera intrinsic parameter matrix K i Get represents the radial distance from the pixel to the principal point, k1 and k2 are the radial distortion coefficients to be optimized; the world coordinate P k =[X k ,Y k ,0] T Transform to the camera coordinate system through the external parameter matrix Project the 3D point in the camera coordinate system onto the normalized plane and calculate the normalized coordinates Calculate the square of the radial distance from the normalized coordinate to the principal point Get the normalized coordinates after correction (x d =x n(1+k1r 2 +k2r 4 ),y d =y n (1+k1r 2 +k2r 4 )); Map the corrected normalized coordinates to the actual pixel coordinate system Among them, (f x ,f y ) is the focal length of the camera; construct the reprojection error function, which is defined as the sum of squared errors of all markers under all viewing angles, The goal is to adjust the radial distortion coefficients k1 and k2 to be optimized through the optimization algorithm to minimize the reprojection error E, where p ik =(u ik ,v ik ) is the actual observation coordinate, is the theoretical projection coordinate; for each marker point, the partial derivatives of the radial distortion coefficients k1 and k2 are calculated through the Jacobian matrix right Find partial derivatives right Find partial derivatives Perform iterative initialization, set the initial distortion coefficients k1 = 0, k2 = 0, the initial damping factor λ = 0.001, and the maximum number of iterations T = 100; perform parameter update Δθ = -(J T J+λI) -1 J T E, where Δθ=[Δk1,Δk2] T After the update, a convergence check is performed. If the reprojection error decreases by E after the update, the update is accepted, k1←k1+Δk1, k2←k2+Δk2, and the damping factor λ←0.1λ is reduced. If the reprojection error increases after the update, the update is rejected and the damping factor λ←10λ is increased. The convergence condition is that the parameter change ||Δθ|| < 10 -6 , error reduction |E new -E old ∣<10 -6 , reaching the maximum number of iterations T.

[0103] Its technical effects are: using a second-order radial distortion model to accurately describe the optical distortion of the lens, and effectively reducing the image deviation caused by lens distortion by optimizing the radial distortion coefficient; constructing a reprojection error function and minimizing its value to ensure that the error of each marking point in the calibration process is minimized under all viewing angles, optimizing the internal and external parameters of the camera, ensuring the accuracy of the visual system, and improving the accuracy of the needle tip positioning of the puncture robot; using the Jacobian matrix to calculate the partial derivatives of the radial distortion coefficient. The introduction of the Jacobian matrix can accelerate convergence during the optimization process, avoiding the slow convergence problem caused by inappropriate initial values ​​or local optimal solutions in traditional methods; through the adjustment of the damping factor, it can effectively avoid excessive updates or premature convergence. If the reprojection error does not decrease significantly after the update, the system will appropriately increase the damping factor to avoid invalid updates.

[0104] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the method of using the particle swarm optimization algorithm to perform a global search on the initial calibration parameters and screen out the optimal parameter set includes: encoding the initial calibration parameters as a particle position vector x = [f x ,f y ,u0,v0,k1,k2,θ R1 ,θ R2 ,θ R3 ,t x ,t y ,t z ], where (f x ,f y ,u0,v0) is the camera intrinsic parameter matrix K i Parameters, (k1, k2) are radial distortion coefficients, is the external parameter rotation matrix R i Euler angle parameterization, (t x ,t y ,t z ) is the translation vector t i Component; According to the physical meaning, set the search space The initial calibration parameters are constrained, where f x and f y Set it near the initial focal length, according to the empirical range of the radial distortion coefficient; set the particle swarm size, the number of particles N particles = 200, set the initial position of the central particle, and the other particles are uniformly and randomly distributed within the parameter range. The initial velocity of the particles is set to 10% of the parameter range; for each particle j, update its velocity v j and position x j , the speed update of the t+1th iteration is Among them, w is the inertia weight, which is used to balance global exploration and local development. The initial value w max=0.9, linearly decreasing to w min =0.4, c1 and c2 are learning factors to control the weight of individual experience and social experience, c1=c2=2, r1 and r2 are random numbers used to increase the randomness of the search, evenly distributed between [0,1], and the position update is If the updated position If the parameter range is exceeded, it is set to the nearest boundary value and the boundary is truncate. The boundary truncation formula is: Record the historical optimal position of each particle as the optimal individual Update; record the individual optimal position of all particles as the global optimal g best Update; when the maximum number of iterations is reached, or the parameter change is less than the preset threshold, such as the focal length f x and f y When the change is less than 0.01%, the iteration is terminated and the global optimal parameter set g is output. best .

[0105] Its technical effects are as follows: the particle swarm optimization algorithm (PSO) can explore the entire parameter space by performing a global search on the initial calibration parameters, avoiding the local optimal solution problem common in traditional optimization methods. PSO does not rely on the selection of the initial solution and can effectively find the global optimal solution, ensuring that the best calibration result is found under multiple parameter combinations; in the particle swarm optimization process, by reasonably setting the search space and range constraints, invalid searches can be reduced, ensuring that particles search within a physically meaningful parameter space; the inertia weight and learning factor in the particle swarm optimization algorithm can be dynamically adjusted according to feedback during the search process to balance global exploration and local development, ensuring that the algorithm can conduct extensive exploration in the initial stage; through boundary truncation, the stability of the algorithm is ensured, and invalid calculations during the search process are avoided.

[0106] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the optimal parameter set is used as input, and the improved differential evolution algorithm is used to optimize the needle tip trajectory fitting error to obtain the optimized kinematic parameters, which includes: the optimal parameter set is p j =[K, d, R, t], where K is the camera intrinsic matrix, d is the radial distortion coefficient, R is the extrinsic rotation matrix, and t is the translation vector; let the three-dimensional trajectory of the needle tip include the point set P = {p1, p2, ..., p i ,...,p N}, where p i =[x i ,y i ,z i ] is the needle tip position at each moment; define the calculation formula of optimal fitness Preal (t) is the true three-dimensional coordinate of the needle tip in the t-th frame image, P pred (t) is the predicted position of the needle tip calculated based on the kinematic model and calibration parameters, N is the total number of time steps; minimize the objective function The fitting error is minimized and the optimized kinematic parameters are obtained.

[0107] Its technical effects are: for the needle tip position at each moment, the optimal fitness calculation formula is used to define the error between the real three-dimensional coordinates of the needle tip in each frame image and the needle tip position predicted by the kinematic model, so that the predicted position of the needle tip is as close to the real position as possible at different time steps; the improved differential evolution algorithm MDE has a strong global optimization capability, which can avoid falling into the local optimal solution, and accurately adjust the kinematic parameters through iterative optimization to minimize the needle tip trajectory fitting error, and continuously provide high-precision needle tip positioning in complex and dynamic environments, such as tissue deformation, blood interference, etc.; minimize the objective function (fitting error function) to minimize the error generated by the needle tip during movement, significantly reduce the deviation and instability in the robot movement, and ensure the accurate tracking and positioning of the needle tip throughout the operation.

[0108] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the construction of a lightweight convolutional neural network, taking the binocular image as input, and obtaining high-precision needle tip coordinates includes: taking the left and right images of the binocular camera as input, normalizing and size-aligning the input images to ensure the spatial consistency of the binocular images; designing a feature extraction backbone network, including an early feature fusion module and a depth-separable convolution block, the early feature fusion module adopts a dual-stream input branch, and the left and right images respectively extract low-level features through independent shallow convolution layers, and the feature maps of the left and right images are spliced ​​in the channel dimension to form multimodal fusion features. Each of the depth-separable convolution blocks contains a depth convolution for extracting features channel by channel and a point-by-point convolution for inter-channel feature fusion, and a depth-separable convolution is used instead of Standard convolution significantly reduces the number of parameters and computational complexity; an attention enhancement network is designed, including a compression excitation module and a spatial attention mask. The compression excitation module compresses the spatial information of the feature map through global average pooling, learns channel weights, and strengthens important feature channels. The spatial attention mask generates a spatial weight map through a convolutional layer to highlight the significant features of the needle tip area; a multi-scale feature fusion network is designed, including a pyramid pooling module, which uses pooling kernels of different scales to extract multi-scale features and enhance the network's sensitivity to tiny displacements of the needle tip. The pooled features are restored to resolution through upsampling and fused with the original feature map; an output layer is designed, including a fully connected regression head, which flattens the final fused feature map into a vector, gradually reduces the dimensionality through two fully connected layers, and outputs the high-precision three-dimensional coordinates of the needle tip. The number of nodes in the output layer is 3.

[0109] Its technical effects are: by normalizing and aligning the size of the input left and right binocular images, the spatial consistency of the images is ensured, the scale differences and offsets between images can be eliminated, and the subsequent feature extraction and coordinate calculation can be based on a unified spatial coordinate system; the early feature fusion module extracts low-level features from the left and right images through independent shallow convolution layers, and splices the feature maps of the left and right images in the channel dimension to form multimodal fusion features, making full use of the different perspective information of the left and right images, enhancing the network's ability to capture spatial information, and thus improving the positioning accuracy of the needle tip coordinates; using depthwise separable convolution instead of traditional convolution, depthwise separable convolution decomposes the standard convolution operation into two steps, depthwise convolution and pointwise convolution, while ensuring the ability to extract features, significantly reducing the computational cost and memory consumption, and greatly reducing the number of parameters and calculations; in the compression excitation module, the spatial information of the feature map is compressed by global average pooling. The spatial attention mask generates a spatial weight map through the convolution layer, highlighting the significant features of the needle tip area, which can effectively reduce the impact of background noise on the needle tip positioning accuracy; the pyramid pooling module extracts multi-scale features through pooling kernels of different scales, enhancing the network's sensitivity to tiny displacements of the needle tip. Multi-scale feature fusion can help the network capture features at different levels in spaces of different scales, especially in needle tip positioning and tiny displacement detection, which can better identify and locate the needle tip position. By upsampling the pooled features and fusing them with the original feature map, the network can maintain high-resolution features and improve positioning accuracy. The output layer of the network is designed as a fully connected regression head, which flattens the final fused feature map into a vector through gradual dimensionality reduction and outputs the high-precision three-dimensional coordinates of the needle tip, so that the network can accurately map image features to the three-dimensional position space of the needle tip.

[0110] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the structured light-assisted three-dimensional reconstruction is performed, the optimized kinematic parameters and the high-precision needle tip coordinates are integrated into the needle tip kinematic model, and the three-dimensional motion trajectory of the needle tip is reconstructed, including: establishing a structured light-assisted system, including a projection module and a binocular vision module, the projection module uses a high-precision blue light or infrared structured light projector to project a preset optical pattern onto the puncture needle and the surrounding area, the binocular vision module uses a synchronously triggered binocular camera to capture scene images under structured light illumination from different perspectives, perform image analysis, and output the three-dimensional point cloud motion trajectory of the needle tip; design a multi-joint chain needle tip kinematic model, according to the mechanical structure of the puncture robot A structure (such as a serial robotic arm) is established, and the joint coordinate system and the link parameters (such as DH parameters) are defined. The position of the needle tip (end effector) is calculated by forward kinematics. The input includes the optimized kinematic parameters and the high-precision needle tip coordinates. The model output is the theoretical three-dimensional coordinates of the needle tip. The joint clearance compensation error term and the thermal deformation compensation error term are introduced into the multi-joint chain needle tip kinematic model to optimize the output accuracy and obtain high-precision three-dimensional coordinates of the needle tip. An extended Kalman filter model is designed, and the high-precision three-dimensional coordinates of the needle tip kinematic model and the three-dimensional point cloud motion trajectory reconstructed by structured light are used as model inputs. The fused needle tip trajectory is fitted with spline interpolation to eliminate high-frequency noise and optimize the trajectory smoothness to obtain the optimized three-dimensional motion trajectory of the needle tip.

[0111] Its technical effects are: structured light assists in projecting optical patterns through high-precision blue light or infrared structured light projectors, and uses synchronously triggered binocular cameras to capture scene images from different perspectives to obtain high-precision three-dimensional point cloud data. Through the generation of depth maps and point cloud reconstruction, the three-dimensional motion trajectory of the needle tip is accurately captured, and the needle tip positioning accuracy is enhanced, especially in complex and dynamic surgical environments; through the multi-joint chain needle tip kinematic model, the mechanical structure and joint coordinate system (such as DH parameters) of the puncture robot are considered to accurately calculate the position and posture of the needle tip. The optimized kinematic parameters are combined with high-precision needle tip coordinates to greatly improve the accuracy of the kinematic model, ensure that the theoretical three-dimensional coordinates of the needle tip output by the model are more consistent with the actual position, and accurately control the robot movement; the introduction of joint gap compensation error terms and thermal deformation compensation error terms further optimizes the output accuracy of the kinematic model. , to compensate for errors caused by factors such as mechanical structure, joint clearance or thermal deformation, and ensure that the robot can maintain high-precision motion control in complex environments, especially high-temperature operating environments; the extended Kalman filter model is combined with the needle tip kinematic model and the three-dimensional point cloud data of structured light reconstruction to eliminate high-frequency noise in the reconstruction process, optimize the smoothness of the trajectory, and filter out noise through spline interpolation fitting to ensure the smoothness of the needle tip motion trajectory, thereby improving the accuracy of needle tip positioning, especially during rapid movement or surgery, and being able to maintain stable and precise operation.

[0112] In a preferred embodiment of the present invention, in the calibration method of the puncture robot based on binocular vision, the step of image analysis of the structured light assisted system includes: designing a light pattern, projecting multiple groups of stripe patterns of different frequencies in sequence by the projector, specifically, low-frequency coarse stripes are used to quickly obtain a wide range of depth information, and high-frequency fine stripes are used to improve the resolution of local details, and according to the complexity of the intraoperative scene, such as tissue deformation and blood occlusion, the projection frequency and pattern type are adaptively adjusted to ensure stable imaging in a dynamic environment; analyzing the stripe image captured by the binocular camera, and extracting the phase value of each point, specifically, the phase value reflects the degree of deformation of the stripes on the surface of the object, and is directly related to the three-dimensional shape of the object surface; combining the phase values ​​of different frequencies to eliminate The blur caused by phase jump is eliminated to generate a continuous absolute phase map; the absolute phase map is converted into accurate three-dimensional depth data by combining the parallax information and phase difference of the binocular camera to generate a high-resolution depth map; each pixel in the high-resolution depth map is converted into a three-dimensional spatial point to form a dense point cloud of the needle tip and surrounding tissue. Specifically, it also includes multi-frame fusion optimization, sliding average or median filtering of continuous multi-frame point clouds, suppression of random noise, spatial alignment, and alignment of point cloud data at different times based on the rigid body motion assumption of the needle tip to ensure trajectory continuity; a grid generation algorithm (such as Poisson reconstruction) is used to convert the dense point cloud into a smooth three-dimensional surface model to highlight the geometric features of the needle tip, and output the three-dimensional point cloud motion trajectory of the needle tip.

[0113] Its technical effect is: through the projection of multi-frequency fringe patterns, a wide range of depth information and high-resolution data of local details can be obtained simultaneously. Low-frequency coarse stripes help to quickly obtain rough depth information of the scene, while high-frequency fine stripes can improve the resolution of local details; the accurate extraction of phase values ​​enables each point in the image to correspond to a specific three-dimensional spatial position, thereby improving the accuracy of three-dimensional reconstruction; by combining phase values ​​of different frequencies, the blurring caused by phase jumps can be eliminated, and a continuous absolute phase map is generated, which effectively avoids the error accumulation during phase unwrapping and ensures the continuity and accuracy of the three-dimensional reconstruction data; combining parallax information and phase difference, the phase map can be converted into accurate three-dimensional depth data through the parallax information of the binocular camera, and the generated high-resolution depth map is the needle tip and surrounding tissue It provides detailed spatial information and effectively captures tiny displacements and detailed features. It effectively suppresses random noise caused by environmental interference through multi-frame fusion optimization, ensures the stability of point cloud data, and aligns point cloud data within multiple time steps. The continuity and accuracy of the needle tip trajectory are guaranteed through spatial registration. Based on the rigid body motion assumption of the needle tip, the point cloud data can accurately reflect the motion trajectory of the needle tip at different time points, ensuring the accuracy and reliability of motion analysis. The dense point cloud is processed using a mesh generation algorithm (such as Poisson reconstruction) to generate a smooth three-dimensional surface model that highlights the geometric features of the needle tip. Through this reconstruction method, the point cloud data is converted into a three-dimensional model with precise geometric features, which greatly improves the reconstruction accuracy of the needle tip position and provides higher-quality visual support for the three-dimensional motion trajectory of the needle tip.

[0114] The second embodiment of the present invention provides a calibration system for a puncture robot based on binocular vision, which includes: a multi-viewpoint geometric calibration module, which is used to design a multi-modal calibration plate and perform multi-viewpoint geometric calibration on the puncture robot; an initial calibration parameter determination module, which is used to construct a polynomial distortion model for the puncture robot based on multi-viewpoint geometric calibration, optimize the radial distortion coefficient, and obtain initial calibration parameters; an optimal parameter set screening module, which is used to use a particle swarm optimization algorithm to perform a global search on the initial calibration parameters and screen out the optimal parameter set; a kinematic parameter optimization module, which is used to use the optimal parameter set as input and adopt an improved differential evolution algorithm to optimize the needle tip trajectory fitting error to obtain optimized kinematic parameters; a convolutional neural network module, which is used to construct a lightweight convolutional neural network, which uses binocular images as input to obtain high-precision needle tip coordinates; a three-dimensional reconstruction module, which is used to perform three-dimensional reconstruction with the assistance of structured light, integrate the optimized kinematic parameters and the high-precision needle tip coordinates into the needle tip kinematic model, and reconstruct the three-dimensional motion trajectory of the needle tip.

[0115] A third embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the calibration method of the puncture robot based on binocular vision as described above is implemented.

[0116] The computer program product of the binocular vision-based puncture robot calibration method and device provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0117] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned calibration method of the puncture robot based on binocular vision, thereby improving the positioning accuracy of the puncture robot and enhancing its stability and real-time performance in complex surgical environments.

[0118] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A calibration method for a puncture robot based on binocular vision, characterized in that: include: Design a multi-modal calibration plate to perform multi-viewpoint geometric calibration on the puncture robot; Based on multi-viewpoint geometric calibration, a polynomial distortion model is constructed for the puncture robot, and the radial distortion coefficient is optimized to obtain initial calibration parameters; The particle swarm optimization algorithm is used to perform a global search on the initial calibration parameters and screen out the optimal parameter set; Taking the optimal parameter set as input, an improved differential evolution algorithm is used to optimize the needle tip trajectory fitting error to obtain optimized kinematic parameters; Construct a lightweight convolutional neural network that uses binocular images as input to obtain high-precision needle tip coordinates; Structured light is used to assist in three-dimensional reconstruction, and the optimized kinematic parameters and the high-precision needle tip coordinates are integrated into the needle tip kinematic model to reconstruct the three-dimensional motion trajectory of the needle tip, specifically including: establishing a structured light-assisted system, including a projection module and a binocular vision module, the projection module uses a high-precision blue light or infrared structured light projector to project a preset optical pattern onto the puncture needle and the surrounding area, the binocular vision module uses a synchronously triggered binocular camera to capture scene images under structured light illumination from different perspectives, perform image analysis, and output the three-dimensional point cloud motion trajectory of the needle tip; designing a multi-joint chain needle tip kinematic model, defining the joint coordinate system and connecting rod parameters according to the mechanical structure of the puncture robot, and calculating the position of the needle tip through forward kinematics, with the input including the optimized kinematic parameters and the high-precision needle tip coordinates, and the model output is the theoretical three-dimensional coordinates of the needle tip; introducing joint gap compensation error terms and thermal deformation compensation error terms into the multi-joint chain needle tip kinematic model to optimize the output accuracy and obtain high-precision three-dimensional coordinates of the needle tip; An extended Kalman filter model is designed, and the high-precision three-dimensional coordinates of the needle tip kinematic model and the three-dimensional point cloud motion trajectory reconstructed by structured light are used as model input. Spline interpolation fitting is performed on the fused needle tip trajectory to eliminate high-frequency noise and optimize the trajectory smoothness to obtain the optimized three-dimensional motion trajectory of the needle tip.

2. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The design of the multimodal calibration plate to perform multi-viewpoint geometric calibration on the puncture robot includes: A black matte carbon fiber plate is selected as a calibration plate, and a periodic fluorescent marker array is embedded on the surface of the calibration plate; Controlling a binocular camera fixed at the end of the robot to photograph the calibration plate from 2N non-coplanar viewing angles, where N is a natural number greater than 0, to obtain a set of continuous frame images, extracting a fluorescent marker from the set of continuous frame images, and locating the central pixel coordinates of the fluorescent marker; Assume that the number The world coordinates of the marker point are , projected to the The pixel coordinates of the perspective image are , construct the projection equation ,in, is the camera intrinsic parameter matrix, Extrinsic rotation matrix, is the translation vector, is the scale factor; The camera intrinsic parameter matrix is ​​obtained by singular value decomposition method. The initial value of Solve to get the external parameter rotation matrix and translation vectors The value of .

3. The calibration method of the puncture robot based on binocular vision according to claim 2, characterized in that: The multi-viewpoint geometric calibration is based on constructing a polynomial distortion model for the puncture robot, optimizing the radial distortion coefficient, and obtaining the initial calibration parameters including: Use the second-order radial distortion model to describe lens distortion ,in, is the distorted pixel coordinate, is the corrected coordinate, The principal point coordinates are obtained from the camera intrinsic parameter matrix Get Indicates the radial distance from the pixel to the principal point, and is the radial distortion coefficient to be optimized; The world coordinates Transform to the camera coordinate system through the external parameter matrix ; Project the 3D point in the camera coordinate system onto the normalized plane and calculate the normalized coordinates ; Calculate the square of the radial distance from the normalized coordinate to the principal point , get the normalized coordinates after correction ; Map the corrected normalized coordinates to the actual pixel coordinate system ,in, is the focal length of the camera; Construct the reprojection error function, which is defined as the sum of squared errors of all markers under all viewing angles, The goal is to adjust the radial distortion coefficient to be optimized through the optimization algorithm and , so that the reprojection error E is minimized, where is the actual observation coordinate, is the theoretical projection coordinate; For each marker point, the radial distortion coefficient is calculated by the Jacobian matrix and The partial derivative of ,right Find partial derivatives ,right Find partial derivatives ; Perform iterative initialization and set the initial distortion coefficient , , initial damping factor , maximum number of iterations ; Update parameters ,in, ; After the update, a convergence check is performed. If the reprojection error is reduced by E after the update, the update is accepted. , , reduce the damping factor If the reprojection error increases after the update, the update is rejected and the damping factor is increased. ; The convergence condition is the parameter change , the error reduction , reaching the maximum number of iterations .

4. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The particle swarm optimization algorithm is used to perform a global search on the initial calibration parameters, and the optimal parameter set obtained by screening includes: Encode the initial calibration parameters as particle position vectors ,in, is the camera intrinsic parameter matrix Parameters, is the radial distortion coefficient, is the extrinsic rotation matrix Euler angle parameterization of , is the translation vector The weight; According to the physical meaning, set the search space , performing range constraints on the initial calibration parameters; Set the particle swarm size and particle number , set the initial position of the central particle, and the other particles are uniformly and randomly distributed within the parameter range. The initial velocity of the particles is set to 10% of the parameter range; For each particle j, update its velocity and location , the speed update of the t+1th iteration is ,in, is the inertia weight, initial value , linearly decreasing to , and is the learning factor, , and is a random number uniformly distributed over Between, the position is updated to ; If the updated position If the parameter range is exceeded, it is set to the nearest boundary value and the boundary is truncate. The boundary truncation formula is: ; Record the historical optimal position of each particle as the optimal individual Make updates; Record the individual optimal positions of all particles as the global optimal Make updates; When the maximum number of iterations is reached or the parameter change is less than the preset threshold, the iteration is terminated and the global optimal parameter set is output. .

5. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The optimal parameter set is used as input, and the improved differential evolution algorithm is used to optimize the needle tip trajectory fitting error to obtain the optimized kinematic parameters including: The optimal parameter set is ,in, is the camera intrinsic parameter matrix, is the radial distortion coefficient, is the extrinsic rotation matrix, is the translation vector; Assume that the three-dimensional trajectory of the needle tip includes the point set ,in, is the needle tip position at each moment; Define the calculation formula for optimal fitness , is the true three-dimensional coordinate of the needle tip in the t-th frame image, is the predicted position of the needle tip calculated based on the kinematic model and calibration parameters, and N is the total number of time steps; Minimize the objective function , so that the fitting error is minimized and the optimized kinematic parameters are obtained.

6. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The method of constructing a lightweight convolutional neural network, taking a binocular image as input, and obtaining high-precision needle tip coordinates includes: Take the left and right images of the binocular camera as input, normalize and align the input images to ensure the spatial consistency of the binocular images; Design a feature extraction backbone network, including an early feature fusion module and a depthwise separable convolution block. The early feature fusion module uses a dual-stream input branch. The left and right images are respectively subjected to independent shallow convolution layers to extract low-level features. The feature maps of the left and right images are then concatenated in the channel dimension to form multimodal fusion features. Each depthwise separable convolution block contains depthwise convolution for channel-by-channel feature extraction and point-by-point convolution for inter-channel feature fusion. Design an attention enhancement network, including a compression excitation module and a spatial attention mask. The compression excitation module compresses the spatial information of the feature map through global average pooling, learns channel weights, and strengthens important feature channels. The spatial attention mask generates a spatial weight map through a convolutional layer to highlight the salient features of the needle tip area. A multi-scale feature fusion network is designed, including a pyramid pooling module. Pooling kernels of different scales are used to extract multi-scale features, enhancing the network's sensitivity to tiny displacements of the needle tip. The pooled features are then upsampled to restore resolution and fused with the original feature map. The output layer is designed, including a fully connected regression head, which flattens the final fused feature map into a vector. The dimensionality is gradually reduced through two fully connected layers to output the high-precision three-dimensional coordinates of the needle tip. The number of nodes in the output layer is 3.

7. The calibration method of the puncture robot based on binocular vision according to claim 1, characterized in that: The steps of image analysis of the structured light auxiliary system include: Design the light pattern and have the projector sequentially project multiple sets of fringe patterns of different frequencies; Analyze the fringe image captured by the binocular camera and extract the phase value of each point; Combine phase values ​​of different frequencies to eliminate ambiguity caused by phase jumps and generate a continuous absolute phase map; Combining the parallax information and phase difference of the binocular camera, the absolute phase map is converted into accurate three-dimensional depth data to generate a high-resolution depth map; Converting each pixel in the high-resolution depth map into a three-dimensional spatial point to form a dense point cloud of the needle tip and surrounding tissue; The dense point cloud is converted into a smooth three-dimensional surface model using a mesh generation algorithm, the geometric features of the needle tip are highlighted, and the output is a three-dimensional point cloud motion trajectory of the needle tip.

8. A calibration system for a puncture robot based on binocular vision, characterized in that: include: Multi-viewpoint geometric calibration module, used to design multi-modal calibration plates and perform multi-viewpoint geometric calibration on the puncture robot; An initial calibration parameter determination module is used to construct a polynomial distortion model for the puncture robot based on multi-viewpoint geometric calibration, optimize the radial distortion coefficient, and obtain initial calibration parameters; The optimal parameter set screening module is used to perform a global search on the initial calibration parameters using the particle swarm optimization algorithm to screen out the optimal parameter set; A kinematic parameter optimization module is used to optimize the needle tip trajectory fitting error using the optimal parameter set as input and to obtain optimized kinematic parameters using an improved differential evolution algorithm; Convolutional neural network module, used to build a lightweight convolutional neural network, using binocular images as input to obtain high-precision needle tip coordinates; A 3D reconstruction module is used to perform 3D reconstruction using structured light as an aid, integrate the optimized kinematic parameters and the high-precision needle tip coordinates into the needle tip kinematic model, and reconstruct the 3D motion trajectory of the needle tip. The operations performed by the 3D reconstruction module specifically include: establishing a structured light assisted system, including a projection module and a binocular vision module, the projection module uses a high-precision blue light or infrared structured light projector to project a preset optical pattern onto the puncture needle and the surrounding area, the binocular vision module uses a synchronously triggered binocular camera to capture scene images under structured light illumination from different perspectives, perform image analysis, and output the 3D point cloud motion trajectory of the needle tip; design a multi-joint chain needle tip kinematic model, define the joint coordinate system and the connecting rod parameters according to the mechanical structure of the puncture robot, calculate the position and posture of the needle tip by forward kinematics, input includes the optimized kinematic parameters and the high-precision needle tip coordinates, and the model output is the theoretical 3D coordinates of the needle tip; introduce joint clearance compensation error terms and thermal deformation compensation error terms into the multi-joint chain needle tip kinematic model to optimize the output accuracy and obtain high-precision 3D coordinates of the needle tip; An extended Kalman filter model is designed, and the high-precision three-dimensional coordinates of the needle tip kinematic model and the three-dimensional point cloud motion trajectory reconstructed by structured light are used as model input. Spline interpolation fitting is performed on the fused needle tip trajectory to eliminate high-frequency noise and optimize the trajectory smoothness to obtain the optimized three-dimensional motion trajectory of the needle tip.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the calibration method of the binocular vision-based puncture robot according to any one of claims 1 to 7 is implemented.

Images