Method and system for staged adjustment of pose of external skeletal fixator based on intelligent mark

Abstract

The invention discloses an external skeletal fixator position and posture staged adjustment method and system based on intelligent marking, and belongs to the field of medical instruments. The method comprises the following steps: firstly, constructing and training a lightweight cross-scale mark generation model, and generating an optimal reference mark for each surface, except a mounting surface, of a three-dimensional marker by using the trained model; projecting each optimal reference mark to electronic paper screens on corresponding surfaces of an upper three-dimensional marker and a lower three-dimensional marker on the external bone fixator; then, a binocular camera is used for collecting images of the upper stereoscopic marker and the lower stereoscopic marker and preprocessing the images; constructing a 2D-3D point pair set matched with the feature points, and estimating a relative pose between an upper fixing ring and a lower fixing ring of the external bone fixator; and finally, calculating a pose error. If the pose error does not exceed the threshold value, the external bone fixator is not adjusted; if the pose error exceeds the threshold value, the adjustment amount of the electric connecting rod is calculated and adjusted; and repeating the steps until the pose error does not exceed the threshold value, and completing pose adjustment. And pose adjustment is carried out in stages, so that the adjustment precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices, and in particular relates to a method and system for adjusting the posture of an external fixator in stages based on intelligent marking. Background Art

[0002] An external fixator is a rehabilitation device used for closed reduction of fractures. It is mainly composed of an upper and lower fixing ring and a Taylor space bracket connecting the two fixing rings. By changing the shape of the Taylor space bracket, the relative posture of the two fixing rings is changed, thereby achieving the purpose of treating fracture displacement, bone nonunion and malformation. Therefore, the accuracy of the posture adjustment of the external fixator directly affects the treatment effect. The fracture treatment process can be divided into three stages: coarse adjustment reduction, fine adjustment correction and maintenance fixation. The goals achieved in different stages are different, and the posture adjustment requirements are also different. The coarse adjustment reduction stage requires rapid determination of the approximate posture of the fracture site and efficient and fast initial alignment. The fine adjustment correction stage requires high-precision posture adjustment at the sub-pixel level. The maintenance fixation stage requires long-term and stable monitoring of the tiny posture changes of the fixing ring.

[0003] The current mainstream posture adjustment of external fixators adopts an open-loop method that combines kinematic modeling with manual operation. Doctors need to manually input the target posture parameters. The system solves the branch adjustment amount of the Taylor space bracket based on complex spatial matrix operations, and then drives the execution module to adjust the length of each connecting rod. Due to the lack of real-time posture feedback mechanism, the system cannot automatically perceive the posture errors generated during the operation. Doctors can only rely on regularly taken X-ray images or monitoring data from external sensors to obtain the reset effect offline. Once a deviation is found, the posture parameters are manually corrected again, thus forming a cyclic adjustment mode of "manual measurement → adjustment → verification". This mode not only has a cumbersome and lengthy operation process and is heavily dependent on the doctor's experience, but also easily causes error accumulation, making the treatment inefficient and difficult to meet the high requirements of modern orthopedic treatment for accuracy and timeliness.

[0004] In the field of computer vision, although marker-based pose estimation techniques have been used for the pose adjustment of external fixators, existing methods still have significant limitations. Most existing marker-based pose estimation methods use static reference markers with fixed and single feature patterns, completely ignoring the different requirements at different stages of the fracture treatment process. They can neither achieve efficient and rapid initial alignment in the rough reduction stage nor meet the high-precision requirements at the sub-pixel level in the fine adjustment stage, let alone capture the tiny pose changes of the fixation ring in a timely manner during the fixation maintenance stage, resulting in difficulties in balancing treatment efficiency and accuracy. In addition, most existing marker-based pose estimation models rely on single-modal data for pose calculation, either only relying on the visual feature recognition and positioning of markers or simply adjusting parameters based on pose errors, making it difficult to comprehensively capture the complex information in the orthopedic treatment scenario. Due to the lack of comprehensive analysis of multiple factors such as the characteristics of treatment stages and real-time environmental changes, the model cannot accurately understand the scene changes when facing complex environments such as occlusion and lighting changes, as well as the special requirements of different treatment stages, resulting in serious deficiencies in the adaptability of marker selection and pose estimation at different treatment stages. This limitation makes traditional pose estimation models difficult to meet the strict requirements of orthopedic rehabilitation treatment for high precision and strong robustness, and unable to provide reliable technical support for clinical treatment. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, the technical problem to be solved by the present invention is to provide a method and system for phased adjustment of the pose of an external fixator based on intelligent markers.

[0006] The present invention adopts the following technical solutions to solve the above technical problems: A method for phased adjustment of the pose of an external fixator based on intelligent markers, comprising the following steps: The first step: construct a lightweight cross-scale marker generation model and train it; Input the feature vector composed of the treatment stage label encoding, the pose error vector, and the real-time scene parameter vector into the lightweight cross-scale marker generation model, use the fully connected layer to generate the adaptation scores of various candidate reference markers, use the Softmax function to convert the adaptation scores into adaptation probabilities, and take the candidate reference marker with the highest adaptation probability as the optimal reference marker; The second step: use the trained lightweight cross-scale marker generation model to generate an optimal reference marker for each surface of the three-dimensional marker except the installation surface in the current treatment stage; project each optimal reference marker onto the electronic paper screens of the corresponding surfaces of the upper and lower three-dimensional markers on the external fixator; The third step: calibrate the binocular camera, use the calibrated binocular camera to collect the images of the upper and lower three-dimensional markers, and preprocess the images; Step 4: Identify the fiducial markers on the surface of the stereo markers in the left and right camera images, and use the fiducial markers with the same ID as the matching fiducial markers; construct a set of 2D-3D point pairs of the matching feature points based on the feature points of the matching fiducial markers; Step 5: Estimate the pose of the external fixator based on the set of 2D-3D point pairs of the matching feature points, and obtain the relative pose between the upper and lower fixing rings of the external fixator; Step 6: Calculate the pose error according to the relative pose between the upper and lower fixing rings of the external fixator and the target pose; if the pose error does not exceed the pose error threshold, the electric link of the external fixator does not need to be adjusted; If the pose error exceeds the pose error threshold, the electric link of the external fixator needs to be adjusted; calculate the adjustment amount of each electric link, and adjust the electric link according to the adjustment amount; Step 7: Return to Step 3, re-acquire the images of the upper and lower stereo markers and perform preprocessing, and continue to execute Steps 4 to 6; repeat this cycle to adjust the pose of the external fixator until the pose error does not exceed the pose error threshold, and complete the pose adjustment of the external fixator.

[0007] Further, each type of candidate fiducial marker is constrained by a size parameter, a feature point distribution density parameter, and an anti-interference threshold parameter; the size parameter of the candidate fiducial marker corresponding to the rough adjustment and reset stage is greater than 5 cm, and the feature point distribution density parameter is less than 0.5 points / cm 2 ; the size parameter of the candidate fiducial marker corresponding to the fine adjustment and correction stage is greater than 2 cm and less than 5 cm, and the feature point distribution density parameter is greater than 1 point / cm 2 ; the size parameter of the candidate fiducial marker corresponding to the maintenance and fixation stage is less than 2 cm, and the anti-interference threshold parameter is greater than 0.8.

[0008] Further, in Step 4, the feature points of the matching fiducial markers are used as candidate matching feature points, the 2D coordinates of the candidate matching feature points in the left and right camera images are extracted, and the 3D coordinates of the candidate matching feature points are calculated by the triangulation method; the 2D coordinates and 3D coordinates of the candidate matching feature points in the left or right camera image are combined into a 2D-3D point pair to obtain a set of 2D-3D point pairs of the candidate matching feature points; the RANSAC algorithm is used to screen the candidate matching feature points to obtain the matching feature points, and then a set of 2D-3D point pairs of the matching feature points is obtained.

[0009] Further, in the fifth step, based on the 2D-3D point pair set of the matched feature points and the intrinsic parameters of the binocular camera, the EPnP algorithm is used to solve the pose of the binocular camera, and the poses of the binocular camera in the upper and lower stereo marker coordinate systems are obtained; the poses of the binocular camera in the upper and lower stereo marker coordinate systems are inversely transformed to obtain the poses of the upper and lower stereo markers in the binocular camera coordinate system; the pose of the upper stereo marker in the binocular camera coordinate system is transformed to the lower stereo marker coordinate system to obtain the relative pose between the upper and lower stereo markers, that is, the relative pose between the upper and lower fixing rings of the external fixator.

[0010] Further, in the sixth step, the coordinates of the connection points of each electric link and the upper fixing ring in the upper stereo marker coordinate system are transformed to the lower stereo marker coordinate system through the target pose between the upper and lower fixing rings of the external fixator, and the target lengths of each electric link are calculated according to the coordinates of the connection points of each electric link and the upper and lower fixing rings in the lower stereo marker coordinate system; similarly, the current lengths of each electric link are calculated according to the relative pose between the upper and lower fixing rings of the external fixator, and the difference between the target length and the current length of the electric link is the adjustment amount.

[0011] Further, the real-time scene parameter vector consists of the distance between the stereo marker and the binocular camera, the incident angle, the ambient light intensity, and the occlusion rate of the stereo marker.

[0012] Further, the preprocessing includes grayscale conversion, binarization, morphological operation, and edge detection.

[0013] Further, the stereo marker is a hollow regular icosahedron, one of its surfaces is used as the mounting surface, and electronic paper screens are provided on the remaining 19 surfaces, and the geometric center of the electronic paper screen coincides with the center of gravity of the surface.

[0014] The present invention also provides an external fixator pose adjustment system, including a visual perception module, a control module, and an execution module; The visual perception module is used to collect stereo marker images in real time; The control module is used to estimate the relative pose between the upper and lower fixing rings of the external fixator, and judge whether it is necessary to adjust the electric link. If necessary, the adjustment amount is calculated; The execution module is used to control the elongation and shortening of each electric link.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention breakthroughly divides the fracture treatment process into three stages: rough adjustment and reduction, fine adjustment and correction, and maintenance and fixation. For the different requirements in different stages, such as the rough adjustment and reduction stage aiming at rapid positioning, the fine adjustment and correction stage focusing on high-precision correction, and the maintenance and fixation stage emphasizing long-term stability monitoring, the corresponding fiducial marks are matched, which can more accurately adapt to the requirements of different treatment stages, improve the accuracy of the pose adjustment of the external fixator, and significantly enhance the treatment effect and efficiency.

[0016] 2. Traditional reduction techniques rely on static marks with fixed features and cannot respond to the dynamic requirements and environmental interferences in different treatment stages, resulting in a gap between visual feedback and mechanical execution. For example, in the rough adjustment and reduction stage, it may be difficult to quickly position due to the fixed size of the markers being restricted by the scene; in the fine adjustment and correction stage, the accuracy is affected by insufficient feature density, and feature point mis-matching is likely to occur in scenes with weak texture and changing viewpoints. The three-dimensional markers of the present invention use an icosahedron as the carrier, through 19 dynamically switchable equilateral triangle e-paper screens, combined with a lightweight cross-scale marker generation model, to generate more adaptable fiducial marks in real time according to the treatment stage, pose error, and environmental parameters (such as distance, illumination, occlusion rate). For example, in the rough adjustment and reduction stage, high-contrast and large-size fiducial marks are generated to expand the recognition range; in the fine adjustment and correction stage, high-density and medium-size fiducial marks are generated to improve the feature point matching accuracy; in the maintenance and fixation stage, fiducial marks with strong anti-interference ability are generated to stably monitor small fluctuations. The fiducial marks achieve accurate feature matching through unique IDs and geometric consistency constraints, avoiding the mis-matching problems of traditional algorithms. The present invention can completely achieve pose estimation based on visual geometric reasoning, significantly enhancing the generality and robustness of the method, and promoting the pose adjustment of the external fixator from an "experience-driven" open-loop operation to a "system-intelligent" closed-loop control.

[0017] 3. Traditional marker-based pose estimation models usually only rely on single-modal data (such as only using the visual features or pose error information of the markers), resulting in insufficient understanding ability of the model for complex scenes and limited adaptability. The lightweight cross-scale marker generation model proposed by the present invention innovatively integrates multi-modal data such as treatment stage labels, pose error vectors, and real-time scene parameters as inputs. By integrating discrete treatment stage information, continuous pose errors, and real-time scene parameters, the model can comprehensively perceive the treatment stage, capture the correlation relationships between different factors, generate more matching fiducial marks for different treatment stages, significantly enhancing the adaptability to complex and changeable clinical scenes and the accuracy of fiducial mark selection, and improving the accuracy of pose adjustment.

[0018] 4. Traditional fiducial marker designs lack systematic theoretical support and are mostly based on experience, making it difficult to quantitatively evaluate the matching degree between the performance of fiducial markers and treatment requirements. The present invention conducts parametric physical constraint design on fiducial markers. By defining geometric dimensions, the distribution density of feature points, and anti-interference threshold parameters, a standardized fiducial marker performance description system is established. Each parameter is closely related to treatment requirements. For example, large sizes correspond to rapid identification, high-density features meet high-precision requirements, and high anti-interference thresholds adapt to complex environments. This parametric design enables the selection and optimization of fiducial markers to break away from experience dependence and achieve scientific design based on theoretical models. Compared with traditional methods, it significantly improves the standardization and clinical applicability of marker design.

[0019] 5. When traditional fiducial marker generation processes tasks at different scales (such as from macroscopic rapid positioning to microscopic precise adjustment), it often fails to balance efficiency and precision due to design limitations. The lightweight cross-scale marker generation model of the present invention has cross-scale capabilities. Through a phased strategy and a dynamic adaptation mechanism, it quickly grasps the overall pose at the macroscopic scale during the coarse adjustment and reset stage to achieve efficient preliminary alignment; switches to the microscopic scale during the fine adjustment and correction stage and uses fine features to achieve high-precision correction; continuously monitors pose changes at the tiny scale during the maintenance and fixation stage. This cross-scale ability enables the present invention to flexibly respond to various scale requirements at different treatment stages and overcomes the performance bottleneck of traditional fiducial marker generation methods in multi-scale tasks.

[0020] 6. Traditional pose adjustment of external fixators for bones relies on an open-loop mode of "manual measurement → adjustment → verification". Doctors need to manually input parameters and detect errors offline, which is cumbersome and inefficient. The present invention constructs a closed-loop adjustment system of "visual perception module → control module → execution module". The visual perception module collects images in real time, the control module automatically estimates the pose, generates adjustment instructions based on the pose error, and the execution module adjusts the electric link according to the adjustment instructions, forming a coherent automated closed-loop control of "perception - decision - execution". Compared with traditional open-loop adjustment methods, it gets rid of the dependence on manual experience and realizes the automated and timely detection and correction of pose deviations during the fracture reduction process.

[0021] 7. Compared with traditional methods for estimating the pose of external fixators for bones that rely on kinematic modeling of Taylor spatial frames, the present invention realizes pose estimation completely based on visual recognition and geometric reasoning, omitting the process of modeling and solving the structural parameters of the Taylor spatial frame, significantly reducing the computational complexity, and enhancing the generality and practicality.

[0022] 8. In existing binocular pose estimation, feature point matching usually relies on traditional image local feature extraction and matching algorithms such as SIFT and ORB. These methods are prone to feature point mismatching problems when facing weak image textures, view changes, or occlusions. In the present invention, fiducial markers are projected onto the e-paper screen on the surface of the stereoscopic marker, and each fiducial marker has a unique ID, which is equivalent to structurally encoding the fiducial markers. By identifying the fiducial markers with the same ID in the left and right camera images and performing feature point matching with position consistency on the identified fiducial markers, the geometric consistency is fully utilized to solve the mismatching problem, improving the robustness and accuracy of feature point matching. Further, the RANSAC algorithm is used to screen out high-quality matching feature points, and these feature points are used to accurately solve the camera pose, which is beneficial to improving the pose estimation accuracy.

[0023] 9. Marker-based monocular pose estimation usually relies on the prior three-dimensional coordinates of feature points and is highly sensitive to the quality of image feature extraction. It is easily affected by factors such as occlusion, distortion, and blur, resulting in a decrease in pose estimation accuracy. The present invention uses the two-dimensional coordinates of feature points in the left and right camera images to calculate the three-dimensional coordinates of feature points through triangulation, and can also achieve accurate and stable pose estimation in complex scenes with unknown structures. Brief Description of the Drawings

[0024] Figure 1 is the overall flowchart of the present invention; Figure 2 is the schematic diagram of the position of the stereoscopic marker of the present invention on the external fixator; Figure 3 is the training framework diagram of the lightweight cross-scale marker generation model of the present invention; Figure 4 is the feature description diagram of the fiducial markers in different treatment stages of the present invention; Figure 5 is the structural diagram of the external fixator pose adjustment system of the present invention. Detailed Embodiments

[0025] The following presents specific embodiments in conjunction with the accompanying drawings. The specific embodiments are only used to further introduce the technical solution of the present invention in detail and do not limit the protection scope of this application.

[0026] The present invention provides a method for stage-by-stage adjustment of the pose of an external fixator based on intelligent markers (hereinafter referred to as the method, see Figures 1-5 ) and includes the following steps: The first step: Construct a lightweight cross-scale marker generation model and train the lightweight cross-scale marker generation model; Collect historical clinical cases, and extract the treatment stage label encoding, pose error vector, real-time scene parameter vector, and optimal reference marker of the cases to form a sample , and several samples form a training set; among them, represents the treatment stage label encoding, and 1, 2, and 3 represent the label encodings of the rough adjustment reset, fine adjustment correction, and maintenance fixation stages respectively; represents the pose error vector, represents the translation error vector, represents the rotation error vector, and the initial value of the pose error vector is a zero vector; represents the real-time scene parameter vector, represents the distance between the stereo marker and the binocular camera; represents the incident angle, that is, the angle between the observation direction and the normal of the currently observed surface of the stereo marker; represents the ambient light intensity, represents the occlusion rate of the stereo marker; is the optimal reference marker, represents the candidate reference marker library.

[0027] According to the actual requirements of the external fixator for bone in different treatment stages, types of candidate reference markers are predefined to form a candidate reference marker library ; among them, represents the th type of candidate reference marker; each type of candidate reference marker is characterized from three aspects: size, feature point distribution density, and anti-interference. Then the th type of candidate reference marker is expressed as: (1) In the formula, represents the feature descriptor; represents the size parameter, which is used to describe the size of the candidate reference marker; represents the feature point distribution density parameter, which is used to describe the feature point distribution of the candidate reference marker; is the anti-interference threshold parameter, which is used to describe the anti-interference ability of the candidate reference marker in a complex environment; As Figure 3 shown, in different treatment stages, the values of the size parameter, feature point distribution density parameter, and anti-interference threshold parameter of the candidate reference marker are all different; In the rough adjustment reset stage ( )Higher efficiency is required. To quickly determine the approximate pose of the fracture site and reduce the distance between the two ends of the broken bone, the model tends to select candidate fiducial markers in the candidate fiducial marker library that have larger sizes and relatively sparse feature point distributions (e.g., size parameter greater than 5 cm and feature point distribution density parameter less than 0.5 points / cm 2 ). These candidate fiducial markers can be quickly identified and located within a large field of view to meet the efficiency requirements of the rough alignment stage. These candidate fiducial markers will obtain higher adaptation scores during this stage.

[0028] Fine adjustment and calibration stage ( ). Higher precision is required. The model tends to select candidate fiducial markers in the candidate fiducial marker library that have medium sizes and relatively dense feature point distributions (e.g., size parameter greater than 2 cm and less than 5 cm, feature point distribution density parameter greater than 1 point / cm 2 ). These candidate fiducial markers can provide rich and accurate feature point information, facilitating high-precision pose adjustment; during the fine adjustment and calibration stage, these candidate fiducial markers will obtain higher adaptation scores, and the adaptation probability is also more in line with the actual requirements.

[0029] Maintenance and fixation stage ( ). It mainly focuses on monitoring the fluctuations of the tiny pose of the fracture site. The model will focus on selecting candidate fiducial markers in the candidate fiducial marker library that have smaller sizes and strong anti-interference capabilities (e.g., size parameter less than 2 cm and anti-interference threshold parameter greater than 0.8). These candidate fiducial markers can continuously and accurately sense tiny pose changes, and obtaining a higher adaptation probability can accurately reflect their applicability.

[0030] As Figure 4 shown, input the feature vector into the lightweight cross-scale marker generation model to predict the fiducial markers and generate the optimal fiducial markers; first, generate the adaptation scores of various candidate fiducial markers through the fully connected layer, as shown in Equation (2): (2) In the formula, represents the adaptation score of the th type of candidate fiducial marker, ReLU represents the activation function, represents the weight matrix of the fully connected layer, represents the feature vector dimension, represents the set of real numbers, represents rows columns of real number matrix; represents the bias vector of the fully connected layer, and the value range of each element in the bias vector is ; The Softmax function calculates the adaptation probabilities of various candidate reference markers through Equation (3), and selects the candidate reference marker with the maximum adaptation probability as the optimal reference marker; (3) In the formula, represents the adaptation probability of the th class of candidate reference markers, ; represents the exponential function with the natural constant e as the base; Based on the candidate reference marker library, a mapping relationship between different treatment stages and reference markers is established through a lightweight cross-scale marker generation model, enabling the model to encode according to the input treatment stage label , pose error vector and real-time scene parameter vector , and more accurately select the optimal reference marker that adapts to the current treatment stage from the candidate reference marker library, fully reflecting the differential selection of reference markers in different treatment stages to adapt to the different adjustment requirements of the external fixator for different treatment stages.

[0031] Step 2: Use the trained lightweight cross-scale marker generation model to generate an optimal reference marker for each surface of the current treatment stage's three-dimensional marker except the installation surface, and project each optimal reference marker onto the electronic paper screens of the corresponding surfaces of the upper and lower three-dimensional markers on the external fixator; Input the current treatment stage label encoding, pose error vector, and real-time scene parameter vector into the trained lightweight cross-scale marker generation model to generate an optimal reference marker for each surface of the three-dimensional marker except the installation surface; project each optimal reference marker onto the electronic paper screens of the corresponding surfaces of the three-dimensional markers on the upper and lower fixing rings of the external fixator, and each reference marker has a unique ID.

[0032] Step 3: Calibrate the binocular camera to obtain the internal and external parameters of the binocular camera; use the calibrated binocular camera to collect images of the upper and lower three-dimensional markers, and the images of the upper and lower three-dimensional markers both include left and right camera images; preprocess the left and right camera images to obtain the edge contours of the three-dimensional markers in the left and right camera images; The preprocessing includes grayscale conversion, binarization, morphological operations, and edge detection; taking the left camera image as an example, convert the left camera image from a color image to a grayscale image, convert the grayscale image to a binary image using an adaptive threshold segmentation method, perform morphological operations on the binary image to remove noise and holes in the image and smooth the image edges; perform edge detection on the image after morphological operations to obtain the edge contour of the three-dimensional marker in the left camera image; similarly, obtain the edge contour of the three-dimensional marker in the right camera image.

[0033] Step 4: Construct a set of 2D-3D point pairs that match feature points; 4-1) Respectively perform polygon fitting on the edge contours of the stereo markers in the left and right camera images to identify the reference markers on the surface of the stereo markers; use the reference markers with the same ID in the left and right camera images as the matching reference markers, use the feature points of the matching reference markers as candidate matching feature points, and extract the 2D coordinates of the candidate matching feature points in the left and right camera images; Construct a reprojection matrix based on the internal and external parameters of the binocular camera as: (4) In the formula, represents the transpose; when is the case, represents the internal parameters of the left camera, represents the external parameters of the left camera; when is the case, represents the internal parameters of the right camera, represents the external parameters of the right camera; represents the rotation matrix, represents the translation vector; Using the reprojection matrix and the 2D coordinates of the candidate matching feature points in the left and right camera images, calculate the 3D coordinates of the candidate matching feature points through triangulation, so as to obtain the 3D coordinates of all candidate matching feature points; the 2D coordinates and 3D coordinates of the candidate matching feature points in the left or right camera image form a set of 2D-3D point pairs, and the 2D-3D point pairs of all candidate matching feature points form a set of 2D-3D point pairs of candidate matching feature points; 4-2) Screen the candidate matching feature points, so as to optimize the set of 2D-3D point pairs of the candidate matching feature points and obtain the set of 2D-3D point pairs of the matching feature points; Adopt the RANSAC algorithm to eliminate low-quality candidate matching feature points. In each iteration, randomly select at least 4 groups of 2D-3D point pairs from the set of 2D-3D point pairs of the candidate matching feature points, and combine the internal parameters of the binocular camera to generate the candidate binocular camera pose using the EPnP algorithm, expressed as: (5) In the formula, represents the candidate binocular camera pose, represents the concatenation symbol, , respectively represent the candidate rotation matrix and the candidate translation vector, represents the homogeneous form of the 2D coordinates of the th candidate matching feature point, represents the homogeneous form of the 3D coordinates of the th candidate matching feature point, Represents the internal parameters of the binocular camera; According to the internal parameters of the binocular camera and the candidate binocular camera poses, reproject the 3D coordinates of the candidate matching feature points, project the candidate matching feature points onto the image plane, and obtain the reprojected 2D coordinates of the candidate matching feature points, denoted as: (6) In the formula, represents the reprojected 2D coordinate of the th candidate matching feature point, represents the reprojection operation; According to the 2D coordinates and reprojected 2D coordinates of the candidate matching feature points, calculate the reprojection error using Equation (7); (7) In the formula, represents the reprojection error of the th candidate matching feature point, represents the 2D coordinate of the th candidate matching feature point in the camera image; If the reprojection error is less than or equal to the set threshold, it is considered that the candidate matching feature point is a high-quality candidate matching feature point, and the candidate matching feature point is taken as the matching feature point; if the reprojection error is greater than the set threshold, it is considered that the candidate matching feature point is a low-quality candidate matching feature point, and it is excluded; traverse all candidate matching feature points to obtain the matching feature points, and then optimize the 2D-3D point pair set of the candidate matching feature points, that is, exclude the 2D-3D point pairs of the low-quality candidate matching feature points to obtain the 2D-3D point pair set of the matching feature points.

[0034] Fifth step: Based on the 2D-3D point pair set of the matching feature points and the internal parameters of the binocular camera, use the EPnP algorithm to accurately solve the pose of the binocular camera to obtain the pose of the binocular camera in the stereo marker coordinate system; the poses of the binocular camera in the upper and lower stereo marker coordinate systems are respectively denoted as and ; among them, , represent the rotation matrix and translation vector of the binocular camera in the upper stereo marker coordinate system, , represent the rotation matrix and translation vector of the binocular camera in the lower stereo marker coordinate system; Perform inverse transformation on the poses of the binocular camera in the upper and lower stereo marker coordinate systems through Equation (8) to obtain the poses of the upper and lower stereo markers in the binocular camera coordinate system; (8) In the formula, when , Represents the pose of the upper stereo marker in the binocular camera coordinate system; when , Represents the pose of the lower stereo marker in the binocular camera coordinate system; , Represents the rotation matrix and translation vector of the stereo marker in the binocular camera coordinate system; Convert the pose of the upper stereo marker in the binocular camera coordinate system to the lower stereo marker coordinate system through Equation (9) to obtain the relative pose between the upper and lower stereo markers, that is, the relative pose between the upper and lower fixing rings of the external fixator, and complete the pose estimation; (9) In the formula, , respectively represent the relative rotation matrix and relative translation vector.

[0035] Step 6: Calculate the pose error through Equation (10) ; (10) In the formula, represents the target pose between the upper and lower fixing rings of the external fixator, , respectively represent the target rotation matrix and target translation vector corresponding to the target pose; represents the translation error vector in the lower stereo marker coordinate system, , , and respectively represent the translation errors along , , axes in the lower stereo marker coordinate system; represents the rotation error matrix in the lower stereo marker coordinate, , , respectively represent the rotation angle deviations around , , axes in the lower stereo marker coordinate system, and are calculated by the following formula: (11) In the formula, represents the element in the th row and th column of the rotation error matrix; Judge whether the pose error exceeds the pose error threshold. The pose error thresholds in different treatment stages are different. If it does not exceed the pose error threshold, that is, Equation (12) is satisfied, the electric link does not need to be adjusted; (12) In the formula, , , respectively represent the translational error thresholds along the , , axes in the lower stereo marker coordinate system, and , , respectively represent the rotational error thresholds around the , , axes in the lower stereo marker coordinate system; If the pose error exceeds the pose error threshold, the electric link needs to be adjusted. The external fixator for bone contains a total of 6 electric links. The upper end of each electric link is connected to the upper fixing ring, and the lower end is connected to the lower fixing ring. Denote the coordinates of the connection points of each electric link to the lower fixing ring in the lower stereo marker coordinate system as , and denote the coordinates of the connection points of each electric link to the upper fixing ring in the upper stereo marker coordinate system as . According to the target pose between the upper and lower fixing rings of the external fixator for bone, calculate the coordinates of the connection points of each electric link to the upper fixing ring in the lower stereo marker coordinate system through formula (13); (13) Calculate the target length of each electric link according to formula (14): (14) Similarly, calculate the current length of each electric link according to the relative pose between the upper and lower fixing rings of the external fixator for bone, and calculate the adjustment amount of each electric link through formula (15); (15) Adjust the electric link according to the adjustment amount, and then adjust the relative pose between the upper and lower fixing rings of the external fixator for bone; when , the electric link elongates, and when , the electric link shortens; Step 7: After the adjustment is completed, return to Step 3, re-collect the upper and lower stereo marker images and perform preprocessing, and continue to execute Steps 4 to 6; repeat this cycle to adjust the pose of the external fixator for bone until the pose error does not exceed the pose error threshold, and complete the pose adjustment of the external fixator for bone.

[0036] As Figure 5As shown in the figure, the present invention also provides a pose adjustment system for an external bone fixator, which includes a visual perception module, a control module, and an execution module; The visual perception module includes a binocular camera and a stereo marker; upper and lower stereo markers are respectively connected to the upper and lower fixing rings of the external bone fixator, and the poses of the stereo markers change synchronously with those of the fixing rings; the stereo marker is a hollow regular icosahedron, with one of its surfaces serving as the mounting surface for the fixing ring, and the remaining 19 surfaces are each provided with an electronic paper screen in the shape of an equilateral triangle. The geometric center of the electronic paper screen coincides with the centroid of the corresponding surface, so that the electronic paper screen completely covers the surface. The electronic paper screen is used to dynamically display reference marks; the binocular camera is used to collect images of the stereo marker in real time; The control module includes a preprocessing unit, a pose estimation unit, and an adjustment judgment unit; the preprocessing unit is used for preprocessing the images of the stereo marker to extract the edge contour of the stereo marker; the pose estimation unit is used for estimating the relative pose between the upper and lower fixing rings of the external bone fixator; the adjustment judgment unit is used for judging whether it is necessary to adjust the electric link, and if so, calculating the adjustment amount; The execution module controls the elongation and shortening of each electric link according to the adjustment amount to realize the pose adjustment of the external bone fixator.

[0037] Those not described in the present invention are applicable to the prior art.

Claims

1. A method for phased adjustment of the pose of an external fixator for bone based on intelligent marking, characterized in that Including the following steps: Step 1: Construct a lightweight cross-scale marker generation model and train it; Input the feature vector composed of the treatment stage label encoding, the pose error vector, and the real-time scene parameter vector into the lightweight cross-scale marker generation model. Use the fully connected layer to generate the adaptation scores of various candidate reference markers, and use the Softmax function to convert the adaptation scores into adaptation probabilities. Take the candidate reference marker with the maximum adaptation probability as the optimal reference marker; Step 2: Use the trained lightweight cross-scale marker generation model to generate an optimal reference marker for each surface of the stereoscopic marker except the installation surface in the current treatment stage; Project each optimal reference marker onto the electronic paper screens on the corresponding surfaces of the upper and lower stereoscopic markers on the external fixator; Step 3: Calibrate the binocular camera, use the calibrated binocular camera to collect images of the upper and lower stereoscopic markers, and preprocess the images; Step 4: Identify the reference markers on the surfaces of the stereoscopic markers in the left and right camera images, and take the reference markers with the same ID as the matching reference markers; construct a 2D-3D point pair set of the matching feature points based on the feature points of the matching reference markers; Step 5: Based on the 2D-3D point pair set of the matching feature points, estimate the pose of the external fixator to obtain the relative pose between the upper and lower fixing rings of the external fixator; Step 6: Calculate the pose error according to the relative pose and the target pose between the upper and lower fixing rings of the external fixator; if the pose error does not exceed the pose error threshold, the electric link of the external fixator does not need to be adjusted; If the pose error exceeds the pose error threshold, the electric link of the external fixator needs to be adjusted; calculate the adjustment amount of each electric link and adjust the electric link according to the adjustment amount; Step 7: Return to Step 3, re-collect the images of the upper and lower stereoscopic markers and preprocess them, and continue to execute Steps 4 to 6; repeat this cycle to adjust the pose of the external fixator until the pose error does not exceed the pose error threshold, and complete the pose adjustment of the external fixator.

2. The method for phased adjustment of the position and orientation of an external fixator for bone based on intelligent markers according to claim 1, wherein Each type of candidate reference marker is constrained by a size parameter, a feature point distribution density parameter, and an anti-interference threshold parameter; the size parameter of the candidate reference marker corresponding to the coarse adjustment reset stage is greater than 5 cm, and the feature point distribution density parameter is less than 0.5 points / cm 2 ; the size parameter of the candidate reference marker corresponding to the fine adjustment correction stage is greater than 2 cm and less than 5 cm, and the feature point distribution density parameter is greater than 1 point / cm 2 ; the size parameter of the candidate reference marker corresponding to the maintenance and fixation stage is less than 2 cm, and the anti-interference threshold parameter is greater than 0.

8.

3. The method for adjusting the pose of the external fixator for bone in stages based on intelligent markers according to claim 1, wherein In Step 4, take the feature points of the matching reference markers as candidate matching feature points, extract the 2D coordinates of the candidate matching feature points in the left and right camera images, and calculate the 3D coordinates of the candidate matching feature points by the triangulation method; form a 2D-3D point pair with the 2D coordinates and the 3D coordinates of the candidate matching feature points in the left or right camera image to obtain a 2D-3D point pair set of the candidate matching feature points; use the RANSAC algorithm to screen the candidate matching feature points to obtain the matching feature points, and then obtain a 2D-3D point pair set of the matching feature points.

4. The method for phased adjustment of the position and pose of an external fixator for bone based on intelligent marking according to claim 1, wherein In Step 5, based on the 2D-3D point pair set of the matching feature points and the internal parameters of the binocular camera, use the EPnP algorithm to solve the pose of the binocular camera to obtain the pose of the binocular camera in the coordinate systems of the upper and lower stereoscopic markers; perform an inverse transformation on the pose of the binocular camera in the coordinate systems of the upper and lower stereoscopic markers to obtain the poses of the upper and lower stereoscopic markers in the coordinate system of the binocular camera; Convert the pose of the upper stereo marker in the binocular camera coordinate system to the lower stereo marker coordinate system to obtain the relative pose between the upper and lower stereo markers, that is, the relative pose between the upper and lower fixing rings of the external fixator.

5. The method for phased adjustment of the pose of the external fixator for bone based on intelligent marking according to claim 1, wherein In the sixth step, convert the coordinates of the connection points of each electric link and the upper fixing ring in the upper stereo marker coordinate system to the lower stereo marker coordinate system through the target pose between the upper and lower fixing rings of the external fixator, and calculate the target length of each electric link according to the coordinates of the connection points of each electric link and the upper and lower fixing rings in the lower stereo marker coordinate system; Similarly, calculate the current length of each electric link according to the relative pose between the upper and lower fixing rings of the external fixator. The difference between the target length and the current length of the electric link is the adjustment amount.

6. The method for phased adjustment of the pose of an external fixator for bone according to claim 1, wherein The real-time scene parameter vector consists of the distance between the stereo marker and the binocular camera, the incident angle, the ambient light intensity, and the stereo marker occlusion rate.

7. The method for phased adjustment of the pose of an external fixator for bone based on intelligent markers according to any one of claims 1 to 6, characterized in that, The preprocessing includes grayscale conversion, binarization, morphological operations, and edge detection.

8. The method for phased adjustment of the position and posture of the external fixator for bone based on intelligent marking according to claim 7, wherein The stereo marker adopts a hollow regular icosahedron, with one surface as the installation surface, and electronic paper screens are arranged on the remaining 19 surfaces. The geometric center of the electronic paper screen coincides with the center of gravity of the surface.

9. An external bone fixator pose adjustment system that adjusts the pose of the external bone fixator according to the method described in claim 1; characterized in that, The system includes a visual perception module, a control module, and an execution module; The visual perception module is used to collect stereo marker images in real time; The control module is used to estimate the relative pose between the upper and lower fixing rings of the external fixator, and judge whether it is necessary to adjust the electric link. If necessary, calculate the adjustment amount; The execution module is used to control the elongation and shortening of each electric link.

Images