Method and system for adjusting the posture of external fixator in stages based on intelligent marking

Abstract

The present invention discloses a method and system for adjusting the posture of an external fixator in stages based on intelligent marking, which belongs to the field of medical devices. First, a lightweight cross-scale marker generation model is constructed and trained, and the trained model is used to generate an optimal reference marker for each surface of the three-dimensional marker except the mounting surface; each optimal reference marker is projected onto the electronic paper screen of the corresponding surface of the upper and lower three-dimensional markers on the external fixator; then, a binocular camera is used to collect the images of the upper and lower three-dimensional markers and perform preprocessing; a 2D-3D point pair set of matching feature points is constructed to estimate the relative posture between the upper and lower fixing rings of the external fixator; finally, the posture error is calculated; if the posture error does not exceed the threshold, the external fixator is not adjusted; if the posture error exceeds the threshold, the adjustment amount of the electric connecting rod is calculated and adjusted; this cycle is repeated until the posture error does not exceed the threshold and the posture adjustment is completed. Posture adjustment is performed in stages to improve the adjustment accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices, and in particular relates to a method and system for adjusting the posture of an external fixator in stages based on intelligent marking. Background Art

[0002] An external fixator is a rehabilitation device used for closed reduction of fractures. It mainly consists of upper and lower fixation rings and a Taylor space bracket connecting the two fixation rings. By changing the shape of the Taylor space bracket, the relative position of the two fixation rings is changed, thereby achieving the purpose of treating fracture displacement, bone nonunion and malunion. Therefore, the accuracy of the external fixator's position adjustment directly affects the treatment effect. The fracture treatment process can be divided into three stages: coarse reduction, fine correction, and maintenance fixation. The goals achieved in different stages are different, and the posture adjustment requirements are also different. The coarse reduction stage requires rapid determination of the approximate posture of the fracture site and efficient and fast initial alignment. The fine correction stage requires high-precision posture adjustment at the sub-pixel level. The maintenance fixation stage requires long-term and stable monitoring of small posture changes of the fixation ring.

[0003] The current mainstream posture adjustment of external fixators uses an open-loop method that combines kinematic modeling with manual operation. Doctors need to manually input the target posture parameters. The system solves the branch adjustment amount of the Taylor space bracket based on complex spatial matrix operations, and then drives the execution module to adjust the length of each connecting rod. Due to the lack of a real-time posture feedback mechanism, the system cannot automatically perceive the posture errors generated during operation. Doctors can only rely on regularly taken X-ray images or monitoring data from external sensors to obtain the reset effect offline. Once a deviation is found, the posture parameters are manually corrected, forming a cyclic adjustment mode of "manual measurement → adjustment → verification". This mode not only has a cumbersome and lengthy operation process and is heavily dependent on the doctor's experience, but also easily leads to error accumulation, resulting in low treatment efficiency and difficulty in meeting the high requirements of modern orthopedic treatment for accuracy and timeliness.

[0004] In the field of computer vision, although marker-based pose estimation technology has been used for the pose adjustment of external fixators, existing methods still have significant limitations. Existing marker-based pose estimation methods mostly use static reference markers, whose characteristic patterns are fixed and single, and completely ignore the differentiated requirements of different stages in the fracture treatment process. They are unable to achieve efficient and fast initial alignment in the coarse adjustment and reduction stage, nor can they achieve sub-pixel high precision requirements in the fine adjustment and correction stage. They are even more unable to capture the slight pose changes of the fixation ring in time during the fixation maintenance stage, resulting in a balance between treatment efficiency and accuracy. In addition, existing marker-based pose estimation models mostly rely on single modal data for pose solution, or rely solely on the visual feature recognition and positioning of the marker, or simply adjust parameters based on pose errors, making it difficult to fully capture the complex information in orthopedic treatment scenarios. Due to the lack of comprehensive analysis of multiple factors such as the characteristics of the treatment stage and real-time environmental changes, the model is unable to accurately understand scene changes when faced with complex environments such as occlusion and lighting changes, as well as the special requirements of different treatment stages. This leads to serious lack of adaptability between marker selection and pose estimation in different treatment stages. This limitation makes it difficult for traditional pose estimation models to meet the strict requirements of orthopedic rehabilitation treatment for high precision and strong robustness, and cannot provide reliable technical support for clinical treatment. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is to provide a method and system for adjusting the posture of an external fixator in stages based on intelligent marking.

[0006] The present invention solves the technical problem by adopting the following technical solutions:

[0007] A method for adjusting the position of an external fixator in stages based on intelligent marking comprises the following steps:

[0008] Step 1: Build a lightweight cross-scale label generation model and train it;

[0009] The feature vector consisting of the treatment phase label encoding, the pose error vector, and the real-time scene parameter vector is input into a lightweight cross-scale marker generation model. The fully connected layer is used to generate the adaptation scores of various candidate fiducial markers. The softmax function is used to convert the adaptation scores into adaptation probabilities. The candidate fiducial marker with the highest adaptation probability is selected as the optimal fiducial marker.

[0010] Step 2: Utilize the trained lightweight cross-scale marker generation model to generate an optimal fiducial marker for each face of the 3D marker in the current treatment phase, excluding the mounting face. Project each optimal fiducial marker onto the corresponding electronic paper screen of the upper and lower 3D markers on the external fixator.

[0011] Step 3: Calibrate the binocular camera, use the calibrated binocular camera to collect the upper and lower stereo marker images, and pre-process the images;

[0012] Step 4: Identify the fiducial markers on the surface of the 3D marker in the left and right camera images, and use the fiducial markers with the same ID as the matching fiducial markers; construct a set of 2D-3D point pairs of matching feature points based on the feature points of the matching fiducial markers;

[0013] Step 5: Based on the 2D-3D point pair set of matched feature points, the external fixator posture is estimated to obtain the relative posture between the upper and lower fixation rings of the external fixator;

[0014] Step 6: Calculate the posture error based on the relative posture between the upper and lower fixation rings of the external fixator and the target posture; if the posture error does not exceed the posture error threshold, the electric connecting rod of the external fixator does not need to be adjusted;

[0015] If the posture error exceeds the posture error threshold, the electric connecting rod of the external fixator needs to be adjusted; the adjustment amount of each electric connecting rod is calculated, and the electric connecting rod is adjusted according to the adjustment amount;

[0016] Step 7: Return to step 3, reacquire the upper and lower stereoscopic marker images and perform preprocessing, and continue with steps 4 to 6. Repeat this cycle to adjust the external fixator's posture until the posture error does not exceed the posture error threshold, completing the external fixator's posture adjustment.

[0017] Furthermore, each candidate fiducial marker is constrained by size parameters, feature point distribution density parameters, and anti-interference threshold parameters; the size parameters of the candidate fiducial markers corresponding to the coarse adjustment reset stage are greater than 5 cm, and the feature point distribution density parameters are less than 0.5 points / cm. 2 The size parameter of the candidate fiducial marker corresponding to the fine-tuning correction stage is greater than 2 cm and less than 5 cm, and the feature point distribution density parameter is greater than 1 point / cm 2 ; Maintain the size parameter of the candidate reference marker corresponding to the fixed stage is less than 2cm, and the anti-interference threshold parameter is greater than 0.8.

[0018] Furthermore, in the fourth step, the feature points of the matching reference marks are used as candidate matching feature points, the 2D coordinates of the candidate matching feature points in the left and right camera images are extracted, and the 3D coordinates of the candidate matching feature points are calculated by triangulation method; the 2D coordinates and 3D coordinates of the candidate matching feature points in the left or right camera image are combined into 2D-3D point pairs to obtain a 2D-3D point pair set of candidate matching feature points; the RANSAC algorithm is used to screen the candidate matching feature points to obtain matching feature points, and then a 2D-3D point pair set of matching feature points is obtained.

[0019] Furthermore, in the fifth step, the EPnP algorithm is used to solve the binocular camera pose based on the 2D-3D point pair set of matching feature points and the internal parameters of the binocular camera to obtain the pose of the binocular camera in the upper and lower stereo marker coordinate systems; the pose of the binocular camera in the upper and lower stereo marker coordinate systems is inversely transformed to obtain the pose of the upper and lower stereo markers in the binocular camera coordinate system; the pose of the upper stereo marker in the binocular camera coordinate system is converted to the lower stereo marker coordinate system to obtain the relative pose between the upper and lower stereo markers, that is, the relative pose between the upper and lower fixation rings of the external fixator.

[0020] Furthermore, in the sixth step, the coordinates of the connection points between each electric connecting rod and the upper fixing ring in the upper three-dimensional marker coordinate system are converted to the lower three-dimensional marker coordinate system through the target posture between the upper and lower fixing rings of the external fixator. The target length of each electric connecting rod is calculated based on the coordinates of the connection points between each electric connecting rod and the upper and lower fixing rings in the lower three-dimensional marker coordinate system. Similarly, the current length of each electric connecting rod is calculated based on the relative posture between the upper and lower fixing rings of the external fixator. The difference between the target length and the current length of the electric connecting rod is the adjustment amount.

[0021] Furthermore, the real-time scene parameter vector is composed of the distance between the stereo marker and the binocular camera, the incident angle, the ambient light intensity, and the occlusion rate of the stereo marker.

[0022] Furthermore, the preprocessing includes grayscale, binarization, morphological operation and edge detection.

[0023] Furthermore, the three-dimensional marker adopts a hollow regular icosahedron, one surface of which is used as a mounting surface, and the remaining 19 surfaces are all provided with electronic paper screens, and the geometric center of the electronic paper screen coincides with the center of gravity of the surface.

[0024] The present invention also provides a posture adjustment system for an external fixator, comprising a visual perception module, a control module and an execution module;

[0025] The visual perception module is used to collect three-dimensional marker images in real time;

[0026] The control module is used to estimate the relative position between the upper and lower fixing rings of the external fixator, and determine whether the electric connecting rod needs to be adjusted, and if so, calculate the adjustment amount;

[0027] The execution module is used to control the extension and contraction of each electric connecting rod.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. This invention innovatively divides the fracture treatment process into three stages: coarse reduction, fine correction, and maintenance fixation. This addresses the differentiated requirements of different stages, such as rapid positioning in the coarse reduction stage, high-precision correction in the fine correction stage, and long-term stability monitoring in the maintenance fixation stage. By matching appropriate reference markers, this invention can more accurately adapt to the requirements of different treatment stages, improve the accuracy of external fixator position adjustment, and significantly enhance treatment effectiveness and efficiency.

[0030] 2. Traditional repositioning technologies rely on static markers with fixed features, which are unable to respond to the dynamic needs and environmental interference of different treatment stages, resulting in a disconnect between visual feedback and mechanical execution. For example, during the coarse repositioning phase, the fixed size of the markers may be limited by the scene and difficult to quickly locate. During the fine correction phase, insufficient feature density affects accuracy, and feature point mismatching is prone to occur in scenes with weak textures or changes in viewing angle. However, the three-dimensional markers of the present invention use a regular icosahedron as a carrier, and through 19 dynamically switchable equilateral triangle electronic paper screens, combined with a lightweight cross-scale marker generation model, more adaptable fiducial markers are generated in real time based on the treatment stage, posture error, and environmental parameters (such as distance, lighting, and occlusion). For example, high-contrast, large-size fiducial markers are generated during the coarse repositioning phase to expand the recognition range. During the fine correction phase, high-density, medium-sized fiducial markers are generated to improve feature point matching accuracy. During the fixed phase, more interference-resistant fiducial markers are generated to stably monitor small fluctuations. The fiducial markers achieve precise feature matching through unique IDs and geometric consistency constraints, avoiding the mismatching problem of traditional algorithms. The present invention can achieve pose estimation entirely based on visual geometric reasoning, significantly enhancing the versatility and robustness of the method, and promoting the pose adjustment of external fixators from "experience-driven" open-loop operation to "system intelligence" closed-loop control.

[0031] 3. Traditional marker-based pose estimation models typically rely on single-modal data (e.g., using only marker visual features or pose error information), resulting in insufficient understanding of complex scenarios and limited adaptability. The lightweight cross-scale marker generation model proposed in this paper innovatively integrates multimodal data such as treatment stage labels, pose error vectors, and real-time scene parameters as input. By integrating discrete treatment stage information, continuous pose errors, and real-time scene parameters, the model can fully perceive the treatment stage, capture the correlation between different factors, and generate more matching benchmark markers for different treatment stages. This significantly enhances the adaptability to complex and changing clinical scenarios, the accuracy of benchmark marker selection, and the precision of pose adjustment.

[0032] 4. Traditional fiducial marker designs lack systematic theoretical support and are mostly based on empirical design, making it difficult to quantitatively evaluate the degree of match between fiducial marker performance and treatment needs. The present invention performs parameterized physical constraint design on fiducial markers, and establishes a standardized fiducial marker performance description system by defining geometric dimensions, feature point distribution density, and anti-interference threshold parameters. Each parameter is closely related to treatment needs, such as large size corresponding to rapid identification, high-density features meeting high-precision requirements, and high anti-interference thresholds adapting to complex environments. This parametric design frees the selection and optimization of fiducial markers from empirical dependence and realizes scientific design based on theoretical models. Compared with traditional methods, it greatly improves the standardization and clinical applicability of marker design.

[0033] 5. Traditional fiducial marker generation often suffers from design limitations when handling tasks at different scales (e.g., from macroscopic rapid positioning to microscopic precise adjustment), making it difficult to balance efficiency and accuracy. The lightweight cross-scale marker generation model of the present invention has cross-scale capabilities. Through a phased strategy and dynamic adaptation mechanism, it quickly grasps the overall posture at the macroscale during the coarse adjustment and reset phase to achieve efficient initial alignment; switches to the microscale during the fine adjustment and correction phase, utilizing fine features for high-precision correction; and continuously monitors tiny-scale posture changes during the fixed maintenance phase. This cross-scale capability enables the present invention to flexibly respond to various scale requirements at different treatment stages, overcoming the performance bottleneck of traditional fiducial marker generation methods in multi-scale tasks.

[0034] 6. The traditional posture adjustment of external fixators relies on the open-loop mode of "manual measurement → adjustment → verification". Doctors need to manually input parameters and detect errors offline, which is cumbersome and inefficient. The present invention constructs a closed-loop adjustment system of "visual perception module → control module → execution module". The visual perception module collects images in real time, and the control module automatically estimates the posture and generates adjustment instructions based on the posture error. The execution module adjusts the electric connecting rod according to the adjustment instructions, forming a coherent automated closed-loop control of "perception-policy-execution". Compared with the traditional open-loop adjustment method, it gets rid of the reliance on manual experience and realizes the automatic and timely detection and correction of posture deviations during fracture reduction.

[0035] 7. Compared with the traditional external fixator posture estimation method that relies on Taylor space bracket kinematic modeling, the present invention realizes posture estimation entirely based on visual recognition and geometric reasoning, omitting the modeling and solution process of Taylor space bracket structural parameters, significantly reducing the computational complexity, and enhancing versatility and practicality.

[0036] 8. In the existing binocular pose estimation, feature point matching usually relies on traditional image local feature extraction and matching algorithms such as SIFT and ORB. These methods are prone to feature point mismatching problems when faced with weak image texture, perspective changes or occlusion. The present invention projects the reference markers onto the electronic paper screen on the surface of the three-dimensional marker. Each reference marker has a unique ID, which is equivalent to structured encoding of the reference markers. By identifying the reference markers with the same ID in the left and right camera images, and matching feature points with consistent positions on the identified reference markers, the geometric consistency is fully utilized to solve the mismatching problem, thereby improving the robustness and accuracy of feature point matching. Furthermore, the RANSAC algorithm is used to screen high-quality matching feature points, and these feature points are used to accurately solve the camera pose, which is conducive to improving the accuracy of pose estimation.

[0037] 9. Marker-based monocular pose estimation typically relies on the prior 3D coordinates of feature points and is highly sensitive to the quality of image feature extraction. It is easily affected by factors such as occlusion, distortion, and blur, which can reduce pose estimation accuracy. This invention uses the 2D coordinates of feature points in the left and right camera images and calculates their 3D coordinates through triangulation. This method enables accurate and stable pose estimation even in complex scenes with unknown structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall flow chart of the present invention;

[0039] Figure 2 Schematic diagram of the position of the three-dimensional marker of the present invention on the external fixator;

[0040] Figure 3 This is a diagram of the training framework of the lightweight cross-scale label generation model of the present invention;

[0041] Figure 4 A diagram depicting the characteristics of the fiducial markers at different treatment stages of the present invention;

[0042] Figure 5 This is a structural diagram of the external fixator posture adjustment system of the present invention. DETAILED DESCRIPTION

[0043] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to further illustrate the technical solutions of the present invention in detail and are not intended to limit the scope of protection of the present application.

[0044] The present invention provides a method for adjusting the position of an external fixator in stages based on intelligent marking (hereinafter referred to as the method, see Figure 1-5 ), including the following steps:

[0045] Step 1: Build a lightweight cross-scale label generation model and train the lightweight cross-scale label generation model;

[0046] Collect historical clinical cases, extract the case's treatment stage label encoding, pose error vector, real-time scene parameter vector and optimal reference marker to form a sample , several samples form a training set; among them, Indicates the label code of the treatment stage, 1, 2, and 3 represent the label codes of the coarse adjustment reset, fine adjustment correction, and maintenance fixation stages respectively; represents the pose error vector, represents the translation error vector, Represents the rotation error vector, and the initial value of the posture error vector is the zero vector; represents the real-time scene parameter vector, Indicates the distance between the stereo marker and the binocular camera; represents the incident angle, that is, the angle between the observation direction and the normal of the current observed surface of the 3D marker; Indicates the ambient light intensity. represents the occlusion rate of the three-dimensional marker; is the optimal benchmark marker, Represents a library of candidate fiducial markers.

[0047] According to the actual demand for external fixators in different treatment stages, pre-defined Classify candidate fiducial markers to form a candidate fiducial marker library ;in, Indicates the Each candidate benchmark marker is characterized by size, feature point distribution density and anti-interference. Class candidate fiducial markers Expressed as:

[0048] (1)

[0049] Where, Represents a feature descriptor; represents the size parameter, which is used to describe the size of the candidate fiducial marker; Represents the feature point distribution density parameter, which is used to describe the distribution of feature points of candidate fiducial markers; is the anti-interference threshold parameter, which is used to describe the anti-interference ability of the candidate fiducial marker in a complex environment;

[0050] like Figure 3 As shown, at different treatment stages, the values ​​of the size parameters, feature point distribution density parameters, and anti-interference threshold parameters of the candidate fiducial markers are different;

[0051] Coarse reset stage ( ) has higher efficiency requirements. In order to quickly determine the approximate position of the fracture site and shorten the distance between the two ends of the broken bone, the model tends to select candidate fiducial markers with larger sizes and relatively sparse feature point distribution (for example, the size parameter is greater than 5 cm and the feature point distribution density parameter is less than 0.5 points / cm). 2 ) candidate fiducial markers, which can be quickly identified and located within a larger field of view to meet the efficiency requirements of the coarse adjustment and reset stage. These candidate fiducial markers will obtain higher adaptation scores in this stage.

[0052] Fine-tuning stage ( ) has high accuracy requirements, and the model tends to select candidate fiducial markers with medium size and relatively dense feature point distribution (for example, the size parameter is greater than 2cm and less than 5cm, and the feature point distribution density parameter is greater than 1 point / cm). 2 ) candidate fiducial markers, which can provide rich and accurate feature point information, facilitating high-precision pose adjustment; in the fine-tuning and correction stage, these candidate fiducial markers will obtain higher adaptation scores, and the adaptation probability will be more in line with actual needs.

[0053] Maintain fixed phase ( ) mainly focuses on monitoring the fluctuations of tiny postures at the fracture site. The model will focus on selecting candidate fiducial markers with smaller size and stronger anti-interference ability (for example, the size parameter is less than 2 cm and the anti-interference threshold parameter is greater than 0.8) in the candidate fiducial marker library. These candidate fiducial markers can continuously and accurately perceive tiny posture changes, and obtain a higher adaptation probability to accurately reflect their applicability.

[0054] like Figure 4 As shown, the feature vector The input is fed into the lightweight cross-scale tag generation model to predict the reference tag and generate the optimal reference tag. First, the adaptation scores of various candidate reference tags are generated through the fully connected layer, as shown in formula (2):

[0055] (2)

[0056] Where, Indicates the The adaptation score of the class candidate benchmark marker, ReLU represents the activation function, represents the weight matrix of the fully connected layer, Represents the feature vector Dimensions, represents the set of real numbers, express OK A real matrix of columns; Represents the bias vector of the fully connected layer. The value range of each element in the bias vector is ;

[0057] The Softmax function calculates the adaptation probability of each candidate benchmark marker through formula (3), and takes the candidate benchmark marker with the largest adaptation probability as the optimal benchmark marker;

[0058] (3)

[0059] Where, Indicates the The fitting probability of the candidate fiducial marker class, ; represents an exponential function with the natural constant e as the base;

[0060] Based on the candidate benchmark marker library, a lightweight cross-scale marker generation model is used to establish the mapping relationship between different treatment stages and benchmark markers, so that the model can encode the input treatment stage label. , pose error vector and the real-time scene parameter vector , more accurately select the optimal fiducial marker suitable for the current treatment stage from the candidate fiducial marker library, fully reflect the differentiated selection of fiducial markers in different treatment stages, and adapt to the different adjustment requirements of external fixators in different treatment stages.

[0061] Step 2: Use the trained lightweight cross-scale marker generation model to generate an optimal fiducial marker for each surface of the 3D marker in the current treatment phase, excluding the mounting surface. Project each optimal fiducial marker onto the corresponding electronic paper screen of the upper and lower 3D markers on the external fixator.

[0062] The label encoding of the current treatment stage, the pose error vector, and the real-time scene parameter vector are input into the trained lightweight cross-scale marker generation model to generate an optimal fiducial marker for each surface of the 3D marker except the mounting surface. Each optimal fiducial marker is projected onto the electronic paper screen of the corresponding surface of the 3D marker on the upper and lower fixation rings of the external fixator. Each fiducial marker has a unique ID.

[0063] Step 3: Calibrate the binocular camera to obtain the intrinsic and extrinsic parameters of the binocular camera; use the calibrated binocular camera to capture the upper and lower stereo marker images, which both include the left and right camera images; preprocess the left and right camera images to obtain the edge contours of the stereo markers in the left and right camera images;

[0064] Preprocessing includes grayscale, binarization, morphological operations and edge detection; taking the left camera image as an example, the left camera image is converted from a color image to a grayscale image, and the adaptive threshold segmentation method is used to convert the grayscale image into a binary image. Morphological operations are performed on the binary image to remove noise and holes in the image and smooth the image edges; edge detection is performed on the image after morphological operations to obtain the edge contour of the three-dimensional marker in the left camera image; similarly, the edge contour of the three-dimensional marker in the right camera image is obtained.

[0065] Step 4: Construct a set of 2D-3D point pairs matching feature points;

[0066] 4-1) Perform polygon fitting on the edge contours of the 3D markers in the left and right camera images to identify the fiducial markers on the surface of the 3D markers. Fiducial markers with the same ID in the left and right camera images are used as matching fiducial markers, and the feature points of the matching fiducial markers are used as candidate matching feature points. The 2D coordinates of the candidate matching feature points in the left and right camera images are extracted.

[0067] Construct a reprojection matrix based on the intrinsic and extrinsic parameters of the binocular camera for:

[0068] (4)

[0069] Where, Indicates transposition; when hour, represents the left camera internal parameter, Represents the left camera external parameter; when hour, represents the intrinsic parameter of the right camera, represents the right camera extrinsic parameter; represents the rotation matrix, represents the translation vector;

[0070] Using the reprojection matrix and the 2D coordinates of the candidate matching feature points in the left and right camera images, the 3D coordinates of the candidate matching feature points are calculated by triangulation, thereby obtaining the 3D coordinates of all candidate matching feature points; the 2D coordinates and 3D coordinates of the candidate matching feature points in the left or right camera image form a set of 2D-3D point pairs, and the 2D-3D point pairs of all candidate matching feature points form a 2D-3D point pair set of candidate matching feature points;

[0071] 4-2) Screening candidate matching feature points, thereby optimizing the 2D-3D point pair set of candidate matching feature points to obtain a 2D-3D point pair set of matching feature points;

[0072] The RANSAC algorithm is used to eliminate low-quality candidate matching feature points. In each iteration, at least four groups of 2D-3D point pairs are randomly selected from the set of candidate matching feature points. Combined with the intrinsic parameters of the binocular camera, the EPnP algorithm is used to generate the candidate binocular camera pose, which is expressed as:

[0073] (5)

[0074] Where, represents the candidate binocular camera pose, Indicates the concatenation character. 、 represent the candidate rotation matrix and candidate translation vector respectively, Indicates the The homogeneous form of the 2D coordinates of candidate matching feature points, Indicates the The homogeneous form of the 3D coordinates of candidate matching feature points, Represents the internal parameters of the binocular camera;

[0075] According to the binocular camera internal parameters and the candidate binocular camera pose, the 3D coordinates of the candidate matching feature points are reprojected, and the candidate matching feature points are projected onto the image plane to obtain the reprojected 2D coordinates of the candidate matching feature points, which are expressed as:

[0076] (6)

[0077] Where, Indicates the The reprojected 2D coordinates of candidate matching feature points, Represents a reprojection operation;

[0078] According to the 2D coordinates of the candidate matching feature points and the reprojected 2D coordinates, the reprojection error is calculated using formula (7);

[0079] (7)

[0080] Where, Indicates the The reprojection error of candidate matching feature points, Indicates the The 2D coordinates of candidate matching feature points in the camera image;

[0081] If the reprojection error is less than or equal to the set threshold, the candidate matching feature point is considered to be a high-quality candidate matching feature point, and the candidate matching feature point is used as the matching feature point; if the reprojection error is greater than the set threshold, the candidate matching feature point is considered to be a low-quality candidate matching feature point and is eliminated; all candidate matching feature points are traversed to obtain matching feature points, and then the 2D-3D point pair set of the candidate matching feature points is optimized, that is, the 2D-3D point pairs of the low-quality candidate matching feature points are eliminated to obtain the 2D-3D point pair set of matching feature points.

[0082] Step 5: Based on the 2D-3D point pair set of matching feature points and the internal parameters of the binocular camera, the EPnP algorithm is used to accurately solve the binocular camera pose and obtain the binocular camera pose in the stereo marker coordinate system; the binocular camera pose in the upper and lower stereo marker coordinate systems are recorded as and ;in, 、 Represents the rotation matrix and translation vector of the binocular camera in the upper stereo marker coordinate system, 、 Represents the rotation matrix and translation vector of the binocular camera in the lower stereo marker coordinate system;

[0083] The position of the binocular camera in the upper and lower stereo marker coordinate systems is inversely transformed by formula (8) to obtain the position of the upper and lower stereo markers in the binocular camera coordinate system;

[0084] (8)

[0085] In the formula, when hour, Represents the position of the upper stereo marker in the binocular camera coordinate system; when hour, Represents the position of the stereo marker in the binocular camera coordinate system; 、 Represents the rotation matrix and translation vector of the stereo marker in the binocular camera coordinate system;

[0086] The pose of the upper stereo marker in the binocular camera coordinate system is converted to the lower stereo marker coordinate system by formula (9), and the relative pose between the upper and lower stereo markers is obtained. , that is, the relative posture between the upper and lower fixing rings of the external fixator, completing the posture estimation;

[0087] (9)

[0088] Where, 、 Represent the relative rotation matrix and relative translation vector respectively.

[0089] Step 6: Calculate the pose error using formula (10) ;

[0090] (10)

[0091] Where, Indicates the target position between the upper and lower fixation rings of the external fixator. 、 Represent the target rotation matrix and target translation vector corresponding to the target pose respectively; represents the translation error vector in the lower stereo marker coordinate system, , 、 and Respectively represent the lower edge of the lower three-dimensional marker coordinate system 、 、 Axis translation error; represents the rotation error matrix in the lower stereo marker coordinates, 、 、 Respectively represent the coordinate system of the lower three-dimensional marker around 、 、 The rotation angle deviation of the axis is calculated by the following formula:

[0092] (11)

[0093] Where, Represents the rotation error matrix Rank Elements of the column;

[0094] Determine whether the posture error exceeds the posture error threshold. The posture error threshold is different in different treatment stages. If it does not exceed the posture error threshold, that is, it satisfies formula (12), the electric connecting rod does not need to be adjusted;

[0095] (12)

[0096] Where, 、 、 Respectively represent the lower edge of the lower three-dimensional marker coordinate system 、 、 The translation error threshold of the axis, 、 、 Respectively represent the coordinate system of the lower three-dimensional marker around 、 、 The rotation error threshold of the axis;

[0097] If the posture error exceeds the posture error threshold, the electric connecting rod needs to be adjusted; the external fixator contains a total of 6 electric connecting rods, the upper end of each electric connecting rod is connected to the upper fixing ring, and the lower end is connected to the lower fixing ring. The coordinates of the connection points between each electric connecting rod and the lower fixing ring in the lower three-dimensional marker coordinate system are marked as The coordinates of the connection points between each electric connecting rod and the upper fixed ring in the upper three-dimensional marker coordinate system are According to the target position between the upper and lower fixing rings of the external fixator, the coordinates of the connection points between each electric connecting rod and the upper fixing ring in the coordinates of the lower three-dimensional marker are calculated by formula (13): ;

[0098] (13)

[0099] Calculate the target length of each electric connecting rod according to formula (14) :

[0100] (14)

[0101] Similarly, the current length of each electric connecting rod is calculated based on the relative position between the upper and lower fixing rings of the external fixator. , calculate the adjustment amount of each electric connecting rod through formula (15) ;

[0102] (15)

[0103] Adjust the electric connecting rod according to the adjustment amount, and then adjust the relative position between the upper and lower fixing rings of the external fixator; When the electric connecting rod is extended, When , the electric connecting rod is shortened;

[0104] Step 7: After the adjustment is completed, return to step 3, reacquire the upper and lower three-dimensional marker images and perform preprocessing, and continue with steps 4 to 6. In this cycle, adjust the external fixator posture until the posture error does not exceed the posture error threshold, completing the external fixator posture adjustment.

[0105] like Figure 5 As shown, the present invention also provides a posture adjustment system for an external fixator, comprising a visual perception module, a control module and an execution module;

[0106] The visual perception module includes a binocular camera and a 3D marker; the upper and lower fixing rings of the external fixator are respectively connected to the upper and lower 3D markers, and the positions of the 3D markers and the fixing rings change synchronously; the 3D marker is a hollow icosahedron, one of which serves as a mounting surface for the fixing ring, and the remaining 19 surfaces are each provided with an equilateral triangular electronic paper screen, the geometric center of the electronic paper screen coincides with the center of gravity of the corresponding surface, so that the electronic paper screen completely covers the surface, and the electronic paper screen is used to dynamically display the reference marker; the binocular camera is used to capture the 3D marker image in real time;

[0107] The control module includes a preprocessing unit, a posture estimation unit, and an adjustment judgment unit; the preprocessing unit is used to preprocess the 3D marker image and extract the edge contour of the 3D marker; the posture estimation unit is used to estimate the relative posture between the upper and lower fixing rings of the external fixator; the adjustment judgment unit is used to determine whether the electric connecting rod needs to be adjusted and, if necessary, calculate the adjustment amount;

[0108] The execution module controls the extension and shortening of each electric connecting rod according to the adjustment amount to achieve the position adjustment of the external fixator.

[0109] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A posture adjustment system for an external fixator, characterized in that: The system includes a visual perception module, a control module and an execution module; The visual perception module is used to collect three-dimensional marker images in real time; The control module is used to estimate the relative position between the upper and lower fixing rings of the external fixator, and determine whether the electric connecting rod needs to be adjusted, and if so, calculate the adjustment amount; The execution module is used to control the extension and contraction of each electric connecting rod; The system adjusts the external fixator position according to the following method, including the following steps: Step 1: Build a lightweight cross-scale label generation model and train it; The feature vector consisting of the treatment phase label encoding, the pose error vector, and the real-time scene parameter vector is input into a lightweight cross-scale marker generation model. The fully connected layer is used to generate the adaptation scores of various candidate fiducial markers. The softmax function is used to convert the adaptation scores into adaptation probabilities. The candidate fiducial marker with the highest adaptation probability is selected as the optimal fiducial marker. Step 2: Utilize the trained lightweight cross-scale marker generation model to generate an optimal fiducial marker for each face of the 3D marker in the current treatment phase, excluding the mounting face. Project each optimal fiducial marker onto the corresponding electronic paper screen of the upper and lower 3D markers on the external fixator. Step 3: Calibrate the binocular camera, use the calibrated binocular camera to collect the upper and lower stereo marker images, and pre-process the images; Step 4: Identify the fiducial markers on the surface of the 3D marker in the left and right camera images, and use the fiducial markers with the same ID as the matching fiducial markers; construct a set of 2D-3D point pairs of matching feature points based on the feature points of the matching fiducial markers; Step 5: Based on the 2D-3D point pair set of matched feature points, the external fixator posture is estimated to obtain the relative posture between the upper and lower fixation rings of the external fixator; Step 6: Calculate the posture error based on the relative posture between the upper and lower fixation rings of the external fixator and the target posture; if the posture error does not exceed the posture error threshold, the electric connecting rod of the external fixator does not need to be adjusted; If the posture error exceeds the posture error threshold, the electric connecting rod of the external fixator needs to be adjusted; the adjustment amount of each electric connecting rod is calculated, and the electric connecting rod is adjusted according to the adjustment amount; Step 7: Return to step 3, reacquire the upper and lower stereoscopic marker images and perform preprocessing, and continue to execute steps 4 to 6; in this cycle, adjust the position of the external fixator until the position error does not exceed the position error threshold, and the position adjustment of the external fixator is completed.

2. The external fixator posture adjustment system according to claim 1, characterized in that: Each type of candidate fiducial marker is constrained by size parameters, feature point distribution density parameters, and anti-interference threshold parameters; the size parameters of the candidate fiducial markers corresponding to the coarse adjustment reset stage are greater than 5 cm, and the feature point distribution density parameters are less than 0.5 points / cm 2 The size parameter of the candidate fiducial marker corresponding to the fine-tuning correction stage is greater than 2 cm and less than 5 cm, and the feature point distribution density parameter is greater than 1 point / cm 2 ; Maintain the size parameter of the candidate reference marker corresponding to the fixed stage is less than 2cm, and the anti-interference threshold parameter is greater than 0.

8.

3. The external fixator posture adjustment system according to claim 1, characterized in that: In the fourth step, the feature points of the matching fiducial markers are used as candidate matching feature points, the 2D coordinates of the candidate matching feature points in the left and right camera images are extracted, and the 3D coordinates of the candidate matching feature points are calculated by triangulation method; the 2D coordinates and 3D coordinates of the candidate matching feature points in the left or right camera image are combined into 2D-3D point pairs to obtain a 2D-3D point pair set of candidate matching feature points; the RANSAC algorithm is used to screen the candidate matching feature points to obtain matching feature points, and then a 2D-3D point pair set of matching feature points is obtained.

4. The external fixator posture adjustment system according to claim 1, characterized in that: In the fifth step, the EPnP algorithm is used to solve the binocular camera pose based on the 2D-3D point pair set of matched feature points and the binocular camera internal parameters, and the pose of the binocular camera in the upper and lower stereo marker coordinate systems is obtained. The pose of the binocular camera in the upper and lower stereo marker coordinate systems is then inversely transformed to obtain the pose of the upper and lower stereo markers in the binocular camera coordinate system. The pose of the upper stereo marker in the binocular camera coordinate system is converted to the lower stereo marker coordinate system to obtain the relative pose between the upper and lower stereo markers, that is, the relative pose between the upper and lower fixing rings of the external fixator.

5. The external fixator posture adjustment system according to claim 1, characterized in that: In the sixth step, the coordinates of the connection points between each electric connecting rod and the upper fixing ring in the upper 3D marker coordinate system are converted to the lower 3D marker coordinate system based on the target position between the upper and lower fixing rings of the external fixator. The target length of each electric connecting rod is calculated based on the coordinates of the connection points between each electric connecting rod and the upper and lower fixing rings in the lower 3D marker coordinate system. Similarly, the current length of each electric connecting rod is calculated according to the relative position between the upper and lower fixing rings of the external fixator, and the difference between the target length and the current length of the electric connecting rod is the adjustment amount.

6. The external fixator posture adjustment system according to claim 1, characterized in that: The real-time scene parameter vector is composed of the distance between the stereo marker and the binocular camera, the incident angle, the ambient light intensity and the occlusion rate of the stereo marker.

7. The external fixator position adjustment system according to any one of claims 1 to 6, characterized in that: The preprocessing includes grayscale, binarization, morphological operation and edge detection.

8. The external fixator posture adjustment system according to claim 7, characterized in that: The three-dimensional marker adopts a hollow regular icosahedron, one surface of which is used as a mounting surface, and the remaining 19 surfaces are all provided with electronic paper screens, and the geometric center of the electronic paper screen coincides with the center of gravity of the surface.

Images