Needle body deformation recognition and needle insertion depth solving method, system, device and medium

Abstract

The specification discloses a needle body deformation recognition and needle insertion depth calculation method, system, device and medium, which synchronously acquires multi-camera images and needle side inertia data, reconstructs a main active light-emitting identification, a visible center line of the needle body and a skin reference surface, establishes a space-time transformation relationship of vision, inertia and a needle body coordinate system and performs fusion estimation, reconstructs an invisible needle segment and a needle tip position according to the visible center line, a needle handle pose, a skin intersection and a needle body length constraint, calculates a deformation pseudo-displacement caused by needle body bending, corrects the needle tip position, and outputs a vertical needle insertion depth and a confidence with a directed distance of the corrected needle tip relative to the skin reference surface, and the embodiment of the specification can reduce calculation errors caused by shielding, inertia drift and rigid body assumption, and is suitable for needle action quantification, teaching evaluation, experimental research and device control.

Description

Technical Field

[0001] This invention relates to the fields of digital measurement of acupuncture techniques, machine vision, inertial sensor data fusion, and deformation reconstruction of flexible and slender instruments. Specifically, it relates to a method, system, device, and medium for needle deformation recognition and needle insertion depth calculation based on visual-inertial fusion. Background Technology

[0002] During acupuncture, the acupuncture needle passes through layers of skin, subcutaneous tissue, fascia, and muscle, encountering axial resistance, lateral constraints, friction, and tissue traction. Due to the small diameter and large length-to-diameter ratio of the acupuncture needle, the needle body may experience slight bending. If the acupuncture needle is considered an absolutely rigid straight rod, and the displacement of the needle handle or exposed reference point is directly taken as the needle tip displacement, the changes and shifts in the reference point's posture caused by the needle body's bending will be mistakenly included in the needle insertion displacement, resulting in spurious deformation displacement. Multi-camera visual tracking can provide absolute spatial observation without cumulative drift, but it is prone to short-term observation loss when the operator's hand obstructs the view, there is specular reflection, the marking is outside the field of view, or the needle contrast is reduced. Inertial measurement units have the advantages of high sampling frequency and independence from external line of sight, but the position and attitude obtained by integrating acceleration and angular velocity have zero bias accumulation. It is difficult to continuously and accurately obtain needle tip motion under conditions of rapid lifting, twisting, and partial occlusion using only visual or inertial information. Current instrument navigation technologies typically approximate the needle or catheter as a rigid body, or only track the marker at the needle handle end. They fail to simultaneously utilize the three-dimensional shape of the visible centerline of the needle body, the inertial attitude of the needle handle, and the dynamic position of the skin reference surface to continuously reconstruct the invisible needle segment after it enters the tissue. Especially when the needle body bends, the displacement of the needle handle reference point is not the same as the actual displacement of the needle tip, resulting in errors in needle insertion depth. Therefore, based on the existing multi-camera vision and inertial measurement, it is necessary to establish a unified spatiotemporal reference, combine the needle handle reference point, the needle body centerline and the skin reference surface, reconstruct the position of the invisible needle segment and needle tip, and compensate for the pseudo-displacement caused by the deformation of the needle body. Summary of the Invention

[0003] This specification provides a method, system, device, and medium for needle deformation recognition and needle insertion depth calculation, which solves the problems of discontinuity and inaccuracy in needle tip position and needle insertion depth calculation caused by visual occlusion, inertial drift, changes in skin reference surface, and the assumption of a rigid needle body. According to a first aspect of the embodiments of this specification, a method for needle deformation recognition and needle insertion depth calculation is provided, including: Simultaneously acquire multi-view images of acupuncture needles, needle handle reference positions, and skin reference areas collected by a multi-camera vision system, as well as acceleration, angular velocity, and attitude data output by an inertial measurement unit fixed at the needle handle or needle body reference position; Establish spatial transformation relationships between the visual global coordinate system, the inertial local coordinate system, and the needle coordinate system, and complete the time registration of visual data and inertial data based on a unified trigger signal or timestamp; The active luminescent visual identifier and the visible center line of the needle body are located from the multi-view images, and the visual three-dimensional coordinates of the needle handle reference point, the three-dimensional point set of the visible center line of the needle body, and the skin reference surface are obtained through multi-view geometric reconstruction. The position, velocity, attitude, zero inertial bias, and needle deformation parameters of the needle handle reference point are used as state variables to fuse and estimate the visual three-dimensional coordinates and inertial data. Based on the three-dimensional point set of the visible center line of the needle body, the position and orientation of the needle handle reference point, and the intersection of the needle body and the skin reference surface, the center line of the needle body is curve-fitted and the curvature distribution is obtained, thereby reconstructing the invisible part of the needle body and the position of the needle tip. The deformation pseudo-displacement is determined based on the difference between the reconstructed needle tip position and the reference needle tip position obtained when the needle body is regarded as a rigid straight rod. The needle tip position is corrected using the deformation pseudo-displacement. The vertical needle insertion depth is output based on the directional distance of the corrected needle tip position relative to the skin reference surface. Optionally, in the step of simultaneously acquiring multi-view images of the acupuncture needle, needle handle reference position, and skin reference area collected by the multi-camera vision system, The multi-camera vision system includes at least three high-speed industrial cameras, and each high-speed industrial camera synchronizes its images through the same hardware trigger signal, unified clock or precise time protocol; the active light-emitting visual identifier includes at least three light-emitting units with non-collinear spatial positional relationship to obtain the three-dimensional position and orientation of the needle holder reference position. Optionally, in the step of locating the active luminescent visual marker and the visible center line of the needle body from the multi-view images, and obtaining the visual three-dimensional coordinates of the needle handle reference point, the three-dimensional point set of the visible center line of the needle body, and the skin reference surface through multi-view geometric reconstruction, The skin reference surface is reconstructed by at least three skin reference points set on the skin surface or a fixed support surface adjacent to the skin; when the number of skin reference points is greater than three, a robust plane fitting is used to obtain the skin reference surface, and the skin reference surface is updated according to the movement of the skin reference points in continuous images. Optionally, in the step of establishing the spatial transformation relationship between the visual global coordinate system, the inertial local coordinate system, and the needle coordinate system, The spatial transformation relationship is determined by synchronous observation of the active luminous visual marker, the inertial measurement unit, and the needle axis under the calibrated attitude. It includes the rotation matrix and translation vector from the inertial local coordinate system to the visual global coordinate system, as well as the installation rotation matrix and installation translation vector from the needle coordinate system to the inertial local coordinate system. Optionally, in the step of fusing and estimating visual three-dimensional coordinates and inertial data using the position, velocity, attitude, zero inertial bias, and needle deformation parameters of the needle handle reference point as state variables, The fusion estimation employs extended Kalman filtering, unscented Kalman filtering, or factor graph optimization. When visual observations are occluded or their quality falls below a preset threshold, the weight of visual observations is reduced, and short-term state prediction is performed using inertial measurement data and the previous time-hand volume deformation parameters. After visual observations are restored, the state during the occlusion period is smoothly corrected using a fixed-duration sliding window. Optionally, in the step of performing curve fitting on the needle centerline and obtaining the curvature distribution based on the three-dimensional point set of the visible centerline of the needle body, the position and orientation of the needle handle reference point, and the intersection of the needle body and the skin reference surface, The three-dimensional point set of the visible center line of the needle body is obtained through sub-pixel positioning of the needle body edge, epipolar constraint matching between cameras, and triangulation. The center line of the needle body is represented by a cubic spline curve, a B-spline curve, or a discrete elastic rod model, and the curvature distribution is calculated based on the first and second derivatives of the curve. Optionally, in the steps of determining the deformation pseudo-displacement based on the difference between the reconstructed needle tip position and the reference needle tip position obtained when the needle body is considered as a rigid straight rod, correcting the needle tip position using the deformation pseudo-displacement, and outputting the vertical needle insertion depth based on the directional distance of the corrected needle tip position relative to the skin reference surface, When reconstructing the invisible part of the needle body, the proximal position boundary and tangential boundary of the needle body centerline are determined by the needle handle reference point and needle handle posture. The boundary position for entering the tissue is determined by the intersection of the needle body and the skin reference surface. The needle tip position is determined by combining the effective length of the acupuncture needle, the continuity of the centerline, and the bending energy constraint. The vertical needle insertion depth is the projection distance of the corrected needle tip position along the normal of the skin reference surface. At the same time, the depth confidence is output by the visual reprojection error, inertial covariance, and curve fitting residual. According to a second aspect of the embodiments of this specification, a needle deformation recognition and needle insertion depth calculation system is provided, including a multi-camera vision acquisition module, an inertial data acquisition module, a skin reference surface acquisition module, a synchronization module, a coordinate calibration module, a fusion filtering module, a deformation recognition module, a deformation compensation module, and a depth calculation module; the multi-camera vision acquisition module is used to output multi-view images of an actively emitting visual marker, the visible center line of the needle, and the skin reference area; the inertial data acquisition module is used to output acceleration, angular velocity, and attitude data; the remaining modules are configured as the needle deformation recognition and needle insertion depth calculation method described above. According to a third aspect of the embodiments of this specification, a computing device is provided, including a storage device and a processor, wherein the storage device is used to store a computer program, and the processor runs the computer program to cause the computing device to perform the steps of the needle deformation recognition and needle insertion depth calculation method. According to a fourth aspect of the embodiments of this specification, a storage medium is provided that stores a computer program used in the computing device, which, when executed by a processor, implements the steps of the needle deformation recognition and needle insertion depth calculation method. This specification provides an embodiment of a method, system, device, and medium for needle deformation recognition and needle insertion depth calculation. Compared with the prior art, the embodiments of this specification have the following advantages: First, the complementary relationship between visual absolute position and inertial high-frequency state improves continuity under occlusion and rapid movement conditions. Second, the visible centerline and curvature distribution of the needle body are incorporated into the fusion solution, avoiding reliance solely on the needle handle reference point. Third, the invisible needle segment is reconstructed through needle handle pose, skin intersection, effective length, and bending constraints to achieve needle tip position estimation. Fourth, deformation pseudo-displacement is explicitly calculated and removed, separating the needle insertion depth from the needle body bending. Fifth, using a dynamic skin reference surface as the depth benchmark reduces the impact of skin surface motion on depth calculation. Sixth, depth confidence can be output, facilitating teaching evaluation, experimental analysis, and equipment safety control. Attached Figure Description To more clearly illustrate the technical solutions in the embodiments or related technologies of this specification, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a flowchart illustrating a method for needle deformation recognition and needle insertion depth calculation provided in one embodiment of this specification. Figure 2 This is a schematic diagram of the structure of a needle deformation recognition and needle insertion depth calculation system provided in one embodiment of this specification; Figure 3 This diagram illustrates the calibration relationships of the visual global coordinate system, inertial local coordinate system, needle coordinate system, and skin reference surface, as provided in one embodiment of this specification. Figure 4 A schematic diagram of the time synchronization and fusion estimation structure for visual and inertial data provided in one embodiment of this specification; Figure 5 This diagram illustrates the extraction of the visible centerline of the needle body, curvature calculation, and reconstruction of the invisible needle segment, as provided in one embodiment of this specification. Figure 6This is a schematic diagram of the geometric relationship between deformation pseudo-displacement compensation and vertical needle depth calculation provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the functional modules of the needle deformation recognition and needle insertion depth calculation system provided in one embodiment of this specification. Figure 8 This is a schematic diagram of the structure of a computing device provided in one embodiment of this specification; Figure 9 This is a schematic diagram of the structure of a storage medium provided in one embodiment of this specification; Figure labeling: 1-High-speed industrial camera; 2-Inertial measurement unit; 3-Actively emitting visual marker; 4-Acupuncture needle; 5-Skin or skin reference surface; 6-Needle deformation center line and curvature; 7-Acquisition motherboard; 8-Skin reference point; 12-Synchronization module; 13-Coordinate calibration module; 14-Fusion filtering module; 15-Deformation recognition module; 16-Deformation compensation module; 17-Depth calculation module. Detailed Implementation The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this specification are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. This specification discloses a method, system, device, and medium for needle deformation recognition and needle insertion depth calculation, which will be described in detail below. Figure 1 This is a flowchart illustrating a method for needle deformation recognition and needle insertion depth calculation provided in one embodiment of this specification. Figure 1 As shown, a method for needle deformation recognition and needle insertion depth calculation includes: S110: Simultaneously acquire multi-view images of the acupuncture needle, needle handle reference position, and skin reference area collected by the multi-camera vision system, as well as acceleration, angular velocity, and attitude data output by the inertial measurement unit fixed at the needle handle or needle body reference position. The multi-camera vision system includes at least three high-speed industrial cameras, and each high-speed industrial camera synchronizes its images through the same hardware trigger signal, unified clock or precise time protocol; the active light-emitting visual identifier includes at least three light-emitting units with non-collinear spatial positional relationship to obtain the three-dimensional position and orientation of the needle holder reference position. S120. Establish the spatial transformation relationship between the visual global coordinate system, the inertial local coordinate system, and the needle body coordinate system, and complete the time registration of visual data and inertial data based on a unified trigger signal or timestamp. The spatial transformation relationship is determined by synchronous observation of the active luminous visual marker, the inertial measurement unit, and the needle axis under the calibrated attitude. It includes the rotation matrix and translation vector from the inertial local coordinate system to the visual global coordinate system, as well as the installation rotation matrix and installation translation vector from the needle coordinate system to the inertial local coordinate system. S130. Locate the active luminescent visual marker and the visible center line of the needle body from the multi-view image, and obtain the visual three-dimensional coordinates of the needle handle reference point, the three-dimensional point set of the visible center line of the needle body, and the skin reference surface through multi-view geometric reconstruction. The skin reference surface is reconstructed by at least three skin reference points set on the skin surface or a fixed support surface adjacent to the skin; when the number of skin reference points is greater than three, a robust plane fitting is used to obtain the skin reference surface, and the skin reference surface is updated according to the movement of the skin reference points in continuous images. S140. Using the position, velocity, attitude, inertial zero bias, and needle deformation parameters of the needle handle reference point as state variables, the visual three-dimensional coordinates and inertial data are fused and estimated. The fusion estimation employs extended Kalman filtering, unscented Kalman filtering, or factor graph optimization. When visual observations are occluded or their quality falls below a preset threshold, the weight of visual observations is reduced, and short-term state prediction is performed using inertial measurement data and the previous time-hand volume deformation parameters. After visual observations are restored, the state during the occlusion period is smoothly corrected using a fixed-duration sliding window. S150. Based on the three-dimensional point set of the visible center line of the needle body, the position and orientation of the needle handle reference point, and the intersection of the needle body and the skin reference surface, the center line of the needle body is curve-fitted and the curvature distribution is obtained, thereby reconstructing the invisible part of the needle body and the position of the needle tip. The three-dimensional point set of the visible center line of the needle body is obtained through sub-pixel positioning of the needle body edge, epipolar constraint matching between cameras, and triangulation. The center line of the needle body is represented by a cubic spline curve, a B-spline curve, or a discrete elastic rod model, and the curvature distribution is calculated based on the first and second derivatives of the curve. S160. Determine the deformation pseudo-displacement based on the difference between the reconstructed needle tip position and the reference needle tip position obtained when the needle body is regarded as a rigid straight rod, and use the deformation pseudo-displacement to correct the needle tip position. Output the vertical needle insertion depth based on the directional distance of the corrected needle tip position relative to the skin reference surface. When reconstructing the invisible part of the needle body, the proximal position boundary and tangential boundary of the needle body centerline are determined by the needle handle reference point and needle handle posture. The boundary position for entering the tissue is determined by the intersection of the needle body and the skin reference surface. The needle tip position is determined by combining the effective length of the acupuncture needle, the continuity of the centerline, and the bending energy constraint. The vertical needle insertion depth is the projection distance of the corrected needle tip position along the normal of the skin reference surface. At the same time, the depth confidence is output by the visual reprojection error, inertial covariance, and curve fitting residual. A needle deformation recognition and insertion depth calculation system includes a multi-camera vision acquisition module, an inertial data acquisition module, a skin reference surface acquisition module, a synchronization module, a coordinate calibration module, a fusion filtering module, a deformation recognition module, a deformation compensation module, and a depth calculation module. The multi-camera vision acquisition module is used to output multi-view images of an actively emitting visual marker, the visible center line of the needle, and the skin reference area. The inertial data acquisition module is used to output acceleration, angular velocity, and attitude data. The remaining modules are configured as described in the needle deformation recognition and insertion depth calculation method. Figure 2 This is a schematic diagram of the structure of a needle deformation recognition and needle insertion depth calculation system provided in one embodiment of this specification. Figure 2 As shown, the system includes multiple high-speed industrial cameras 1, an inertial measurement unit 2, an active light-emitting visual marker 3, an acupuncture needle 4, skin or a skin reference surface 5, a needle deformation center line and curvature 6, and a data acquisition motherboard 7. The multiple high-speed industrial cameras 1 cover the needle handle, the visible portion of the needle body, and the skin reference area from different perspectives. The inertial measurement unit 2 and the active light-emitting visual marker 3 are positioned on or near the needle handle at the needle body reference position, and are fixedly mounted to the acupuncture needle 4. The number of high-speed industrial cameras 1 is preferably 3 to 6, with a frame rate of 100 to 1000 frames per second; the sampling rate of the inertial measurement unit 2 can be 200 to 2000 Hz. The acquisition motherboard 7 generates a hardware trigger signal or receives a unified clock signal and assigns timestamps to the camera frames and inertial samples. The active luminous visual marker 3 can use three or more luminous units with a known spatial layout, or it can use luminous strips or luminous dot matrices with directional encoding. Skin reference points 8 are set around the target needle insertion area. Skin reference points 8 can be identifiable printed marks, reflective marks, actively luminescent points, or stable feature points automatically selected based on skin texture. At least three non-collinear skin reference points 8 are used to establish the skin reference surface 5; when the number of reference points is greater than three, random sampling consistency or robust least squares is used for plane fitting. Figure 3 This diagram illustrates the calibration relationships of the visual global coordinate system, inertial local coordinate system, needle coordinate system, and skin reference surface, as provided in one embodiment of this specification. Figure 3 As shown, a global visual coordinate system is established. Inertial local coordinate system and needle coordinate system First, the intrinsic parameters, distortion parameters, and extrinsic parameters between the high-speed industrial cameras 1 are calibrated using a calibration board. Then, the acupuncture needles 4 are placed in at least two different and stably identifiable calibration postures, while simultaneously acquiring the three-dimensional position of the active light-emitting visual marker 3, the posture of the inertial measurement unit 2, and the direction of the needle axis. When transforming a point in the inertial local coordinate system to the visual global coordinate system, a rotation matrix and a translation vector are used: When transforming a point in the needle coordinate system to the inertial local coordinate system, the installation rotation matrix and installation translation vector are used: The above rotation matrix and translation vector are obtained by minimizing the visual reprojection error and posture residual through multiple sets of calibration postures. Skin reference plane 5 is represented by a point on the reference plane and a unit normal vector; its plane equation is: The normal direction is determined during the initialization phase by observing the direction with the camera, the initial axis of the acupuncture needle, or manually specifying it; during continuous acquisition, the reference point position and unit normal vector are updated based on the skin reference point 8. Figure 4 This is a schematic diagram illustrating the time synchronization and fusion estimation structure of visual and inertial data provided in one embodiment of this specification. Figure 4 As shown, camera images and inertial data each carry a hardware trigger number or a timestamp. Synchronization module 12 interpolates inertial samples to the image time, or uses image observations as asynchronous update inputs to fusion filtering module 14. The maximum time difference after synchronization can be set according to the action frequency, for example, no more than 2 milliseconds or no more than one inertial sampling period. The state vector used by the fusion filtering module 14 includes at least the needle handle reference point position, velocity, attitude quaternion, accelerometer bias, gyroscope bias, and needle body deformation parameters. The needle handle reference point position, velocity, and attitude are used to describe the needle handle motion; accelerometer zero bias and gyroscope zero bias are used to suppress inertial integral drift; and needle body deformation parameters are used to characterize the centerline shape. Inertial measurement is used for state prediction, the three-dimensional coordinates and attitude of the active luminous visual identifier 3 are used for position and attitude updates, and the needle body centerline shape is used for deformation parameter updates. When the active luminous visual marker 3 is obscured by a hand or the visual reprojection error exceeds a threshold, the fusion filtering module 14 reduces the weight of visual observation and performs short-term prediction using inertial data, the velocity from the previous moment, and the deformation parameters from the previous moment. After visual recovery, the position, attitude, inertial zero bias, and deformation parameters are jointly optimized within a sliding window composed of the most recent frames to avoid abrupt state jumps. Figure 5 This diagram illustrates the extraction of the visible center line of the needle body, curvature calculation, and reconstruction of the invisible needle segment, as provided in one embodiment of this specification. Figure 5 As shown, background suppression, edge detection, and sub-pixel center localization are performed on the visible area of ​​the needle in each camera image to obtain the two-dimensional centerline of the needle in each viewpoint. A correspondence is established based on the epipolar constraints between cameras, and triangulation is performed to obtain the three-dimensional point set of the visible centerline of the needle. For reflections or localized defects, the continuity of the centerline and the needle shank orientation can be used to supplement the corresponding points. The centerline of the needle body is represented as a three-dimensional curve with arc length as the parameter, and its curvature is calculated according to the following formula: Curve fitting can be performed using cubic splines, B-splines, Bezier curves, piecewise circular arcs, or discrete elastic rod models. When the needle segment entering the tissue is invisible, the position of the needle handle reference point is used as the proximal position boundary, the needle handle attitude determined by the inertial measurement unit 2 and the active luminous visual marker 3 is used as the proximal tangential boundary, and the intersection of the needle body centerline and the skin reference surface 5 is used as the tissue entry boundary. Constraints are also added regarding the effective length of the acupuncture needle, curve position continuity, tangential continuity, and minimum bending energy. By solving the constraint curve model, the positions of the invisible needle segment and needle tip are obtained. . Figure 6 This is a schematic diagram illustrating the geometric relationship between deformation pseudo-displacement compensation and vertical needle depth calculation provided in one embodiment of this specification. Figure 6 As shown, under the rigid body assumption, the reference needle tip position is obtained using the needle handle reference point, the unit vector of the needle body axis, and the effective length of the acupuncture needle: The needle tip position is obtained after deformation reconstruction. The difference between the reference needle tip position and the deformation-reconstructed needle tip position is defined as the deformation pseudo-displacement. The deformation-compensated needle tip position is obtained by correcting the rigid body reference needle tip position using deformation pseudo-displacement: The vertical needle insertion depth is the directional distance between the deformed needle tip position and the skin reference surface 5. When the actual insertion length along the centerline of the needle body is required, the arc length of the centerline between the skin intersection and the needle tip can be integrated. Depth confidence is determined based on camera reprojection error, number of visual observations, inertial state covariance, centerline fitting residual, and skin reference surface fitting residual. When the confidence is below a threshold, a low confidence flag can be output or the use of depth for automatic control can be paused. Figure 7 This is a schematic diagram of the functional modules of a needle deformation recognition and needle insertion depth calculation system provided in one embodiment of this specification. Figure 7 As shown, the system includes a multi-camera vision acquisition module, an inertial data acquisition module, a skin reference surface acquisition module, a synchronization module 12, a coordinate calibration module 13, a fusion filtering module 14, a deformation recognition module 15, a deformation compensation module 16, and a depth calculation module 17. Each module can be implemented in the acquisition motherboard 7, an industrial control computer, an edge computing device, or a server. The multi-camera vision acquisition module outputs multi-view images; the inertial data acquisition module outputs acceleration, angular velocity, and attitude; the skin reference surface acquisition module outputs... and Synchronization module 12 completes time correspondence; coordinate calibration module 13 completes coordinate transformation; fusion filtering module 14 outputs the fused pose of the needle handle reference point; deformation recognition module 15 outputs the needle body centerline and curvature; deformation compensation module 16 outputs deformation pseudo-displacement and corrected needle tip position; depth calculation module 17 outputs vertical needle insertion depth, centerline insertion length and confidence level. The active luminescent visual identifier 3 can be replaced with a reflective identifier, a coded pattern, or a micro-feature on the needle surface; the skin reference point 8 can be a marker point or a natural texture; the fusion algorithm can employ extended Kalman filtering, unscented Kalman filtering, particle filtering, or factor graph optimization; the deformation model can be calibrated according to the acupuncture needle material and diameter. Any technical solution that achieves the same purpose by using equivalent spatiotemporal calibration, deformation reconstruction, and compensation methods based on the original vision-inertial fusion, needle deformation recognition, and needle insertion depth calculation methods falls under the technical concept of this invention. Figure 8 This is a schematic diagram of the structure of a computing device provided in one embodiment of this specification. Figure 8As shown, a computing device 800 includes a storage device 810 and a processor 820. The storage device 810 stores a computer program, and the processor 820 runs the computer program to enable the computing device 800 to perform the steps of the needle deformation recognition and needle insertion depth calculation method. Figure 9 This is a schematic diagram of the structure of a storage medium provided in one embodiment of this specification. For example... Figure 9 As shown, a storage medium 900 stores a computer program 910 used in the computing device. When the computer program 910 is executed by a processor, it implements the steps of the needle deformation recognition and needle insertion depth calculation method. In summary, the embodiments of this specification provide a method, system, device, and medium for needle deformation recognition and needle insertion depth calculation. The method sequentially completes data acquisition, spatiotemporal calibration, visual observation reconstruction, visual-inertial fusion, needle deformation reconstruction, deformation pseudo-displacement compensation, and needle insertion depth output. It effectively solves the problems of discontinuity and inaccuracy in needle tip position and needle insertion depth calculation caused by visual occlusion, inertial drift, changes in skin reference surface, and the assumption of a rigid needle body. The data output of the embodiments in this specification is used for acupuncture action quantification, teaching evaluation, experimental research, robot reproduction or equipment feedback control, and does not directly output disease diagnosis conclusions or treatment plans. Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention. Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for needle deformation recognition and needle insertion depth calculation, characterized in that, include: Simultaneously acquire multi-view images of acupuncture needles, needle handle reference positions, and skin reference areas collected by a multi-camera vision system, as well as acceleration, angular velocity, and attitude data output by an inertial measurement unit fixed at the needle handle or needle body reference position; Establish spatial transformation relationships between the visual global coordinate system, the inertial local coordinate system, and the needle coordinate system, and complete the time registration of visual data and inertial data based on a unified trigger signal or timestamp; The active luminescent visual identifier and the visible center line of the needle body are located from the multi-view images, and the visual three-dimensional coordinates of the needle handle reference point, the three-dimensional point set of the visible center line of the needle body, and the skin reference surface are obtained through multi-view geometric reconstruction. The position, velocity, attitude, zero inertial bias, and needle deformation parameters of the needle handle reference point are used as state variables to fuse and estimate the visual three-dimensional coordinates and inertial data. Based on the three-dimensional point set of the visible center line of the needle body, the position and orientation of the needle handle reference point, and the intersection of the needle body and the skin reference surface, the center line of the needle body is curve-fitted and the curvature distribution is obtained, thereby reconstructing the invisible part of the needle body and the position of the needle tip. The deformation pseudo-displacement is determined based on the difference between the reconstructed needle tip position and the reference needle tip position obtained when the needle body is regarded as a rigid straight rod. The needle tip position is corrected using the deformation pseudo-displacement. The vertical needle insertion depth is output based on the directional distance of the corrected needle tip position relative to the skin reference surface.

2. The method according to claim 1, characterized in that, In the step of simultaneously acquiring multi-view images of the acupuncture needle, needle handle reference position, and skin reference area collected by the multi-camera vision system, The multi-camera vision system includes at least three high-speed industrial cameras, and each high-speed industrial camera synchronizes its images through the same hardware trigger signal, unified clock or precise time protocol. The active luminescent visual identifier includes at least three luminescent units with non-collinear spatial relationships to obtain the three-dimensional position and orientation of the needle handle reference position.

3. The method according to claim 1, characterized in that, In the steps of locating the active luminescent visual marker and the visible center line of the needle body from the multi-view images, and obtaining the visual three-dimensional coordinates of the needle handle reference point, the three-dimensional point set of the visible center line of the needle body, and the skin reference surface through multi-view geometric reconstruction,... The skin reference surface is reconstructed by at least three skin reference points set on the skin surface or a fixed support surface adjacent to the skin; when the number of skin reference points is greater than three, a robust plane fitting is used to obtain the skin reference surface, and the skin reference surface is updated according to the movement of the skin reference points in continuous images.

4. The method according to claim 1, characterized in that, In the step of establishing the spatial transformation relationship between the visual global coordinate system, the inertial local coordinate system, and the needle body coordinate system, The spatial transformation relationship is determined by synchronous observation of the active luminous visual marker, the inertial measurement unit, and the needle axis under the calibrated attitude. It includes the rotation matrix and translation vector from the inertial local coordinate system to the visual global coordinate system, as well as the installation rotation matrix and installation translation vector from the needle coordinate system to the inertial local coordinate system.

5. The method according to claim 1, characterized in that, In the step of fusing and estimating visual three-dimensional coordinates and inertial data using the position, velocity, attitude, zero inertial bias, and needle deformation parameters of the needle handle reference point as state variables... The fusion estimation employs extended Kalman filtering, unscented Kalman filtering, or factor graph optimization. When visual observations are occluded or their quality is below a preset threshold, the weight of visual observations is reduced, and short-term state prediction is performed using inertial measurement data and the previous time-hand volume deformation parameters. After visual observation is restored, the state during the occlusion period is smoothly corrected by using a sliding window of fixed duration.

6. The method according to claim 1, characterized in that, In the step of curve fitting and obtaining the curvature distribution of the needle centerline based on the three-dimensional point set of the visible centerline of the needle body, the position and orientation of the needle handle reference point, and the intersection of the needle body and the skin reference surface, The three-dimensional point set of the visible center line of the needle body is obtained through sub-pixel positioning of the needle body edge, epipolar constraint matching between cameras, and triangulation. The center line of the needle body is represented by a cubic spline curve, a B-spline curve, or a discrete elastic rod model, and the curvature distribution is calculated based on the first and second derivatives of the curve.

7. The method according to claim 1, characterized in that, In the steps described above, the deformation pseudo-displacement is determined based on the difference between the reconstructed needle tip position and the reference needle tip position obtained when the needle body is considered as a rigid straight rod; the needle tip position is corrected using the deformation pseudo-displacement; and the vertical needle insertion depth is output based on the directional distance of the corrected needle tip position relative to the skin reference surface. When reconstructing the invisible part of the needle body, the proximal position boundary and tangential boundary of the needle body centerline are determined by the needle handle reference point and needle handle posture. The boundary position for entering the tissue is determined by the intersection of the needle body and the skin reference surface. The needle tip position is determined by combining the effective length of the acupuncture needle, the continuity of the centerline, and the bending energy constraint. The vertical needle insertion depth is the projection distance of the corrected needle tip position along the normal of the skin reference surface. At the same time, the depth confidence is output by the visual reprojection error, inertial covariance, and curve fitting residual.

8. A needle deformation recognition and needle insertion depth calculation system, characterized in that, The system includes a multi-camera vision acquisition module, an inertial data acquisition module, a skin reference surface acquisition module, a synchronization module, a coordinate calibration module, a fusion filtering module, a deformation recognition module, a deformation compensation module, and a depth calculation module. The multi-camera vision acquisition module is used to output multi-view images of an active luminous visual marker, the visible center line of the needle, and the skin reference area. The inertial data acquisition module is used to output acceleration, angular velocity, and attitude data. The remaining modules are configured to perform the method described in any one of claims 1 to 7.

9. A computing device, characterized in that, The device includes a storage device and a processor, the storage device being used to store a computer program, and the processor running the computer program to cause the computing device to perform the steps of the method according to any one of claims 1-7.

10. A storage medium, characterized in that, It stores a computer program used in the computing device of claim 9, which, when executed by a processor, implements the steps of the method of any one of claims 1-7.

Images