Mechanical arm puncture method and system based on optical tracking

Abstract

The invention discloses a mechanical arm puncture method and system based on optical tracking, and the method comprises the steps: collecting high-dynamic light field data, and calculating a surface deformation field; surface motion feature vectors are extracted, and a dynamic breathing phase model is constructed; performing model training on the dynamic breathing phase model by adopting an adversarial training strategy, and outputting a predicted phase; constructing a dynamic security domain according to the predicted phase; searching an optimal path in the dynamic safety domain by adopting a self-adaptive algorithm, and outputting a joint angle sequence of the mechanical arm; measuring the strain distribution of the needle body in real time, calculating the stress of the needle tip, constructing a hybrid observation model, and outputting a force feedback signal. The system comprises an optical data acquisition module, a dynamic breathing phase model construction module, a prediction phase output module, a dynamic security domain construction module, an optimal path search module, a force feedback module and a correction module. The defects of a traditional puncture method in the aspects of precision, adaptability and safety can be effectively overcome, and the success rate of puncture is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arm puncture, and particularly to a robotic arm puncture method and system based on optical tracking. Background Art

[0002] With the development of artificial intelligence, robotics, and optical tracking technologies, computer-aided puncture systems have been widely used in precision operation scenarios, such as intelligent manufacturing, experimental automation, and micro-operation tasks. In recent years, the research focus of robotic arm puncture technology has mainly been concentrated on aspects such as precise positioning, motion compensation, force feedback control, and safety improvement. Traditional puncture methods rely on the preset trajectory of the robotic arm, but it is difficult to ensure high precision and high stability in a dynamic environment. The optical tracking technology combined with a real-time calculation model provides a new solution for intelligent puncture in complex scenarios.

[0003] Although important progress has been made in optical tracking and robot control technologies, the existing puncture systems still face the following technical challenges:

[0004] (1) The dynamic changes on the target surface affect the positioning accuracy. Traditional puncture methods based on static calibration are difficult to adapt to the dynamic deformation of the target surface. For example, due to environmental factors such as small displacements or periodic movements caused by breathing, it is easy to cause positioning errors.

[0005] (2) The motion prediction is unstable and lacks real-time performance. Existing motion prediction models based on classical signal processing methods have poor adaptability to complex environments and are difficult to cope with non-linear or highly dynamic motion patterns.

[0006] (3) The path planning and safety are insufficient. Existing methods usually adopt preset path planning, which is difficult to adapt to sudden environmental changes, resulting in a decline in the stability of the puncture operation.

[0007] (4) The force feedback control accuracy is limited. Most existing force feedback control methods rely on simple force sensors, which are difficult to accurately quantify the force on the puncture needle tip and affect the precise control of the puncture process. Summary of the Invention

[0008] In view of this, the purpose of the embodiments of the present invention is to provide a robotic arm puncture method and system based on optical tracking, which can effectively overcome the deficiencies of traditional puncture methods in terms of accuracy, adaptability, and safety, improve the success rate of puncture, have high precision, be safe and reliable, and have a high degree of automation.

[0009] The embodiments of the present invention are implemented as follows:

[0010] A robotic arm puncture method based on optical tracking, which includes:

[0011] Collect high-dynamic light field data through a high-frequency adjustable multi-spectral ring light source installed at the end of the robotic arm, and calculate the surface deformation field.

[0012] Extract the surface motion feature vector, use the surface motion feature vector as the input of the temporal convolutional network, and construct a dynamic breathing phase model for predicting the breathing phase within the next Δt time.

[0013] Adopt an adversarial training strategy to train the dynamic breathing phase model and output the predicted phase.

[0014] Construct a dynamic safety region according to the predicted phase.

[0015] Use an adaptive algorithm to search for the optimal path within the dynamic safety region and output the robotic arm joint angle sequence.

[0016] Measure the needle body strain distribution in real time, calculate the force on the needle tip, combine it with the surface deformation field, construct a hybrid observation model, and output a force feedback signal.

[0017] Feed the force feedback signal back to the adaptive algorithm to correct the boundary of the dynamic safety region in real time.

[0018] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the step of collecting high-dynamic light field data through a high-frequency adjustable multi-spectral ring light source installed at the end of the robotic arm and calculating the surface deformation field includes:

[0019] Install a high-frequency adjustable multi-spectral ring light source at the end of the robotic arm. The light source covers near-infrared wavelengths and visible wavelengths, and each group of light sources flashes in sequence according to a preset frequency.

[0020] Capture the reflected light signal on the tissue surface through a dual high-speed event camera.

[0021] Adopt pulse code modulation technology to bind the turn-on timing of light sources with different wavelengths to the exposure timing of the dual high-speed event camera, and separate and obtain the light field information of each channel through a demodulation algorithm.

[0022] Calculate the surface deformation field where λ is the light source wavelength, α λ is the tissue surface reflection coefficient, β λ is the tissue absorption coefficient, d(x, y) is the target surface depth value, and (x, y) is the pixel position in the image coordinate system.

[0023] Its technical effects are as follows: By means of a high-frequency adjustable multi-spectral light source that covers the near-infrared and visible light bands and adapts to the optical characteristics of different tissues; adopting two high-speed event cameras that can capture surface deformations with a time resolution of microseconds, improving the perception accuracy of fast movements; adopting pulse code modulation technology to bind the turn-on timing of light sources with different wavelengths to the exposure timing of the high-speed event cameras, enabling effective separation of multi-spectral information and avoiding the problem of optical signal aliasing of traditional RGB or single-wavelength light sources; using continuous light field information to calculate the deformation field, the motion state of dynamic tissues can be sensed in real time, enabling the puncture path planning to adapt to changes and improving the stability and safety of punctures.

[0024] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the extracting of the surface motion feature vector and using the surface motion feature vector as the input of the temporal convolutional network to construct a dynamic breathing phase model for predicting the breathing phase within the future Δt time includes:

[0025] Extracting a motion feature vector from the data of the surface deformation field where p i (t) is the motion trajectory of the feature point P i in three-dimensional space, represents the rate of deformation change in the time direction, represents the deformation gradient in the horizontal direction, represents the deformation gradient in the vertical direction.

[0026] Using the surface motion feature vector V(t) to construct the input matrix X = [V(t - T), V(t - T + 1),..., V(t)] of the temporal convolutional network, where T is the length of the historical time window.

[0027] Constructing a dynamic breathing phase model for predicting the breathing phase within the future Δt time, including a 1D convolutional layer, an extended causal convolutional layer, and a fully connected layer. The fully connected layer outputs the predicted breathing phase, and the prediction formula is

[0028] Its technical effects are as follows: Extracting motion feature vectors from the surface deformation field data reduces the dependence on additional sensors and avoids physical contact interference; The combination of the motion trajectories of feature points, the deformation change rate, the horizontal gradient, and the vertical gradient comprehensively describes the dynamic changes on the tissue surface, making the respiratory phase prediction more accurate; The combination of the 1D convolutional layer and the extended causal convolutional layer enables the network to effectively model long-term dependence relationships, capture subtle dynamic features in the respiratory cycle, and improve the prediction accuracy within the future Δt time. By using a temporal convolutional network, the extended causal convolutional layer can expand the receptive field through the dilation factor, enabling the model to capture long-term temporal dependence information without increasing the computational amount, realizing a farther respiratory phase prediction, with strong parallel computing capabilities. Compared with the RNN series methods, TCN can be calculated faster during the inference stage, ensuring the real-time performance of puncture.

[0029] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the model training of the dynamic respiratory phase model using the adversarial training strategy, and the output prediction phase includes:

[0030] Using the adversarial training strategy, the generator outputs a phase prediction, and the discriminator is used to distinguish between the real respiratory signal and the prediction result. The loss function uses Where, is the predicted value, is the real value, and λ is the weight coefficient.

[0031] Using the trained model to output the predicted phase

[0032] Its technical effects are as follows: Introducing the adversarial training strategy, through continuous optimization of the generator, the predicted respiratory phase is closer to the real signal. At the same time, the discriminator provides feedback, prompting the model to generate more reliable phase predictions; Since the loss function combines the prediction error and the adversarial loss, it ensures that the model can not only learn short-term accuracy but also optimize the global prediction stability; Through the adversarial training strategy, the model can learn different types of respiratory patterns during the training process and adapt to various individual differences, improving the adaptability to different patients. The discriminator continuously confronts the generator, making its output closer to the real respiratory data, and can make stable predictions even in the face of unseen respiratory patterns; The introduction of the discriminator enables the model to automatically learn and ignore noise factors, prompting the generator to focus on the key features that really affect the respiratory phase, improving the anti-interference ability of the prediction. Through the optimization of the adversarial loss, the adaptability of the model to complex environmental variables is enhanced, and the error caused by external environmental changes is reduced.

[0033] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, constructing the dynamic safety domain according to the predicted phase includes:

[0034] Set the phase safety radius according to the predicted phase

[0035] Construct a dynamic safety region in three-dimensional space where p is the spatial point coordinate, p target is the target puncture point.

[0036] Its technical effect is as follows: By predicting the phase to construct a dynamic safety region, it ensures that the puncture path is always adjusted around the safe area of respiratory movement, avoiding the influence of tissue offset caused by breathing on puncture accuracy; setting the phase safety radius enables the puncture point to always be within the feasible path range under different respiratory phases of the patient, ensuring that the puncture point fluctuates within the minimum error range and improving the targeting accuracy. The present invention adopts a method for constructing a dynamic safety region based on predicted phase, which can adaptively adjust the size and position of the safety region according to the individual respiratory pattern of the patient, making the puncture strategy more flexible. The setting of the dynamic safety region ensures that when the robotic arm performs puncture, it can always stay within the safe range, avoiding puncturing into non-target areas due to respiratory displacement; combined with the adaptive path optimization algorithm, if the robotic arm predicts that it is about to exceed the safety region, it automatically adjusts the path to ensure that the puncture is performed within the safe range, improving the surgical safety.

[0037] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the use of an adaptive algorithm to search for the optimal path within the dynamic safety region and output the robotic arm joint angle sequence includes:

[0038] Adopt an adaptive algorithm to initialize a rapidly exploring random tree within the dynamic safety region Ω(t), and set the starting point as the current position q of the end of the robotic arm start , and the end point as the target position p target .

[0039] Perform random sampling within the dynamic safety region Ω(t) to generate an initial path candidate set where the path point P i follows the safety constraint P i ∈Ω(t).

[0040] Optimize the initial path candidate set and select the optimal path P * .

[0041] Adopt the fifth-order polynomial interpolation method to generate the joint angle sequence θ * corresponding to the smooth trajectory of the optimal path P i (t) = a0 + a1t + a2t 2 + a3t 3 + a4t 4 + a5t 5, where a0, a1, a2, a3, a4, and a5 are the coefficients of the fifth-degree polynomial, and t is the current moment of the robotic arm's movement.

[0042] Its technical effects are as follows: The Rapidly-Exploring Random Tree (RRT) method can quickly generate a set of candidate paths within the dynamic safety domain and efficiently search for feasible solutions in complex environments, ensuring the computational efficiency of path planning. By using an adaptive algorithm to dynamically adjust the search strategy, it improves the convergence speed of the optimal path and ensures finding a feasible solution in a short time. Random sampling within the dynamic safety domain ensures that the generated path points are always within the feasible region, preventing the robotic arm from exceeding the safety boundary and enhancing the safety of the puncture process. The path optimization algorithm is used to eliminate unreasonable paths and, through smooth interpolation, avoid sudden movements, improving the feasibility and safety of the path. Quintic polynomial interpolation is used to ensure smooth transitions of the path in both time and space, reducing the impact on the robotic arm joints, optimizing the motion trajectory, and making the puncture action smoother. Through high-order polynomial interpolation, it can ensure the continuity of the joint angle's velocity, acceleration, and jerk, avoiding unnecessary jitter and vibration during the execution of the robotic arm.

[0043] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the real-time measurement of the needle body strain distribution, calculation of the tip force, and combination with the surface deformation field to construct a hybrid observation model, the output force feedback signal includes:

[0044] Integrate Fiber Bragg Grating (FBG) sensors or strain gauge arrays on the puncture needle body to obtain the strain at multiple measurement points i where Δλ i is the drift of the grating center wavelength, k is the strain sensitivity coefficient of the fiber Bragg grating, and λ B is the Bragg wavelength of the fiber Bragg grating, to obtain the strain distribution matrix E = [ε1, ε2,..., ε n , where n is the number of fiber Bragg grating sensors or strain gauges.

[0045] Calculate the bending moment on the tip of the needle where I is the moment of inertia of the needle body cross-section, x i is the position of the i-th sensor, and L is the total length of the needle body.

[0046] Calculate the tangential force on the tip of the needle where r is the distance from the tip of the needle to the point of action of the bending moment.

[0047] Calculate the resultant force on the tip of the needle where F n is the tissue reaction force along the axial direction of the needle body.

[0048] Combine the surface deformation field to calculate the corrected tip force where γ is the deformation coupling coefficient, is the gradient of the surface deformation field.

[0049] Construct a hybrid observation model where α is the force feedback weight and β is the deformation feedback weight, is the time change rate of the surface deformation field.

[0050] Take the output of the hybrid observation model H(t) as the force feedback signal F feedback Output.

[0051] Its technical effects are as follows: Calculate the strain through the grating wavelength drift, and combine the high-sensitivity characteristics of the fiber Bragg grating to achieve precise perception of the deformation of the needle body; Calculate the bending moment, tangential force and resultant force, and accurately calculate the force on the tip of the needle, so that the system can timely obtain the change information of the puncture resistance, thereby improving the real-time performance and accuracy of the force feedback. Calculate and correct the force on the tip of the needle through the surface deformation field gradient, that is, consider the influence of the externally observable tissue deformation on the puncture resistance, and improve the accuracy of the force estimation; The deformation coupling coefficient combines the surface deformation field information to correct the force on the tip of the needle, making the force feedback more accurate and comprehensive. Construct a hybrid observation model, integrating force feedback and deformation feedback. The force feedback weight ensures the contribution of the force sensing measurement data, and the deformation feedback weight adjusts the force feedback according to the tissue surface deformation, improving the adaptability; The time change rate of the surface deformation field improves the dynamic response ability of the feedback, enabling it to adjust the force control strategy during the puncture process in real time. Use the force feedback signal to guide the puncture force control, which can adjust the puncture strategy when the tissue resistance increases abnormally, such as when contacting high-density tissue or abnormal tissue structure, to avoid tissue damage or puncture failure; Through the corrected calculation of the force on the tip of the needle, the robotic arm can dynamically adjust the puncture force to ensure the smooth progress of the puncture process and avoid deviation or damage to the surrounding tissue caused by sudden resistance changes; When approaching the target, the feedback signal can be used to achieve fine-tuning control to ensure the accuracy of the final puncture point and improve the success rate of the puncture operation.

[0052] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the feedback of the force feedback signal to the adaptive algorithm to real-time correct the boundary of the dynamic safety domain includes:

[0053] When the force feedback signal F feedback exceeds the set threshold F th at this time, adjust the dynamic safety domain, and the adjustment amount is ΔΩ(t) = -γ(F feedback - F th ), where γ is the safety domain adjustment coefficient.

[0054] The corrected dynamic safety domain is Ω′(t) = Ω(t) + ΔΩ(t). If the force feedback signal F feedback> the set threshold F th , the dynamic safety region is shrunk. If the force feedback signal F feedback < the set threshold F th , the dynamic safety region is expanded.

[0055] Based on the corrected dynamic safety region, the optimal path is recalculated.

[0056] Its technical effect is as follows: The safety region is dynamically adjusted based on the force feedback signal. When the force feedback signal exceeds the set threshold, if the force feedback signal is too large, such as when encountering high-density tissue or abnormal resistance, the system shrinks the safety region to prevent accidental puncture or tissue damage; if the force feedback signal is too small, such as when the tissue is soft and the puncture resistance is small, the safety region is expanded to increase the freedom of path search and make the puncture more efficient. By adjusting the safety region in real-time through force feedback, if abnormally high resistance is detected, such as when contacting hard tissue or bone, the safety region is automatically shrunk to avoid further penetration and cause damage; if low resistance is detected, such as when passing through soft tissue, the safety region is appropriately expanded to improve the puncture efficiency and reduce unnecessary adjustment and computational burden. A closed-loop mechanism of force feedback - dynamic safety region - path recalculation is adopted, making the puncture path always dynamically optimized among safe, low-resistance, and efficient paths, and improving the system's self-adjustment ability. Combining real-time force feedback to adjust the safety region enables the puncture robot to dynamically correct the path under the influence of tissue deformation, ensuring that the puncture process is always within the safe range, improving the puncture efficiency when the safety region is expanded, and avoiding tissue damage when the safety region is shrunk, maximizing the adaptation to the patient's physiological movement and increasing the puncture success rate.

[0057] A robotic arm puncture system based on optical tracking, which includes:

[0058] An optical data acquisition module, which is used to collect high-dynamic light field data through a high-frequency adjustable multi-spectral ring light source installed at the end of the robotic arm and calculate the surface deformation field.

[0059] A dynamic breathing phase model construction module, which is used to extract surface motion feature vectors, use the surface motion feature vectors as the input of a temporal convolutional network, and construct a dynamic breathing phase model for predicting the breathing phase within the next Δt time.

[0060] A predicted phase output module, which is used to train the dynamic breathing phase model using an adversarial training strategy and output the predicted phase.

[0061] A dynamic safety region construction module, which is used to construct a dynamic safety region according to the predicted phase.

[0062] An optimal path search module, which is used to search for the optimal path within the dynamic safety region using an adaptive algorithm and output the robotic arm joint angle sequence.

[0063] A force feedback module is used to measure the strain distribution of the needle body in real time, calculate the force on the needle tip, construct a hybrid observation model in combination with the surface deformation field, and output a force feedback signal.

[0064] A correction module is used to feedback the force feedback signal to the adaptive algorithm to correct the boundary of the dynamic safety region in real time.

[0065] A computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, it implements the method for robotic arm puncture based on optical tracking as described above.

[0066] The beneficial effects of the embodiments of the present invention are as follows:

[0067] The present invention adopts a high-frequency adjustable multi-spectral annular light source and a dual high-speed event camera to accurately collect the surface deformation field, and combines a dynamic breathing phase prediction model to effectively predict breathing motion. Compared with traditional systems based on a single light source or ordinary cameras, it can obtain more comprehensive tissue surface dynamic change data; the breathing phase is predicted through a temporal convolutional network, reducing puncture deviation caused by the patient's spontaneous breathing motion and improving the positioning accuracy of the robotic arm.

[0068] The present invention uses an adaptive algorithm to search for the optimal path within the predicted dynamic safety region, effectively avoiding trajectory deviation caused by patient breathing, body position changes, etc.; a stable joint angle sequence is generated by combining the rapidly-exploring random tree (RRT) with B-spline curve smoothing, making the robotic arm move smoothly and avoiding tissue damage caused by sudden trajectory adjustments; compared with traditional static path planning algorithms, it has better dynamic adaptability and can optimize the path in real time to adapt to complex environments.

[0069] The hybrid observation model of the present invention combines a fiber Bragg grating sensor or a strain gauge array to accurately measure the strain distribution of the needle body and calculate the force on the needle tip in real time; the puncture force is adjusted through a force feedback mechanism to avoid over-puncturing or tissue damage caused by tissue hardness changes or patient movement; a dynamic safety region boundary adaptive adjustment mechanism is set, and when the force feedback signal exceeds the set threshold, the safety region range is automatically adjusted to ensure that the tissue will not be over-squeezed or damaged during the puncture process. Description of the Drawings

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0071] Figure 1 It is a flowchart of the method for robotic arm puncture based on optical tracking of the present invention. Detailed implementation manners

[0072] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and illustrated herein can be arranged and designed in a variety of different configurations.

[0073] Please refer to Figure 1 , a first embodiment of the present invention provides a robotic arm puncture method based on optical tracking, which includes: collecting high-dynamic light field data through a high-frequency adjustable multi-spectral ring light source installed at the end of the robotic arm, and calculating the surface deformation field; extracting surface motion feature vectors, using the surface motion feature vectors as the input of a temporal convolutional network, and constructing a dynamic breathing phase model for predicting the breathing phase within the next Δt time; training the dynamic breathing phase model using an adversarial training strategy to output a predicted phase; constructing a dynamic safety region according to the predicted phase; using an adaptive algorithm to search for an optimal path within the dynamic safety region and output a robotic arm joint angle sequence; measuring the needle body strain distribution in real time, calculating the force on the needle tip, and combining the surface deformation field to construct a hybrid observation model to output a force feedback signal; feeding back the force feedback signal to the adaptive algorithm to correct the boundary of the dynamic safety region in real time.

[0074] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the step of collecting high-dynamic light field data through a high-frequency adjustable multi-spectral ring light source installed at the end of the robotic arm and calculating the surface deformation field includes: installing a high-frequency adjustable multi-spectral ring light source at the end of the robotic arm, the light source covering near-infrared wavelengths and visible light wavelengths, and each group of light sources flashing in sequence according to a preset frequency; capturing the reflected light signal on the tissue surface through two high-speed event cameras; using pulse code modulation technology to bind the on-time sequence of light sources with different wavelengths to the exposure time sequence of the two high-speed event cameras, and separating and obtaining the light field information of each channel through a demodulation algorithm; calculating the surface deformation field where λ is the light source wavelength, α λ is the tissue surface reflection coefficient, β λ is the tissue absorption coefficient, d(x, y) is the target surface depth value, and (x, y) is the pixel position in the image coordinate system.

[0075] Its technical effects are as follows: By means of a multi-spectral light source with adjustable high frequency, covering the near-infrared and visible light bands, it adapts to the optical characteristics of different tissues; adopting two high-speed event cameras, it can capture surface deformations with a time resolution of microseconds, improving the perception accuracy of fast movements; adopting pulse code modulation technology, binding the turning-on time sequences of light sources with different wavelengths to the exposure time sequences of high-speed event cameras, enabling effective separation of multi-spectral information and avoiding the problem of optical signal aliasing of traditional RGB or single-wavelength light sources; calculating the deformation field using continuous light field information, it can perceive the motion state of dynamic tissues in real time, enabling the puncture path planning to adapt to changes, and improving the stability and safety of punctures.

[0076] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, for extracting the surface motion feature vector and using the surface motion feature vector as the input of a temporal convolutional network to construct a dynamic breathing phase model for predicting the breathing phase within the future Δt time, it includes: extracting the motion feature vector from the data of the surface deformation field. where p i (t) is the motion trajectory of the feature point P i in three-dimensional space. represents the deformation change rate in the time direction, represents the deformation gradient in the horizontal direction, represents the deformation gradient in the vertical direction; using the surface motion feature vector V(t) to construct the input matrix X of the temporal convolutional network = [V(t - T), V(t - T + 1),..., V(t)], where T is the length of the historical time window; constructing a dynamic breathing phase model for predicting the breathing phase within the future Δt time, including a 1D convolutional layer, an extended causal convolutional layer, and a fully connected layer, the fully connected layer outputs the predicted breathing phase, and the prediction formula is

[0077] Its technical effects are as follows: Extracting the motion feature vector from the surface deformation field data reduces the dependence on additional sensors and avoids physical contact interference; the combination of the motion trajectory of the feature point, the deformation change rate, the horizontal gradient, and the vertical gradient comprehensively describes the dynamic changes on the tissue surface, making the breathing phase prediction more accurate; the combination of the 1D convolutional layer and the extended causal convolutional layer enables the network to effectively model long-term dependence relationships, capture subtle dynamic features in the breathing cycle, and improve the prediction accuracy within the future Δt time. Adopting a temporal convolutional network, where the extended causal convolutional layer can expand the receptive field through the dilation factor, enabling the model to capture long-term sequence dependence information without increasing the computational amount, realizing a farther breathing phase prediction, with strong parallel computing ability. Compared with RNN series methods, TCN can be calculated faster in the inference stage, ensuring puncture real-time performance.

[0078] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the model training of the dynamic breathing phase model using the adversarial training strategy, and the output of the predicted phase includes: using the adversarial training strategy, the generator outputs a phase prediction, and the discriminator is used to distinguish the real breathing signal from the prediction result. The loss function uses Wherein, is the predicted value, is the real value, and λ is the weight coefficient; the trained model is used to output the predicted phase

[0079] The technical effect is as follows: By introducing the adversarial training strategy and continuous optimization by the generator, the predicted breathing phase is closer to the real signal. At the same time, the discriminator provides feedback to prompt the model to generate more reliable phase predictions; since the loss function combines the prediction error and the adversarial loss, it ensures that the model can not only learn short-term accuracy but also optimize the global prediction stability; through the adversarial training strategy, the model can learn different types of breathing patterns during training and adapt to various individual differences, improving the adaptability to different patients. The discriminator continuously confronts the generator to make its output closer to the real breathing data, and can make stable predictions even in the face of unseen breathing patterns; the introduction of the discriminator enables the model to automatically learn and ignore noise factors, prompting the generator to focus on the key features that truly affect the breathing phase and improving the anti-interference ability of the prediction. Through the optimization of the adversarial loss, the adaptability of the model to complex environmental variables is enhanced, and the error caused by external environmental changes is reduced.

[0080] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, constructing a dynamic safety region according to the predicted phase includes: setting a phase safety radius according to the predicted phase Constructing a dynamic safety region in three-dimensional space Where p is the spatial point coordinate, and p target is the target puncture point.

[0081] Its technical effects are as follows: By predicting the phase to construct a dynamic safety region, it ensures that the puncture path is always adjusted around the safe area of respiratory movement, avoiding the influence of tissue displacement caused by breathing on puncture accuracy; setting the phase safety radius makes the puncture point always within the feasible path range at different respiratory phases of the patient, ensuring that the puncture point fluctuates within the minimum error range and improving the targeting accuracy. The present invention adopts a method for constructing a dynamic safety region based on predicted phase, which can adaptively adjust the size and position of the safety region according to the individual respiratory pattern of the patient, making the puncture strategy more flexible. The setting of the dynamic safety region ensures that when the robotic arm performs puncture, it can always stay within the safe range, avoiding puncturing into non-target areas due to respiratory displacement; combined with the adaptive path optimization algorithm, if the robotic arm predicts that it is about to exceed the safety region, it automatically adjusts the path to ensure that the puncture is performed within the safe range, improving the surgical safety.

[0082] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the use of the adaptive algorithm to search for the optimal path within the dynamic safety region and output the robotic arm joint angle sequence includes: using the adaptive algorithm to initialize a rapidly-exploring random tree within the dynamic safety region Ω(t), setting the starting point as the current position q of the end of the robotic arm start , and the end point as the target position p target ; performing random sampling within the dynamic safety region Ω(t) to generate an initial path candidate set wherein, the path point P i obeys the safety constraint P i ∈Ω(t); optimizing the initial path candidate set to select the optimal path P * ; using the quintic polynomial interpolation method to generate the joint angle sequence θ * corresponding to the smooth trajectory of the optimal path P i (t) = a0 + a1t + a2t 2 + a3t 3 + a4t 4 + a5t 5 , where a0, a1, a2, a3, a4, and a5 are the coefficients of the quintic polynomial respectively, and t is the current moment of the robotic arm movement.

[0083] Specifically, smoothing the optimal path P * with a B-spline curve to generate a smooth path, discretizing the smooth path into a series of target points, and obtaining the robotic arm joint angle sequence through inverse kinematics solution.

[0084] Its technical effects are as follows: The fast-expanding random tree method can quickly generate a set of candidate paths within the dynamic safety domain and efficiently search for feasible solutions in complex environments, ensuring the computational efficiency of path planning. It dynamically adjusts the search strategy through an adaptive algorithm to improve the convergence speed of the optimal path and ensure finding a feasible solution in a short time. Random sampling is carried out within the dynamic safety domain to ensure that the generated path points are always located in the feasible region, preventing the robotic arm from exceeding the safety boundary and enhancing the safety of the puncture process. A path optimization algorithm is adopted to eliminate unreasonable paths and, through smooth interpolation, avoid abrupt movements, improving the feasibility and safety of the path. Quintic polynomial interpolation is used to ensure smooth transitions of the path in terms of time and space, reducing the impact on the robotic arm joints, optimizing the motion trajectory, and making the puncture action smoother. Through high-order polynomial interpolation, it can ensure the continuity of the angular velocity, acceleration, and jerk of the joint angles, avoiding unnecessary jitters and vibrations during the execution of the robotic arm.

[0085] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, for the real-time measurement of the strain distribution of the needle body, calculation of the force on the needle tip, and construction of a hybrid observation model in combination with the surface deformation field, the output force feedback signal includes: integrating fiber Bragg grating sensors or strain gauge arrays on the puncture needle body to obtain the strain at multiple measurement points i. where, Δλ i is the drift of the grating center wavelength, k is the strain sensitivity coefficient of the fiber Bragg grating, and λ B is the Bragg wavelength of the fiber Bragg grating, to obtain the strain distribution matrix E = [ε1, ε2,..., ε n , where n is the number of fiber Bragg grating sensors or strain gauges; calculate the bending moment on the needle tip where, I is the moment of inertia of the needle body cross-section, x i is the position of the i-th sensor, and L is the total length of the needle body; calculate the tangential force on the needle tip where, r is the distance from the needle tip to the bending moment action point; calculate the resultant force on the needle tip where, F n is the tissue reaction force along the axial direction of the needle body, obtained by integrating the compressive strain distribution; in combination with the surface deformation field, calculate the corrected force on the needle tip where, γ is the deformation coupling coefficient, is the surface deformation field gradient; construct a hybrid observation model where, α is the force feedback weight, β is the deformation feedback weight, is the time change rate of the surface deformation field; use the output of the hybrid observation model H(t) as the force feedback signal F feedback for output.

[0086] Its technical effects are as follows: By calculating the strain through the grating wavelength drift and combining with the high-sensitivity characteristics of the fiber Bragg grating, the precise perception of the deformation of the needle body is realized; the bending moment, tangential force and resultant force are calculated to accurately deduce the force on the needle tip, enabling the system to timely obtain the change information of the puncture resistance, thereby improving the real-time performance and accuracy of the force feedback. By calculating the surface deformation field gradient to correct the force on the needle tip, that is, considering the influence of the externally observable tissue deformation on the puncture resistance, the accuracy of the force estimation is improved; the deformation coupling coefficient combines the surface deformation field information to correct the force on the needle tip, making the force feedback more accurate and comprehensive. A hybrid observation model is constructed to fuse the force feedback and deformation feedback. The force feedback weight ensures the contribution of the force sensing measurement data, and the deformation feedback weight adjusts the force feedback according to the tissue surface deformation to improve the adaptability; the time change rate of the surface deformation field improves the dynamic response ability of the feedback, enabling it to adjust the force control strategy during the puncture process in real time. Using the force feedback signal to guide the puncture force control can adjust the puncture strategy when the tissue resistance increases abnormally, such as when contacting high-density tissue or abnormal tissue structure, to avoid tissue damage or puncture failure; by correcting the calculation of the force on the needle tip, the robotic arm can dynamically adjust the puncture force to ensure the smooth progress of the puncture process and avoid deviation or damage to the surrounding tissue caused by sudden resistance changes; when approaching the target point, the feedback signal can be used to achieve fine-tuning control to ensure the accuracy of the final puncture point and improve the success rate of the puncture operation.

[0087] In a preferred embodiment of the present invention, in the above-mentioned robotic arm puncture method based on optical tracking, the feedback of the force feedback signal to the adaptive algorithm to real-time correct the boundary of the dynamic safety domain includes: when the force feedback signal F feedback exceeds the set threshold F th , the dynamic safety domain is adjusted, and the adjustment amount is ΔΩ(t)=-γ(F feedback -F th ), where γ is the safety domain adjustment coefficient; the corrected dynamic safety domain is Ω′(t)=Ω(t)+ΔΩ(t). If the force feedback signal F feedback > the set threshold F th , the dynamic safety domain is shrunk. If the force feedback signal F feedback < the set threshold F th , the dynamic safety domain is expanded; based on the corrected dynamic safety domain, the optimal path is recalculated.

[0088] Its technical effects are as follows: The safety domain is dynamically adjusted based on the force feedback signal. When the force feedback signal exceeds the set threshold, if the force feedback signal is too large, such as when encountering high-density tissue or abnormal resistance, the system shrinks the safety domain to prevent accidental puncture or tissue damage; if the force feedback signal is too small, such as when the tissue is soft and the puncture resistance is small, the safety domain is expanded to increase the freedom of path search and make the puncture more efficient. By adjusting the safety domain through real-time force feedback, if an abnormally high resistance is detected, such as when contacting hard tissue or bone, the safety domain is automatically shrunk to avoid damage caused by continuous penetration; if a low resistance is detected, such as when passing through soft tissue, the safety domain is appropriately expanded to improve the puncture efficiency and reduce unnecessary adjustment and computational burden. A closed-loop mechanism of force feedback-dynamic safety domain-path recalculation is adopted, enabling the puncture path to be dynamically optimized between safe, low-resistance, and efficient paths, and improving the system's self-adjustment ability. Combining real-time force feedback to adjust the safety domain enables the puncture robot to dynamically correct the path under the influence of tissue deformation, ensuring that the puncture process is always within the safe range, improving the puncture efficiency when the safety domain is expanded, and avoiding tissue damage when the safety domain is shrunk, adapting to the patient's physiological movements to the greatest extent and increasing the puncture success rate.

[0089] The second embodiment of the present invention provides a robotic arm puncture system based on optical tracking, including: an optical data acquisition module for collecting high-dynamic light field data through a high-frequency adjustable multi-spectral annular light source installed at the end of the robotic arm and calculating the surface deformation field; a dynamic breathing phase model construction module for extracting surface motion feature vectors and using the surface motion feature vectors as the input of a temporal convolutional network to construct a dynamic breathing phase model for predicting the breathing phase within the next Δt time; a predicted phase output module for training the dynamic breathing phase model using an adversarial training strategy and outputting the predicted phase; a dynamic safety domain construction module for constructing a dynamic safety domain based on the predicted phase; an optimal path search module for searching for the optimal path within the dynamic safety domain using an adaptive algorithm and outputting a robotic arm joint angle sequence; a force feedback module for measuring the needle body strain distribution in real time, calculating the force on the needle tip, and constructing a hybrid observation model in combination with the surface deformation field to output a force feedback signal; and a correction module for feeding back the force feedback signal to the adaptive algorithm to correct the boundary of the dynamic safety domain in real time.

[0090] The third embodiment of the present invention provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the robotic arm puncture method based on optical tracking as described above.

[0091] A computer program product of the robotic arm puncture method and device based on optical tracking provided by an embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the foregoing method embodiments. For specific implementation, reference can be made to the method embodiments and will not be elaborated herein.

[0092] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the above-mentioned robotic arm puncture method based on optical tracking, thereby effectively overcoming the deficiencies of traditional puncture methods in terms of accuracy, adaptability, and safety, improving the success rate of puncture, with high precision, safety and reliability, and high automation.

[0093] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program code.

[0094] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or make equivalent replacements for some of the technical features; and these modifications, changes, or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A robotic arm puncture method based on optical tracking, characterized in that, Comprising: A high-frequency adjustable multi-spectral annular light source installed at the end of the robotic arm is used to collect high-dynamic light field data and calculate the surface deformation field; Extract the surface motion feature vector, use the surface motion feature vector as the input of the temporal convolutional network, and construct a dynamic breathing phase model for predicting the breathing phase within the next Δt time; Adopt an adversarial training strategy to train the dynamic breathing phase model and output the predicted phase; Construct a dynamic safety domain according to the predicted phase; Use an adaptive algorithm to search for the optimal path within the dynamic safety domain and output the robotic arm joint angle sequence; Measure the needle body strain distribution in real time, calculate the force on the needle tip, combine with the surface deformation field, construct a hybrid observation model, and output a force feedback signal; Feed the force feedback signal back to the adaptive algorithm to dynamically correct the boundary of the dynamic safety domain in real time.

2. The robotic arm puncture method based on optical tracking according to claim 1, wherein The process of using the high-frequency adjustable multi-spectral annular light source installed at the end of the robotic arm to collect high-dynamic light field data and calculate the surface deformation field includes: Install a high-frequency adjustable multi-spectral annular light source at the end of the robotic arm. The light source covers near-infrared and visible light wavelengths, and each group of light sources flashes in sequence according to a preset frequency; Capture the reflected light signal on the tissue surface through a dual high-speed event camera; Adopt pulse code modulation technology to bind the turn-on timing of light sources with different wavelengths to the exposure timing of the dual high-speed event camera, and separate and obtain the light field information of each channel through a demodulation algorithm; Calculate the surface deformation field where λ is the wavelength of the light source, α λ is the tissue surface reflection coefficient, β λ is the tissue absorption coefficient, d(x,y) is the target surface depth value, and (x,y) is the pixel position in the display image coordinate system.

3. The robotic arm puncture method based on optical tracking according to claim 1, characterized in that The process of extracting the surface motion feature vector, using the surface motion feature vector as the input of the temporal convolutional network, and constructing a dynamic breathing phase model for predicting the breathing phase within the next Δt time includes: Extract the motion feature vector from the data of the surface deformation field where p i (t) is the motion trajectory of the feature point P i in the three-dimensional space, represents the rate of deformation change in the time direction, represents the deformation gradient in the horizontal direction, represents the deformation gradient in the vertical direction; Use the surface motion feature vector V(t) to construct the input matrix X of the temporal convolutional network = [V(t - T), V(t - T + 1),..., V(t)], where T is the length of the historical time window; Construct a dynamic respiratory phase model for predicting the respiratory phase within the next Δt time, including a 1D convolutional layer, an extended causal convolutional layer, and a fully connected layer. The fully connected layer outputs the predicted respiratory phase, and the prediction formula is 4. The robotic arm puncture method based on optical tracking according to claim 1, wherein, The process of adopting an adversarial training strategy to train the dynamic breathing phase model and output the predicted phase includes: An adversarial training strategy is adopted for the generator to output phase prediction and for the discriminator to distinguish between real respiratory signals and prediction results. The loss function uses where is the predicted value, is the true value, and λ is the weight coefficient; Output the predicted phase using the trained model 5. The robotic arm puncture method based on optical tracking according to claim 1, wherein, The process of constructing a dynamic safety domain according to the predicted phase includes: Set the phase safety radius according to the predicted phase Construct a dynamic safety domain in three-dimensional space where p is the coordinate of a spatial point, and p target is the target puncture point.

6. The robotic arm puncture method based on optical tracking according to claim 5, characterized in that, The process of using an adaptive algorithm to search for the optimal path within the dynamic safety domain and output the robotic arm joint angle sequence includes: Initialize a rapidly-exploring random tree (RRT) within the dynamic safety region Ω(t) using an adaptive algorithm, with the starting point set as the current end-effector position q of the robotic arm start , and the ending point set as the target position p target ; Random sampling is performed within the dynamic safety domain Ω(t) to generate an initial path candidate set where the path point P i obeys the safety constraint P i ∈Ω(t); Optimize the initial path candidate set and select the optimal path P * ; Using the fifth-order polynomial interpolation method, generate the optimal path P * The joint angle sequence θ i (t) = a0 + a1t + a2t 2 + a3t 3 + a4t 4 + a5t 5 , where a0, a1, a2, a3, a4, and a5 are the coefficients of the fifth-order polynomial, and t is the current time of the robotic arm movement.

7. The robotic arm puncture method based on optical tracking according to claim 6, characterized in that, The process of measuring the needle body strain distribution in real time, calculating the force on the needle tip, combining with the surface deformation field, constructing a hybrid observation model, and outputting a force feedback signal includes: Integrate a fiber Bragg grating sensor or a strain gauge array on the puncture needle body to obtain the strain at multiple measurement points i where, Δλ i is the drift of the grating center wavelength, k is the strain sensitivity coefficient of the fiber Bragg grating, and λ B is the Bragg wavelength of the fiber Bragg grating, and the strain distribution matrix E = [ε1, ε2,..., ε n is obtained, where n is the number of fiber Bragg grating sensors or strain gauges; Calculate the bending moment on the tip of the needle where I is the moment of inertia of the needle body cross-section, x i is the position of the i-th sensor, and L is the total length of the needle body; Calculate the tangential force on the tip of the needle where r is the distance from the tip of the needle to the point of action of the bending moment; Calculate the resultant force on the needle tip where F n is the tissue reaction force along the axial direction of the needle body; Calculate the corrected tip force in combination with the surface deformation field where γ is the deformation coupling coefficient and ∇S is the gradient of the surface deformation field Construct a hybrid observation model where α is the force feedback weight and β is the deformation feedback weight, is the time change rate of the surface deformation field; Use the output of the hybrid observation model H(t) as the force feedback signal F feedback Output.

8. The robotic arm puncture method based on optical tracking according to claim 7, characterized in that The process of feeding the force feedback signal back to the adaptive algorithm to dynamically correct the boundary of the dynamic safety domain in real time includes: When the force feedback signal F feedback exceeds the set threshold F th , adjust the dynamic safety domain, and the adjustment amount is ΔΩ(t) = -γ(F feedback - F th ), where γ is the safety domain adjustment coefficient; The corrected dynamic safety domain is Ω′(t) = Ω(t) + ΔΩ(t). If the force feedback signal F feedback > the set threshold F th , then contract the dynamic safety domain. If the force feedback signal F feedback < the set threshold F th , then expand the dynamic safety domain; Based on the corrected dynamic safety domain, recalculate the optimal path.

9. A robotic arm puncture system based on optical tracking, characterized in that, Comprising: An optical data acquisition module for collecting high-dynamic light field data and calculating the surface deformation field through a high-frequency adjustable multi-spectral annular light source installed at the end of the robotic arm; A dynamic breathing phase model construction module for extracting the surface motion feature vector, using the surface motion feature vector as the input of the temporal convolutional network, and constructing a dynamic breathing phase model for predicting the breathing phase within the next Δt time; A predicted phase output module for training the dynamic breathing phase model using an adversarial training strategy and outputting the predicted phase; A dynamic safety domain construction module for constructing a dynamic safety domain according to the predicted phase; An optimal path search module, which is used to search for an optimal path within the dynamic safety domain by using an adaptive algorithm and output a sequence of robotic arm joint angles; A force feedback module, which is used to measure the strain distribution of the needle body in real time, calculate the force on the needle tip, combine with the surface deformation field, construct a hybrid observation model, and output a force feedback signal; A correction module, which is used to feedback the force feedback signal to the adaptive algorithm and correct the boundary of the dynamic safety domain in real time.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the optical tracking-based robotic arm puncture method according to any one of claims 1 to 8.

Images