Puncture tissue identification and transmembrane event control method and system based on force sense and image fusion

Abstract

The invention discloses a puncture tissue recognition and transmembrane event control method and system based on force sense and image fusion, and the method comprises the steps: collecting a force signal at the tail end of a puncture needle, a real-time ultrasonic image sequence and the pose data of a mechanical arm, and carrying out the timestamp alignment to construct a synchronous data set; on the basis, tissue stiffness, a force change rate, extreme value features, texture features, envelope boundary features and optical flow displacement features are extracted, the tissue level where the puncture needle is located is recognized through multi-modal fusion, and a membrane penetrating event is detected in combination with the force sense extreme value features and envelope boundary fracture; according to the tissue level and the transmembrane event, event driving control strategies such as constant-speed propulsion, deceleration early warning, rotation assistance, transmembrane braking and path compensation are automatically triggered, so that the puncture process is accurately regulated and controlled. The accuracy of puncture tissue recognition and the reliability of transmembrane event judgment can be remarkably improved, and the safety and stability of puncture operation are improved.

Description

Technical Field

[0001] This invention relates to the field of medical robots and intelligent interventional therapy, specifically to a method and system for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion. It can be applied to percutaneous puncture, biopsy sampling, ablation therapy, renal intervention, and other minimally invasive surgical procedures that require precise tissue layer recognition and safety control. Background Technology

[0002] In recent years, with the development of surgical robots and intelligent sensing technologies, the application of force sensing and medical imaging in puncture path planning and safety control has gradually attracted attention. Minimally invasive procedures such as percutaneous puncture, biopsy sampling, tumor ablation, and renal intervention typically require doctors to precisely deliver the puncture needle to the target tissue location under real-time image guidance. However, traditional puncture operations rely heavily on the surgeon's experience, visual judgment, and subjective perception of touch, resulting in significant human uncertainty. When the needle tip crosses different tissue layers or penetrates the organ capsule, the lack of an effective objective identification mechanism can easily lead to over-probing, deviation, or damage to surrounding normal tissue, thus affecting the safety and treatment outcome of the procedure.

[0003] Force signals can reflect the mechanical properties of tissues and are often used to detect tissue boundaries and identify membrane perforation events. However, a single force signal is easily affected by factors such as individual differences, changes in posture, tissue lesions, and tissue degeneration caused by repeated punctures, resulting in large fluctuations in characteristics and insufficient recognition stability.

[0004] Ultrasound imaging, as the most commonly used real-time imaging method in clinical practice, has advantages such as being radiation-free and having a high frame rate. However, its image quality is affected by factors such as operator experience, probe angle, and acoustic shadowing. For the identification of tissue layers, hyperechoic structures may appear blurred, broken, or subject to noise interference; for the dynamic deformation of soft tissue in front of the needle tip, it is difficult to establish an accurate prediction model based solely on frame-by-frame images. Therefore, a single imaging modality is still insufficient to meet the requirements of high robustness and high recognition accuracy for refined puncture control.

[0005] Existing research attempts to use force signals or image features for monitoring the puncture process, but lacks deep integration of the two, especially in key steps such as tissue hierarchy identification, membrane penetration event prediction, and path deviation compensation. It fails to provide a unified, real-time, and controllable perception-decision-execution mechanism. Furthermore, most existing systems employ rule-driven or single-channel criteria, lacking the ability to jointly infer multimodal information, making them prone to misjudgment or delayed judgment in complex tissue structures or dynamic tissue deformation scenarios.

[0006] In the prior art, application CN113400304A discloses a control method that combines force, displacement, and visual information during robotic endotracheal intubation. This method is based on a standard intubation path and an oral biomechanical model. It obtains the corresponding points in the standard path through visual image mapping, reads the displacement and force information of the robotic device, uses a virtual gripper method to determine the safety of the current area, and uses parallel PID and threshold control to adjust the movement speed of the robotic arm. Thus, it comprehensively considers mechanical and visual information during intubation to achieve safety and efficiency in the laryngoscope and endotracheal tube insertion process.

[0007] However, this technical solution is mainly aimed at endotracheal intubation and focuses on the overall safety control of catheter posture and path. Although it involves the mixed use of force and visual signals, it does not utilize ultrasound imaging to observe the internal structure of soft tissue in real time, nor does it combine force signals and ultrasound images to jointly identify and control the soft tissue layers in front of the puncture needle, capsule penetration events, and puncture path deviation. Therefore, it is difficult to meet the application requirements of percutaneous puncture, renal intervention, etc., which require precise tissue layer identification and capsule penetration safety control. Summary of the Invention

[0008] This invention aims to overcome the problems of existing puncture procedures, such as reliance on operator experience, instability of single-modal perception, susceptibility of ultrasound imaging to noise and obstruction, and insufficient accuracy in tissue layer identification. It provides a method and system for puncture tissue identification and control based on force and image fusion. This method simultaneously acquires force signals from the puncture needle tip, real-time ultrasound image sequences, and robotic arm pose data. Based on multimodal information synchronization, and combined with features such as tissue stiffness, force change rate, texture characteristics, tissue boundaries, and optical flow displacement, it achieves real-time identification of each tissue layer during the puncture process and makes a reliable determination when capsule penetration occurs. Furthermore, this invention uses an event-driven control strategy to automatically adjust the puncture speed, rotation mode, and path compensation under different tissue layers and risk events, enabling the puncture process to have higher stability, safety, and intelligence, significantly improving the operational accuracy and clinical safety in percutaneous puncture, biopsy sampling, and interventional treatment.

[0009] A method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion includes the following steps: S1. Signal Acquisition and Synchronization: Acquire force signals from the tip of the puncture needle, real-time ultrasound image sequences, and pose data from the robotic arm encoder. Timestamp-align signals at different sampling frequencies to form a synchronized dataset. S2. Feature Extraction: Calculate tissue stiffness, force change rate, and extreme value features of force change rate during the puncture process based on force signals; extract texture features and tissue boundary features based on ultrasound image sequences, and use optical flow algorithm to obtain optical flow displacement of the tissue in front of the needle tip; S3, Tissue Hierarchy Recognition: The tissue stiffness, force change rate and extreme value features are weighted and fused with texture features and tissue boundary features. The tissue hierarchy where the puncture needle is located is identified according to the preset judgment rules. The tissue hierarchy includes the skin layer, fat layer, muscle layer, capsule layer and renal parenchyma layer. S4. Perforation event recognition: When extreme features are detected simultaneously with the fracture of the hyperechoic capsule boundary in the ultrasound image, it is determined that the puncture needle has perforated the capsule. S5. Event-driven control: The puncture control strategy is automatically adjusted based on tissue level and membrane penetration event. The control strategy includes constant speed advancement, rotation assistance, deceleration warning, membrane penetration braking, and path compensation based on optical flow displacement.

[0010] Preferably, the method for extracting tissue stiffness in step S2 is as follows: , Where k(t) represents tissue stiffness, ΔF(t) is the force change during the sampling period, and Δx(t) is the corresponding puncture displacement change.

[0011] Preferably, in step S2, the extreme value feature is identified by monitoring the maximum rate of increase of the force change rate F′(t) and the subsequent maximum rate of decrease.

[0012] Preferably, in step S2, the texture features extracted based on the ultrasound image sequence are used to extract contrast, energy, or homogeneity parameters through a gray-level co-occurrence matrix to distinguish between the fat layer, muscle layer, and renal parenchyma.

[0013] Preferably, the optical flow algorithm in step S2 is used to estimate the displacement change of the tissue in front of the needle tip between adjacent image frames. In step S2, the optical flow algorithm calculates the image gradient to satisfy the formula: I x u+I y v+I t =0 Where I x and I y These represent the spatial gradients of the image in the x and y directions, respectively; I t denoted as the gradient of the image in the time dimension; u and v are the components of the optical flow displacement of the pixel in the x and y directions, respectively; the optical flow displacement is used to characterize the amount of displacement of the tissue in front of the needle tip between adjacent image frames.

[0014] Preferably, the determination of the tissue level in step S3 is based on the following formula: P=α·P force +β·P image Where P force P is the probability of judgment obtained based on force perception characteristics. imageThe probability of judgment is obtained based on image features, and α and β are dynamically adjustable weight coefficients.

[0015] Preferably, in step S5, a deceleration warning is triggered when the distance between the high-echo membrane boundary and the needle tip is detected to be lower than a preset threshold; and path compensation control is triggered when the optical flow displacement is detected to exceed the compensation threshold.

[0016] During the puncture operation of this invention, the end effector of the robotic arm carries the puncture needle and gradually advances it towards the target area. The system first acquires force signals, ultrasound image sequences, and robotic arm pose information in real time during the puncture process through a force sensing unit, an image acquisition unit, and a pose acquisition unit. A signal synchronization module then unifies the multi-source data to the same time reference. Subsequently, the feature extraction module calculates tissue stiffness, force change rate, and extreme value features based on the force signals, and extracts tissue texture, boundary contours, and optical flow displacement in front of the needle tip through ultrasound images, thereby forming a multimodal feature set reflecting the current tissue state.

[0017] Based on this, the tissue-level recognition module performs joint inference of multimodal features according to preset fusion rules, enabling real-time identification of tissues such as skin, fat, muscle, capsule, and renal parenchyma. When the capsule boundary of the preceding tissue breaks in the image and the force signal produces a characteristic extreme value, the membrane penetration event recognition module immediately determines whether the membrane has penetrated. Subsequently, the control execution module automatically triggers corresponding control strategies based on the tissue state and event outcome, providing deceleration warnings when approaching the capsule, performing braking control at the moment of membrane penetration, and performing path compensation based on optical flow estimation results when there is a risk of deviation, thereby enabling the entire puncture process to have continuous perception, real-time discrimination, and dynamic control capabilities.

[0018] The beneficial effects of this invention are as follows: (1) By fusing force features with ultrasound image features in real time, this invention significantly reduces misjudgment caused by the instability of single-modal information and improves the identification accuracy of different tissue levels.

[0019] (2) The present invention utilizes the extreme value characteristics of the force change rate and the fracture phenomenon of the membrane boundary in the ultrasound image for dual verification, making the identification of membrane penetration event more reliable and reducing the risk of overshoot.

[0020] (3) The present invention uses an event-driven control strategy to automatically decelerate when approaching the membrane, perform braking at the moment of membrane penetration, and compensate for the offset trend in real time according to the optical flow displacement, which can effectively avoid damage to deep structures and improve puncture safety.

[0021] (4) Due to the robustness of the multimodal fusion feature of the present invention, it can effectively cope with image quality fluctuations, puncture angle changes or tissue mechanical differences, making the system widely applicable to kidney puncture, liver puncture, biopsy sampling and other percutaneous intervention scenarios.

[0022] (5) This invention automatically identifies tissue status and intelligently controls the movement of the robotic arm through an algorithm, making the operation process more standardized and improving the success rate and safety of beginners and operators with limited experience. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of a puncture tissue recognition and membrane perforation event control method based on force sensing and image fusion; Figure 2 This is a schematic diagram of a framework for a puncture tissue recognition and membrane perforation event control system based on force sensing and image fusion. Figure 3 This is a schematic diagram illustrating the multimodal feature extraction and fusion of a puncture tissue recognition and membrane perforation event control method based on force perception and image fusion. Figure 4 This is a schematic diagram illustrating the extreme value characteristics of the force change rate during membrane penetration events in a method for puncture tissue recognition and membrane penetration event control based on force perception and image fusion. Figure 5 This is a schematic diagram of a hyperechoic capsule boundary fracture during a puncture event, which is a method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion. Figure 6 This is a schematic diagram of an event-driven control strategy for a puncture tissue recognition and membrane perforation event control method based on force perception and image fusion. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Example 1

[0025] like Figure 1 As shown, the puncture tissue identification and control method based on force and image fusion provided in this embodiment includes core steps such as signal acquisition and synchronization, feature extraction, tissue level identification, membrane penetration event identification, and event-driven control.

[0026] This method first acquires multimodal data generated in real time during the puncture process through a force acquisition unit, an image acquisition unit, and a pose acquisition unit. Then, the signal synchronization module timestamps the signals of different types and sampling frequencies to form a continuous synchronous dataset for subsequent analysis.

[0027] After acquiring synchronized data, the method proceeds to the feature extraction stage. For the force feedback channel, it extracts stiffness, rate of force change, and corresponding extreme features reflecting the tissue's mechanical response. For the image channel, it extracts tissue texture features, capsule boundary features, and optical flow displacement features of the tissue in front of the needle tip. Subsequently, based on these features and a multimodal fusion mechanism, the system identifies the current tissue layer of the puncture needle, including different tissue structures such as the skin layer, fat layer, muscle layer, capsule layer, and renal parenchyma layer.

[0028] As the puncture needle gradually approaches the capsule layer... Figure 1 The method described enters the membrane perforation event identification stage, which determines whether a membrane perforation event has occurred by monitoring typical extreme patterns of force change rate and fracture changes of the membrane boundary in ultrasound images. Upon identification of a membrane perforation event, the system immediately enters event-driven control mode, automatically adjusting the puncture strategy based on the current event type and tissue level, such as deceleration, braking, rotational assistance, or path compensation, to ensure the safety and accuracy of the puncture operation.

[0029] The entire process achieves dynamic coordination of multimodal perception, intelligent recognition and adaptive control, enabling the puncture process to maintain high stability and reliability in complex tissue structures and dynamic deformation environments.

[0030] like Figure 2 As shown, the puncture tissue recognition and control system used in this embodiment includes multiple functional units such as a force sensing acquisition unit, an image acquisition unit, a pose acquisition unit, a signal synchronization module, a feature extraction module, a tissue level recognition module, a membrane penetration event recognition module, and a control execution module.

[0031] After the system starts, the robotic arm first performs a homing and self-calibration operation to establish a stable world coordinate system and ensure that the spatial position and orientation of the puncture needle can be accurately acquired. The image acquisition unit initializes the ultrasound probe, including setting the imaging depth, focal position, and gain, to ensure that the target tissue presents a clear echo structure in the image.

[0032] The force sensing acquisition unit enters real-time acquisition mode, continuously recording the axial force signal of the puncture needle during its advancement. The pose acquisition unit, using the robotic arm's end effector as a reference point, acquires the position and orientation data of the puncture needle in real time. Since the sampling frequencies of these signals are not consistent, the signal synchronization module uses a unified time reference, interpolation calculation, and timestamp calibration to align the three types of signals and generate a time-continuous synchronized dataset. This synchronized dataset ensures that force sensing features, image features, and pose changes correspond to each other at the same point in time, providing a basis for subsequent... Figure 3 The multimodal feature extraction and fusion demonstrated provide a consistent and reliable data foundation.

[0033] pass Figure 2The system structure and initialization steps shown in this embodiment ensure the accuracy of multimodal signal acquisition, the integrity of data association, and the system stability before the puncture task is executed, laying a solid foundation for the implementation of the entire method.

[0034] like Figure 3 As shown, after signal acquisition and synchronization are completed, this invention performs multimodal feature extraction on force signals and ultrasound image sequences, and achieves tissue hierarchy recognition based on fusion rules. First, tissue stiffness features are extracted from the force channel.

[0035] In this invention, the method for extracting tissue stiffness is as follows: ; Where k(t) represents the tissue stiffness at time t, which is used to characterize the resistance of the tissue to the puncture needle along the current puncture path; ΔF(t) represents the change in force signal within a preset sampling period, that is, the increment of axial force of the puncture needle within the current sampling period; Δx(t) represents the displacement change in the puncture direction within the same sampling period, that is, the distance the puncture needle advances along the needle axis during that time period.

[0036] The force signal F(t) is the axial force at the end of the puncture needle acquired by the force sensing acquisition unit at time t, and the displacement signal x(t) is the position of the puncture needle in the puncture direction acquired by the pose acquisition unit at time t.

[0037] The stiffness characteristics calculated by the above formula can reflect the differences in mechanical response of different tissue layers during puncture, providing a basis for distinguishing between the fat layer, muscle layer, capsule layer and renal parenchyma layer.

[0038] In addition to stiffness characteristics, this invention also extracts the extreme value characteristics of the force change rate from the force signal. The force change rate F′(t) represents the rate of change of the force signal with time, and can be obtained by differentiating or numerically differentiating the continuous force signal F(t), used to characterize how quickly the force increases or decreases with time.

[0039] When F′(t) shows a significant upward peak in a short period of time and is followed by a significant downward abrupt change, it can be used to characterize the typical mechanical mode in the membrane penetration process. The specific extreme value extraction method is described in words in the embodiments, and is not limited to a specific numerical calculation format.

[0040] In the image channel, this invention takes the region of interest in front of the needle tip as the analysis object and extracts texture features and tissue boundary features from the ultrasound image sequence.

[0041] Texture features can be used to calculate parameters such as contrast, energy, and homogeneity based on the gray-level co-occurrence matrix, which can be used to distinguish different tissue types such as fat, muscle, and parenchyma; tissue boundary features are used to detect and track the continuity and positional changes of hyperechoic capsule boundaries, providing a basis for subsequent transmembrane event recognition.

[0042] To characterize the relative displacement of tissue between adjacent image frames, this invention further employs an optical flow algorithm to calculate the optical flow displacement in the region in front of the needle tip. The optical flow estimation satisfies the following optical flow constraint equation: ; Among them, I x and I y These represent the spatial gradients of image grayscale in the horizontal and vertical directions, respectively, reflecting the spatial variation of local grayscale; I t The grayscale value represents the rate of change of the image grayscale over time, reflecting the change in brightness between adjacent frames over time; u and v represent the components of the optical flow displacement of a pixel in the horizontal and vertical directions, respectively, describing the direction and magnitude of the pixel's movement between adjacent frames.

[0043] This invention obtains the overall displacement and displacement trend of tissue by statistically analyzing the optical flow vector in a local area in front of the needle tip, and determines whether there is tissue displacement caused by factors such as breathing or pressing, providing a basis for subsequent path compensation control.

[0044] After extracting force and image features, this invention fuses multimodal features to improve the stability and accuracy of tissue hierarchy recognition. Specifically, based on the features of the force channel (including stiffness, rate of change of force, and its extreme value features), a set of tissue category determination probabilities is obtained, denoted as P. force Based on the features of the image channels (including texture features, capsule boundary features, and optical flow-related features), another set of tissue category determination probabilities is obtained, denoted as P. image .

[0045] This invention uses the following weighted fusion rule to calculate the final fusion determination probability: ; Where P represents the probability of determining the organizational hierarchy after fusion, which can be the probability distribution corresponding to each candidate organizational category; P force P represents the probability of a judgment based on force characteristics. image This represents the probability of judgment based on image features; α and β are fusion weight coefficients used to adjust the relative importance of the force channel and the image channel in the final judgment. They are usually in the range of 0 to 1. α and β can be set independently, and it is preferable to satisfy the normalization constraint.

[0046] In practical applications, when the image quality is good, the value of β can be appropriately increased; when the ultrasound image is affected by noise or the probe contact is unstable, the value of α can be appropriately increased to enhance the reliance on force information.

[0047] In a specific implementation, the weighting coefficients α and β can be adaptively set based on image quality indicators and force signal stability indicators.

[0048] For example, the system can evaluate the current image quality based on indicators such as the signal-to-noise ratio of the ultrasound image, the continuity score of the capsule boundary, or the image gray-level variance. When the indicator is higher than a preset threshold, the image channel weight β is increased; when there is obvious speckle noise, blurred boundaries, or unstable probe contact in the image, resulting in a decrease in echo quality, β is decreased and the force perception channel weight α is increased accordingly.

[0049] Simultaneously, the system can also assess the reliability of the force sensory channel by monitoring the fluctuation amplitude of the force sensory signal within a short time window or the stability during repeated punctures. When the force signal changes smoothly and the extreme value characteristics are clear, α is increased; when the force signal fluctuates abnormally due to posture changes or tissue inhomogeneity, α is appropriately decreased and β is increased.

[0050] By using the weight adjustment method based on multi-source quality assessment, the contributions of force information and image information to tissue hierarchy determination can be dynamically balanced under different imaging conditions and tissue environments, thereby improving the stability and robustness of the fusion determination results.

[0051] By selecting the maximum value of the fusion probability P, the tissue level identification result corresponding to the current time point can be obtained, which can be used for subsequent membrane penetration event identification and event-driven control.

[0052] like Figure 4 and Figure 5 As shown, this invention identifies membrane penetration events based on the joint identification of dual-modal features of the force sensing channel and the image channel.

[0053] In the force perception channel, the presence of membrane-penetration-related mechanical patterns is identified by monitoring the extreme characteristics of the force change rate. The force change rate F′(t), as a measure of the rate of change of the force signal over time, typically corresponds to a gradual increase in tissue resistance during puncture. However, at the moment of membrane penetration, the resistance suddenly decreases due to the puncture of the membrane, resulting in a significant abrupt change in F′(t). This invention identifies the extreme rising value (maximum rate of rise) and the subsequent extreme falling value (maximum rate of fall) of F′(t) within a specified time window, using both as force perception criteria for membrane penetration events.

[0054] The rising extreme value of the rate of change of force characterizes the resistance accumulation stage before the membrane is stretched to its limit, while the falling extreme value reflects the instantaneous unloading process after the membrane ruptures. The sequential appearance of these two characteristics constitutes a typical force perception pattern of the membrane penetration process, which has good identifiability.

[0055] In image channels, such as Figure 5 As shown, this invention utilizes changes in the membrane boundary in ultrasound images as image evidence of membrane penetration events.

[0056] The capsule usually appears as a continuous, regular hyperechoic line on ultrasound images, and the boundary maintains its overall continuity as the puncture needle approaches; however, at the moment of puncture, the local continuity is broken or the strength is significantly reduced as the needle tip breaks through the capsule.

[0057] This invention detects the change in connectivity of the capsule boundary in adjacent image frames and uses the transition from continuous to fragmented hyperechoic capsule boundary as a criterion for transmembrane events in images. This feature can effectively reflect the abrupt changes in tissue morphology caused by the interaction between the needle tip and the capsule.

[0058] To improve the accuracy of recognition, this invention employs a joint determination method using force events and image events. Force events and image events are represented as binary states, where 1 indicates that the event has occurred and 0 indicates that the event has not occurred.

[0059] When both the extreme value feature on the force side and the capsule rupture feature on the image side occur in time (i.e., both events are 1), the puncture needle is considered to have penetrated the capsule. If only one side of the event occurs or neither occurs, it is not considered a capsule penetration event.

[0060] This dual-modal joint determination method can avoid misjudgments caused by relying solely on force signals or image signals, thereby improving the reliability and robustness of membrane penetration recognition.

[0061] like Figure 6 As shown, after obtaining the tissue level identification results and the membrane penetration event identification results, the present invention adopts an event-driven control strategy to adaptively adjust the puncture behavior of the robotic arm in order to improve the safety and accuracy of the puncture process.

[0062] During the normal puncture phase, if tissue level identification indicates that the needle tip is located in superficial tissue and there is no risk event, the control execution module drives the robotic arm to advance the puncture needle at a preset constant speed along the puncture direction, allowing it to smoothly enter the target tissue. When force features show a significant increase in tissue resistance, but image features have not yet indicated the presence of nearby critical structures, the system can automatically trigger a rotation assist mode. This mode applies a slow rotation to the puncture needle while maintaining the advancement speed, reducing local frictional resistance and improving puncture smoothness.

[0063] When tissue hierarchy identification results or image features indicate that the puncture needle is too close to the capsule or other critical tissue structures, the control execution module enters a deceleration warning mode. This automatically reduces the advance speed, making the puncture process more stable and reducing the risk of accidental puncture due to excessive speed. When the membrane puncture event identification module determines that a membrane puncture event has occurred, the invention immediately triggers a membrane puncture braking strategy, causing the robotic arm to stop advancing along the puncture direction for a short period to prevent the needle tip from continuing to penetrate deeper tissues after puncturing the membrane. This braking process can be achieved by adjusting the robotic arm's motion commands, setting the maximum allowable displacement, or introducing control damping.

[0064] Furthermore, this invention utilizes optical flow displacement features to identify the movement trend of tissue in adjacent image frames. When the optical flow features indicate that tissue in front of the needle tip has moved and may cause the puncture path to deviate from the target area, the control execution module can make minor adjustments to the trajectory of the robotic arm based on the direction and amplitude of the optical flow, achieving path compensation control. This compensation can effectively address target deviations caused by factors such as breathing, probe pressure fluctuations, or tissue deformation, thereby ensuring that the puncture needle always points towards the predetermined target position.

[0065] Through the event-driven control mechanism described above, the present invention can automatically switch control strategies according to different events during the puncture process, which not only improves puncture accuracy but also significantly enhances operational safety.

[0066] In practical applications, the method of the present invention can be used for kidney biopsy, liver biopsy, or other ultrasound-guided percutaneous puncture procedures.

[0067] The following describes the implementation process of this invention using a kidney biopsy as an example. The operator first positions the patient in a suitable position and places the ultrasound probe above the target kidney. Simultaneously, the robotic arm's end effector grips the puncture needle and probe to establish the puncture initiation position. After system startup, the force acquisition unit, image acquisition unit, and pose acquisition unit begin synchronously acquiring force signals, ultrasound image sequences, and robotic arm pose data. The signal synchronization module performs time alignment on the above data, forming a synchronized dataset that can be used for feature extraction.

[0068] During the puncture, the feature extraction module updates force features, texture features, capsule boundary features, and optical flow features in real time, while the tissue layer identification module determines the current tissue layer of the puncture needle based on the fused probabilities. When the system detects that the puncture needle is approaching the capsule, it automatically triggers a deceleration warning to reduce the advancement speed to ensure the controllability of the puncture process. When puncture occurs simultaneously... Figure 4 The extreme characteristics of the rate of change of force shown are Figure 5 When the capsule ruptures as shown, the membrane penetration event recognition module determines that the membrane penetration event has occurred, and the control execution module immediately performs a membrane penetration braking operation to stop the puncture needle from advancing further after breaking through the capsule, thereby avoiding damage to deep tissues.

[0069] If the tissue structure in front of the needle tip shifts during the puncture process due to factors such as respiration or tissue deformation, this invention detects the shift trend through optical flow characteristics and promptly executes path compensation control to realign the puncture trajectory with the original target. Throughout the process, the system continuously updates the characteristics of each channel and dynamically selects control strategies to ensure safe, controllable, and accurate punctures. Example 2

[0070] This embodiment provides an implementation of the present invention applicable to in vitro simulation training systems or animal experimental environments, further illustrating that the method and system of the present invention can achieve stable operation in different application scenarios. This embodiment is consistent with the basic principle of Embodiment 1, but differentiated optimizations have been made in signal processing details, characteristic parameter settings, and control strategy applications, further enhancing the feasibility of the present invention.

[0071] like Figure 2 As shown, the system structure used in this embodiment is basically the same as that in Embodiment 1, including a force sensing acquisition unit, an image acquisition unit, a pose acquisition unit, a signal synchronization module, a feature extraction module, a tissue level recognition module, a membrane penetration event recognition module, and a control execution module. After starting the system, the robotic arm first completes its own zero-position calibration to establish a stable world coordinate reference system.

[0072] Subsequently, the image acquisition unit sets an appropriate ultrasound imaging depth according to operational requirements to ensure the target tissue is within the imaging focal range. The axial direction of the puncture needle is automatically identified by the pose acquisition unit and serves as the reference direction for subsequent multimodal feature extraction. Furthermore, each acquisition unit sets its corresponding sampling period according to the system's master clock. The signal synchronization module uses the ultrasound image timestamp as the primary time reference, interpolating and aligning the force and pose signals to ensure a synchronized dataset with consistent time. This initialization process enables the system to maintain stable operation in the in vitro model environment, providing high-quality foundational data for subsequent multimodal analysis.

[0073] like Figure 1 As shown, the puncture tissue identification and control method in this embodiment still includes steps such as signal acquisition and synchronization, feature extraction, tissue level identification, membrane penetration event identification, and event-driven control. However, due to the certain differences between the simulated tissue structure in vitro and real tissue, this embodiment dynamically adjusts the processing method of some data information to adapt to the characteristics of large fluctuations in tissue resistance distribution and image quality in the simulated environment. Through unified processing of the acquired signals, this embodiment can still achieve the same theoretical process and execution mechanism as Embodiment 1, thereby ensuring the consistency and transferability of the method of the present invention in various application scenarios.

[0074] like Figure 3As shown, this embodiment analyzes multimodal features based on synchronous data. The extraction of force features is still based on the tissue stiffness formula k(t)=ΔF(t) / Δx(t), where ΔF(t) represents the change in force signal between adjacent sampling periods during puncture, and Δx(t) represents the corresponding change in puncture displacement. Both are used to describe the mechanical response of the simulated tissue to the puncture needle.

[0075] In this embodiment, the force signal is smoothed to enhance robustness in simulated environments with high noise levels. The force change rate F′(t) is still calculated using the rate of change of the force signal in the time domain and is used to reflect the trend of resistance changes during puncture.

[0076] Image feature extraction still focuses on the region of interest (ROI) in front of the needle tip. Texture parameters are obtained by analyzing the gray-level co-occurrence matrix of the ROI to represent the detailed differences in the internal structure of the simulated tissue. In this embodiment, because the simulated tissue has stronger material consistency, the variation in texture features may be weaker. Therefore, an image boundary continuity index is added to improve the ability to distinguish between the capsule layer and the solid layer.

[0077] Furthermore, the optical flow displacement characteristics are obtained through the optical flow constraint equation I. x u+I y v+I t =0 is obtained through calculation, where I x and I y I represents the spatial gradient of an image. t The brightness changes of the image over time are represented by u and v, which represent the horizontal and vertical displacements of the pixels, respectively. By tracking the optical flow field in the image sequence, this embodiment can effectively monitor positional changes in tissue caused by external forces, probe pressure, or the elasticity of the model structure.

[0078] Based on the aforementioned force and image features, this embodiment still employs a probabilistic fusion method for tissue hierarchy identification, with the fusion formula being P=α. Pforce +β Pimage Here, α and β are weighting factors for different modalities, which can be dynamically adjusted according to the quality of the simulated image or the stability of the force signal. In this embodiment, during the simulation training process, the force perception weight is increased when the image signal is interfered with by noise, and the image weight is increased when the force signal is unstable, thereby ensuring the stability of tissue hierarchy recognition.

[0079] like Figure 4As shown, this embodiment still utilizes the extreme value characteristics of the rate of change of force as an important criterion for membrane penetration events on the force perception side. Although the mechanical properties of simulated tissue may differ from those of real tissue, a trend of gradual increase in the force signal can still be observed as the puncture needle approaches the capsule layer, and a significant rapid change occurs upon breaching the capsule surface. This embodiment identifies the transient characteristics of the force signal reflecting membrane penetration behavior by detecting the rising extreme value and the sudden drop in F′(t) over a short period of time. Due to the high mechanical consistency of the model material, this embodiment appropriately relaxes the time correlation between the rising peak and the falling peak to enhance the algorithm's adaptability to the model environment.

[0080] like Figure 5 As shown, in the image channel, the capsule layer still appears as a highly echogenic boundary. When the puncture needle approaches the capsule, this boundary remains continuous in the image, but when the needle tip penetrates the capsule, the capsule boundary breaks or disappears in a local area of ​​the image. This embodiment identifies the transition from continuity to breakage by comparing the morphological changes of the capsule boundary in adjacent image frames, thereby achieving the determination of membrane penetration events on the image side. Since the simulated model may result in less obvious boundary echoes than real tissue, this embodiment combines boundary intensity changes and morphological changes for joint judgment, enhancing the stability of membrane penetration event detection on the image side.

[0081] The final determination of a membrane penetration event still relies on a joint judgment mechanism of dual-modal events. When the force side detects an extreme value feature and the image side detects a membrane rupture feature, the system determines that a membrane penetration event has occurred; if only one side meets the criteria or neither side meets the criteria, it is not considered a membrane penetration event. Through the above joint judgment criteria, this embodiment effectively avoids the risk of misjudgment that may be caused by a single modal signal.

[0082] like Figure 6 As shown, this embodiment uses an event-driven mechanism to automatically control the puncture process. Although the resistance distribution and image quality of the simulated tissue material differ from those in the real scene, the logic of control execution is completely consistent.

[0083] When tissue level identification indicates that the puncture needle is in superficial tissue, the system maintains a constant speed of advancement. When force feedback indicates increased resistance or image information shows that the structure around the needle tip is dense, the system can enter a rotation-assisted mode to improve puncture smoothness through moderate rotation.

[0084] When the system detects that the tissue layer is close to the capsule or that the image side is too close to the capsule, it automatically executes a deceleration warning to ensure the controllability of the membrane penetration process. After the membrane penetration event is successfully identified, this embodiment quickly executes membrane penetration braking control, causing the robotic arm to stop advancing along the puncture direction, thereby preventing the puncture needle from penetrating too deeply and affecting the stability of the simulation training.

[0085] If the target area moves during the puncture process due to model elasticity or external force, the optical flow displacement characteristics can reflect this change in a timely manner. The system then performs path compensation to realign the puncture trajectory with the predetermined target, ensuring the consistency and repeatability of the experimental results.

[0086] In a simulated training scenario, the operator adjusts the end effector of the robotic arm to the puncture initiation position and starts the system of the present invention, so that the force sensing, image and pose acquisition units begin to collect data synchronously.

[0087] Subsequently, the feature extraction module calculates multimodal features in real time and updates the tissue layer identification results. During the advancement of the puncture needle, as the needle tip gradually approaches the simulated capsule layer, the system automatically decelerates and prompts the operator to pay attention to the membrane penetration process.

[0088] when Figure 4 The extreme characteristics of the rate of change of force shown are Figure 5 When the capsule rupture features shown appear simultaneously, the membrane penetration event recognition module determines that a membrane penetration event has occurred, and the control execution module immediately triggers a braking action to stop the puncture needle from penetrating too deeply into the model. When the movement of the model or probe causes a change in the tissue position in the image, the system promptly uses optical flow information for path compensation, allowing the needle tip to be re-aligned with the target. This process verifies the stability and effectiveness of the method of the present invention in simulated training scenarios and demonstrates that the present invention has good scene adaptability.

[0089] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion, characterized in that, Includes the following steps: S1. Signal Acquisition and Synchronization: Acquire force signals from the tip of the puncture needle, real-time ultrasound image sequences, and pose data from the robotic arm encoder. Timestamp-align signals at different sampling frequencies to form a synchronized dataset. S2. Feature extraction: Calculate the tissue stiffness, force change rate, and extreme value features of the force change rate during the puncture process based on the force signal; Texture features and tissue boundary features are extracted based on the ultrasound image sequence, and optical flow displacement of the tissue in front of the needle tip is obtained using an optical flow algorithm; S3, Tissue layer identification: The tissue stiffness, the force change rate and the extreme value feature are weighted and fused with the texture feature and the tissue boundary feature, and the tissue layer at which the puncture needle is currently located is identified according to the preset judgment rules. The tissue layer includes the skin layer, fat layer, muscle layer, capsule layer and renal parenchyma layer. S4. Perforation event identification: When the extreme value feature is detected at the same time as the fracture phenomenon of the hyperechoic capsule boundary in the ultrasound image, it is determined that the puncture needle has caused a capsule penetration event. S5. Event-driven control: The puncture control strategy is automatically adjusted based on the tissue level and the membrane penetration event. The control strategy includes constant speed advancement, rotation assistance, deceleration warning, membrane penetration braking, and path compensation based on the optical flow displacement.

2. The method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion according to claim 1, characterized in that, The method for extracting tissue stiffness in step S2 is as follows: , where k(t) represents the tissue stiffness, ΔF(t) is the force change during the sampling period, and Δx(t) is the corresponding puncture displacement change.

3. The method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion according to claim 1, characterized in that, The extreme value feature mentioned in step S2 is identified by monitoring the maximum rate of increase of the force change rate F′(t) and the subsequent maximum rate of decrease.

4. The method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion according to claim 1, characterized in that, In step S2, the texture features extracted from the ultrasound image sequence are used to extract contrast, energy, or homogeneity parameters through a gray-level co-occurrence matrix to distinguish the fat layer, the muscle layer, and the renal parenchyma.

5. The method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion according to claim 1, characterized in that, The optical flow algorithm described in step S2 is used to estimate the displacement change of the tissue in front of the needle tip between adjacent image frames.

6. The method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion according to claim 1, characterized in that, The determination of the organizational level in step S3 is based on the following formula: P=α·P force +β·P image Where P force P is the probability of judgment obtained based on force perception characteristics. image The probability of judgment is obtained based on image features, and α and β are dynamically adjustable weight coefficients.

7. The method for puncture tissue recognition and membrane perforation event control based on force sensing and image fusion according to claim 1, characterized in that, In step S5, a deceleration warning is triggered when the distance between the high-echo membrane boundary and the needle tip is detected to be lower than a preset threshold; path compensation control is triggered when the optical flow displacement is detected to exceed a compensation threshold.

8. A puncture tissue recognition and membrane perforation event control system based on force sensing and image fusion, characterized in that, include: Force acquisition unit: used to acquire force signals at the tip of the puncture needle and to transmit the force signals to the signal synchronization module; Image acquisition unit: used to acquire real-time ultrasound image sequences and to transmit the ultrasound image sequences to the signal synchronization module; Pose acquisition unit: used to acquire pose data of the robotic arm and to transmit the pose data to the signal synchronization module; Signal synchronization module: used to receive the force signal transmitted by the force acquisition unit, to receive the ultrasound image sequence transmitted by the image acquisition unit, to receive the pose data transmitted by the pose acquisition unit, to perform time alignment of signals with different sampling frequencies based on the sampling time of the force signal, the ultrasound image sequence and the pose data to generate a synchronized data set, and to transmit the data set to the feature extraction module. Feature extraction module: used to receive the data set transmitted by the signal synchronization module, used to calculate tissue stiffness, force change rate, and extreme features of force change rate, used to extract image texture features, tissue boundary features, and optical flow displacement, used to transmit the tissue stiffness, the force change rate, and the extreme features to the membrane penetration event recognition module, and used to transmit the texture features, the tissue boundary features, and the optical flow displacement to the tissue level recognition module; Tissue hierarchy identification module: used to receive the tissue stiffness, force change rate and its extreme value features, texture features, tissue boundary features and optical flow displacement transmitted by the feature extraction module, and to identify the current tissue hierarchy of the puncture needle based on multimodal feature fusion rules, and to transmit the identification results to the control execution module; Perforation event recognition module: used to receive the tissue stiffness, the force change rate and the extreme value feature transmitted by the feature extraction module, used to determine the perforation event when the extreme value feature and the capsule boundary breakage are detected at the same time, and used to transmit the determination result to the control execution module; Control execution module: Used to receive the results transmitted by the tissue hierarchy identification module and the membrane penetration event identification module, and to execute constant speed propulsion, rotation assistance, deceleration warning, membrane penetration braking and path compensation control according to the results.

9. A puncture tissue recognition and membrane perforation event control system based on force sensing and image fusion according to claim 8, characterized in that, The feature extraction module is configured to calculate the tissue stiffness based on the relationship between the force signal and the puncture displacement, in order to characterize the mechanical response characteristics of different tissues.

10. A puncture tissue recognition and membrane perforation event control system based on force sensing and image fusion according to claim 8, characterized in that, The feature extraction module is configured to calculate the optical flow displacement based on the spatial and temporal gradients of the ultrasound image sequence, which is used to characterize the displacement change of the tissue in front of the needle tip between adjacent image frames.

Images