Remote robot ultrasonic system and control method thereof

Abstract

The invention provides a remote robot ultrasonic system and a control method thereof, and belongs to the technical field of medical robots and ultrasonic detection. The system comprises a master terminal subsystem and a slave terminal subsystem which are connected through a network. The slave end subsystem is deployed on the patient side, comprises a six-axis mechanical arm, an ultrasonic acquisition instrument, a six-dimensional force / torque sensor and a slave end control computer, and is used for acquiring force sense, pose and ultrasonic image data and resolving rheological parameters of contact tissues in real time. The main terminal subsystem is deployed on a doctor side, comprises a force feedback operation device, a first main terminal control computer, a second main terminal control computer and a mixed reality display device, and is used for realizing local force field reconstruction and mixed reality scene rendering and display. The control method comprises a system initialization stage and a cycle control stage. According to the method, the influence of network time delay is reduced through parameterized force transmission, deep fusion of force sense and visual information is realized, and intelligent operation guidance based on image quality is provided, so that the stability of remote ultrasonic operation is improved.

Description

Technical Field

[0001] This invention relates to the field of medical robots and ultrasound testing technology, and in particular to a remote robotic ultrasound system and its control method. Background Technology

[0002] Remote ultrasound robot systems, by remotely replicating the doctor's scanning techniques, provide effective ultrasound diagnostic support in areas with limited medical resources, and have significant clinical value. However, ultrasound examinations heavily rely on the operator's precise control over the probe's contact force, trajectory, and angle to obtain clear diagnostic images. This necessitates that remote systems address core challenges such as the realism of force-sensory interaction, the intuitiveness of information presentation, and intelligent assistance during the operational process.

[0003] Current technical solutions mainly explore force feedback and visual enhancement, but significant limitations remain. Regarding force feedback, mainstream methods rely on the long-distance transmission of high-frequency raw force signals, making system performance extremely sensitive to network latency and jitter. This can easily lead to unstable feedback forces or even oscillations, making reliable application in complex network environments difficult. In terms of information presentation, visual information (such as ultrasound images and anatomical diagrams) and tactile information (such as the magnitude and direction of contact force) are usually displayed separately through different devices or interfaces. This forces operators to frequently switch their gaze and attention between different information sources, resulting in high cognitive load and poor immersion. Regarding operational guidance, existing navigation or assistance functions are mostly based on static geometric constraints or preset paths, lacking adaptability to the dynamic process of ultrasound imaging. They cannot intelligently guide operators to adjust probe posture to optimize imaging results based on real-time image quality.

[0004] Therefore, existing remote ultrasound robot systems have shortcomings in real-time stability of force feedback, fusion and presentation of multimodal information, and dynamic intelligent guidance, which restrict their operational efficiency and clinical usability. There is an urgent need for a comprehensive technical solution that can collaboratively address issues such as delayed force feedback, separation of visual and tactile information, and lack of intelligent guidance. Summary of the Invention

[0005] The purpose of this invention is to provide a remote robotic ultrasound system and its control method to solve the problems in the prior art, such as poor stability of force feedback due to network latency, high cognitive load on operators caused by the separation of visual and tactile information display, and lack of intelligent guidance based on real-time imaging quality. This enables remote ultrasound scanning operations with low latency, high immersion, and intelligent assistance.

[0006] To achieve the above objectives, the present invention provides a remote robotic ultrasound system, comprising a master terminal system and a slave terminal system connected via a network; Deploying the terminal system to the patient side includes: A six-axis robotic arm is used to carry and control the pose of the ultrasonic probe; An ultrasound acquisition device is used to acquire real-time ultrasound images; A six-dimensional force / torque sensor is installed between the end effector of a six-axis robotic arm and an ultrasonic probe to collect contact force and torque information in real time. The slave control computer is used to control the movement of the six-axis robotic arm, collect data from the six-dimensional force / torque sensor and ultrasonic acquisition instrument, and run environmental parameter identification algorithms to calculate the rheological parameters of the contact tissue in real time. The main terminal system is deployed on the doctor's side and includes: Force feedback operating device, with posture input and force output functions; The first master control computer is used to acquire input signals from the force feedback operating device, send motion commands to the slave control computer, receive rheological parameters and ultrasonic image data from the slave control computer, and run a local force field reconstruction algorithm to output force feedback to the force feedback operating device. Mixed reality display devices; The second master computer controls the running and rendering of the mixed reality interactive scene.

[0007] Preferably, the environmental parameter identification algorithm for the slave control computer is configured to: calculate the rheological parameters of the contact tissue, including instantaneous stiffness and viscosity coefficient, in real time based on the contact force data collected by the six-dimensional force / torque sensor and the pose data of the six-axis robotic arm; and transmit the rheological parameters to the master terminal system via the network at a first preset frequency.

[0008] Preferably, the local force field reconstruction algorithm run by the first master control computer is configured to: reconstruct the force field based on the received rheological parameters and the local displacement collected in real time by the force feedback operating device. and speed Calculates and outputs force feedback locally in real time. The calculation formula is: ; in, Indicates a reference position.

[0009] Preferably, the software system running on the second master control computer includes the following functional modules: The force vector visualization module is used to convert the received force information into a visual graphic overlay display; The ultrasound image visualization module is used to render real-time ultrasound images as spatially correlated augmented reality projections. The virtual fixture generation module is used to generate a virtual force field for operation guidance. The virtual force field is synthesized from the positioning guidance force based on the preoperatively calibrated target position and the quality-optimized guidance force based on the real-time ultrasound image quality assessment results.

[0010] Preferably, the force vector visualization module is configured to generate visualizations in at least one of the following forms: Vector arrow-shaped force indicators, whose direction, length, and color respectively encode the direction, magnitude, and safety status of the force; Linear instrument-type force indicator, whose fill height and color map normalize force values ​​and safety status; Force-thermal maps, drawn on the patient's body surface model using color-coded and radius-coded circular markers; HUD (Head-Up Display) shows force values ​​at a fixed position in the field of view.

[0011] Preferably, the ultrasound image visualization module is configured to: render real-time ultrasound images as semi-transparent augmented reality projection slices, the augmented reality projection slices being rigidly connected in space to the virtual ultrasound probe model; and to perform quality assessment on the ultrasound images, generating image quality color bars that reflect imaging confidence.

[0012] Preferably, the virtual fixture generation module is configured to: generate a positioning guidance force based on the deviation between the target scanning position obtained from preoperative calibration and the current position of the six-axis robotic arm end effector; simultaneously, generate a quality optimization guidance force to guide the probe pose toward the direction of image quality improvement based on the average confidence level and effective imaging area ratio calculated in real time by the ultrasound image visualization module; and synthesize the positioning guidance force and the quality optimization guidance force into a virtual fixture guidance force field according to preset weights. ; ; in, This indicates the position guiding force generated by the positioning fixture. This represents the quality optimization guidance force generated by the image quality servo. and These represent the weighting coefficients of the corresponding guiding force components.

[0013] The present invention also provides a control method for a remote robot ultrasonic system, comprising the following steps: Step S1: System initialization phase, establish communication connection between master terminal system and slave terminal system, and initialize each hardware device; Step S2, the loop control phase, after the system initialization is completed, periodically executes slave data acquisition and processing, master-slave data exchange, master-slave local force feedback and mixed reality rendering, and motion control based on master-slave operation.

[0014] Preferably, step S1 includes: Step S101: Establish a communication connection, start the slave control computer and the first master control computer, and establish a bidirectional communication link between the two through the DDS communication mechanism of the ROS2 framework. Step S102, Equipment initialization, includes: initializing the six-axis robotic arm, ultrasonic acquisition device, and six-dimensional force / torque sensor of the slave terminal system; and initializing the force feedback operation device and mixed reality display device of the master terminal system; Step S103: Coordinate system calibration. Perform spatial calibration of the coordinate system of the mixed reality display device and the base coordinate system of the six-axis robotic arm to establish the transformation relationship between the two.

[0015] Preferably, step S2 includes: Step S201: Data acquisition from the end: The end control computer acquires force data from the six-dimensional force / torque sensor, end pose data of the six-axis robotic arm, and real-time ultrasonic images from the ultrasonic acquisition device. Step S202: Environmental parameter identification. Based on the collected force and pose data, the slave control computer runs an environmental parameter identification algorithm to calculate the rheological parameters of the contact tissue in real time. Step S203: Data transmission, the calculated rheological parameters, real-time ultrasound images and probe pose data are transmitted from the slave control computer to the first master control computer; Step S204, Local Force Field Reconstruction and Force Feedback Output: The first master control computer calculates the force feedback value by running the local force field reconstruction algorithm based on the received rheological parameters and the local displacement and velocity collected from the force feedback operation device, and outputs the force feedback value to the force feedback operation device. Step S205: Motion command generation and transmission. The first master control computer generates motion control commands based on the pose input signal of the force feedback operating device and sends them to the slave control computer to control the movement of the six-axis robotic arm. Step S206: Mixed Reality Scene Rendering. The second master control computer executes the rendering task, including: generating force visualization graphics through the force vector visualization module; rendering augmented reality ultrasound projection and image quality color bars through the ultrasound image visualization module; and generating a virtual clamp guiding force field through the virtual clamp generation module. Step S207: Mixed Reality Display Output. The second master control computer transmits the rendered mixed reality scene to the mixed reality display device for display.

[0016] Therefore, the present invention employs the aforementioned remote robot ultrasonic system and its control method, and the beneficial technical effects are as follows: (1) This invention abandons the traditional direct transmission method of original high-frequency force signals and innovatively adopts a local force field reconstruction mechanism based on parameter transmission. Specifically, the slave end calculates the rheological parameters (such as instantaneous stiffness and viscosity coefficient) of the contact tissue in real time through an environmental parameter identification algorithm, and transmits these low-frequency parameter packets to the master end; the master end then uses the received parameters, combined with the locally acquired operation pose information, to calculate and generate force feedback locally. This method decouples the calculation of force feedback from the high-latency network transmission, so that the force response felt by the doctor is almost instantaneous, fundamentally eliminating the problems of force feedback lag, jitter, and even system oscillation caused by network latency. At the same time, the amount of transmitted data is greatly reduced, enhancing the robustness to network jitter and packet loss, enabling the system to operate stably in complex wide area network environments.

[0017] (2) This invention uses mixed reality (MR) technology to visualize and encode force information, and integrates it with ultrasound images in the same spatial field of view. The system provides various intuitive force visualization forms such as vector arrows, linear instruments, and force heat maps, and uses color gradients (such as green-yellow-red) to map the safety status. At the same time, real-time ultrasound images are projected into the operating field of view in the form of semi-transparent slices with spatial location association. This design transforms the originally separate tactile channel information into an enhanced element of the visual channel, allowing doctors to simultaneously perceive the magnitude, direction, safety status, and real-time imaging effect of the contact force without frequently switching their gaze between the screen, force feedback device, and patient site. This significantly reduces the cognitive load of multitasking and improves the immersion and efficiency of the operation.

[0018] (3) The virtual fixture technology based on image quality servoing proposed in this invention surpasses traditional navigation methods based on static geometric constraints. The system can not only generate positioning guidance forces pointing towards the target scanning area based on preoperative calibration, but also dynamically generate quality optimization guidance forces to optimize probe pose based on the quality assessment results of real-time ultrasound images (such as average confidence). The virtual guidance force field synthesized by both can intelligently assist doctors in actively adjusting the probe to a better imaging posture while rapidly locating the target. This mechanism, which integrates image quality feedback into the operation guidance closed loop, enables the system to adapt to individual differences in different patient tissues and dynamic changes during the scanning process, effectively assisting doctors in improving the success rate of the first scan. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall architecture of the remote robot ultrasound system provided in an embodiment of the present invention; Figure 2 This is a flowchart of a remote robot ultrasonic system control method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram showing the display effect of the force vector visualization module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of augmented reality projection for an ultrasound image visualization module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the working principle of the virtual fixture generation module provided in an embodiment of the present invention.

[0020] Figure Labels 101. Six-axis robotic arm; 102. Ultrasonic acquisition device; 103. Six-dimensional force / torque sensor; 104. Slave control computer; 201. Force feedback operation device; 202. First master control computer; 203. Mixed reality display device; 204. Second master control computer; 205. Force vector visualization module; 206. Ultrasonic image visualization module; 207. Virtual fixture generation module. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example 1 This embodiment provides a remote robotic ultrasound system, the overall architecture of which is as follows: Figure 1 As shown, the system consists of two main parts: a master terminal system and a slave terminal system connected via a network. It aims to build a "visual-tactile digital twin" to achieve deep integration of visual and tactile information, thereby reducing the cognitive load of remote ultrasonic scanning operations.

[0024] 1. From the terminal system.

[0025] The terminal system is deployed at the patient's location and is responsible for performing physical scanning actions and collecting raw data. Its specific components are as follows: The six-axis robotic arm 101 serves as the system's actuator, carrying the ultrasonic probe and enabling precise pose (position and attitude) control of it in three-dimensional space. The robotic arm connects to the slave control computer 104 via an Ethernet interface to receive motion control commands.

[0026] Ultrasound acquisition unit 102: Includes a color Doppler ultrasound diagnostic instrument and a linear array probe fixed to the end effector of a six-axis robotic arm 101. The linear array probe is fixedly connected to the end effector of the six-axis robotic arm 101 via a flange structure, and is used to scan the patient and acquire real-time ultrasound images. The ultrasound images are transmitted to the slave control computer 104 via a video acquisition card.

[0027] Six-dimensional force / torque sensor 103: Installed in series between the end effector of the six-axis robotic arm and the ultrasonic probe, it is used to acquire three-dimensional force and torque information generated by the contact between the probe and the body surface at a high frequency (not less than 500Hz) in real time. This sensor communicates with the slave control computer 104 via Ethernet.

[0028] The slave control computer 104 runs the robot operating system ROS2 and serves as the control core of the slave end. It is responsible for receiving motion commands from the master end and driving the six-axis robotic arm 101, synchronously acquiring force sensor data and ultrasonic images, and running the environmental parameter identification module 106. Based on the acquired force data, probe position, and velocity, the environmental parameter identification module 106 uses a Kalman filter to calculate the mechanical property parameters of the contact tissue in real time, i.e., rheological parameters, including instantaneous stiffness. Viscosity coefficient Relaxation time constant, etc.

[0029] 2. Main terminal system.

[0030] The main subsystem is deployed at the doctor's end, providing an operating interface, force feedback, and fusion information display. Its specific components are as follows: Force feedback manipulation device 201: Provides doctors with an interactive interface for six-degree-of-freedom pose input and three-degree-of-freedom force output. Doctors remotely control probe movement by manipulating its end effector and receive force signals fed back by the system. This device connects to the first master control computer 202 via a USB interface.

[0031] The first master control computer 202 also runs the ROS2 system. It is responsible for acquiring the pose input signals of the force feedback operating device 201 and sending corresponding motion control commands to the slave device via the network. Simultaneously, it receives the mechanical parameters transmitted from the slave device and runs a local force field reconstruction algorithm based on local operating information (displacement). ,speed ) and received mechanical parameters ( , ), calculate and output force feedback values ​​in real time. Force feedback device. The calculation formula is: ; in, Indicates a reference position.

[0032] Mixed Reality Display Device 203: This embodiment uses a Microsoft HoloLens 2 head-mounted display to present doctors with a virtual-real fusion operation scenario, including: an interactive menu bar, force vector visualization graphics, ultrasound image projection, and virtual gripper guidance signs.

[0033] The second master control computer 204 runs the Unity real-time rendering engine, responsible for the construction and rendering of mixed reality interactive scenes. The software system running on it includes core functional modules: Force Vector Visualization Module 205 Figure 3 As shown: The force vector visualization module 205 is used to convert the force information collected from the slave end into intuitive visual graphics and overlay them onto the mixed reality scene. This invention provides the following four configurable visualization methods, which operators can select or combine according to specific operating scenarios and personal preferences: 1) Vector Arrow Force Indicator: In the mixed reality display device 203, this indicator overlays and renders a dynamic three-dimensional force vector arrow on the virtual ultrasonic probe model to intuitively express the magnitude and direction of the contact force.

[0034] Its visual encoding rules are as follows: Directional coding: The direction of the arrow represents the direction vector of the current contact resultant force, which helps doctors determine whether the probe is subjected to excessive lateral force, thereby assessing the risk of slippage.

[0035] Length encoding: The length of the arrow is linearly related to the magnitude of the force; the greater the force, the longer the arrow. The system presets a maximum display length, which is the upper limit of the stress sensor's range.

[0036] Color and Status Coding: The color of the arrow is used to map the force value relative to a preset safety threshold, represented by a three-segment gradient from green, yellow to red. Specifically, it is green when the force value is within the safe range; it gradually turns yellow when the force value approaches the warning threshold; and when the force value exceeds a preset danger threshold (e.g., 10N, which may cause patient discomfort), the arrow turns red and triggers a flashing warning animation to provide a strong visual alert.

[0037] 2) Linear Instrument-Type Force Indicator: This indicator renders force information directly onto the virtual model surface of the ultrasound probe in the form of a dashboard. Compared to independent vector arrows, this method avoids additional 3D object occlusion and visual redundancy in complex mixed reality scenes.

[0038] Its visual encoding rules are as follows: Fill height encoding: The fill height of the instrument directly maps to the normalized force value, with a value ranging from 0 to 1. By default, it displays the force magnitude along the probe axis (i.e., the contact direction), but it can also be configured to display the magnitude of the three-dimensional resultant force.

[0039] Color coding: The color of the filled area is used to indicate the safety threshold status, using a three-segment gradient color band from blue, green to red, corresponding to the three status intervals of safe, warning and danger, respectively.

[0040] Reference line markings: Warning height reference lines and danger height reference lines are fixed on the instrument as intuitive visual references to help doctors quickly determine the safe range of the current force value.

[0041] 3) Force Heat Map: The force heat map "draws" force information in the form of a heat map on a virtual body surface model of the area to be scanned, forming an intuitive visualization of force distribution.

[0042] Its implementation mechanism is as follows: Contact point detection: Using Raycast technology, a detection ray is projected downwards from the virtual ultrasound probe position, and the coordinates of the intersection point between the ray and the patient's body surface model are calculated to accurately pinpoint the current contact position.

[0043] Color coding: At the detected contact point location, a circular thermal marker is rendered. The ring's color uses a gradient mapping from blue, green to red, representing a continuous change in force value from small to large.

[0044] Size coding: The radius of the ring is dynamically adjusted according to the force value; the greater the force, the larger the radius. The system also sets warning radii and danger radii as visual reference thresholds.

[0045] Historical Trajectory Function: Enables trajectory drawing, using texture mapping technology to draw the force movement trajectory on the patient's surface model in real time, creating a persistent thermal image. This function helps doctors review the force distribution history throughout the scanning process, facilitating operational review and scan quality assessment.

[0046] 4) HUD (Heads-Up Display): The HUD displays force information in digital and text form at a fixed position in the doctor's field of vision (such as the edge of the field of vision), serving as an effective supplement to the above-mentioned graphical display methods.

[0047] Its displayed content mainly includes: The precise numerical reading of the current force value.

[0048] The upper and lower limits of the recommended range are reference values.

[0049] Force unit identifier.

[0050] Its display features include support for Billboard display mode, meaning the orientation of the display panel will always be dynamically adjusted to face the user's viewpoint, thus ensuring that all text information remains upright and readable when the doctor turns their head.

[0051] Ultrasound image visualization module 206 Figure 4As shown: This module is used to present real-time ultrasound images in a mixed reality environment and provide real-time visual evaluation of image quality. Specifically, it includes two core functional components: augmented reality ultrasound projection and an image quality color bar.

[0052] 1) Augmented Reality Ultrasound Projection: This function overlays a vertical slice of real-time ultrasound image onto the mixed reality display device 203 below the virtual ultrasound probe model. This allows doctors to view the ultrasound imaging results intuitively and spatially while observing the actual operating scene.

[0053] Its rendering has the following characteristics: Spatial positioning: The position and orientation of the ultrasound image slice plane are rigidly connected to the virtual ultrasound probe and change synchronously with the movement and rotation of the probe, thereby accurately simulating the spatial geometric relationship between the image and the probe in real ultrasound imaging.

[0054] Transparent rendering: Image slices are rendered using a semi-transparent material, allowing doctors to observe real scene elements (such as the patient's body surface or anatomical landmarks) that are obscured by the ultrasound image to a certain extent, thus avoiding the virtual image completely blocking the operating field of vision.

[0055] Display mode: Supports Billboard display mode, which means that the image plane can automatically adjust its orientation to always face the user's viewpoint, thereby ensuring that doctors can obtain clear, frontal ultrasound images from any viewing angle and guaranteeing the best image readability.

[0056] 2) Image Quality Color Bar: The image quality color bar function performs online quality assessment of real-time ultrasound images and presents the assessment results instantly in an intuitive color-coded manner, helping doctors quickly determine whether the current imaging quality meets the requirements.

[0057] Its implementation mechanism is as follows: Sampling mechanism: Each frame of ultrasound image slices is uniformly sampled along the column direction, and the average brightness value of the pixels in each sampled column is extracted. This brightness value is then used as the image quality index for that local area. Generally, a higher brightness value indicates better penetration and reflection of ultrasound signals in that area, resulting in higher imaging confidence.

[0058] Color mapping: The sampled brightness values ​​are mapped to visual color codes using a three-segment gradient color band from red, yellow to green, corresponding to three quality levels: low confidence, medium confidence, and high confidence, respectively. The generated color bars are displayed along the lateral edges (such as the top or bottom) of the ultrasound image slices, thus forming an intuitive lateral distribution indicator of image quality.

[0059] Temporal smoothing: A linear interpolation (Lerp) algorithm is used to smooth the color values ​​of each sampling point between adjacent video frames in the time domain. This process effectively eliminates color flickering caused by image noise or instantaneous signal fluctuations, significantly improving the stability and continuity of the color bar's visual presentation.

[0060] Dynamic range normalization: The system detects the minimum and maximum brightness values ​​of the entire current frame of the ultrasound image in real time and dynamically adjusts the input brightness range of the color map accordingly. This adaptive mechanism enables real-time contrast enhancement, ensuring that the color bars provide clear and effective quality differentiation under different patients, different imaging parameters, or different tissue characteristics.

[0061] Virtual fixture generation module 207 Figure 5 As shown: Used to generate a guiding force field for intelligent operation.

[0062] 1) Positioning clamp: The positioning clamp generates a positional guiding force based on the preoperative calibration and registration results, which is designed to help doctors move the ultrasound probe quickly and accurately to the predetermined target scanning area.

[0063] Its working principle is as follows: Target location acquisition: During the preoperative preparation stage, spatial calibration and registration algorithms are used to determine the three-dimensional coordinates of the target anatomical structure in the mixed reality coordinate system. .

[0064] Horizontal guiding force: when the current position of the ultrasound probe With the target location When a deviation exists in the horizontal plane (XY plane), the system generates a virtual guiding force pointing towards the target position. This guiding force assists the doctor in moving the probe across the patient's body surface (X, Y directions) towards the target location. The magnitude of the guiding force is proportional to the deviation between the current and target locations, and its direction always points towards the target location.

[0065] Contact direction positioning: Once the probe's horizontal position is close to the target, the doctor applies appropriate pressure in the contact direction (Z-axis). During this stage, the doctor uses force feedback provided by the system to perceive the tissue's depth and mechanical response, thus achieving precise positioning of the scan depth. Afterward, the doctor can focus on fine-tuning the probe's posture and further optimizing the ultrasound image quality.

[0066] 2) Image quality servo: The image quality servo function evaluates the quality of ultrasound imaging in real time and uses the evaluation results to dynamically guide the generation of virtual fixture force fields, thereby achieving intelligent guidance based on imaging effects.

[0067] Its implementation mechanism includes: Confidence Map Calculation: The system calculates a confidence map for each frame of real-time ultrasound image. This confidence map reflects the imaging quality level of each region in the image, and its calculation is mainly based on the brightness distribution characteristics of the image along the column direction.

[0068] Quality metric extraction: Global image quality assessment metrics are extracted from the generated confidence map. These metrics may include parameters such as average confidence, the proportion of effective imaging area, and the proportion of sound-shadow area.

[0069] Guidance strategy generation: Based on the real-time changes in the aforementioned image quality indicators, the system dynamically adjusts the direction and magnitude of the guiding force generated by the virtual fixture. Its core purpose is to guide the doctor to adjust the probe's pose in a direction that improves image quality.

[0070] 3) Virtual Fixture Force Field Synthesis: The virtual fixture module vector synthesizes the position guiding force generated by the aforementioned positioning fixture with the quality optimization guiding force generated by the image quality servo, generating the final, unified virtual fixture guiding force field. .

[0071] Its synthesis strategy is defined by the following formula: ; in, This indicates the position guiding force generated by the positioning fixture. This indicates the quality optimization guidance force generated by the image quality servo. and These represent the weighting coefficients of the corresponding guiding force components. These two coefficients can be dynamically adjusted according to different operational stages (such as the coarse positioning stage or the fine optimization stage) and the doctor's personal operational preferences, thereby achieving flexible and adaptive operational guidance.

[0072] 3. System communication architecture.

[0073] The system components are connected in the following ways: Remote communication between master and slave: The slave control computer 104 and the first master control computer 202 establish bidirectional communication through the ROS2 DDS (Data Distribution Service) mechanism to transmit motion commands, mechanical parameter packets, ultrasound image streams and status data.

[0074] Local communication between master terminals: The first master terminal control computer 202 and the second master terminal control computer 204 are connected via a local area network TCP connection to achieve data sharing.

[0075] Mixed Reality Rendering Communication: The second master control computer 204 and the mixed reality display device 203 are connected via WiFi and use the Holographic Remoting protocol to transmit low-latency rendered video streams and interactive data.

[0076] like Figure 2 As shown, the control method of the above system is as follows: Step S1: System initialization phase. Establish communication connection between master terminal system and slave terminal system, and initialize each hardware device.

[0077] Step S101: Establish a communication connection, start the slave control computer 104 and the first master control computer 202, and establish a bidirectional communication link between the two through the DDS communication mechanism of the ROS2 framework. Step S102, device initialization, includes: initializing the six-axis robotic arm 101, ultrasonic acquisition device 102 and six-dimensional force / torque sensor 103 of the slave terminal system; and initializing the force feedback operation device 201 and mixed reality display device 203 of the master terminal system; Step S103: Coordinate system calibration. Perform spatial calibration of the coordinate system of the mixed reality display device 203 and the base coordinate system of the six-axis robotic arm 101 to establish the transformation relationship between the two.

[0078] Step S2, the loop control phase, after the system initialization is completed, periodically executes slave data acquisition and processing, master-slave data exchange, master-slave local force feedback and mixed reality rendering, and motion control based on master-slave operation.

[0079] Step S201: Data acquisition from the end: The end control computer 104 acquires force data from the six-dimensional force / torque sensor 103, end pose data from the six-axis robotic arm 101, and real-time ultrasonic images from the ultrasonic acquisition instrument 102. Step S202: Environmental parameter identification. Based on the collected force and pose data, the slave control computer 104 runs the environmental parameter identification algorithm to calculate the rheological parameters of the contact tissue in real time. Step S203: Data transmission, the calculated rheological parameters, real-time ultrasound images and probe pose data are transmitted from the slave control computer 104 to the first master control computer 202; Step S204, Local Force Field Reconstruction and Force Feedback Output: The first master control computer 202 calculates the force feedback value by running the local force field reconstruction algorithm based on the received rheological parameters and the local displacement and velocity collected from the force feedback operation device 201, and outputs the force feedback value to the force feedback operation device 201. Step S205: Motion command generation and transmission. The first master control computer 202 generates motion control commands based on the pose input signal of the force feedback operation device 201 and sends them to the slave control computer 104 to control the movement of the six-axis robotic arm 101. Step S206: Mixed Reality Scene Rendering. The second master control computer 204 executes the rendering task, including: generating force visualization graphics through the force vector visualization module 205; rendering augmented reality ultrasound projection and image quality color bars through the ultrasound image visualization module 206; and generating a virtual clamp guiding force field through the virtual clamp generation module 207. Step S207: Mixed Reality Display Output. The second master control computer 204 transmits the rendered mixed reality scene to the mixed reality display device 203 for display.

[0080] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0081] Therefore, the present invention adopts the above-mentioned remote robot ultrasound system and its control method, realizes local low-latency force perception reconstruction at the master end by transmitting rheological parameters, presents force perception visualization information and ultrasound images through mixed reality fusion, and generates intelligent guidance by combining real-time image quality assessment. In this way, the problems of poor force feedback stability, separation of visual and tactile information and lack of intelligence in operation guidance are solved in a coordinated manner at the system level, which effectively improves the operation efficiency, immersion and imaging quality of remote ultrasound scanning.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A remote robotic ultrasound system, characterized in that, This includes master and slave terminal systems connected via a network; Deploying the terminal system to the patient side includes: A six-axis robotic arm is used to carry and control the pose of the ultrasonic probe; An ultrasound acquisition device is used to acquire real-time ultrasound images; A six-dimensional force / torque sensor is installed between the end effector of a six-axis robotic arm and an ultrasonic probe to collect contact force and torque information in real time. The slave control computer is used to control the movement of the six-axis robotic arm, collect data from the six-dimensional force / torque sensor and ultrasonic acquisition instrument, and run environmental parameter identification algorithms to calculate the rheological parameters of the contact tissue in real time. The main terminal system is deployed on the doctor's side and includes: Force feedback operating device, with posture input and force output functions; The first master control computer is used to acquire input signals from the force feedback operating device, send motion commands to the slave control computer, receive rheological parameters and ultrasonic image data from the slave control computer, and run a local force field reconstruction algorithm to output force feedback to the force feedback operating device. Mixed reality display devices; The second master computer controls the running and rendering of the mixed reality interactive scene.

2. The remote robotic ultrasound system according to claim 1, characterized in that, The environmental parameter identification algorithm for the slave control computer is configured to: calculate the rheological parameters of the contact tissue in real time, including instantaneous stiffness and viscosity coefficient, based on the contact force data collected by the six-dimensional force / torque sensor and the pose data of the six-axis robotic arm; and transmit the rheological parameters to the master terminal system via the network at a first preset frequency.

3. The remote robotic ultrasound system according to claim 2, characterized in that, The local force field reconstruction algorithm running on the first master control computer is configured to: based on the received rheological parameters and the local displacement collected in real time by the force feedback operating device. and speed Calculates and outputs force feedback locally in real time. The calculation formula is: ； in, Indicates a reference position.

4. The remote robotic ultrasound system according to claim 1, characterized in that, The software system running on the second master control computer includes the following functional modules: The force vector visualization module is used to convert the received force information into a visual graphic overlay display; The ultrasound image visualization module is used to render real-time ultrasound images as spatially correlated augmented reality projections. The virtual fixture generation module is used to generate a virtual force field for operation guidance. The virtual force field is synthesized from the positioning guidance force based on the preoperatively calibrated target position and the quality-optimized guidance force based on the real-time ultrasound image quality assessment results.

5. A remote robotic ultrasound system according to claim 4, characterized in that, The force vector visualization module is configured to generate visualizations in at least one of the following forms: Vector arrow-shaped force indicators, whose direction, length, and color respectively encode the direction, magnitude, and safety status of the force; Linear instrument-type force indicator, whose fill height and color map normalize force values ​​and safety status; Force-thermal maps, drawn on the patient's body surface model using color-coded and radius-coded circular markers; HUD (Head-Up Display) shows force values ​​at a fixed position in the field of view.

6. A remote robotic ultrasound system according to claim 4, characterized in that, The ultrasound image visualization module is configured to: render real-time ultrasound images as semi-transparent augmented reality projection slices, which are rigidly connected to the virtual ultrasound probe model in space; and to perform quality assessment on the ultrasound images, generating image quality color bars that reflect the imaging confidence level.

7. A remote robotic ultrasound system according to claim 4, characterized in that, The virtual fixture generation module is configured to: generate a positioning guidance force based on the deviation between the target scanning position obtained from preoperative calibration and the current position of the six-axis robotic arm end; and simultaneously, generate a quality optimization guidance force to guide the probe pose toward the direction of image quality improvement based on the average confidence level and effective imaging area ratio index calculated in real time by the ultrasound image visualization module. The positioning guidance force and the quality optimization guidance force are combined according to preset weights to form a virtual fixture guidance force field. ; ； in, This indicates the position guiding force generated by the positioning fixture. This represents the quality optimization guidance force generated by the image quality servo. and These represent the weighting coefficients of the corresponding guiding force components.

8. A control method for a remote robotic ultrasonic system, characterized in that, Includes the following steps: Step S1: System initialization phase, establish communication connection between master terminal system and slave terminal system, and initialize each hardware device; Step S2, the loop control phase, after the system initialization is completed, periodically executes slave data acquisition and processing, master-slave data exchange, master-slave local force feedback and mixed reality rendering, and motion control based on master-slave operation.

9. The control method for a remote robot ultrasonic system according to claim 8, characterized in that, Step S1 includes: Step S101: Establish a communication connection, start the slave control computer and the first master control computer, and establish a bidirectional communication link between the two through the DDS communication mechanism of the ROS2 framework. Step S102, Equipment initialization, includes: initializing the six-axis robotic arm, ultrasonic acquisition device, and six-dimensional force / torque sensor of the slave terminal system; and initializing the force feedback operation device and mixed reality display device of the master terminal system; Step S103: Coordinate system calibration. Perform spatial calibration of the coordinate system of the mixed reality display device and the base coordinate system of the six-axis robotic arm to establish the transformation relationship between the two.

10. The control method for a remote robot ultrasonic system according to claim 8, characterized in that, Step S2 includes: Step S201: Data acquisition from the end: The end control computer acquires force data from the six-dimensional force / torque sensor, end pose data of the six-axis robotic arm, and real-time ultrasonic images from the ultrasonic acquisition device. Step S202: Environmental parameter identification. Based on the collected force and pose data, the slave control computer runs an environmental parameter identification algorithm to calculate the rheological parameters of the contact tissue in real time. Step S203: Data transmission, the calculated rheological parameters, real-time ultrasound images and probe pose data are transmitted from the slave control computer to the first master control computer; Step S204, Local Force Field Reconstruction and Force Feedback Output: The first master control computer calculates the force feedback value by running the local force field reconstruction algorithm based on the received rheological parameters and the local displacement and velocity collected from the force feedback operation device, and outputs the force feedback value to the force feedback operation device. Step S205: Motion command generation and transmission. The first master control computer generates motion control commands based on the pose input signal of the force feedback operating device and sends them to the slave control computer to control the movement of the six-axis robotic arm. Step S206: Mixed Reality Scene Rendering. The second master control computer executes the rendering task, including: generating force visualization graphics through the force vector visualization module; rendering augmented reality ultrasound projection and image quality color bars through the ultrasound image visualization module; and generating a virtual clamp guiding force field through the virtual clamp generation module. Step S207: Mixed Reality Display Output. The second master control computer transmits the rendered mixed reality scene to the mixed reality display device for display.

Images