Spatial information driven embodied ultrasonic robotic autonomous scanning method

Abstract

This invention presents a spatial information-driven autonomous scanning method for embodied ultrasound robots, belonging to the field of ultrasound motion planning. Existing ultrasound scanning methods based on handheld operation and offline planning generally suffer from drawbacks such as reliance on operator experience, high operational workload, insufficient real-time adjustment capabilities, and poor repeatability, leading to limited examination efficiency, difficulty in ensuring result consistency, and continuously increasing pressure on the supply of high-quality medical resources. The proposed method integrates spatial positioning, environmental perception, and scanning strategy generation to achieve online closed-loop control of the scanning path, probe posture, and contact state. During the examination, it can adaptively adjust in real time based on tissue deformation, changes in patient position, and image quality feedback, thereby significantly reducing reliance on individual physician techniques and improving scanning standardization and result stability.

Description

Technical Field

[0001] This invention belongs to the field of ultrasonic detection motion planning, and particularly relates to a spatial information-driven autonomous scanning method for embodied ultrasonic robots. Background Technology

[0002] Choosing the appropriate scanning path is crucial for visualizing target objects or locating lesions in human tissues during ultrasound examinations. Existing autonomous scanning technologies can be broadly categorized into offline planning and online planning. Offline planning is a commonly used method in robotic ultrasound systems. The algorithm primarily involves first reconstructing the scanning area in 3D and then planning the scanning path on top of it. Hennersperger et al. validated the feasibility of autonomously transferring planned scanning paths to the current settings based on registration between MRI and 3D surface point clouds acquired by a Kinect camera. Jiang et al. proposed a non-rigid registration based on a skeleton map, which is performed between cartilage point clouds extracted from a tomographic template and the patient's ultrasound image. To further improve registration accuracy, they introduced a dense skeleton map to replace manually designed skeleton key points. However, these methods all require pre-setting the scanning location and region, and their generalization and flexibility fall far short of the needs of actual ultrasound examinations, without significantly reducing the workload of ultrasound physicians.

[0003] Online methods can generate more flexible trajectories than offline methods and can adjust in real time according to changes in the actual situation. Milletari et al. proposed a preliminary work using a deep DQN architecture to autonomously find the correct acoustic window for cardiac examination in real time. Jiang et al. proposed a method that enables an ultrasound robot to automatically perform ultrasound screening of tubular structures based solely on real-time ultrasound image feedback. By activating a U-Net network, cross-sectional vascular cavities are continuously segmented from ultrasound images, and the centerline direction of the local vessels is estimated in real time and the robot moves in that direction. However, these methods are limited in that they require a large amount of data for training for each scanning task, and their effectiveness decreases when switching to a different site. Furthermore, they are only applicable to relatively simple scenarios and cannot handle long-process ultrasound scanning tasks with complex structures.

[0004] Unlike existing methods, imitation learning methods based on the transformer architecture have demonstrated the ability to learn long-range contextual information in other fields. They can also promptly perceive and react to changes in the target. Furthermore, their end-to-end design allows them to learn the potential relationships between inputs and outputs across modalities. Works such as RT-1, InstructRL, GATO, and BC-Z were among the first to apply this architecture to robotics. Instead of the traditional object-centric perspective, they learn and reason from a motion-centric perspective. However, these models are mostly designed for simple actions like grasping and pushing / pulling, and have not been applied to complex scenarios such as ultrasonic testing. Moreover, these models directly predict end-effector positions based on 2D image or video input. In entirely new environments, they require extensive demonstrations to fit the space and perform inference, resulting in high training costs. The introduction of 3D feature information enables models to understand three-dimensional spatial information; works of this type include PerAct, C2F-ARM, and Act3D. PerAct and C2F-ARM perceive 3D space by voxelizing the robot's motion space, while Act3D improves computational efficiency by combining 2D features with depth information through a pre-trained model. While these methods acquire 3D information, they neglect the potential mapping relationship between the probe orientation and the superficial surface of the human body. This information is a core factor determining image quality in ultrasound scanning tasks. Summary of the Invention

[0005] This invention aims to address three core challenges faced by ultrasound robots in autonomously planning scans of multiple body parts: first, the lack of real-time adjustment and optimization capabilities when the human body experiences local vibrations or deformations; second, the lack of three-dimensional spatial information in existing RGB image-based methods, resulting in poor accuracy of vertical synchronous scanning of the skin at the contact point by the probe; and third, the weak perception of motion direction in existing methods, making them prone to getting stuck in local loops during long-cycle scanning tasks. To address these challenges, this invention proposes a spatial information-driven transformer architecture-based online autonomous scanning method for ultrasound robots. This method enables highly robust, high-precision, fully autonomous, real-time, vertically aligned, and high-quality detection of complex human body structures during long-cycle scanning tasks.

[0006] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0007] A spatial information-driven autonomous scanning method for an embodied ultrasound robot includes the following steps:

[0008] Step 1: Acquire sensor data input; the input information includes point cloud data generated by a camera that follows the movement of the robotic arm through a pseudo point cloud generation algorithm, and 6DOF motion information of the ultrasonic scanning probe placed at the end of the robotic arm;

[0009] Step 2: Point cloud feature extraction and embedding representation; The original point cloud frame is voxelized, then uniformly transformed to the same world coordinate system, and finally a block-based point cloud mask modeling strategy is used to embed the point cloud to obtain point cloud features;

[0010] Step 3: Spatial action temporal perception and multimodal coupling; First, the 6DOF action information is encoded to obtain action features; then, the point cloud features and action features are input into the spatial action collaborative perception module, and a dual-branch interactive neural network structure is used to couple and model the point cloud features and action features in the same feature space to obtain multimodal observation information;

[0011] Step 4: Historical information embedding; Based on the continuous temporal attention embedding strategy, aggregate point cloud-action coordination information within the historical time window, and output a fused feature vector containing spatiotemporal evolution context;

[0012] Step 5: Using the fused feature vector and multimodal observation information within a continuous time window as conditional inputs, model and predict the robot's future action sequence through a policy network;

[0013] Furthermore, step 1 is detailed as follows:

[0014] During the training phase, point cloud data, probe pose, and corresponding ultrasound image quality information are collected. The quality of the collected ultrasound images and the accuracy of the trajectory are scored, and low-scoring data are removed. Subsequently, the point cloud data and motion data are converted into the data format given by the model and a dataset is constructed.

[0015] Furthermore, step 2 is detailed as follows:

[0016] Step 2.1: Voxelize the original point cloud frame using the stacked voxel feature encoding method;

[0017] Step 2.2: Calculation of voxel features weighted by reflection intensity; point cloud data from different camera coordinate systems are uniformly transformed to the same world coordinate system. During the coordinate transformation process, the reflection intensity of the point cloud is introduced for weighted processing.

[0018] Step 2.3: Point cloud mask modeling strategy; First, randomly select a sub-point cloud from the original point cloud, then determine its j adjacent sub-point clouds to construct a spatially continuous region; subsequently, use this continuous region as the mask region, whose index set... Represented as Among them, the mask index set The number of elements inside is determined by the mask ratio. The product of the total number of sub-point clouds, g, determines the outcome.

[0019] Next, all point cloud features within the masked region are replaced with learnable predefined mask embedding vectors; the final point cloud data used for feature extraction is represented as follows:

[0020]

[0021] This represents the point cloud feature vector at the i-th position in the input sequence. The spatiotemporal location encoding vector represents the i-th position.

[0022] Furthermore, the spatial motion collaborative perception module couples and models point cloud features and motion features in the same feature space through a cross-attention layer.

[0023] Furthermore, step 4 is detailed as follows:

[0024] First, a temporal sequence is constructed: the feature sequence within a finite historical time window before the current moment is obtained, and a temporal tensor containing point cloud feature sets and action feature sets is constructed.

[0025] Then, collaborative feature encoding is performed: the point cloud features and action features at the same time are concatenated to generate the collaborative perception vector at that time.

[0026] Next, attention weighting enhancement is performed: the influence weight of the collaborative perception vector at different historical moments on the current decision is calculated using the self-attention mechanism, and the information within the window is aggregated by weighted summation;

[0027] The final output is a fused feature vector containing the spatiotemporal evolution context, which is used to guide the generation of actions in the next time step.

[0028] Furthermore, step 5 is detailed as follows:

[0029] Latent variables are obtained by using an approximate posterior distribution based on fused feature vectors and multimodal observation information; the multimodal observation information and latent variables are input into the policy network to generate future action sequence blocks, which serve as candidate execution trajectories for the next stage;

[0030] The future action sequence blocks are weighted and fused using an exponential decay method to obtain the final execution trajectory;

[0031] During the training phase, the predicted action is compared with the actual action, and the prediction error is calculated using a loss function. The loss function introduces a lower bound of evidence as an optimization objective, and indirectly optimizes the distribution of latent variables and model parameters by maximizing the log-likelihood of the observed data.

[0032] Furthermore, during the training phase, end-to-end optimization is performed using the following total loss function:

[0033]

[0034] in, The operator represents the mathematical expectation under the posterior distribution. Representing latent variables, Represents the fused feature vector. Represents multimodal observation information, Indicates by parameters The conditional generative decoding distribution fitted by the policy network with weights; This represents the Kullback-Leibler divergence operator. Indicates by parameters The posterior variational inference distribution fitted by the weighted neural network. As a standard prior boundary for latent space parameter regularization This represents the hyperparameter balance coefficient.

[0035] The beneficial effects of this invention are as follows:

[0036] Existing ultrasound scanning methods based on handheld operation and offline planning generally face pain points such as reliance on operator experience, high operational workload, insufficient real-time adjustment capabilities, and poor repeatability, leading to limited examination efficiency, difficulty in ensuring result consistency, and continuously increasing pressure on the supply of high-quality medical resources. To address these issues, this invention proposes a spatial information-driven embodied ultrasound robot autonomous scanning method. By integrating spatial positioning, environmental perception, and scanning strategy generation, it achieves online closed-loop control of the scanning path, probe posture, and contact state. During the examination, it can adaptively adjust in real time based on tissue deformation, changes in patient position, and image quality feedback, thereby significantly reducing reliance on individual physician techniques and improving scanning standardization and result stability.

[0037] From a social perspective, this method helps improve the accessibility and uniformity of ultrasound examinations: on the one hand, by lowering the operational threshold, it reduces the rigid dependence of primary care institutions on senior ultrasound physicians, promoting the downward flow of high-quality diagnostic capabilities; on the other hand, by improving the consistency and repeatability of examinations, it reduces missed scans and duplicate examinations, improving diagnostic and treatment safety and the patient's medical experience. Simultaneously, robotic autonomous scanning can significantly reduce the musculoskeletal load and occupational injury risks caused by physicians holding the probe for extended periods and making repeated adjustments, allowing physicians to devote more energy to critical image interpretation and clinical decision-making, thus improving the overall quality of medical services.

[0038] From an economic perspective, this invention can effectively increase the throughput of a single device and the operational efficiency of a department, shorten examination time, and reduce rework rates, resulting in direct savings in labor costs and increased output per unit time. In multi-departmental collaborative scenarios, it can reduce queuing and bedside examination scheduling costs, improving the utilization rate of hospital resources. At the industry level, this method promotes the upgrade of ultrasound equipment from a "tool-type" to an "intelligent and automated system," driving collaborative innovation in robotics, navigation and positioning, intelligent control, and medical imaging algorithms. This leads to replicable product and service models with broad application prospects in scenarios such as health check centers, emergency bedside care, telemedicine, and primary care screening, comprehensively improving the operational efficiency of the healthcare system and social health benefits. Attached Figure Description

[0039] Figure 1 This is a diagram of the action space collaborative perception structure of the present invention;

[0040] Figure 2 This is a schematic diagram of the experimental platform in a specific embodiment of the present invention;

[0041] Figure 3 This is a structural design drawing of an ultrasonic probe.

[0042] Figure 4 This is a schematic diagram of the contact force change curve during carotid artery scanning.

[0043] Figure 5 This is a schematic diagram of the joint coordinate system and base coordinate system of the robotic arm. Detailed Implementation

[0044] To better understand the purpose, structure, and function of this invention, the following detailed description, in conjunction with the accompanying drawings, provides a spatial information-driven autonomous scanning method for an embodied ultrasonic robot.

[0045] like Figure 1 As shown, the ultrasonic scanning robot system mainly consists of data input, point cloud feature extraction, spatial motion collaborative perception, historical time sequence embedding, and motion generation. The input information includes spatial information of observed superficial human tissue and motion information of the robotic arm's end effector. Secondly, point cloud feature extraction achieves compression and representation of the massive point cloud while retaining key information by extracting 3D point cloud data. Then, the point cloud features and motion features output from the motion encoder are input to the spatial motion collaborative perception module to couple the two modalities together. Next, the model combines current information and historical time sequence information to perceive the motion process, predict and generate the next sequence of motion blocks, and then hand them over to the controller for execution. The specific steps are as follows:

[0046] Step 1: Sensor data input; The system input information includes environmental information of the target area during the scanning process and the robot's own motion information. It mainly includes dense point cloud generated by the RGB-D camera that follows the movement of the robotic arm through a pseudo point cloud generation algorithm, 6DOF information (including position and attitude) of the ultrasonic scanning probe placed at the end of the robotic arm, and optional velocity / joint state information.

[0047] During the training phase, experienced ultrasound physicians remotely controlled a robotic arm to simulate the ultrasound scanning process in a hospital, collecting point cloud data, probe pose, and corresponding ultrasound image quality information. The collected ultrasound images were scored based on their quality and trajectory accuracy, and low-scoring data were discarded. The point cloud data and motion data were then converted into the HDF5 format provided by the model to construct a dataset.

[0048] Furthermore, before data acquisition, hardware calibration and coordinate system setup are required; specifically, such as... Figure 5 As shown, an external and internal parameter calibration relationship is established between the RGB-D camera and the robotic arm, and a rigid body transformation matrix is ​​established between the world coordinate system, the camera coordinate system, the robotic arm base coordinate system, and the end effector coordinate system; during execution, point clouds from different perspectives (first person and third person) are uniformly transformed to the world coordinate system.

[0049] Step 2: Point cloud feature extraction and embedding representation.

[0050] Step 2.1: 3D point clouds can provide spatial structural information that RGB images lack; however, directly inputting raw point cloud data into the model for processing will impose a significant computational burden, thereby significantly reducing computational efficiency. To address this issue, this invention employs a series of point cloud downsampling and structuring techniques to reduce computational complexity while preserving key information.

[0051] Specifically, the original point cloud frames are voxelized using the Stacked Voxel Feature Encoding (SVFE) method. This process first divides each frame of point cloud data into equally spaced voxel units according to a preset spatial resolution.

[0052] Subsequently, the centroid of the point cloud data within each voxel unit is calculated, and the corresponding entire voxel block is represented by the centroid, thereby achieving a spatial dimensionality reduction representation of the point cloud data.

[0053] Step 2.2: Calculation of Voxel Features Weighted by Reflection Intensity. Given that the point cloud data originates from sensor devices with different viewpoints, including a first-person view camera fixed to the robotic arm and a third-person view camera positioned to the side of the robotic arm, it is necessary to uniformly transform the point cloud data from different camera coordinate systems to the same world coordinate system. During the coordinate transformation process, point cloud reflection intensity information is introduced as a weighting factor to enhance the representation capability of key regions.

[0054] By introducing reflection intensity into the voxel calculation process for weighting, points with higher reflection intensity occupy higher weights in feature representation. This effectively avoids the loss of important information while keeping the point cloud data lightweight, and ensures that the extracted voxel features are positively correlated with the reflection intensity information.

[0055] Step 2.3: Surface Perception Ability; During actual testing, it was found that the model is highly sensitive to subtle changes in point cloud data. Even small changes in local details can significantly affect the model's output. For example, in a breast scan task, the orientation of the human head in the point cloud (e.g., face upwards or sideways) can influence the model's judgment result. This factor itself is not directly related to the target determination task and theoretically should not affect the decision result. Therefore, improving the accuracy and generalization ability of the point cloud feature extraction process is of great significance.

[0056] Based on the above problems and drawing on the masking modeling ideas in BERT and BEiT, this invention proposes a point masking modeling strategy to ensure that the feature vectors input to subsequent networks can accurately represent the essential features of the point cloud, thereby reducing the interference of task-irrelevant factors on model decisions.

[0057] Unlike the random masking method used in BERT, this invention employs a block-based masking strategy. Specifically, firstly, a sub-point cloud is randomly selected from the original point cloud, and then its j neighboring sub-point clouds are determined to construct a spatially continuous region; subsequently, this continuous region is used as the masking region, and its index set... Represented as

[0058] Among them, the mask index set The number of elements inside is determined by the mask ratio. Total number of point clouds The product determines the product.

[0059] Next, all point cloud features within the masked region are replaced with learnable predefined mask embedding vectors. Finally, the point cloud data used for feature extraction can be represented as:

[0060]

[0061] This represents the point cloud feature vector at the i-th position in the input sequence. Represents the set of mask regions, The spatiotemporal location encoding vector represents the i-th position.

[0062] Step 3: Spatial Motion Sequence Perception and Multimodal Coupling; Previous work often focused on learning motion, merely treating environmental information as a trigger signal, neglecting the implicit mapping relationship between the two. When a doctor holds a handheld ultrasound scanner, they plan the optimal overall scanning trajectory through their spatial perception, which implies an "ultrasound language" that combines environmental and motion information—this is what the model should learn. Based on this idea, this module is specifically designed to enhance the robot's ability to learn the potential mapping relationship between environmental spatial information and motion sequences.

[0063] Specifically, the 6DOF motion information of the robotic arm's end effector is first encoded to obtain motion features, which are used to characterize the current motion state and operational trend. The point cloud features and motion features are then input into a spatial motion collaborative perception module. This module employs a dual-branch interactive neural network structure, using an internal cross-attention layer to couple and model the two heterogeneous data in the same feature space, thereby deeply learning the implicit mapping relationship between the spatial structure of the environment and the motion sequence. For example... Figure 1 As shown, this module simulates the "observation-thinking-decision" logic chain of ultrasound physicians, combining visual and dynamic processes: it uses point cloud information to simulate the spatial constraints obtained by human eye observation, uses robotic arm movements to simulate the physical feedback of handheld devices, and uses a neural network as the logical core to fit the expert's judgment process. The module ultimately outputs multimodal observation information, which will serve as input information for generating latent variables in step 5. This output enables the model-generated action sequence to dynamically perceive environmental changes, better aligning with expert teaching data, and significantly improving the accuracy of path planning and the quality of acquired ultrasound image data in ultrasound examination tasks.

[0064] Step 4: Embedding Historical Information; Previous research has shown that even small prediction errors can be gradually amplified during the execution phase, leading to significant deviations between model predictions and actual motion states. This phenomenon is particularly prominent in tasks with long execution cycles, such as ultrasonic scanning. Prediction errors accumulate continuously in the sequence of actions, easily causing significant "error accumulation" problems, which in turn cause positional drift in the robot during task execution, severely affecting task stability and execution accuracy. Furthermore, if the trajectory prediction process relies solely on local observation information at the current moment, the robot often struggles to accurately infer its stage position within the overall task flow, resulting in action decision confusion in areas with highly similar spatial or state characteristics. In such cases, the robot may deviate from its intended action or even fall into a local loop, ultimately preventing the task from being completed successfully.

[0065] To address the aforementioned issues, this invention introduces a Context-aware Temporal Action Embedding (CTAE) strategy. By incorporating global temporal context information during action prediction, it effectively mitigates the problem of prediction error accumulation and improves the trajectory consistency and decision robustness of the robot in long-term tasks.

[0066] The overall process of the continuous-time attention embedding strategy differs from traditional methods that rely solely on instantaneous observations at the current moment for action prediction. CTAE aggregates spatial-action co-information within a finite historical time window of the input, enabling the model to simultaneously perceive the direction of motion and its evolution over time. The specific process is as follows:

[0067] First, a temporal sequence is constructed: the feature sequence within a finite historical time window L before the current time t is obtained, and a temporal tensor containing the point cloud spatial feature set S and the action feature set A is constructed.

[0068] Then, collaborative feature encoding is performed: spatial features and action features at the same time are concatenated to generate a collaborative perception vector at that time.

[0069] Next, attention weighting enhancement is performed: the influence weight of the collaborative perception vector at different historical moments on the current decision is calculated using the self-attention mechanism, and the information within the window is aggregated by weighted summation.

[0070] The final output is a fused feature vector containing the spatiotemporal evolution context, which is used to guide the generation of actions in the next time step.

[0071] By explicitly encoding short-term motion history, the model can not only capture local temporal consistency but also more accurately determine its task state throughout the entire scan process. This capability is particularly important for tasks with repetitive motion patterns, such as reciprocating scans, effectively avoiding motion regression, path oscillation, or local looping caused by ambiguous state recognition, thereby significantly improving the stability and robustness of trajectory prediction.

[0072] Step 5: Action Generation Based on Continuous-Time Environmental Information; Inspired by the decoding and modeling of latent variables in Conditional Variational Auto-Encoders, this invention proposes an action generation method based on continuous-time environmental information. This method uses multimodal observation information within a continuous time window as conditional input to model and predict the robot's future action sequences.

[0073] Specifically, in the generation of latent variable z, historical action features and historical point cloud features, combined with multimodal features at the current moment, are introduced through unified modeling. This allows the latent variable z to simultaneously encode spatial structure information and temporal action patterns. After obtaining the latent variable, it is reconstructed through a decoding network to generate the corresponding predicted action sequence. The main steps include:

[0074] Based on fusion feature vectors Combined with multimodal observation information, using approximate posterior distribution The latent variable z and the multimodal observation information obtained in step 3 The policy network generates future action sequence blocks (containing 6DOF poses or incremental poses at several consecutive time points) as candidate execution trajectories for the next stage.

[0075]

[0076] in, This represents a block of predicted future action sequences. Represents the policy network.

[0077] Multiple future predicted action sequence blocks are weighted and fused to obtain the execution action. The weights of each future action sequence block adopt an exponential decay form, so that recent predictions contribute more to the current execution, thereby improving the smoothness of the action output and reducing jitter.

[0078] Based on the predicted pose information, the robot is controlled to perform an ultrasonic scanning task: the fused execution action is sent to the robotic arm controller to drive the ultrasonic probe to move and complete the scan; the system simultaneously acquires a new frame of point cloud and the current end pose, and enters the next cycle steps S1 to S5 to achieve online closed-loop autonomous scanning.

[0079] During the training phase, the predicted action is compared with the actual action, and the prediction error is calculated using a loss function. To effectively train the model, this invention introduces the Evidence Lower Bound (ELBO) as an optimization objective. By maximizing the log-likelihood of the observed data, the latent variable distribution and the parameters of the generative model are indirectly optimized. The mathematical derivation is as follows:

[0080]

[0081]

[0082]

[0083] The input data is from express, This represents the posterior variational inference distribution. This represents the conditional generative decoding distribution; This represents the Kullback-Leibler divergence operator, used to quantify the inconsistency between the posterior and prior distributions;

[0084] Based on the above derivation and formula transformation, the variational optimization loss function expression of the space-action collaborative attention enhancement module can be obtained as follows:

[0085] in, Represents the spatiotemporal observation buffer at the current moment. Previous time-series data of multimodal environmental observation characteristics within a limited historical time window, Represented by parameters The conditional generative decoding distribution fitted by the policy network with weights. Represented by parameters The posterior variational inference distribution fitted by the weighted neural network is responsible for encoding high-dimensional multimodal operational observations into a latent space distribution. The representative performs the first step within the potential space. The specific latent variable sample points obtained by the next random resampling. The fractional term represents the total number of samples randomly resampled within the latent space. This represents the expected mean value processing performed on these sample points.

[0086] To jointly optimize the space-action coordination module and the CTAE module, this invention satisfies physical constraints while maximizing the lower bound of evidence, and defines the total loss function expression for end-to-end optimization as follows:

[0087]

[0088] in, The representative is the standard prior boundary for latent space parameter regularization. The operator representing the mathematical expectation under the posterior distribution, while Then, it represents the hyperparameter balance coefficient that adjusts the weights between the variational reconstruction term (first term) and the standard regularization term (second term).

[0089] The spatial information-driven autonomous scanning method for embodied ultrasound robots proposed in this invention has significant technical advantages over existing technologies and has been fully validated in experiments with medical silicone dummies and real volunteers. Specific experimental details and related control experiments will be described in detail below.

[0090] Experimental platform establishment: Schematic diagram of the mechanical structure of the key hardware of the ultrasonic robot, the ultrasonic probe, is shown below. Figure 3 As shown, it includes an ultrasonic probe, joint motors, force sensors, and contact force adjustment springs. Through its specially designed ultrasonic robot hardware, the robot can execute the motion commands given by the model with high precision.

[0091] like Figure 2 As shown, the entire experimental scenario was divided into two parts: teleoperation data acquisition and model inference robot execution. As illustrated in the left figure, firstly, the six-dimensional pose control of the controller was calculated using a VR headset, and then a topic was published via ROS to instruct the robotic arm to execute the corresponding action. During this process, the pose of the robotic arm's end effector and point cloud data from two perspectives were simultaneously recorded. Then, hundreds of sets of data from different body parts were collected to establish a dataset. After model training, testing was conducted on a simulated human body model and real volunteers.

[0092] To ensure data diversity and effectively reduce the risk of model overfitting, this paper constructs a large-scale robotic ultrasound scanning dataset, containing over 600 complete scan trajectories. The dataset was obtained through extensive real-world experiments on medical silicone mannequins and human volunteers.

[0093] Specifically, data collection was conducted on five medical-grade silicone mannequins and eight real volunteers. In the mannequin-based experiments, to avoid the model relying on fixed spatial priors, we intentionally translated and rotated the mannequins during testing, making their pose relative to the robotic arm significantly different from that during training, thus constructing test scenarios with a deviation from the training distribution. To further evaluate the generalization ability of the proposed method, a leave-one-out evaluation strategy based on the subjects was adopted in the real-person experiments. Data from six volunteers was used for model training, while data from the remaining two volunteers were not used for training and were only used for testing. This setup directly examines the model's performance under conditions of unfamiliar target individuals and individual differences.

[0094] Through large-scale data collection, dynamic perturbation settings in dummy experiments, and a test protocol based on subject segmentation, the constructed dataset achieves sufficient coverage in terms of the diversity of scan structures. Therefore, the experimental results in this paper can realistically reflect the generalization performance of the model in real-world deployment scenarios.

[0095] In practical tests, point cloud features were extracted by inputting voxelization data into a point cloud feature extraction model. However, the extraction results using point cloud features directly were poor. This was because the dataset size of the ultrasound robot collected from real-world data was small, leading to poor training performance. Collecting more real-world data, however, was prohibitively expensive. Considering that the training of point cloud feature extraction does not need to be limited to ultrasound scenarios and that numerous publicly available point cloud datasets exist, this work introduces a transfer learning strategy. It improves training performance through a "pre-training on public datasets + optimization on ultrasound data" approach. Specifically, PointNet++ is first pre-trained on the publicly available point cloud dataset ModelNet40. Then, during training with real-world data, the pre-trained weights are loaded, and the parameters of the first 70% of layers are frozen, preventing updates during backpropagation. The remaining 30% of layers are then updated during training. This ensures that the point cloud extraction model first learns the "low-level features" from the public dataset and then learns the "ultrasound semantic features" from the ultrasound data and the transformer model.

[0096] The experiment focused on ultrasound scanning tasks of the carotid artery, breast, abdomen, and deep vein thrombosis (DVT), and compared them with cutting-edge algorithms ACOTG, ACT, and US-EDRL. ACOTG is a traditional offline ultrasound algorithm based on point cloud input, ACT is a transformer-based robot end-to-end output algorithm based on RGB image input, and US-EDRL is an image-based ResNet+LSTM end-to-end ultrasound algorithm.

[0097] In multi-site experiments, the proposed method outperformed other comparative algorithms in overall performance. The following section will analyze each evaluation index in detail. The specific data are shown in Table 1. TP and DWT reflect the robot's completeness in the task execution process and the learning effect of the taught trajectory, which directly determine whether there will be problems such as missed scans or incorrect scans during ultrasound scanning. The experimental results show that the end-to-end method can output more accurate movements than traditional offline algorithms such as ACOTG by learning expert teaching data. In contrast, US-EDRL has been used for small-scale scanning tasks and performs poorly when facing long-cycle scanning tasks such as breast and carotid artery, and ACT is prone to regression problems. In comparison, the proposed method achieved completion rates of 93.7% and 89.8% in breast and carotid artery scanning tasks, respectively. This is because the transformer architecture can better model long-distance dependencies and grasp the direction of motion progress through CTAE. Furthermore, AOE reflects the model's ability to adjust the probe's posture. If the probe is not perpendicular to the contact point, it will lead to insufficient contact between the probe and the skin, resulting in partially blurred or even missing ultrasound images. Among these, the ACOTG method, based on point cloud input, achieves better results by constructing a 3D map to plan the scanning posture. This is why we did not use ACT or US-EDRL, which use image input. To further improve the directional accuracy of posture adjustment, this paper constructs a potential mapping relationship through spatial motion collaborative perception, achieving significant results. This indicator is improved by up to 59.5%. Adjustment speed is also crucial. High latency in motion optimization will lead to a decrease in scanning effect. This is the main challenge faced by multimodal input Transformer models, especially when dealing with large-scale point cloud input. Therefore, the point cloud processing module needs to solve this core problem—improving computational efficiency through methods such as voxelization, so that the Operation Response Latency (ORL) reaches the same level as traditional offline planning methods. In summary, through the above optimization measures, the ultrasound robot's task execution process is closer to the expert's level, thus obtaining the highest evaluation in the Ultrasound Image Clarity Score (USIC).

[0098] Table 1: Comparison of Experimental Results and Performance Indicators

[0099]

[0100] by Figure 4Taking the carotid artery scanning task shown as an example, (a) represents the initial stage of the task where the probe gradually approaches the neck region, (b)–(d) correspond to the robot performing the lateral scanning process, and (e)–(g) represent the longitudinal scanning stage. From the actual execution images, it can be observed that throughout the scanning process, the distal ultrasound probe maintains stable and approximately perpendicular close contact with the skin tissue. Simultaneously, the obtained ultrasound images show a complete imaging area and clear structure, indicating that the acoustic coupling conditions are effectively maintained during the movement.

[0101] Safety and comfort are key evaluation metrics for robotic ultrasonic scanning systems (RUSS). For example... Figure 4 The force sensor data shown in the figure below indicates that the normal contact force remained stably within the range of approximately 8 ± 2 N for most of the task execution. This force range ensures sufficient contact between the probe and the skin tissue without causing significant pressure on sensitive areas such as the neck, thus balancing image quality and subject comfort. It should be noted that this stable contact force adjustment effect is mainly due to the adaptive stiffness physical interaction control method introduced in this paper. This method can adjust the virtual stiffness online according to the real-time contact state and external disturbances, thereby achieving safe and compliant normal interaction control.

[0102] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A spatial information-driven autonomous scanning method for an embodied ultrasonic robot, characterized in that, Includes the following steps: Step 1: Acquire sensor data input; the input information includes point cloud data generated by a camera that follows the movement of the robotic arm through a pseudo point cloud generation algorithm, and 6DOF motion information of the ultrasonic scanning probe placed at the end of the robotic arm; Step 2: Point cloud feature extraction and embedding representation; The original point cloud frame is voxelized, then uniformly transformed to the same world coordinate system, and finally a block-based point cloud mask modeling strategy is used to embed the point cloud to obtain point cloud features; Step 3: Spatial action temporal perception and multimodal coupling; First, the 6DOF action information is encoded to obtain action features; then, the point cloud features and action features are input into the spatial action collaborative perception module, and a dual-branch interactive neural network structure is used to couple and model the point cloud features and action features in the same feature space to obtain multimodal observation information; Step 4: Historical information embedding; Based on the continuous temporal attention embedding strategy, aggregate point cloud-action coordination information within the historical time window, and output a fused feature vector containing spatiotemporal evolution context; Step 5: Using the fused feature vector and multimodal observation information within a continuous time window as conditional inputs, the robot's future action sequence is modeled and predicted through a policy network.

2. The spatial information-driven autonomous scanning method for a embodied ultrasonic robot according to claim 1, characterized in that, Step 1 is described in detail as follows: During the training phase, point cloud data, probe pose, and corresponding ultrasound image quality information are collected. The quality of the collected ultrasound images and the accuracy of the trajectory are scored, and low-scoring data are removed. Subsequently, the point cloud data and motion data are converted into the data format given by the model and a dataset is constructed.

3. The spatial information-driven autonomous scanning method for a embodied ultrasonic robot according to claim 1, characterized in that, Step 2 is described in detail below: Step 2.1: Voxelize the original point cloud frame using the stacked voxel feature encoding method; Step 2.2: Calculation of voxel features weighted by reflection intensity; point cloud data from different camera coordinate systems are uniformly transformed to the same world coordinate system. During the coordinate transformation process, the reflection intensity of the point cloud is introduced for weighted processing. Step 2.3: Point cloud mask modeling strategy; First, randomly select a sub-point cloud from the original point cloud, then determine its j adjacent sub-point clouds to construct a spatially continuous region; subsequently, use this continuous region as the mask region, whose index set... Represented as Among them, the mask index set The number of elements inside is determined by the mask ratio. The product of the total number of sub-point clouds, g, determines the outcome. Next, all point cloud features within the masked region are replaced with learnable predefined mask embedding vectors; the final point cloud data used for feature extraction is represented as follows: This represents the point cloud feature vector at the i-th position in the input sequence. The spatiotemporal location encoding vector represents the i-th position.

4. The spatial information-driven autonomous scanning method for a embodied ultrasonic robot according to claim 3, characterized in that, The spatial motion collaborative perception module uses a cross-attention layer to couple and model point cloud features and motion features in the same feature space.

5. The spatial information-driven autonomous scanning method for a embodied ultrasonic robot according to claim 4, characterized in that, Step 4 is described in detail below: First, a temporal sequence is constructed: the feature sequence within a finite historical time window before the current moment is obtained, and a temporal tensor containing point cloud feature sets and action feature sets is constructed. Then, collaborative feature encoding is performed: the point cloud features and action features at the same time are concatenated to generate the collaborative perception vector at that time. Next, attention weighting enhancement is performed: the influence weight of the collaborative perception vector at different historical moments on the current decision is calculated using the self-attention mechanism, and the information within the window is aggregated by weighted summation; The final output is a fused feature vector containing the spatiotemporal evolution context, which is used to guide the generation of actions in the next time step.

6. The spatial information-driven autonomous scanning method for a embodied ultrasonic robot according to claim 5, characterized in that, Step 5 is described in detail below: Latent variables are obtained by using an approximate posterior distribution based on fused feature vectors and multimodal observation information; the multimodal observation information and latent variables are input into the policy network to generate future action sequence blocks, which serve as candidate execution trajectories for the next stage; The future action sequence blocks are weighted and fused using an exponential decay method to obtain the final execution trajectory; During the training phase, the predicted action is compared with the actual action, and the prediction error is calculated using a loss function. The loss function introduces a lower bound of evidence as an optimization objective, and indirectly optimizes the distribution of latent variables and model parameters by maximizing the log-likelihood of the observed data.

7. The spatial information-driven autonomous scanning method for a embodied ultrasonic robot according to claim 6, characterized in that, During the training phase, end-to-end optimization is performed using the following total loss function: in, The operator represents the mathematical expectation under the posterior distribution. Representing latent variables, Represents the fused feature vector. Represents multimodal observation information, Indicates by parameters The conditional generative decoding distribution fitted by the policy network with weights; This represents the Kullback-Leibler divergence operator. Indicates by parameters The posterior variational inference distribution fitted by the weighted neural network. As a standard prior boundary for latent space parameter regularization This represents the hyperparameter balance coefficient.

Images