Robot ultrasound scan trajectory data generation method and system based on video world model

Abstract

This invention relates to the technical field of medical image processing, and discloses a method and system for generating robot ultrasound scanning trajectory data based on a video world model. The data generation method, applied to an ultrasound scanning robot control system, specifically includes the following steps: S11: Receiving a teaching data collection request, collecting and synchronizing a small batch of multimodal real teaching data from ultrasound expert-operated robots, selecting a pre-trained text-video generation model on video data as the basic video world model, and fine-tuning the parameters of the basic video world model using LoRA and real teaching data. This invention, by employing a pre-trained video generation model and utilizing LoRA technology for efficient fine-tuning with small samples, can learn and reproduce the kinematic and visual interaction patterns in actual operation, generating massive amounts of high-fidelity synthetic video sequences covering various body types, pathologies, and anatomical variations at extremely low real data costs.

Description

Technical Field

[0001] This invention relates to the technical field of medical image processing, and more specifically, to a method and system for generating robot ultrasound scanning trajectory data based on a video world model. Background Technology

[0002] Ultrasound imaging, as a safe, real-time, and radiation-free medical imaging technique, is widely used in diagnosis and monitoring in various clinical settings, including cardiology, abdominal surgery, and obstetrics. However, the quality of traditional ultrasound examinations highly depends on the operator's professional experience and technique. The operator must hold the probe, precisely locate specific anatomical structures on the patient's body surface, and continuously adjust the probe's position, posture, and pressure to obtain clear and standardized diagnostic images. This places extremely high demands on the operator's skills and presents problems such as significant differences between operators and fatigue from prolonged operation.

[0003] To address these issues, robot-assisted ultrasound scanning systems have emerged. These systems use robotic arms to replace human hands in holding and moving the probe, potentially achieving standardization and automation of the scanning operation. However, existing methods for planning robot scanning paths have the following limitations: 1. The robotic arm is manually guided by an experienced ultrasound physician or remotely operated via a master-slave device to record a standard scanning trajectory. The robot can only repeat these fixed trajectories afterward, resulting in high data acquisition costs, requiring valuable time from senior physicians, and a cumbersome teaching process, making it difficult to collect large-scale data covering various pathologies and body types. 2. When the teaching trajectory is highly fixed to a specific standard human model or the patient being taught, the replay of the fixed trajectory often fails to obtain the ideal acoustic window when facing new patients with individual differences in body shape, weight, and anatomical structure, resulting in decreased image quality or missed diagnosis. 3. Simulating human tissue and ultrasound physical processes in a virtual environment to generate a large amount of training data presents a huge gap between simulation and reality. Accurately simulating the complex mechanical interaction between the ultrasound probe and human soft tissue, as well as the propagation and imaging physical processes of ultrasound waves, is extremely difficult. Models trained in simulation often experience a sharp decline in performance when applied to real human bodies. Moreover, constructing high-fidelity biomechanical and acoustic models for different organs and patient types requires a significant investment of professional knowledge and engineering expertise.

[0004] In recent years, large-scale video generation models have gained a profound understanding of the dynamic laws of the physical world by pre-training on massive amounts of Internet video data. However, how to transform this powerful video generation capability into training data that can be learned by robots, contains precise motion and force control information, and is suitable for ultrasonic scanning remains an unresolved technical problem.

[0005] Therefore, there is an urgent need in the field for a general-purpose scanning robot system that can generate massive, diverse, and high-fidelity robotic ultrasonic scanning trajectory data using only a small amount of expert teaching data, thereby training the robot to understand scanning tasks, adapt to individual differences, and have the ability to adjust in real time. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for generating robot ultrasound scanning trajectory data based on a video world model. By employing a pre-trained video generation model and utilizing LoRA technology for efficient fine-tuning with small samples, it can learn and reproduce the kinematic and visual interaction patterns in actual operation, generating a massive amount of high-fidelity synthetic video sequences covering various body types, pathologies, and anatomical variations at extremely low real data cost, aiming to solve the problems in the prior art.

[0007] This invention is implemented as follows: a method for generating ultrasonic scanning trajectory data for robots based on a video world model, applied to the control system of an ultrasonic scanning robot, specifically includes the following steps: S11: Receive teaching data collection request, collect and synchronize multimodal real teaching data of a small batch of ultrasound expert operation robot, select a text-video generation model pre-trained on video data as the basic video world model, and fine-tune the parameters of the basic video world model using LoRA and real teaching data. S12: Obtain diverse initial condition input pairs, input the initial condition pairs in batches into the fine-tuned video world model for autoregressive long sequence inference, and generate a large-scale high-fidelity synthetic video sequence. Each synthetic video sequence visually and realistically simulates the movement of the robotic ultrasound probe on the surface of a virtual patient according to clinical logic. S13: Construct a pseudo-label generation model to process the synthetic video sequence, so that the synthetic video sequence presents the corresponding dynamic change process of ultrasound image that conforms to the anatomical and acoustic laws, infer the corresponding robot action sequence and contact force sequence, and generate visual trajectory data. S14: Obtain a policy network that outputs robot actions and force control commands. Use visual trajectory data and its corresponding pseudo-label pairs as the core training dataset. Make it output the action sequence and force control sequence within a preset time period in the future, as well as the difference between the pseudo-labels at the corresponding time in the visual trajectory data, so that the policy network can generate scanning trajectories and contact forces that adapt to individual differences. S15: Deploy the scanning strategy network into the ultrasound scanning robot control system. When actually scanning the patient, acquire images from external cameras, video streams from the ultrasound equipment, and receive natural language or standardized instructions from the operator in real time. Combined with the robot's own state feedback, these are input into the deployed strategy network. Based on the real-time multimodal perception information, perform forward calculations to generate the optimal robot action sequence and force control sequence online, and then hand them over to the underlying controller for execution.

[0008] Furthermore, in S11, a teaching data collection request is received, and a small batch of multimodal real teaching data of the ultrasound expert operating robot is collected and synchronized, including: Upon receiving the teaching data collection request, the multimodal data acquisition process is initiated to initialize and configure the hardware and environment, including calibrating and enabling multiple external high-definition cameras located in the operating area, establishing a video stream transmission interface with the ultrasonic equipment, ensuring that the data log function of the robot control system is enabled, and verifying the communication and zero point of the contact force / torque sensor. When ultrasound experts operate the robot to perform scanning tasks, the data streams of all modalities are captured synchronously. Through hardware trigger signals, it is ensured that each frame of external view video, ultrasound image sequence, and robot status data packet is stamped with a uniform timestamp. The robot status data package includes the six-dimensional pose of the robotic arm end effector, six-dimensional contact force / torque readings, and servo control commands.

[0009] Further, in S12, diverse initial condition input pairs are obtained, and these initial condition pairs are batch-input into the fine-tuned video world model for autoregressive long-sequence inference, including: The initial visual frame for each input pair is sampled from real patient data, high-fidelity anatomical model renderings, and historical frames enhanced by elastic deformation and illumination simulation. The input pair generates corresponding text instructions from a structured instruction library. The time of each initial moment is strictly aligned with the external view video frame and the real-time ultrasound image video frame, and then stitched together to form a unified grid-like multi-view visual input. The batch-organized initial conditions are input in parallel into the fine-tuned video world model. The video world model uses the generated output of the previous frame as the autoregressive input condition for the next frame, and is guided by the semantic text instructions throughout the process to gradually infer and generate a long sequence of future frames.

[0010] Furthermore, the video world model uses the generated output of the previous frame as the autoregressive input condition for the next frame, and is guided by consistent textual instruction semantics, including: In each autoregressive inference step, the video world model is conditionally generated with two core inputs: one is the video frame generated in the previous time step, which includes the probe position, the state of contact with the body surface, and the grid view of the corresponding ultrasound image; the other is the encoded semantic vector of the initial input text instructions. The denoising network inside the video world model extracts spatiotemporal features from the previous frame and uses the text semantic vector as a global condition, injecting it into the feature generation process through a cross-attention mechanism to predict the visual content of the next frame that conforms to the instruction intent. The video world model maintains a dynamic latent state memory throughout the autoregressive loop. This latent state memory accumulates contextual information from historical frame sequences and continuously interacts with fixed textual instruction conditions.

[0011] Furthermore, in S13, a pseudo-label generation model is constructed to process the synthesized video sequence, causing the synthesized video sequence to exhibit a corresponding dynamic change process of ultrasound images that conforms to anatomical and acoustic laws, including: A pseudo-label generation model is constructed using an inverse dynamics scheme. A deep neural network model is trained using a small amount of real teaching data collected in the first stage as a training set. The deep neural network model takes two sets of multimodal visual frames that are consecutive in time as input. The difference in visual state is analyzed through an encoder-regressor architecture, and the action sequence and force control sequence that the robot end effector should execute within the observation interval are directly regressed. After training, the trained pseudo-label generation model is applied to a large number of synthetic video sequences without action labels generated in the second stage. A sliding window with a fixed step size is used to input the continuous visual frames within the window into the pseudo-label generation model. The pseudo-label generation model outputs pseudo-action and force sequences for each window. Through splicing and smoothing, pseudo-robot action label sequences and pseudo-contact force label sequences that run through the entire duration of each synthetic video are generated.

[0012] Furthermore, in S14, visual trajectory data and its corresponding pseudo-label pairs are used as the core training dataset, enabling it to output action sequences and force control sequences within a preset future time period, including: Based on the generated visual trajectory data and its corresponding pseudo-action and force label pairs, a core dataset for training the scanning strategy network is constructed. Each training sample integrates the current multi-view external camera image, real-time ultrasound image frame, encoded high-level language instruction vector describing the scanning target, as well as the robot's current joint angle, end pose, and six-dimensional force sensing state. The supervision label is the robot action sequence and contact force control sequence within a preset time window in the future. Visual trajectory data and its corresponding pseudo-labels are input into the policy network. The policy network, conditioned on the aforementioned integrated multimodal state tensor, learns through forward propagation to generate action and force sequences that are closest to the future real pseudo-labels. During training, by minimizing the difference between the predicted sequence and the pseudo-label sequence, the policy network learns the ability to proactively plan the optimal control commands that are safe, smooth, and can achieve the scanning target based on real-time multimodal perception information.

[0013] Furthermore, in S15, the robot's own state feedback is combined with the input to the deployed policy network. Based on real-time multimodal perception information, forward calculation is performed to generate the optimal robot action sequence and force control sequence online, which are then executed by the underlying controller, including: The system continuously collects and integrates video streams from multiple fixed external cameras, real-time images output by ultrasonic equipment, language command text from the operating interface, and joint encoder data and end-effector six-dimensional force sensor readings from the robot controller via a high-speed data bus. All multimodal data is processed through a unified state synchronization interface and converted into a standardized tensor format required by the policy network as input for each control cycle. After receiving the real-time input, the strategy network, based on the aforementioned multimodal state, uses its internal diffusion model for denoising and sampling or feedforward network calculation to predict the optimal robot end effector sequence and contact force sequence within a future time window. This sequence is then sent to the robot's underlying motion and force control controller in real time to drive the probe through the entire process from initial approach, safe contact, scanning along the planned path, to finally stabilizing on the target plane.

[0014] Furthermore, in S11, LoRA and real teaching data are used to fine-tune the parameters of the basic video world model, including: We selected a text-video generation model pre-trained on a large-scale general video dataset, embedded a trainable LoRA adaptation layer into the fully connected layer of the text-video generation model, and froze the original parameters. The base model embedded in the LoRA adaptation layer is fine-tuned and trained using small batches of real multimodal teaching data collected and synchronized, learning to predict the grid-like multi-view visual content of the next frame based on the previous frame and text instructions. Supervised learning enables the gradual learning and reproduction of the unique kinematic constraints of a specific ultrasound scanning robot in real-world operation, the visual dynamic patterns of probe movement on the body surface, and the complex visual interaction representations generated by the probe contacting and pressing different soft tissues.

[0015] Compared with existing technologies, the robot ultrasonic scanning trajectory data generation method and system based on video world model provided by this invention have the following advantages: 1. By employing a pre-trained video generation model and utilizing LoRA technology for efficient fine-tuning with small samples, it can learn and reproduce the kinematic and visual interaction patterns in actual operation. It generates a massive amount of high-fidelity synthetic video sequences covering various body types, pathologies, and anatomical variations at extremely low real data cost, thus eliminating the dependence on large-scale and expensive teaching data. Furthermore, by utilizing the prior knowledge of physical laws of the generative model, it bypasses high-fidelity biomechanical simulation engineering, achieving a leap from limited real data to unlimited synthetic training data. 2. By using pseudo-label generation technology and multimodal diffusion strategy network, visual sequences are transformed into executable robot control knowledge, which can directly respond to advanced clinical commands to achieve target-driven scanning. It also has strong individualized adaptability, can dynamically adjust probe posture and pressure based on real-time ultrasound images to optimize the acoustic window, and at the same time ensure the smoothness and safety of the generated trajectory. It can autonomously complete the entire scanning task and adapt to different clinical scenarios and robot platforms with high flexibility and robustness.

[0016] A robot ultrasonic scanning trajectory data generation system based on a video world model is used to execute the aforementioned robot ultrasonic scanning trajectory data generation method. The system includes: The data acquisition and processing module is used to acquire and synchronize multimodal real teaching data; The video world model generation module is used to fine-tune the basic video world model and generate high-fidelity synthetic video sequences based on diverse initial conditions. The pseudo-tag generation module is used to process the synthesized video sequence and generate corresponding pseudo-tags for the robot action sequence and contact force sequence; The policy training module is used to train a policy network that generates scanning trajectories and contact forces using synthetic videos and their pseudo-labels as a dataset. The strategy execution and scanning control module is used to deploy the trained policy network to the robot control system and generate and execute control commands based on real-time multimodal perception information.

[0017] Specifically, the pseudo-tag generation module includes: The model training unit is used to train an inverse dynamics model using real teaching data. The model takes continuous multimodal visual frames as input and regresses the action and force sequence between two frames. The inference generation unit is used to input the trained inverse dynamics model with consecutive visual frames of the synthesized video and recursively infer the complete sequence of pseudo-actions and force labels through a sliding window. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the robot ultrasonic scanning trajectory data generation method based on a video world model proposed in this invention. Figure 2 This is a flowchart illustrating the process of fine-tuning the parameters of the basic video world model using LoRA and real teaching data in the robot ultrasonic scanning trajectory data generation method based on the video world model proposed in this invention. Figure 3 This is a schematic diagram of the robot ultrasonic scanning trajectory data generation system based on a video world model proposed in this invention. Figure 4 This is a schematic diagram of the pseudo-label generation module in the robot ultrasonic scanning trajectory data generation system based on a video world model proposed in this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] The implementation of the present invention will be described in detail below with reference to specific embodiments.

[0021] In the accompanying drawings of this embodiment, the same or similar reference numerals correspond to the same or similar components. In the description of this invention, it should be understood that if terms such as "upper," "lower," "left," and "right" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting this invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0022] Reference Figure 1-2 As shown, a method for generating robot ultrasonic scanning trajectory data based on a video world model, applied to the control system of an ultrasonic scanning robot, specifically includes the following steps: S11: Receive teaching data collection request, collect and synchronize small batches of multimodal real teaching data of ultrasound expert operation robot, select a pre-trained text-video generation model on video data as the basic video world model, and use LoRA and real teaching data to fine-tune the parameters of the basic video world model. This fine-tuning process aims to enable the general basic model to quickly absorb the specific visual and dynamic patterns in the field of ultrasound scanning, thereby obtaining a domain-adaptive model core that can understand and generate videos that conform to the professional scene. This includes receiving teaching data collection requests, collecting and synchronizing small batches of multimodal real-world teaching data from the ultrasound expert operating robot, including: Upon receiving the teaching data collection request, the multimodal data acquisition process is initiated to initialize and configure the hardware and environment. This includes calibrating and enabling multiple external high-definition cameras located in the operating area, establishing a video stream transmission interface with the ultrasonic equipment, ensuring that the data log function of the robot control system is enabled, and verifying the communication and zero point of the contact force / torque sensor. This ensures the consistency of the time and space reference of the multi-source heterogeneous data at the starting point of the acquisition, laying a solid foundation for subsequent data alignment and fusion. When ultrasound experts operate the robot to perform scanning tasks, the data streams of all modalities are captured synchronously. Through hardware trigger signals, it is ensured that each frame of external view video, ultrasound image sequence, and robot status data packet is stamped with a unified timestamp. This high-precision synchronization mechanism effectively ensures the strict spatiotemporal correspondence between visual observation, control commands, and mechanical feedback, which is a key prerequisite for building high-quality, learnable teaching data. The robot status data package includes the six-dimensional pose of the robotic arm end effector, six-dimensional contact force / torque readings, and servo control commands. S12: Obtain diverse initial condition input pairs, input the initial condition pairs in batches into the fine-tuned video world model for autoregressive long sequence inference, and generate large-scale high-fidelity synthetic video sequences. Each synthetic video sequence visually and realistically simulates the movement of the robotic ultrasound probe on the virtual patient's body surface according to clinical logic. Using the fine-tuned video world model as a powerful "visual simulator", training data covering a wide range of anatomical variations and scanning scenarios is automatically and in batches synthesized according to diverse task instructions and the patient's initial state, thereby solving the core bottleneck of scarcity and insufficient coverage of real data. This includes obtaining diverse initial condition input pairs and batch inputting these pairs into a fine-tuned video world model for autoregressive long-sequence inference, including: The initial visual frame for each input pair is sampled from real patient data, high-fidelity anatomical model renderings, and historical frames enhanced by elastic deformation and illumination simulation. The input pair generates corresponding text instructions from a structured instruction library. The time of each initial moment is strictly aligned with the external view video frame and the real-time ultrasound image video frame, and then stitched together to form a unified grid-like multi-view visual input. This ensures that the initial conditions received by the model visually contain a complete interactive context (surface and interior) and semantically have a clear task objective, providing sufficient information for generating high-quality, goal-oriented video sequences. The initial conditions, organized in batches, are input into the fine-tuned video world model in parallel. The video world model uses the output of the previous frame as the autoregressive input condition for the next frame, and is guided by the semantics of the text instructions throughout the process. It gradually infers and generates a long sequence of future frames. Parallel batch processing significantly improves data generation efficiency. The combination of autoregressive generation and continuous guidance from instruction conditions ensures the coherence, consistency, and strong correlation with the high-level task objectives in the dynamic evolution of the long video sequence. S13: A pseudo-label generation model is constructed to process the synthetic video sequence, causing the synthetic video sequence to present a corresponding dynamic change process of ultrasound images that conforms to anatomical and acoustic laws. The corresponding robot action sequence and contact force sequence are inferred, and visual trajectory data is generated. The executable robot control signals are "reverse reasoned" for the synthetic video without action annotations, thereby transforming the "demonstration video" rich in visual change information into "action-vision" pairing data that can be used for supervised learning of robot strategies, thus bridging the gap from visual generation to action learning. Specifically, a pseudo-label generation model is constructed to process the synthesized video sequence, causing the synthesized video sequence to exhibit a corresponding dynamic change process of ultrasound images that conforms to anatomical and acoustic principles, including: A pseudo-label generation model is constructed using an inverse dynamics approach. A deep neural network model is trained using a small amount of real teaching data collected in the first stage as the training set. The deep neural network model takes two sets of multimodal visual frames that are consecutive in time as input. Through an encoder-regressor architecture, it analyzes the differences in visual states and directly regresses the action sequence and force control sequence that the robot's end effector should execute within the observation interval. This inverse dynamics model learns the mapping relationship from "visual state change" to "action that produces the change". It captures the statistical regularity of this mapping using limited real data, providing generalizable reasoning ability for the generation of labels for infinitely synthesized videos. After training, the trained pseudo-label generation model is applied to a large number of synthetic video sequences without action labels generated in the second stage. A sliding window with a fixed step size is used to input continuous visual frames within the window into the pseudo-label generation model. The pseudo-label generation model outputs pseudo-action and force sequences for each window. Through splicing and smoothing, pseudo-robot action label sequences and pseudo-contact force label sequences are generated for each synthetic video throughout its entire duration. The recursive application of the sliding window achieves dense labeling of the entire long video sequence. Post-processing ensures the temporal smoothness and physical executability of the generated pseudo-label sequences, thus constituting a high-quality policy training objective. S14: Obtain a policy network that outputs robot actions and force control commands. Use visual trajectory data and its corresponding pseudo-label pairs as the core training dataset. This allows the network to output action sequences and force control sequences within a preset time period, as well as the differences between these sequences and the pseudo-labels at the corresponding moments in the visual trajectory data. This enables the policy network to generate scanning trajectories and contact forces that adapt to individual differences. By utilizing the massive "vision-action" pairing data produced in the previous steps, a policy network capable of end-to-end perception and decision-making is trained. Essentially, this allows the robot to learn to observe multimodal environments, understand task commands, and imitate "expert behavior" defined by the video world model and the inverse dynamics model, thereby gaining the ability to perform intelligent scanning in the real world. S15: Deploy the scanning strategy network into the ultrasound scanning robot control system. During actual patient scanning, acquire external camera images, ultrasound equipment video streams, and receive natural language or standardized instructions from the operator in real time. Combined with the robot's own state feedback, these are input into the deployed strategy network. Based on real-time multimodal perception information, perform forward calculations to generate the optimal robot action sequence and force control sequence online, which are then executed by the underlying controller. By combining the trained strategy network with the robot hardware, an autonomous intelligent agent capable of perception, thinking, and action is formed, enabling real-time, adaptive, and intelligent ultrasound scanning of patients.

[0023] In S11 of this embodiment, LoRA and real teaching data are used to fine-tune the parameters of the basic video world model, including: We selected a text-video generation model pre-trained on a large-scale general video dataset, embedded a trainable LoRA adaptation layer into the fully connected layer of the text-video generation model, and froze the original parameters to achieve efficient parameter fine-tuning. While preserving the model's general world knowledge and generation capabilities to a great extent, we introduced specific modeling capabilities for ultrasound scanning tasks with extremely low training costs. The basic model embedded with the LoRA adaptation layer is fine-tuned and trained using small batches of real multimodal teaching data collected and synchronized. The model learns to predict the grid-like multi-view visual content of the next frame based on the previous frame and text instructions. The training process drives the model to internalize the dynamic laws of probe-tissue interaction and the causal logic of ultrasound imaging, so that the generated video is not only visually realistic, but also reasonable in physical and clinical logic. Supervised learning enables the gradual learning and reproduction of the unique kinematic constraints of a specific ultrasound scanning robot in real-world operation, the visual dynamic patterns of probe movement on the body surface, and the complex visual interaction representations generated by the probe contacting and pressing different soft tissues.

[0024] In S12 of this embodiment, the video world model uses the generated output of the previous frame as the autoregressive input condition for the next frame, and is guided by the semantics of the text instructions that run throughout, including: In each autoregressive inference step, the video world model is conditionally generated with two core inputs: one is the video frame generated in the previous time step, which includes the probe position, the state of contact with the body surface, and the grid view of the corresponding ultrasound image; the other is the encoded semantic vector of the initial input text instructions. The denoising network inside the video world model extracts spatiotemporal features from the previous frame and uses the text semantic vector as a global condition, injecting it into the feature generation process through a cross-attention mechanism to predict the visual content of the next frame that conforms to the instruction intent. The video world model maintains a dynamic latent state memory throughout the autoregressive loop. The latent state memory accumulates contextual information from historical frame sequences and continuously interacts with fixed textual instruction conditions.

[0025] In S14 of this embodiment, visual trajectory data and its corresponding pseudo-label pairs are used as the core training dataset, which outputs action sequences and force control sequences within a preset time period, including: Based on the generated visual trajectory data and its corresponding pseudo-action and force label pairs, a core dataset for training the scanning strategy network is constructed. Each training sample integrates the current multi-view external camera image, real-time ultrasound image frame, encoded high-level language instruction vector describing the scanning target, as well as the robot's current joint angle, end pose, and six-dimensional force sensing state. The supervision label is the robot action sequence and contact force control sequence within a preset time window in the future. Visual trajectory data and its corresponding pseudo-labels are input into the policy network. The policy network, conditioned on the aforementioned integrated multimodal state tensor, learns through forward propagation to generate action and force sequences that are closest to the future real pseudo-labels. During training, by minimizing the difference between the predicted sequence and the pseudo-label sequence, the policy network learns the ability to proactively plan the optimal control commands that are safe, smooth, and can achieve the scanning target based on real-time multimodal perception information.

[0026] In S15 of this embodiment, the robot's own state feedback is input into the deployed policy network. Based on real-time multimodal perception information, forward calculation is performed to generate the subsequent optimal robot action sequence and force control sequence online, which are then executed by the underlying controller. This includes: The system continuously collects and integrates video streams from multiple fixed external cameras, real-time images output by ultrasonic equipment, language command text input from the operating interface, and joint encoder data and end-effector six-dimensional force sensor readings fed back by the robot controller via a high-speed data bus. All multimodal data is processed through a unified state synchronization interface and converted into a standardized tensor format required by the strategy network as input for each control cycle. The real-time data processing flow is the hardware and software foundation that ensures the strategy network can make correct decisions based on accurate and timely environmental and self-state information, and is the key to achieving closed-loop control. After receiving the real-time input, the policy network, based on the aforementioned multimodal state, predicts the optimal robot end effector sequence and contact force sequence within a future time window through its internal diffusion model denoising sampling or feedforward network calculation. This sequence is then sent to the robot's underlying motion and force control controller in real time for execution, driving the probe to complete the entire process from initial approach, safe contact, scanning along the planned path, to final stabilization on the target section. The online inference of the policy network transforms high-level semantic instructions and complex multimodal perception into specific, executable low-level control instructions. Through model predictive control and other concepts, it achieves robust and compliant operation in dynamic environments through rolling optimization, ultimately completing the clinical scanning task.

[0027] This technical solution employs a pre-trained video generation model and utilizes LoRA technology for efficient fine-tuning with small samples. It can learn and reproduce the kinematic and visual interaction patterns in actual operation, generating a massive amount of high-fidelity synthetic video sequences covering various body types, pathologies, and anatomical variations at extremely low real data cost. This eliminates the dependence on large-scale and expensive teaching data and, by utilizing the prior knowledge of physical laws of the generative model, bypasses high-fidelity biomechanical simulation engineering, achieving a leap from limited real data to unlimited synthetic training data. Reference Figure 3-4 As shown, a robot ultrasonic scanning trajectory data generation system based on a video world model is used to execute the aforementioned robot ultrasonic scanning trajectory data generation method. The system includes: The data acquisition and processing module is used to acquire and synchronize multimodal real teaching data; The video world model generation module is used to fine-tune the basic video world model and generate high-fidelity synthetic video sequences based on diverse initial conditions. The pseudo-tag generation module is used to process the synthesized video sequence and generate corresponding pseudo-tags for the robot action sequence and contact force sequence; The policy training module is used to train a policy network that generates scanning trajectories and contact forces using synthetic videos and their pseudo-labels as a dataset. The strategy execution and scanning control module is used to deploy the trained strategy network to the robot control system. It generates and executes control commands based on real-time multimodal perception information. Through pseudo-label generation technology and multimodal diffusion strategy network, it transforms visual sequences into executable robot control knowledge. It can directly respond to advanced clinical commands to achieve target-driven scanning and has strong individualized adaptability. It can dynamically adjust the probe posture and pressure based on real-time ultrasound images to optimize the acoustic window, while ensuring the smoothness and safety of the generated trajectory. It can autonomously complete the entire scanning task and adapt to different clinical scenarios and robot platforms with high flexibility and robustness.

[0028] Specifically, the pseudo-tag generation module includes: The model training unit is used to train the inverse dynamics model using real teaching data. The model takes continuous multimodal visual frames as input and regresses the action and force sequence between two frames. The inference generation unit is used to input the continuous visual frames of the synthetic video into a pre-trained inverse dynamics model. Through a sliding window, it recursively infers complete pseudo-action and force label sequences. By using a pre-trained video generation model and employing LoRA technology for efficient fine-tuning with small samples, it can learn and reproduce the kinematic and visual interaction patterns in actual operation. It generates a massive amount of high-fidelity synthetic video sequences covering various body types, pathologies, and anatomical variations at extremely low real data cost. This eliminates the dependence on large-scale and expensive teaching data. Furthermore, by utilizing the prior knowledge of physical laws of the generative model, it bypasses high-fidelity biomechanical simulation engineering and achieves a leap from limited real data to unlimited synthetic training data.

[0029] This invention is embodied in the following four aspects: 1. By integrating multimodal inputs such as ultrasound images, body surface visual information, task prompts, and robot status, and extracting semantic expression vectors through a pre-trained visual language model, this mechanism achieves deep understanding of task semantics and goal-driven trajectory guidance. Compared to traditional methods relying on vision or manual rules, this mechanism possesses stronger generalization ability and interpretability.

[0030] 2. To address the difficulty of acquiring ultrasound teaching data, a unified multimodal data synchronization scheme is proposed, and pseudo-label generation technology is introduced, including image coding prediction and inverse dynamics modeling, to expand the utilization efficiency of unlabeled or semi-labeled data and provide rich and multi-level data support for strategy training.

[0031] 3. An action generation module based on a diffusion model is employed to generate continuous trajectory sequences through progressive denoising. Compared to direct action regression methods, this approach offers better smoothness and stability, and is particularly suitable for trajectory reconstruction and semantic alignment tasks under conditions of small sample training.

[0032] 4. While keeping the main parameters of the pre-trained model frozen, only the diffusion strategy module and action decoder are fine-tuned, and task adaptation is completed by combining high-quality, limited teaching trajectories. This method significantly reduces training costs, improves deployment efficiency across organs and tasks, and has good prospects for clinical application and engineering.

[0033] It can achieve the following significant technical effects and advantages: 1. Only a very small number (e.g., 10-50) of expert teaching trajectories are needed to generate tens of thousands of high-quality training data, enabling precise planning for complex ultrasound scanning tasks and significantly reducing data acquisition costs.

[0034] 2. Introduce language prompts to enable the robot to understand advanced commands such as "find standard sections" and "scan the entire organ," and automatically generate scanning paths that meet clinical requirements based on the commands, achieving intelligent "point-and-scan" operation.

[0035] 3. By training on diverse synthetic data, the strategy model learns to cope with different body shapes and anatomical variations, and can dynamically adjust the probe posture and pressure based on real-time visual feedback (especially ultrasound images) during real scans to obtain the optimal acoustic window.

[0036] 4. The diffusion model generation method ensures the smoothness and physical feasibility of the trajectory, avoiding abrupt trajectory changes. Simultaneously, the modeling and generation of contact forces ensures the scanning process is safe and comfortable for the patient.

[0037] 5. The modular design and lightweight fine-tuning scheme facilitate the rapid transfer of this method to different ultrasound sites (such as from the abdomen to the heart) or different types of robotic platforms, demonstrating good clinical applicability and engineering prospects.

[0038] In this embodiment, the entire operation process can be controlled by a computer to provide signal feedback and implement the steps sequentially. These are all conventional knowledge in current automation control, and will not be elaborated on in this embodiment.

[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for robot ultrasound scan trajectory data generation based on a video world model, characterized by, Applied to the control system of an ultrasonic scanning robot, the specific steps include: S11: Receive teaching data collection request, collect and synchronize multimodal real teaching data of a small batch of ultrasound expert operation robot, select a text-video generation model pre-trained on video data as the basic video world model, and fine-tune the parameters of the basic video world model using LoRA and real teaching data. S12: Obtain diverse initial condition input pairs, input the initial condition pairs in batches into the fine-tuned video world model for autoregressive long sequence inference, and generate a large-scale high-fidelity synthetic video sequence. Each synthetic video sequence visually and realistically simulates the movement of the robotic ultrasound probe on the surface of a virtual patient according to clinical logic. S13: Construct a pseudo-label generation model to process the synthetic video sequence, so that the synthetic video sequence presents the corresponding dynamic change process of ultrasound image that conforms to the anatomical and acoustic laws, infer the corresponding robot action sequence and contact force sequence, and generate visual trajectory data. S14: Obtain a policy network that outputs robot actions and force control commands. Use visual trajectory data and its corresponding pseudo-label pairs as the core training dataset. Make it output the action sequence and force control sequence within a preset time period in the future, as well as the difference between the pseudo-labels at the corresponding time in the visual trajectory data, so that the policy network can generate scanning trajectories and contact forces that adapt to individual differences. S15: Deploy the scanning strategy network into the ultrasound scanning robot control system. When actually scanning the patient, acquire images from external cameras, video streams from the ultrasound equipment, and receive natural language or standardized instructions from the operator in real time. Combined with the robot's own state feedback, these are input into the deployed strategy network. Based on the real-time multimodal perception information, perform forward calculations to generate the optimal robot action sequence and force control sequence online, and then hand them over to the underlying controller for execution.

2. The robot ultrasound scan trajectory data generation method based on a video world model of claim 1, wherein, In S11, a teaching data collection request is received, and a small batch of multimodal real teaching data for the ultrasound expert operating robot is collected and synchronized, including: Upon receiving the teaching data collection request, the multimodal data acquisition process is initiated to initialize and configure the hardware and environment, including calibrating and enabling multiple external high-definition cameras located in the operating area, establishing a video stream transmission interface with the ultrasonic equipment, ensuring that the data log function of the robot control system is enabled, and verifying the communication and zero point of the contact force / torque sensor. When ultrasound experts operate the robot to perform scanning tasks, the data streams of all modalities are captured synchronously. Through hardware trigger signals, it is ensured that each frame of external view video, ultrasound image sequence, and robot status data packet is stamped with a uniform timestamp. The robot status data package includes the six-dimensional pose of the robotic arm end effector, six-dimensional contact force / torque readings, and servo control commands.

3. The robot ultrasound scan trajectory data generation method based on a video world model of claim 2, wherein, In S12, diverse initial condition input pairs are obtained, and these initial condition pairs are batch-input into the fine-tuned video world model for autoregressive long-sequence inference, including: The initial visual frame for each input pair is sampled from real patient data, high-fidelity anatomical model renderings, and historical frames enhanced by elastic deformation and illumination simulation. The input pair generates corresponding text instructions from a structured instruction library. The time of each initial moment is strictly aligned with the external view video frame and the real-time ultrasound image video frame, and then stitched together to form a unified grid-like multi-view visual input. The batch-organized initial conditions are input in parallel into the fine-tuned video world model. The video world model uses the generated output of the previous frame as the autoregressive input condition for the next frame, and is guided by the semantic text instructions throughout the process to gradually infer and generate a long sequence of future frames.

4. The robot ultrasound scan trajectory data generation method based on a video world model of claim 3, wherein, The video world model uses the generated output of the previous frame as the autoregressive input condition for the next frame, and is guided by semantic textual instructions throughout, including: In each autoregressive inference step, the video world model is conditionally generated with two core inputs: one is the video frame generated in the previous time step, which includes the probe position, the state of contact with the body surface, and the grid view of the corresponding ultrasound image; the other is the encoded semantic vector of the initial input text instructions. The denoising network inside the video world model extracts spatiotemporal features from the previous frame and uses the text semantic vector as a global condition, injecting it into the feature generation process through a cross-attention mechanism to predict the visual content of the next frame that conforms to the instruction intent. The video world model maintains a dynamic latent state memory throughout the autoregressive loop. This latent state memory accumulates contextual information from historical frame sequences and continuously interacts with fixed textual instruction conditions.

5. The robot ultrasound scan trajectory data generation method based on a video world model of claim 4, wherein, In S13, a pseudo-label generation model is constructed to process the synthesized video sequence, causing the synthesized video sequence to exhibit a corresponding dynamic change process of ultrasound images that conforms to anatomical and acoustic laws, including: A pseudo-label generation model is constructed using an inverse dynamics scheme. A deep neural network model is trained using a small amount of real teaching data collected in the first stage as a training set. The deep neural network model takes two sets of multimodal visual frames that are consecutive in time as input. The difference in visual state is analyzed through an encoder-regressor architecture, and the action sequence and force control sequence that the robot end effector should execute within the observation interval are directly regressed. After training, the trained pseudo-label generation model is applied to a large number of synthetic video sequences without action labels generated in the second stage. A sliding window with a fixed step size is used to input the continuous visual frames within the window into the pseudo-label generation model. The pseudo-label generation model outputs pseudo-action and force sequences for each window. Through splicing and smoothing, pseudo-robot action label sequences and pseudo-contact force label sequences that run through the entire duration of each synthetic video are generated.

6. The method for generating robot ultrasonic scanning trajectory data based on a video world model as described in claim 5, characterized in that, In S14, visual trajectory data and its corresponding pseudo-label pairs are used as the core training dataset, enabling it to output action sequences and force control sequences within a preset time period, including: Based on the generated visual trajectory data and its corresponding pseudo-action and force label pairs, a core dataset for training the scanning strategy network is constructed. Each training sample integrates the current multi-view external camera image, real-time ultrasound image frame, encoded high-level language instruction vector describing the scanning target, as well as the robot's current joint angle, end pose, and six-dimensional force sensing state. The supervision label is the robot action sequence and contact force control sequence within a preset time window in the future. Visual trajectory data and its corresponding pseudo-labels are input into the policy network. The policy network, conditioned on the aforementioned integrated multimodal state tensor, learns through forward propagation to generate action and force sequences that are closest to the future real pseudo-labels. During training, by minimizing the difference between the predicted sequence and the pseudo-label sequence, the policy network learns the ability to proactively plan the optimal control commands that are safe, smooth, and can achieve the scanning target based on real-time multimodal perception information.

7. The method for generating robot ultrasonic scanning trajectory data based on a video world model as described in claim 6, characterized in that, In S15, along with the robot's own state feedback, the data is input into the deployed policy network. Based on real-time multimodal perception information, forward computation is performed to generate the optimal robot action sequence and force control sequence online, which are then executed by the underlying controller. This includes: The system continuously collects and integrates video streams from multiple fixed external cameras, real-time images output by ultrasonic equipment, language command text from the operating interface, and joint encoder data and end-effector six-dimensional force sensor readings from the robot controller via a high-speed data bus. All multimodal data is processed through a unified state synchronization interface and converted into a standardized tensor format required by the policy network as input for each control cycle. After receiving the real-time input, the strategy network, based on the aforementioned multimodal state, uses its internal diffusion model for denoising and sampling or feedforward network calculation to predict the optimal robot end effector sequence and contact force sequence within a future time window. This sequence is then sent to the robot's underlying motion and force control controller in real time to drive the probe through the entire process from initial approach, safe contact, scanning along the planned path, to finally stabilizing on the target plane.

8. The method for generating robot ultrasonic scanning trajectory data based on a video world model as described in claim 7, characterized in that, In S11, LoRA and real teaching data are used to fine-tune the parameters of the basic video world model, including: We selected a text-video generation model pre-trained on a large-scale general video dataset, embedded a trainable LoRA adaptation layer into the fully connected layer of the text-video generation model, and froze the original parameters. The base model embedded in the LoRA adaptation layer is fine-tuned and trained using small batches of real multimodal teaching data collected and synchronized, learning to predict the grid-like multi-view visual content of the next frame based on the previous frame and text instructions. Supervised learning enables the gradual learning and reproduction of the unique kinematic constraints of a specific ultrasound scanning robot in real-world operation, the visual dynamic patterns of probe movement on the body surface, and the complex visual interaction representations generated by the probe contacting and pressing different soft tissues.

9. A robot ultrasonic scanning trajectory data generation system based on a video world model, characterized in that, The system is used to perform the robot ultrasonic scanning trajectory data generation method according to any one of claims 1-8, the system comprising: The data acquisition and processing module is used to acquire and synchronize multimodal real teaching data; The video world model generation module is used to fine-tune the basic video world model and generate high-fidelity synthetic video sequences based on diverse initial conditions. The pseudo-tag generation module is used to process the synthesized video sequence and generate corresponding pseudo-tags for the robot action sequence and contact force sequence; The policy training module is used to train a policy network that generates scanning trajectories and contact forces using synthetic videos and their pseudo-labels as a dataset. The strategy execution and scanning control module is used to deploy the trained policy network to the robot control system and generate and execute control commands based on real-time multimodal perception information.

10. The robot ultrasonic scanning trajectory data generation system based on a video world model as described in claim 9, characterized in that, The pseudo-tag generation module includes: The model training unit is used to train an inverse dynamics model using real teaching data. The model takes continuous multimodal visual frames as input and regresses the action and force sequence between two frames. The inference generation unit is used to input the trained inverse dynamics model with consecutive visual frames of the synthesized video and recursively infer the complete sequence of pseudo-actions and force labels through a sliding window.

Images