Ultrasound language imitation learning control method and system of deep reinforcement learning

Abstract

This invention relates to the field of robotics and discloses a deep reinforcement learning-based ultrasound language imitation learning control method and system. Through lightweight model architecture design, community-driven data training, and asynchronous inference optimization, this invention achieves high success rate and high precision in ultrasound scanning tasks, while lowering the hardware deployment threshold and promoting the widespread application of intelligent ultrasound technology in primary healthcare institutions. It effectively solves the problems of excessively large model size, high deployment costs, weak generalization ability, and insufficient real-time performance of existing vision, language, and motion models in ultrasound robot applications.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically, to a deep reinforcement learning-based ultrasonic language imitation learning control method and system. Background Technology

[0002] With the widespread application of robotics in the medical service field, the requirements for the autonomous decision-making and motion planning capabilities of ultrasound robots are increasing. Visual-language multimodal input, which integrates visually perceived environmental information with natural language task instructions, has become a core technical path for robots to understand complex scenes and perform diverse tasks. Related research has formed an important technical branch called visual-language-motion (VLA) models.

[0003] Most existing VLA models are adapted and optimized based on large-scale pre-trained visual language models. By integrating visual observations, language commands, and robot propagation perception information, they achieve the predictive generation of action sequences. These models, leveraging the rich world knowledge and cross-modal correlation capabilities inherent in pre-trained visual language models, have made some progress in basic robot operations such as grasping, placement, and assembly. For example, some existing technologies use the Transformer architecture to build end-to-end models, directly encoding visual image features and language commands and inputting them into a decoder to output continuous or discrete robot action commands, achieving a direct mapping from perception to action.

[0004] However, existing technologies still face many unresolved issues in practical ultrasound detection applications. First, the contradiction between model size and deployment cost is prominent. Current high-performance VLA models typically contain billions of parameters, requiring massive computing resources for training and relying on high-performance GPU hardware for inference, making them difficult to deploy on consumer devices or low-cost robotic platforms, severely limiting the technology's widespread application. Second, there are limitations in acquiring and adapting training data. Existing models largely rely on standardized datasets built by academic institutions or industry. While these datasets are well-labeled, their scenario coverage is limited, and their acquisition costs are high, ignoring the massive heterogeneous data resources generated by community-level low-cost robotic platforms. This results in insufficient generalization ability of the models in real-world complex environments.

[0005] Existing technologies also have shortcomings in terms of the real-time performance and accuracy of motion planning. Traditional VLA models employ a synchronous inference mode, requiring the prediction of complete action blocks before execution. This results in a significant delay between action execution and environmental perception, leading to insufficient responsiveness in dynamically changing environments. Furthermore, some models use a single attention mechanism or a simple action generation strategy, making it difficult to balance the smoothness of action sequences with the accuracy of task execution. In scenarios requiring precise manipulation, such as ultrasonic testing, this can easily lead to motion jitter or task failure. In addition, the multimodal input fusion methods of existing models are not optimized enough. The integration of visual features, language commands, and robot state information lacks an efficient collaborative mechanism, resulting in an insufficient understanding of complex task commands and an incomplete perception of environmental changes, thus affecting the rationality and reliability of motion planning.

[0006] While some studies have attempted to lower the deployment threshold through model compression and lightweight architecture design, or to improve motion prediction quality using techniques such as diffusion models and autoregressive generation, a mature solution that balances model size, inference efficiency, and task performance has yet to be established. How to construct a lightweight, high-performance, and highly generalizable visual-language multimodal input model to achieve accurate, real-time, and low-cost deployment of ultrasonic robot motion planning has become a pressing technical challenge in this field. Summary of the Invention

[0007] To overcome the technical problems of existing vision, language, and motion models in ultrasound robot applications, such as excessive model size, high deployment cost, weak generalization ability, and insufficient real-time performance, this invention provides a deep reinforcement learning-based ultrasound language imitation learning control method and system. Through lightweight model architecture design, community-driven data training, and asynchronous inference optimization, it achieves high success rate and high precision execution of ultrasound scanning tasks, while lowering the hardware deployment threshold and promoting the widespread application of intelligent ultrasound technology in primary healthcare institutions.

[0008] The technical solution of this invention is as follows:

[0009] In a first aspect, the present invention provides a deep reinforcement learning-based ultrasonic language imitation learning control method, comprising the following steps:

[0010] Step S1: Complete hardware deployment, software environment configuration, and system parameter calibration;

[0011] Step S2: Filter and integrate community-level robot datasets, and after preprocessing, form a standardized training dataset adapted to ultrasound scanning tasks;

[0012] Step S3: Use the standardized training dataset to jointly train the compact pre-trained visual language module and the streaming matching action expert module to obtain a lightweight visual, language, and action model.

[0013] Step S4: Standardize the multimodal observation data of the ultrasound scanning scene to generate a model input sequence in a unified format. The multimodal observation data includes image data, language command data, and sensor data.

[0014] Step S5: Input the model input sequence into the lightweight vision, language, and action model to generate ultrasound scan action blocks and store them in the action sequence, while updating the action sequence through an asynchronous inference mechanism;

[0015] Step S6: The action execution layer drives the robotic arm and ultrasonic probe to perform scanning according to the action sequence, and simultaneously provides real-time feedback of execution status data;

[0016] Step S7: Based on the execution status data and real-time observation data fed back during the scanning process, dynamically correct the action parameters, and simultaneously record effective execution data to incrementally optimize the lightweight vision, language, and action models;

[0017] Step S8: After completing the scan, generate a scan report.

[0018] As a preferred embodiment of the present invention, step S2 specifically includes the following sub-steps:

[0019] Step S21: Select community-level robot datasets from the community platform and integrate them to form a training dataset. The selection criteria include simultaneously containing RGB images and corresponding motion sequence data, having ≥50 episodes, and having data quality that meets the requirements for ultrasound scene adaptation.

[0020] Step S22: Sample representative images of preset frames for each episode of the training dataset and input them into the visual language model to generate standardized natural language instructions;

[0021] Step S23: Normalize the camera view of all community-level robot datasets in the training dataset;

[0022] Step S24: Adjust the brightness and contrast of the RGB images in the training dataset and apply Gaussian blur. Add small Gaussian noise with σ=0.01 to the action sequence data in the training dataset. Perform synonym replacement and sentence rewriting on the natural language instructions in the training dataset to generate multiple versions of natural language instructions.

[0023] As a preferred embodiment of the present invention, step S3 specifically includes the following sub-steps:

[0024] Step 31: Configure the training environment;

[0025] Step 32, Module Training: Construct the compact pre-trained visual language module based on the visual encoder and language decoder, and train it using the standardized training dataset with the image feature extraction accuracy and natural language instruction understanding accuracy as optimization objectives.

[0026] Step 33: Initialize the parameters of the streaming matching action expert module, and train it using the streaming matching loss function with the optimization objectives of action sequence smoothness and action prediction accuracy.

[0027] Step 34: Connect the trained compact pre-trained visual language module with the streaming matching action expert module to form the lightweight visual, language, and action model, and perform joint training using the standardized training dataset with the success rate of ultrasound scanning task execution as the optimization objective.

[0028] As a preferred embodiment of the present invention, step S4 specifically includes the following sub-steps:

[0029] Step S41: Acquire image data through a camera, and perform resolution adjustment, noise filtering, and visual token compression on the acquired image data;

[0030] Step S42: Collect natural language commands input by medical staff through the command input area of ​​the interaction layer, and perform command preprocessing, BPE word segmentation, and sequence completion and truncation processing on the collected natural language commands.

[0031] Step S43: Collect sensor data through the sensor acquisition module, and perform data cleaning, data normalization and single token mapping processing on the collected sensor data;

[0032] Step S44: The processed image data, the natural language instructions, and the sensor data are concatenated in the order of visual tokens, language instruction tokens, and sensor tokens to form the model input sequence in a unified format.

[0033] As a preferred embodiment of the present invention, step S5 specifically includes the following sub-steps:

[0034] Step S51: The intelligent decision layer receives the model input sequence and inputs it into the lightweight vision, language, and action model to generate an initial action block containing multiple control steps; at the same time, the initial action block is sent to the action execution layer, the action execution layer initializes the action sequence, and stores the initial action block in the action sequence.

[0035] Step S52: The action execution layer executes actions according to the action sequence and monitors the number of remaining actions in the queue in real time;

[0036] Step S53: When the proportion of the remaining actions in the action sequence is lower than the preset proportion, collect the current multimodal observation data, and send it to the intelligent decision layer after standardization processing;

[0037] Step S54: After receiving the processed current multimodal observation data, the intelligent decision-making layer runs the lightweight vision, language, and action model to generate new action blocks.

[0038] Step S55: The new action block is asynchronously sent to the action execution layer, and the action execution layer updates the action sequence through the overlapping region aggregation strategy and continues to execute the action.

[0039] As a preferred embodiment of the present invention, step S6 specifically includes the following sub-steps:

[0040] Step S61: The robotic arm performs scanning actions in accordance with the action instructions in the action sequence and the ultrasonic probe.

[0041] Step S62: The execution status data during the scanning process is fed back to the intelligent decision-making layer and the interaction layer in real time; the execution status data includes actual position, actual angle, position error, angle error and pressure sensor data.

[0042] As a preferred embodiment of the present invention, step S7 specifically includes the following sub-steps:

[0043] Step S71: During the scanning process, monitor the actual position and angle of the end effector of the robotic arm in real time, compare them with the target value, and calculate the position error and angle error; if the position error or angle error is greater than the preset error, correct the action command in the next control step.

[0044] Step S72: During the scanning process, the pressure sensor continuously monitors the contact pressure. If the contact pressure exceeds the preset pressure range, the robotic arm automatically adjusts its height.

[0045] Step S73: Record the effective execution data during the scanning process in real time. The effective execution data includes the multimodal observation data, the ultrasound scanning action block, the execution status data, and the task execution result.

[0046] Step S74: Set incremental training trigger conditions. When the accumulated effective execution data reaches a certain amount or the system runs continuously for a preset time, perform incremental optimization on the lightweight vision, language, and action models.

[0047] As a preferred embodiment of the present invention, step S8 specifically includes the following sub-steps:

[0048] Step S81: After the scan is completed, the robotic arm carries the ultrasonic probe back to the initial position and automatically saves the image data, motion parameters and pressure data during the scan process, and generates the scan report;

[0049] Step S82: Determine the image quality based on the scan report. If it is not up to standard, a second scan can be triggered according to the user's instructions, and steps S4 to S7 can be executed again.

[0050] Secondly, the present invention provides a deep reinforcement learning ultrasonic language imitation learning control system for executing any of the above-mentioned deep reinforcement learning ultrasonic language imitation learning control methods.

[0051] As a preferred embodiment of the present invention, the system includes:

[0052] The action execution layer is used to perform ultrasound scanning actions;

[0053] The multimodal perception layer is used to collect multimodal observation data of the ultrasound scanning scene, including a camera and a sensor data acquisition module. The camera provides global monitoring, probe close-up and organ positioning perspectives respectively, and the sensor data acquisition module simultaneously acquires the joint status and pressure data of the robotic arm.

[0054] The intelligent decision-making layer is used to realize multimodal information fusion, model training and asynchronous action planning, including a lightweight vision, language and action model and an asynchronous inference scheduling module. The lightweight vision, language and action model consists of a compact pre-trained vision and language module and a streaming matching action expert module.

[0055] The interaction layer is used for human-computer collaboration and system monitoring.

[0056] The intelligent decision-making layer establishes bidirectional communication with the action execution layer, the multimodal perception layer, and the interaction layer, forming a closed-loop system of perception-decision-execution-feedback.

[0057] According to the above-described solution, the beneficial effects of this invention are as follows:

[0058] This invention achieves high success rate and high precision in ultrasound scanning tasks through lightweight model architecture design, community-driven data training, and asynchronous inference optimization. At the same time, it lowers the hardware deployment threshold, promotes the widespread application of intelligent ultrasound technology in primary healthcare institutions, and effectively solves the problems of excessive model size, high deployment cost, weak generalization ability, and insufficient real-time performance of existing vision, language, and motion models in ultrasound robot applications. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the ultrasonic language imitation learning control method using deep reinforcement learning in one embodiment of the present invention.

[0060] Figure 2 for Figure 1 A detailed flowchart of step S2 is shown below;

[0061] Figure 3 for Figure 1 A detailed flowchart of step S3 is shown below;

[0062] Figure 4 for Figure 1 A detailed flowchart of step S4 is shown below;

[0063] Figure 5 for Figure 1 A detailed flowchart of step S5 is shown below;

[0064] Figure 6 for Figure 1 A detailed flowchart of step S6 is shown below;

[0065] Figure 7 for Figure 1 A detailed flowchart of step S7 is shown below;

[0066] Figure 8 for Figure 1 A detailed flowchart of step S8 is shown below;

[0067] Figure 9 This is a schematic diagram of the structure of a deep reinforcement learning ultrasonic language imitation learning control system according to an embodiment of the present invention;

[0068] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0069] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] It should be noted that the terms "comprising" and "having," and any variations thereof, in the specification and claims of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0071] like Figure 1 As shown, one embodiment of the present invention provides a deep reinforcement learning-based ultrasonic language imitation learning control method, comprising the following steps:

[0072] Step S1: Complete the deployment of hardware equipment, configuration of software environment and calibration of system parameters, and establish accurate communication links and coordinate mapping relationships between modules to provide stable and reliable hardware support and software foundation for subsequent data acquisition, model training and action execution.

[0073] Specifically, the hardware deployment includes:

[0074] 1) Robotic arm module: A 6-DOF industrial-grade robotic arm (load ≥ 5kg, repeatability ≤ ±0.02mm) is selected. The end effector is equipped with a customized ultrasound probe holder with a built-in pressure sensor (measurement range 0-50N, accuracy ±0.1N) to monitor the contact pressure between the probe and the patient's body surface in real time, avoiding excessive pressure that could cause patient discomfort or probe damage. The robotic arm base is equipped with a horizontal adjustment device to ensure stable posture during operation and reduce the impact of vibration on ultrasound scanning.

[0075] 2) Visual Acquisition Module: Deploys three high-resolution industrial cameras (2592×1944 resolution, 30fps frame rate), corresponding to a global monitoring view camera (installed on the top of the scanning room, covering the entire operating area, used to locate the patient's position and the global movement of the robotic arm), a probe close-up view camera (integrated on the wrist of the robotic arm, 5-8cm away from the probe, capturing the probe's posture and details of the scanning area), and an organ positioning view camera (installed on the side of the scanning bed, focusing on the target scanned organ area, assisting in the identification of organ boundaries); the lenses of the three cameras are low-distortion lenses (distortion rate ≤1%), and white balance and exposure calibration are performed to ensure stable image color and brightness under different lighting conditions.

[0076] 3) Control and Computing Module: Equipped with an industrial control computer (CPU: Intel Core i9-14900K, RAM: 64GB, SSD: 2TB) to run motion planning algorithms and robot control programs; if inference speed needs to be improved, an external consumer-grade GPU (NVIDIA RTX 4080, 16GB VRAM) can be connected to the industrial control computer via a PCIe 4.0 interface to meet the computational needs of real-time model inference; it is also equipped with an emergency stop button and emergency stop circuit, which can immediately cut off the power supply when abnormal robot arm movement or excessive pressure is detected to ensure operational safety.

[0077] Specifically, the software environment configuration includes:

[0078] 1) Operating System Installation: Install the Linux Ubuntu 22.04 LTS operating system on the industrial control computer, configure the RT_PREEMPT real-time kernel, and control the system scheduling latency to within 1ms by modifying kernel parameters (such as HZ=1000, PREEMPT_IRQ_FAST=1) to meet the real-time requirements of robot control; disable redundant services such as automatic system updates, hibernation, and screen savers to reduce system resource consumption; install a firewall and open necessary ports (such as EtherCAT communication port, ROS 2 communication port, and TCP / IP interaction port) to ensure network security.

[0079] 2) Driver Installation: Install the official driver provided by the robotic arm manufacturer (supporting the EtherCAT protocol), and configure the robotic arm device address (default 0x01) and communication cycle (1ms); install the driver provided by the camera manufacturer (pylonSDK 7.3) to ensure that the camera can be recognized by the system and supports functions such as image acquisition, exposure adjustment, and white balance setting; install the pressure sensor driver and configure the sampling frequency (100Hz) and data format (16-bit binary); after all drivers are installed, perform a connectivity test to ensure that the device can respond normally to control commands and data acquisition requests.

[0080] 3) Algorithm dependency library configuration: Based on the Python 3.10 environment, an independent virtual environment (named ultrasound_ml) is created using conda to avoid dependency conflicts; PyTorch 2.2.0 (supporting CUDA 12.1) is installed in the virtual environment for model training and inference; OpenCV 4.9.0 is installed for image preprocessing (resolution adjustment, noise filtering, token compression); PyQt5 5.15.9 is installed for human-computer interaction interface development; ROS 2 Humble is installed for inter-module data communication; and numerical computing and image processing libraries such as NumPy 1.26.0, SciPy 1.11.3, and Pillow 10.2.0 are also installed to ensure the normal operation of the algorithm module; after all dependency libraries are installed, the availability of the libraries is verified through test scripts.

[0081] Specifically, system parameter calibration includes:

[0082] 1) Robotic Arm Accuracy Calibration: A laser tracker (model: Leica AT960-MR) with a measurement accuracy of ±0.01mm was used to calibrate the motion parameters of each joint of the robotic arm. The robotic arm was controlled to move at 100 preset target points (covering the entire workspace). The laser tracker measured the actual position of the end effector in real time and compared it with the theoretical position. The error value of each joint at different angles was recorded. An error compensation model (using a polynomial fitting algorithm) was established through the robotic arm control software to compensate for the joint motion angles in real time, ultimately ensuring that the positioning accuracy of the end probe is ≤±0.05mm. At the same time, the motion response speed of the robotic arm under different loads (0-5kg, simulating different probe weights) was tested, and the motion acceleration parameter was adjusted to 0.5m / s² to avoid attitude fluctuations caused by high-speed movement.

[0083] 2) Vision System Calibration: Using a 12×9 checkerboard calibration board with a grid side length of 20mm, intrinsic and extrinsic parameters of the three cameras were calibrated. The calibration board was placed within the camera's field of view in at least 10 different poses and at least 5 different positions, and more than 20 sets of calibration images were acquired. The camera's focal length, principal point coordinates, distortion coefficients, and other intrinsic parameters were calculated using OpenCV's calibrateCamera function, ensuring that the intrinsic parameter calibration error was ≤0.5 pixels. Then, the relative position and attitude extrinsic parameters between the three cameras were obtained using ROS 2's multicamera_calibrator tool. Hand-eye calibration was performed using an "eye-on-hand" mode. The calibration board was fixed to the scanning bed, and the robotic arm was controlled to move the probe's near-field camera, acquiring 30 sets of images of the calibration board at different positions. The transformation matrix between the camera coordinate system and the robotic arm coordinate system was calculated using the Tsai-Lenz algorithm, ensuring that the coordinate transformation error was ≤±0.1mm.

[0084] 3) Pressure sensor calibration: Place the pressure sensor on a standard pressure test bench (accuracy ±0.01N) and apply standard pressures of 0N, 10N, 20N, 30N, 40N, and 50N respectively. Record the sensor output value and the standard pressure value. Establish a calibration curve (linear fitting) using the least squares method to correct the sensor output data and ensure that the pressure measurement error is ≤±0.1N. After calibration, write the calibration curve parameters into the sensor driver program to achieve real-time calibration.

[0085] Step S2: Filter and integrate community-level robot datasets. After preprocessing, a standardized training dataset adapted to ultrasound scanning tasks is formed, providing high-quality, multi-scenario data support for the model to learn ultrasound operation logic.

[0086] Specifically, such as Figure 2 As shown, step S2 includes the following sub-steps:

[0087] Step S21: Dataset Filtering and Integration

[0088] 1) Data source selection: Select community-level robot datasets from community platforms (such as Hugging Face platform) and integrate them to form training datasets; the selection scope covers the operation data of low-cost robot platforms such as SO100 / SO101 and Franka Panda. These datasets are characterized by high scenario diversity, large data volume and low acquisition cost.

[0089] 2) Selection Criteria: Datasets are selected according to three core criteria: First, they must contain both RGB images and corresponding motion sequence data (motion sequences must include parameters such as joint angles and movement speeds) to ensure the integrity of multimodal input; Second, the number of episodes must be ≥50, and each episode must contain a complete task execution process (from initialization to task completion) to ensure sufficient data volume; Third, the data quality must meet the requirements for ultrasound scene adaptation, i.e., it must include actions similar to ultrasound operations such as fine grasping, angle adjustment, and smooth movement, avoiding the introduction of noise from irrelevant data.

[0090] 3) Dataset Integration: Several datasets (e.g., 481) that met the requirements were ultimately selected and integrated to form a training dataset of 22.9 Kepisodes and 10.6M frames. Simultaneously, all data underwent format standardization: image formats from different datasets were converted to JPG (90% compression quality), motion sequence data were converted to CSV format (fields including timestamp, joint angles 1-6, joint velocities 1-6, and end-effector position), and language instruction data were converted to TXT format. Files were named according to the rule "dataset number-episode number-frame number" (e.g., D001-E005-F0200.jpg), and a dataset index table was created for easy management and retrieval.

[0091] Step S22, Task Labeling Optimization

[0092] 1) Representative image sampling: For each episode of the training dataset, representative images of preset frames (e.g., 10 frames) are uniformly sampled in time series, including the probe initial position frame, target position frame, key operation frames (e.g., angle adjustment, pressure change), etc., to fully cover the key nodes of the task execution process.

[0093] 2) Standardized instruction generation: The sampled representative images are input into the visual language model to generate standardized natural language instructions;

[0094] 3) Labeling and Association: The generated standardized natural language instructions are associated and stored with the corresponding episode data. An "instruction.txt" file is created in the root directory of each episode to store the standardized instructions for that episode. At the same time, an instruction field is added to the dataset index table to establish the correspondence between images, action sequences and language instructions, ensuring that multimodal data can be acquired synchronously during model training.

[0095] Step S23: Camera perspective normalization

[0096] Camera view normalization processing for all community-level robot datasets in the training dataset:

[0097] 1) Viewpoint Classification Mapping: To address the issue of inconsistent camera naming in the community dataset (such as "top", "wrist", "side", "front", etc.), the camera installation location, shooting range, and purpose of each dataset were manually analyzed to uniformly map all camera views into three standardized viewpoints: OBS_IMAGE_1 (top view, corresponding to the global monitoring viewpoint in ultrasound operation), OBS_IMAGE_2 (wrist view, corresponding to the viewpoint of the ultrasound probe's built-in camera), and OBS_IMAGE_3 (side view, corresponding to the auxiliary monitoring viewpoint).

[0098] 2) Redundant perspective handling: For unused redundant perspectives in the dataset (such as local close-up perspectives and irrelevant environment perspectives), they are directly discarded during the data preprocessing stage, and only the image data of the three standardized perspectives are retained; for some datasets that are missing a certain perspective, they are marked as missing, and the model automatically ignores the perspective data during training, ensuring that the visual input format of the model is consistent on different datasets and improving the cross-scene generalization ability.

[0099] Step S24: Data Augmentation Processing

[0100] To improve the robustness and generalization ability of the model, a triple data augmentation strategy was designed to address the specific characteristics of ultrasound operation scenarios:

[0101] 1) Image Enhancement: Multi-dimensional enhancement of RGB images in the training dataset to simulate image changes under different clinical environments: I. Brightness Adjustment: Randomly adjust the brightness of RGB images within ±20% to simulate changes in scanning room lighting (such as lights being turned on / off, changes in light outside the window); II. Contrast Adjustment: Randomly adjust the contrast of RGB images within ±15% to enhance the model's ability to recognize image details (such as organ boundaries, probe contours); III. Gaussian Blur: Randomly blur RGB images using a Gaussian filter with σ≤0.5 to simulate image blurring caused by ultrasound equipment imaging noise, probe surface contamination, or patient hair; IV. Random Cropping: Randomly crop within 10% of the RGB image edges to simulate slight shifts in camera viewpoint.

[0102] 2) Motion Enhancement: Add tiny Gaussian noise (σ=0.01) to the motion sequence data in the training dataset to simulate minor disturbances during robot execution (such as wear on robotic arm joints, environmental vibration, and pressure fluctuations), enhancing the model's robustness to operational errors and making the trained model more stable in practical applications; at the same time, the motion sequences are time-scaled (scaling ratio 0.8-1.2 times) to simulate motion execution scenarios at different operating speeds;

[0103] 3) Instruction Enhancement: Synonym replacement and sentence rewriting are performed on the natural language instructions in the training dataset to generate multiple versions of natural language instructions, improving the model's ability to understand instructions with different expressions; Synonym replacement: Core verbs (such as "move", "adjust", "locate", "place") and locative words (such as "above", "below", "left", "right") in natural language instructions are replaced with synonyms; Sentence rewriting: Active sentences are rewritten as passive sentences (such as "move the ultrasound probe to the liver area" → "the ultrasound probe was moved to the liver area"), or the word order is adjusted to keep the meaning of the instruction unchanged; 3-5 multiple versions of natural language instructions are generated for each natural language instruction, which are stored in association with the original data, and one of the multiple versions of natural language instructions is randomly selected as input during training.

[0104] Step S3: Jointly train the compact pre-trained visual language module and the streaming matching action expert module using a standardized training dataset to obtain a lightweight visual, language, and action model. The lightweight visual, language, and action model has the core ability to parse ultrasonic task instructions from multimodal inputs, perceive scene information, and generate precise operation actions, providing algorithmic support for subsequent action planning.

[0105] Specifically, such as Figure 3 As shown, step S3 includes the following sub-steps:

[0106] Step 31: Configure the training environment

[0107] 1) Hardware configuration: The training cluster consists of four consumer-grade GPUs (NVIDIA RTX 4090, 16GB VRAM), which are used to achieve high-speed communication between GPUs (100GB / s bandwidth) through NVLink technology; the industrial control computer is equipped with an Intel Core i9-14900K CPU (24 cores and 32 threads), 64GB DDR5 memory, and a 2TB NVMe SSD to meet the needs of data reading and model calculation; the training cluster is equipped with a 6000W redundant power supply to ensure stable power supply during training.

[0108] 2) Training tool optimization: Kernel optimization was performed using the torch.compile function of PyTorch 2.2.0 to accelerate the model training process, improving training speed by approximately 30%; a distributed training framework (DDP) was adopted to evenly distribute the dataset across 4 GPUs by episode, enabling parallel training and improving training efficiency; training checkpoints were set to save model weights and training logs every 1000 iterations to avoid data loss due to training interruptions; mixed precision training (bfloat16) was adopted to reduce GPU memory usage while ensuring training accuracy, allowing a single GPU to process larger batches of data (batch size=32).

[0109] Step 32: Construction and Training of Compact Pre-trained Visual Language Module (VLM)

[0110] A compact pre-trained visual language module was constructed based on the SigLIP visual encoder and the SmolLM2 language decoder. The optimization objectives were image feature extraction accuracy (validated through an image classification task) and natural language instruction understanding accuracy (validated through an instruction matching task). Training was performed using a standardized training dataset. The optimizer was AdamW with a learning rate of 1e-5, weight decay of 1e-4, and 5000 training iterations. During training, the layer skipping strategy was adjusted (to verify the performance differences of retaining 12, 14, 16, and 18 layers) and the visual token compression parameters were adjusted (to test the feature extraction effect of 32, 64, and 128 tokens). The optimal configuration of retaining the first 16 layers and 64 visual tokens per frame was ultimately determined, ensuring that the model achieved a key feature extraction accuracy ≥95% with minimal computational cost.

[0111] Step 33: Construction and Training of the Streaming Matching Action Expert Module

[0112] The Flow Matching action expert module is built using the Transformer architecture with conditional flow matching.

[0113] The parameters of the streaming matching action expert module were initialized (using Xavier uniform initialization), and training was performed using the streaming matching loss function, with the optimization objectives being action sequence smoothness (standard deviation of the first derivative of the action sequence) and action prediction accuracy (mean squared error with the expert action sequence). The optimizer used was AdamW with a learning rate of 5e-6, weight decay of 1e-4, and 8000 training iterations. During training, the parameters of the interleaved attention mechanism were optimized by monitoring the smoothness of the action sequence and the action prediction accuracy (e.g., setting the number of attention heads to 8 and the attention dropout rate to 0.1), resulting in an improvement of action sequence smoothness of over 25% and an action prediction accuracy of ≥90%.

[0114] The streaming matching loss function is defined as:

[0115] ;

[0116] In the formula, a loss function with respect to parameter θ is defined. Used to train models over multiple time spans The prediction consistency is as follows. The meanings of the variables are as follows:

[0117] : Current action distribution Conditional distribution of future action sequences The joint expectations; The action at the current moment is determined by the agent based on observations. Obtained by sampling; The current observation includes information such as visual, verbal, or sensor input. Starting from time t, spanning [time range] The future sequence of actions or behaviors; : A prediction or estimate of a given sequence of observations and future actions by a model (such as a neural network) controlled by parameter θ; : Reference values ​​provided by the objective function or teacher model, used as supervision signals (e.g., true rewards or known environmental outputs); Euclidean norm squared, used to measure the deviation between the prediction and the target.

[0118] In the Transformer block of the streaming action expert module, an alternating structure of cross-attention (CA) and causal self-attention (SA) is adopted to replace the single attention mechanism in existing technologies. Specifically, the CA layer uses the fusion features of a compact pre-trained visual language module as keys and values, and action tokens as queries, achieving precise guidance of action generation by multimodal features. The SA layer employs causal masking to ensure that each action token in the action block depends only on its preceding token, avoiding leakage of future action information and enhancing the temporal continuity of the action sequence. Compared to pure CA or pure SA structures, the alternating structure of cross-attention (CA) and causal self-attention (SA) improves the smoothness of the action sequence and the accuracy of action prediction.

[0119] Step 34, Joint Training

[0120] The pre-trained compact visual language module is concatenated with the streaming matching action expert module to form a lightweight visual, language, and action model. The model is jointly trained using a standardized training dataset, with the success rate of ultrasound scanning task execution as the optimization objective. The joint training specifically includes:

[0121] 1) Training parameter settings:

[0122] The optimizer used is AdamW with a learning rate of 3e-6, a weight decay of 1e-4, 10,000 training iterations, and a batch size of 16. An early stopping strategy is adopted: if the success rate of the validation set task does not improve for 500 consecutive iterations, training is stopped and the optimal model weights are selected.

[0123] 2) Training process monitoring: During training, the model is tested on an independent validation set (containing 2K episodes) every 500 iterations to evaluate the success rate of the ultrasound scanning task, the accuracy of action prediction, the accuracy of image feature extraction, and other indicators. At the same time, the training loss of the model (total loss, compact pre-trained visual language module loss, and streaming matching action expert module loss) is monitored to ensure that the loss decreases steadily and there are no gradient explosion or gradient vanishing problems.

[0124] 3) Model optimization and adjustment: Based on the test results of the validation set, the model parameters are iteratively adjusted, such as adjusting the weight coefficients of multimodal fusion, the hidden layer dimension of the action expert module, and the parameters of the attention mechanism, so that the model can achieve a task execution success rate of ≥85% on the validation set, the parameter size is controlled within 0.45B, and the total training time is about 30k GPU hours.

[0125] Step S4: Standardize the multimodal observation data (image data, language command data, and sensor data) of the ultrasound scanning scene to generate a model input sequence in a unified format, ensuring that the input data can be effectively parsed by the lightweight vision, language, and motion models.

[0126] Specifically, such as Figure 4 As shown, step S4 includes the following sub-steps:

[0127] Step S41, Image Data and Processing

[0128] Image data was acquired using three industrial cameras at a frequency of 30fps. The image format was RGB with an original resolution of 2592×1944. Automatic exposure and white balance adjustments were automatically triggered during acquisition to ensure stable image quality. Simultaneously, the acquired image data was processed as follows: 1) Resolution Adjustment: The image resolution was uniformly adjusted to 512×512 using bilinear interpolation to ensure no image distortion; 2) Noise Filtering: Gaussian filtering was applied to the adjusted images with a standard deviation σ=0.3, effectively removing high-frequency noise while preserving key details such as organ boundaries and probe positions; 3) Visual Token Compression: The 512×512 resolution image was converted into 64 visual tokens through an 8×8 grid pixel rearrangement operation. Each token is a 768-dimensional feature vector (extracted using a SigLIP visual encoder).

[0129] Step S42: Language Command Acquisition and Processing

[0130] The system collects natural language commands input by medical staff through the command input area of ​​the interaction layer. Simultaneously, the collected natural language commands are processed as follows: 1) Command preprocessing: Removing redundant characters such as spaces and punctuation marks, and standardizing capitalization (for English commands); 2) BPE segmentation: Using the Byte-Pair Encoding (BPE) segmentation algorithm to segment the preprocessed commands, with a segmentation dictionary size of 32k; 3) Sequence padding and truncation: Processing the segmented token sequences, padding sequences shorter than 64 tokens with zeros (using [PAD] as the padding token), and truncation sequences longer than 64 tokens, ultimately forming a sequence with a fixed length of 64 tokens.

[0131] Step S43: Sensor Data Acquisition and Processing

[0132] Sensor data is collected through a sensor acquisition module, including data from robotic arm joint angles, motion speed, acceleration, and pressure sensors, at a frequency of 50Hz. The collected sensor data is then processed as follows: 1) Data cleaning: outliers in the sensor data are removed (using the 3σ criterion) to prevent them from affecting model inference; 2) Data normalization: the robotic arm joint angles (range -π / 2), motion speed (range -0.5-0.5 m / s), and pressure data (range 0~50 N) are mapped to the range [-1, 1] to eliminate the influence of dimensional differences on the model; 3) Single-token mapping: the normalized sensor data (dimension 13, including 6 joint angles, 6 joint speeds, and 1 pressure value) is mapped to a single-dimensional token (dimension 512) through a linear projection layer, maintaining consistency with the dimensions of visual tokens and language command tokens.

[0133] Step S44: Multimodal data fusion

[0134] The processed image data, natural language commands, and sensor data are concatenated in the order of visual tokens, language command tokens, and sensor tokens to form a model input sequence with a uniform format (e.g., dimension 512, sequence length 129). Simultaneously, to help the model better understand the structure and meaning of the multimodal data, positional and modal encodings are added to the concatenated model input sequence: positional encoding uses sine and cosine coding to distinguish the temporal positions of different tokens; modal encoding uses one-hot coding to distinguish between visual, language, and sensor tokens.

[0135] Step S5: Input the model input sequence into the lightweight vision, language, and action model to generate ultrasound scan action blocks and store them in the action sequence. At the same time, update the action sequence through an asynchronous inference mechanism to ensure the continuity and real-time performance of the action sequence, providing a dynamic solution for accurate execution.

[0136] Specifically, such as Figure 5 As shown, step S5 includes the following sub-steps:

[0137] Step S51: Initial Action Block Generation

[0138] The intelligent decision-making layer receives the model input sequence and inputs it into the lightweight vision, language, and motion model to generate an initial action block containing multiple control steps (typically 50 control steps). The action instructions for each control step include parameters such as the target angle, movement speed, and dwell time (default 0ms) of each joint of the robotic arm. The action block is in JSON array format. At the same time, the initial action block is sent to the action execution layer. The action execution layer initializes the action sequence (using Python's deque data structure) and stores the initial action block in the action sequence. The maximum length of the action sequence is the same as the number of control steps (typically 50).

[0139] Step S52, Action Execution and Sequence Monitoring

[0140] The action execution layer executes actions according to the action sequence, monitors the number of remaining actions in the queue in real time, calculates the percentage of remaining actions (number of remaining actions / total action block length, i.e., number of remaining actions / 50), and feeds back the sequence status (number of remaining actions, percentage of remaining actions) to the intelligent decision layer in real time at a frequency of 10Hz.

[0141] Step S53, Asynchronous Prediction Trigger

[0142] When the proportion of remaining actions in the action sequence is lower than the preset proportion (e.g., 0.7), the multimodal perception layer collects the current multimodal observation data (including image data, language command data, and sensor data), and sends it to the intelligent decision layer after standardization processing. After receiving the processed current multimodal observation data, the intelligent decision layer runs the lightweight vision, language, and action model to generate a new action block, and sends the new action block asynchronously to the action execution layer.

[0143] Step S54, Action Sequence Update

[0144] After receiving a new action block, the action execution layer averages the overlapping portion of the existing remaining actions with the new action block (i.e., the part of the new action block that coincides with the timing of the remaining actions, typically 15-35 control steps) to form a transitional action, avoiding abrupt changes in actions. Then, the transitional action is concatenated with the non-overlapping portion of the new action block to update the action sequence, and the action continues to be executed, ensuring the continuity and smoothness of the action sequence. For example, if the sequence has 20 remaining actions and the new action block contains 50 actions, the first 20 actions of the new action block are averaged with the remaining actions to form a transitional action, and then the last 30 actions of the new action block are concatenated to form a new action sequence (length 50).

[0145] Step S6: The action execution layer drives the robotic arm and ultrasonic probe to perform scanning according to the action sequence, and at the same time provides real-time feedback of the execution status data to provide real-time feedback for subsequent dynamic adjustments.

[0146] Specifically, such as Figure 6As shown, step S6 specifically includes the following sub-steps:

[0147] Step S61: The robotic arm performs scanning actions in accordance with the action instructions in the action sequence and the ultrasonic probe, controlling the frequency at 50Hz to ensure that the action instructions for each control step can be executed accurately.

[0148] Step S62: The execution status data during the scanning process (including actual position, actual angle, position error, angle error and pressure sensor data, etc.) is fed back to the intelligent decision layer and interaction layer in real time for dynamic adjustment and status monitoring.

[0149] Step S7: Based on the execution status data and real-time observation data fed back during the scanning process, dynamically correct the action parameters, and at the same time record the effective execution data to incrementally optimize the lightweight vision, language and action models, thereby improving the model's generalization ability in real-world scenarios.

[0150] Specifically, such as Figure 7 As shown, step S7 includes the following sub-steps:

[0151] Step S71: During the scanning process, monitor the actual position and angle of the robotic arm end in real time, compare them with the target value, and calculate the position error and angle error. If the position error or angle error is greater than the preset error (e.g., position error > 0.5 mm or angle error > 5°), then correct the action command in the next control step to ensure execution accuracy.

[0152] Step S72: During the scanning process, the pressure sensor continuously monitors the contact pressure. If the contact pressure exceeds the preset pressure range (e.g., 10±0.2N), the robotic arm automatically adjusts the height (e.g., rises or falls by 0.05cm) to maintain stable pressure.

[0153] Step S73: Record the effective execution data during the scanning process in real time. The effective execution data includes multimodal observation data, ultrasound scanning action blocks, execution status data, and task execution results (success / failure). The effective execution data is named according to the rule of patient ID-scanning date-scanning site and stored on the industrial control computer hard drive and cloud server to ensure data security and traceability.

[0154] Step S74: Set incremental training trigger conditions. When the accumulated effective execution data reaches a certain amount (e.g., 1 Kepisode) or the system runs continuously for a preset time (e.g., 72 hours), perform incremental optimization on the lightweight vision, language, and action models. After the incremental training is completed, generate a new model weight file and record the version information through a model version management tool (e.g., Git). At the same time, automatically deploy the new model weights to the intelligent decision layer to replace the old model. The deployment process does not affect the current scanning task, ensuring the continuous operation of the system. Meanwhile, retain the old model weights to support version rollback and avoid performance degradation of the new model.

[0155] Step S8: After completing the scan, generate a scan report.

[0156] Specifically, such as Figure 8 As shown, step S8 includes the following sub-steps:

[0157] Step S81: After completing the scan, the robotic arm returns to its initial position carrying the ultrasound probe and automatically saves the image data, motion parameters, and pressure data from the scanning process. Based on this data, a standardized scan report is generated. The scan report includes the scan area, execution accuracy, and image quality assessment results. Image quality assessment: The ultrasound image is quantitatively evaluated using indicators such as sharpness, contrast, and artifact level, and a quality grade (Excellent / Good / Pass / Fail) is given.

[0158] Step S82: Determine the image quality based on the scan report. If it is not up to standard, a second scan can be triggered according to the user's instructions, and steps S4 to S7 can be executed again.

[0159] In summary, the deep reinforcement learning-based ultrasonic language imitation learning control method of the present invention has the following technical effects:

[0160] 1) Significantly Improved Success Rate of Ultrasound Scanning Tasks: Through a lightweight vision, language, and motion model architecture design and community-driven data training, the reliability of ultrasound scanning task execution is greatly improved. The staggered attention mechanism and streaming matching motion generation strategy adopted by the model effectively solve the problems of insufficient motion continuity and inadequate multimodal information fusion in traditional models, enabling the ultrasound probe to stably respond to the coordinated guidance of language commands and visual observations, reducing task failures caused by motion deviations or command comprehension errors. Furthermore, pre-training on multiple community-level datasets allows the model to cover more heterogeneous scenarios similar to ultrasound operations. When facing different equipment layouts and operational differences, the success rate decrease is far less than that of models trained on traditional standardized datasets, fully verifying the high reliability of this invention in ultrasound scanning tasks.

[0161] 2) The accuracy of ultrasound scanning meets medical-grade requirements.

[0162] Regarding the accuracy of ultrasound scanning, this invention leverages the refined feature extraction capabilities of the compact pre-trained visual language module and the smooth motion generation mechanism of the motion expert module to achieve high-precision operation and angle control. In simulated ultrasound probe positioning tasks, the model exhibits minimal error in controlling the position and angle of the ultrasound probe's end effector, meeting the core requirements of medical ultrasound scanning for accurate target organ positioning and avoiding missed or repeated scans. This accuracy advantage stems from two technological optimizations: firstly, the compact pre-trained visual language module, through a specific strategy, compresses visual tokens while preserving key details such as organ boundaries and probe position in the ultrasound image, providing precise visual basis for motion planning; secondly, the motion expert module employs a streaming matching training mechanism, modeling a continuous motion distribution manifold to avoid motion jitter caused by traditional regression loss, resulting in smoother probe movement and angle adjustment processes and reducing positioning deviations caused by motion fluctuations. Compared to traditional models, this invention significantly improves accuracy and effectively reduces the risk of operational errors during ultrasound scanning.

[0163] 3) Enhanced stability of ultrasound image quality

[0164] This invention indirectly improves the stability of image quality during ultrasound scanning by optimizing multimodal input fusion and motion planning logic. Because the model can precisely control the probe's position, angle, and contact force with the patient's skin, it avoids problems such as image blurring and increased artifacts caused by probe misalignment, improper angle, or uneven pressure.

[0165] 4) Positive impacts on society and the economy

[0166] From a societal perspective, the lightweight architecture design of this invention significantly lowers the hardware barrier for intelligent ultrasound robots, enabling small and medium-sized medical institutions, primary healthcare centers, and even remote medical points to introduce intelligent ultrasound scanning technology without investing in expensive high-end computing equipment and customized systems. Furthermore, the high degree of automation in the model reduces the reliance on medical staff experience for ultrasound operations. The robot can be guided through standardized scanning procedures using natural language commands, reducing the workload of medical staff while avoiding service quality fluctuations caused by human error, thus improving the standardization and accessibility of medical ultrasound services.

[0167] From an economic perspective, this invention achieves significant cost savings through multi-dimensional technological optimization. On one hand, model training and deployment costs are drastically reduced: the computational resources required during training are significantly less than those of traditional large models; the inference phase supports low-cost hardware deployment, significantly reducing equipment procurement costs and greatly alleviating the equipment investment pressure on medical institutions. On the other hand, the model improves ultrasound scanning efficiency: the asynchronous inference mechanism shortens task completion time compared to traditional synchronous inference, increasing the number of scans that can be completed within a fixed time window, indirectly improving the work efficiency of ultrasound equipment and medical staff, reducing patient waiting time, and allowing limited medical resources to serve a wider population.

[0168] like Figure 9 As shown, the present invention also provides a deep reinforcement learning-based ultrasonic language imitation learning control system, including...

[0169] The motion execution layer 1, as the direct carrier of the ultrasound scanning motion, is used to perform the ultrasound scanning motion. It includes a six-degree-of-freedom industrial-grade robotic arm, an ultrasound probe, and a pressure sensor. The repeatability of the robotic arm is ≤ ±0.02mm, and the pressure sensor monitors the contact pressure between the ultrasound probe and the human body surface in real time with a measurement accuracy of ±0.1N.

[0170] Multimodal perception layer 2 is used to collect multimodal observation data of ultrasound scanning scene, including three industrial cameras and sensor data acquisition module. The cameras provide global monitoring, probe close-up and organ positioning perspectives respectively, and the sensor data acquisition module simultaneously acquires the joint status and pressure data of robotic arm.

[0171] The intelligent decision layer 3 is used to realize multimodal information fusion, model training and asynchronous action planning. It includes lightweight vision, language and action models and asynchronous inference scheduling modules. The lightweight vision, language and action models are composed of a compact pre-trained vision and language module and a streaming matching action expert module. The asynchronous inference scheduling module decouples action block prediction and action execution to solve the action delay problem caused by traditional synchronous inference, realizes parallel processing and ensures the real-time performance of action planning.

[0172] Interaction layer 4 is used for human-machine collaboration and system monitoring, including human-machine interaction interface and status monitoring and alarm module. The human-machine interaction interface supports natural language command input and data visualization, and the status monitoring and alarm module provides real-time feedback on abnormalities and triggers emergency stop procedures.

[0173] Among them, the intelligent decision-making layer 3 establishes bidirectional communication with the action execution layer 1, the multimodal perception layer 2, and the interaction layer 4, forming a closed-loop system of perception-decision-execution-feedback.

[0174] The ultrasonic language imitation learning control system based on deep reinforcement learning provided in this application can execute the above-described method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0175] like Figure 10 The diagram shown is a schematic representation of the structure of an electronic device that implements the ultrasonic language imitation learning control method for deep reinforcement learning according to the present invention.

[0176] The electronic device may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a deep reinforcement learning ultrasonic language imitation learning control program.

[0177] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a deep reinforcement learning ultrasonic language imitation learning control program) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0178] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a deep reinforcement learning ultrasonic language imitation learning control program, but also to temporarily store data that has been output or will be output.

[0179] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0180] Communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), and optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0181] Figure 10 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 10 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0182] For example, a power supply, although not shown, may also include a power source (such as a battery) to power various components. Preferably, the power supply can be logically connected to at least one processor 10 via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power sources, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated further here.

[0183] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0184] The deep reinforcement learning ultrasonic language imitation learning control program stored in the memory 11 of the electronic device is a combination of multiple computer programs. The specific implementation method of the processor 10 for the above computer programs can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0185] Furthermore, if the modules / units integrated into an electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, a computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0186] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following functions: Figure 1 The flowchart of the deep reinforcement learning ultrasonic language imitation learning control method is shown.

[0187] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0188] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0189] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0190] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0191] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0192] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0193] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0194] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

[0195] The present invention has been described above with reference to the accompanying drawings. Obviously, the implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A deep reinforcement learning-based ultrasonic language imitation learning control method, characterized in that, Includes the following steps: Step S1: Complete hardware deployment, software environment configuration, and system parameter calibration; Step S2: Filter and integrate community-level robot datasets, and after preprocessing, form a standardized training dataset adapted to ultrasound scanning tasks; Step S3: Use the standardized training dataset to jointly train the compact pre-trained visual language module and the streaming matching action expert module to obtain a lightweight visual, language, and action model. Step S4: Standardize the multimodal observation data of the ultrasound scanning scene to generate a model input sequence in a unified format. The multimodal observation data includes image data, language command data, and sensor data. Step S5: Input the model input sequence into the lightweight vision, language, and action model to generate ultrasound scan action blocks and store them in the action sequence, while updating the action sequence through an asynchronous inference mechanism; Step S6: The action execution layer drives the robotic arm and ultrasonic probe to perform scanning according to the action sequence, and simultaneously provides real-time feedback of execution status data; Step S7: Based on the execution status data and real-time observation data fed back during the scanning process, dynamically correct the action parameters, and simultaneously record effective execution data to incrementally optimize the lightweight vision, language, and action models; Step S8: After completing the scan, generate a scan report; Step S3 specifically includes the following sub-steps: Step 31: Configure the training environment; Step 32, Module Training: Construct the compact pre-trained visual language module based on the visual encoder and language decoder, and train it using the standardized training dataset with the image feature extraction accuracy and natural language instruction understanding accuracy as optimization objectives. Step 33: Initialize the parameters of the streaming matching action expert module, and train it using the streaming matching loss function with the optimization objectives of action sequence smoothness and action prediction accuracy. Step 34: Connect the trained compact pre-trained visual language module with the streaming matching action expert module to form the lightweight visual, language, and action model, and perform joint training using the standardized training dataset with the success rate of ultrasound scanning task as the optimization objective. The streaming matching loss function is defined as follows: ； In the formula, a loss function with respect to parameter θ is defined. Used to train models over multiple time spans The predictive consistency is shown above, with the meanings of each variable as follows: : Current action distribution Conditional distribution of future action sequences The joint expectations; The action at the current moment is determined by the agent based on observations. Obtained by sampling; The current observation, including visual, verbal, or sensor input information; Starting from time t, spanning [time range] The future sequence of actions or behaviors; : A prediction or estimate of a given sequence of observations and future actions by a model controlled by parameter θ; The reference value provided by the objective function or teacher model is used as a supervision signal; Euclidean norm squared, used to measure the deviation between a prediction and a target; Step S4 specifically includes the following sub-steps: Step S41: Acquire image data through a camera, and perform resolution adjustment, noise filtering, and visual token compression on the acquired image data; Step S42: Collect natural language commands input by medical staff through the command input area of ​​the interaction layer, and perform command preprocessing, BPE word segmentation, and sequence completion and truncation processing on the collected natural language commands. Step S43: Collect sensor data through the sensor acquisition module, and perform data cleaning, data normalization and single token mapping processing on the collected sensor data; Step S44: The processed image data, the natural language instructions, and the sensor data are concatenated in the order of visual tokens, language instruction tokens, and sensor tokens to form the model input sequence in a unified format; Step S5 specifically includes the following sub-steps: Step S51: The intelligent decision layer receives the model input sequence and inputs it into the lightweight vision, language, and action model to generate an initial action block containing multiple control steps; at the same time, the initial action block is sent to the action execution layer, the action execution layer initializes the action sequence, and stores the initial action block in the action sequence. Step S52: The action execution layer executes actions according to the action sequence and monitors the number of remaining actions in the queue in real time; Step S53: When the proportion of the remaining actions in the action sequence is lower than the preset proportion, collect the current multimodal observation data, and send it to the intelligent decision layer after standardization processing; Step S54: After receiving the processed current multimodal observation data, the intelligent decision-making layer runs the lightweight vision, language, and action model to generate new action blocks. Step S55: The new action block is asynchronously sent to the action execution layer, and the action execution layer updates the action sequence through the overlapping region aggregation strategy and continues to execute the action; Step S6 specifically includes the following sub-steps: Step S61: The robotic arm performs scanning actions in accordance with the action instructions in the action sequence and the ultrasonic probe. Step S62: During the scanning process, the execution status data is fed back to the intelligent decision-making layer and the interaction layer in real time; the execution status data includes actual position, actual angle, position error, angle error and pressure sensor data; Step S7 specifically includes the following sub-steps: Step S71: During the scanning process, monitor the actual position and angle of the end effector of the robotic arm in real time, compare them with the target value, and calculate the position error and angle error; if the position error or angle error is greater than the preset error, correct the action command in the next control step. Step S72: During the scanning process, the pressure sensor continuously monitors the contact pressure. If the contact pressure exceeds the preset pressure range, the robotic arm automatically adjusts its height. Step S73: Record the effective execution data during the scanning process in real time. The effective execution data includes the multimodal observation data, the ultrasound scanning action block, the execution status data, and the task execution result. Step S74: Set incremental training trigger conditions. When the accumulated effective execution data reaches a certain amount or the system runs continuously for a preset time, perform incremental optimization on the lightweight vision, language, and action models.

2. The ultrasonic language imitation learning control method based on deep reinforcement learning according to claim 1, characterized in that, Step S2 specifically includes the following sub-steps: Step S21: Select community-level robot datasets from the community platform and integrate them to form a training dataset. The selection criteria include simultaneously containing RGB images and corresponding motion sequence data, having ≥50 episodes, and having data quality that meets the requirements for ultrasound scene adaptation. Step S22: Sample representative images of preset frames for each episode of the training dataset and input them into the visual language model to generate standardized natural language instructions; Step S23: Normalize the camera view of all community-level robot datasets in the training dataset; Step S24: Adjust the brightness and contrast of the RGB images in the training dataset and apply Gaussian blur. Add small Gaussian noise with σ=0.01 to the action sequence data in the training dataset. Perform synonym replacement and sentence rewriting on the natural language instructions in the training dataset to generate multiple versions of natural language instructions.

3. The ultrasonic language imitation learning control method based on deep reinforcement learning according to claim 1, characterized in that, Step S8 specifically includes the following sub-steps: Step S81: After the scan is completed, the robotic arm carries the ultrasonic probe back to the initial position and automatically saves the image data, motion parameters and pressure data during the scan process, and generates the scan report; Step S82: Determine the image quality based on the scan report. If it is not up to standard, a second scan can be triggered according to the user's instructions, and steps S4 to S7 can be executed again.

4. A deep reinforcement learning ultrasonic language imitation learning control system for executing the deep reinforcement learning ultrasonic language imitation learning control method as described in any one of claims 1 to 3.

5. The ultrasonic language imitation learning control system based on deep reinforcement learning according to claim 4, characterized in that, include: The action execution layer is used to perform ultrasound scanning actions; The multimodal perception layer is used to collect multimodal observation data of the ultrasound scanning scene, including a camera and a sensor data acquisition module. The camera provides global monitoring, probe close-up and organ positioning perspectives respectively, and the sensor data acquisition module simultaneously acquires the joint status and pressure data of the robotic arm. The intelligent decision-making layer is used to realize multimodal information fusion, model training and asynchronous action planning, including a lightweight vision, language and action model and an asynchronous inference scheduling module. The lightweight vision, language and action model consists of a compact pre-trained vision and language module and a streaming matching action expert module. The interaction layer is used for human-computer collaboration and system monitoring. The intelligent decision-making layer establishes bidirectional communication with the action execution layer, the multimodal perception layer, and the interaction layer, forming a closed-loop system of perception-decision-execution-feedback.

Images