An ultrasound system, method, and product based on a multi-modal embodied large model

Abstract

The application provides an ultrasound system, method and product based on a multimodal embodied large model, and relates to the technical field of ultrasound acquisition. The system comprises: an information acquisition module configured to acquire ultrasound information of a position of a mechanical arm at a current time through a plurality of sensors, wherein the ultrasound information comprises an RGB image, point cloud data, an ultrasound image, a text instruction and 6-dimensional force estimation data; a feature extraction module configured to perform feature extraction to obtain ultrasound features; a multimodal embodied large model configured to output mechanical arm state features and mechanical arm translation and rotation features according to the ultrasound features; the mechanical arm state features represent whether the mechanical arm is in an abnormal state at the current time, and the mechanical arm translation and rotation features represent translation and rotation actions required to be performed by the mechanical arm at the current time; and an action decoder configured to decode the mechanical arm state features and the mechanical arm translation and rotation features to generate ultrasound actions to guide the mechanical arm to move according to the ultrasound actions at the current time.

Description

Technical Field

[0001] This application relates to the field of ultrasound acquisition technology, and in particular to an ultrasound system, method and product based on a multimodal embodied large model. Background Technology

[0002] Ultrasound imaging using robotic arms is an emerging and important task in the medical field, aiming to replace handheld ultrasound scanning methods. Compared with traditional manual operation, automated ultrasound scanning using robotic arms offers higher accuracy and consistency, while also reducing the operator's workload and improving work efficiency.

[0003] However, the complex and diverse structures of human organs, along with significant individual differences (such as height, weight, age, and sex), often make it difficult for robotic arm-based ultrasound imaging methods to accurately locate the ideal cross-sectional position of the target organ in practical applications. This results in a low success rate and an inability to meet the complex and ever-changing clinical needs. Therefore, there is an urgent need to develop an ultrasound system, method, and product based on a multimodal embodied large model to improve the reliability of embodied robotic arm-based ultrasound scanning. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide an ultrasound system, method and product based on a multimodal embodied large model, in order to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this application provides an ultrasound system based on a multimodal embodied large model, the system comprising:

[0006] The information acquisition module is used to acquire ultrasonic information of the current position of the robotic arm through multiple sensors. The ultrasonic information includes: RGB image, point cloud data, ultrasonic image, text command, and 6-dimensional force estimation data.

[0007] The feature extraction module is used to extract features from the ultrasound information to obtain ultrasound features, which include: RGB image features, point cloud data features, ultrasound image features, text command tokens, and force estimation features.

[0008] A multimodal embodied large model is used to output robotic arm state features and robotic arm translation and rotation features based on the ultrasonic features; wherein, the robotic arm state features indicate whether the robotic arm is in an abnormal state at the current moment, and the robotic arm translation and rotation features indicate the translation and rotation actions that the robotic arm needs to perform at the current moment.

[0009] An action decoder is used to decode the state characteristics and translational / rotation characteristics of the robotic arm to generate ultrasonic motions, which guide the robotic arm to move according to the ultrasonic motions at the current moment.

[0010] In one possible implementation, the plurality of sensors include:

[0011] A depth camera is used to acquire the RGB image and the point cloud data. The RGB image represents an image containing three-dimensional morphological information of the detected object, and the point cloud data represents scene point cloud information containing the spatial relationship between the robotic arm and the detected object.

[0012] An ultrasonic probe, which is fixed on the robotic arm, is used to acquire ultrasonic images at the location of the robotic arm;

[0013] A text input device for receiving text commands input by a user;

[0014] A force sensor is used to collect the 6-dimensional force estimation data, which represents the magnitude of the force applied by the ultrasonic probe to the object being tested.

[0015] In one possible implementation, the feature extraction module is configured to:

[0016] The RGB image is scaled to a resolution of 224x224. Using a first visual feature extraction model, a 768-dimensional image feature vector is extracted from the RGB image. The extracted image feature vector is then passed through a fully connected layer to obtain a 4096-dimensional RGB image feature.

[0017] In the base coordinate system of the robotic arm, the encoder of the PCT model extracts 1024-dimensional three-dimensional feature information from the point cloud data. The extracted three-dimensional feature information is then transformed through a fully connected layer to obtain 4096-dimensional point cloud data features.

[0018] The ultrasound image is scaled to a resolution of 224x224. Using a second visual feature extraction model, a 768-dimensional ultrasound image feature vector is extracted from the ultrasound image. The extracted ultrasound image feature vector is then passed through a fully connected layer to obtain a 4096-dimensional ultrasound image feature.

[0019] Using a large language model, the text instructions are converted into text instruction tokens with a feature dimension of 1024;

[0020] The 6-dimensional force estimation data is transformed by a fully connected layer to obtain the 4096-dimensional force estimation features.

[0021] In one possible implementation, the first visual feature extraction model and the second visual feature extraction model are the same visual feature extraction model.

[0022] In one possible implementation, the feature extraction module and the multimodal embodied large model are trained according to the following steps:

[0023] Acquire training samples, which are ultrasonic information of the robotic arm's location obtained through the multiple sensors, and training sample labels; wherein, the ultrasonic information of the samples includes: sample RGB image, sample point cloud data, sample ultrasonic image, sample text command, and sample 6-dimensional force estimation data; the training sample labels indicate the state of the robotic arm at the time of acquisition of the training samples, and the movement action performed by the robotic arm in the next moment;

[0024] The training samples are input into the feature extraction module to extract the ultrasonic features of the samples. The ultrasonic features of the samples include: RGB image features of the samples, point cloud data features of the samples, ultrasonic image features of the samples, text command tokens of the samples, and force estimation features of the samples.

[0025] The ultrasonic features of the sample are input into the multimodal embodied large model to obtain the output results, which include: sample robotic arm state features and sample robotic arm translation and rotation features.

[0026] Calculate the loss function value based on the output and the training sample labels;

[0027] Based on the loss function value, the parameters of the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, the large language model, and the multimodal embodied large model in the feature extraction module are updated.

[0028] Repeat the above steps until the preset number of training iterations or the loss function value converges, then end the training.

[0029] In one possible implementation, the robotic arm state characteristics include at least one of the following:

[0030] The first robotic arm state characteristic indicates whether the robotic arm has reached the predetermined target position at the current moment;

[0031] The second robotic arm state characteristic indicates whether the robotic arm has detached from the surface of the target object at the current moment;

[0032] The third robotic arm status characteristic indicates whether there is any abnormality in the relative position of the robotic arm with respect to the target at the current moment.

[0033] In one possible implementation, the multimodal embodied large model further includes a storage module for storing historical ultrasound features; the step of outputting robotic arm state features and robotic arm translation and rotation features based on the ultrasound features includes:

[0034] Retrieve historical ultrasound features from multiple moments in the storage module;

[0035] An ultrasound feature sequence is generated based on the historical ultrasound features at the multiple time points and the currently input ultrasound features;

[0036] Based on the ultrasonic feature sequence, the state characteristics and translational / rotation characteristics of the robotic arm are determined;

[0037] The storage module is also used to store the currently input ultrasound features as historical ultrasound features.

[0038] A second aspect of this application also provides an ultrasound method, executed by the ultrasound system based on a multimodal embodied large model described in the first aspect of this application, the method comprising:

[0039] The ultrasonic information of the current position of the robotic arm is obtained through multiple sensors. The ultrasonic information includes: RGB image, point cloud data, ultrasonic image, text command, and 6-dimensional force estimation data.

[0040] The ultrasonic information is processed by a feature extraction module to obtain ultrasonic features, which include: RGB image features, point cloud data features, ultrasonic image features, text command tokens, and force estimation features.

[0041] Using a multimodal embodied large model, based on the ultrasonic features, the robot arm state features and robot arm translation and rotation features are output; wherein, the robot arm state features indicate whether the robot arm is in an abnormal state at the current moment, and the robot arm translation and rotation features indicate the translation and rotation actions that the robot arm needs to perform at the current moment.

[0042] The motion decoder decodes the state features and translational / rotation features of the robotic arm to generate ultrasonic motions, which guide the robotic arm to move according to the ultrasonic motions at the current moment.

[0043] A third aspect of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the ultrasound method described in the second aspect of this application.

[0044] The fourth aspect of this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the ultrasonic method described in the second aspect of this application.

[0045] The fifth aspect of this application also provides a computer program product that, when run on an electronic device, causes a processor to execute the steps of the ultrasonic method as described in the second aspect of this application.

[0046] This application provides an ultrasonic system based on a multimodal embodied large model. The system includes: an information acquisition module for acquiring ultrasonic information of the current position of a robotic arm through multiple sensors, the ultrasonic information including: RGB images, point cloud data, ultrasonic images, text commands, and 6-dimensional force estimation data; a feature extraction module for extracting ultrasonic features from the ultrasonic information, the ultrasonic features including: RGB image features, point cloud data features, ultrasonic image features, text command tokens, and force estimation features; a multimodal embodied large model for outputting robotic arm state features and robotic arm translational and rotational features based on the ultrasonic features; wherein the robotic arm state features indicate whether the robotic arm is in an abnormal state at the current moment, and the robotic arm translational and rotational features indicate the translational and rotational actions that the robotic arm needs to perform at the current moment; and an action decoder for decoding the robotic arm state features and the robotic arm translational and rotational features to generate ultrasonic actions to guide the robotic arm to move according to the ultrasonic actions at the current moment.

[0047] The specific beneficial effects are as follows:

[0048] On the one hand, the embodiments of this application utilize an information acquisition module to acquire ultrasonic information through multiple sensors (ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text commands, and 6-dimensional force estimation data), thereby comprehensively utilizing the advantages of various sensors, making up for the shortcomings of a single sensor, ensuring the comprehensiveness of data acquisition, thereby improving the accuracy of the acquired ultrasonic information, which is beneficial for subsequent model calculations to obtain more accurate ultrasonic actions.

[0049] On the other hand, this application embodiment utilizes a multimodal embodied large model to fuse multimodal feature information from different sensors (including RGB image features, point cloud data features, ultrasonic image features, text command tokens, and force estimation features) for comprehensive analysis and decision-making. It outputs robotic arm state features (whether the robotic arm is in an abnormal state at the current moment) and robotic arm translation and rotation features (the translational and rotational actions the robotic arm needs to perform at the current moment). Based on the output results, it guides the movement of the robotic arm, achieving intelligent control of the robotic arm. By integrating multimodal ultrasonic feature information, the ultrasonic imaging scanning process becomes more accurate, efficient, and safe, improving the quality and efficiency of ultrasonic scanning. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the structure of an ultrasound system based on a multimodal embodied large model provided in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of the processing flow of an ultrasound system provided in an embodiment of this application;

[0053] Figure 3 This is a flowchart of the steps of an ultrasonic method provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0056] Ultrasound imaging using robotic arms is an emerging and important task in the medical field, aiming to replace handheld ultrasound scanning methods. Compared with traditional manual operation, automated ultrasound scanning using robotic arms offers higher accuracy and consistency, while also reducing the operator's workload and improving work efficiency.

[0057] In related technologies, robotic arm-based ultrasound image scanning is typically accomplished through two main methods: rule-based and ultrasound image-based visual neural networks. Rule-based methods rely on pre-set operating procedures, with the robotic arm scanning according to predetermined trajectories and parameters. This method performs well in simple, fixed operating environments, but due to the complex and diverse structures of human organs and significant individual differences (such as height, weight, age, and gender), rule-based methods often fail to accurately locate the ideal cross-sectional position of the target organ in practical applications, resulting in a low success rate. To overcome this limitation, ultrasound image-based visual neural network methods have emerged. This method uses deep learning technology and is trained with a large amount of ultrasound image data, enabling the robotic arm to perform scanning more intelligently. Although this method improves scanning accuracy and generalization ability to some extent, its single signal source still leads to a high failure rate when dealing with special patients (such as obese patients or disabled patients).

[0058] Furthermore, in practice, the scanning process relies not only on ultrasound images but also on a combination of sensory information to determine and adjust the procedure. For example, while a human performs ultrasound manually, they observe the patient's position and posture, analyze the ultrasound images in real time, sense the contact strength between the probe and the skin, and combine this with the patient's verbal feedback to determine the optimal probe position and posture. This comprehensive use of multimodal information greatly improves the accuracy and effectiveness of the scan. Therefore, how to enable robotic arms to acquire and process multimodal information in automated ultrasound scanning, much like human operation, has become a key research direction. Although automated robotic arm-based medical ultrasound imaging technology has begun to take shape, current technology is still in the initial exploratory stage and has not yet adopted targeted optimization measures, making it difficult to cope with the complex and ever-changing clinical needs.

[0059] In view of the above problems, this application proposes an ultrasound system, method, and product based on a multimodal embodied large model to improve the reliability of ultrasound scanning based on an embodied robotic arm. The following, in conjunction with the accompanying drawings, provides a detailed description of an ultrasound system based on a multimodal embodied large model provided by this application through some embodiments and application scenarios.

[0060] The first aspect of this application provides an ultrasound system based on a multimodal embodied large model, referring to... Figure 1 , Figure 1 A schematic diagram of the structure of an ultrasound system based on a multimodal embodied large model is provided for an embodiment of this application, as shown below. Figure 1 As shown, the system includes:

[0061] The information acquisition module is used to acquire ultrasonic information of the current position of the robotic arm through multiple sensors. The ultrasonic information includes: RGB image, point cloud data, ultrasonic image, text command, and 6-dimensional force estimation data.

[0062] The feature extraction module is used to extract features from the ultrasound information to obtain ultrasound features, which include: RGB image features, point cloud data features, ultrasound image features, text command tokens, and force estimation features.

[0063] A multimodal embodied large model is used to output robotic arm state features and robotic arm translation and rotation features based on the ultrasonic features; wherein, the robotic arm state features indicate whether the robotic arm is in an abnormal state at the current moment, and the robotic arm translation and rotation features indicate the translation and rotation actions that the robotic arm needs to perform at the current moment.

[0064] An action decoder is used to decode the state characteristics and translational / rotation characteristics of the robotic arm to generate ultrasonic motions, which guide the robotic arm to move according to the ultrasonic motions at the current moment.

[0065] In the embodiments of this application, reference is made to Figure 2 , Figure 2 A schematic diagram of the processing flow of an ultrasound system is shown, such as Figure 2 As shown, each time the robotic arm performs an ultrasonic acquisition action, it controls multiple sensors to simultaneously acquire data. The information acquisition module obtains the perceived data (i.e., the ultrasonic information at the current location of the robotic arm) through these sensors. During application, each time the robotic arm moves, it automatically performs an ultrasonic acquisition, obtaining the ultrasonic image and related information at the corresponding location. The information acquisition module in this system comprehensively utilizes the advantages of various sensors, compensating for the shortcomings of a single sensor, ensuring the comprehensiveness of data acquisition, and thus improving the accuracy of the acquired ultrasonic information.

[0066] In one possible implementation, the plurality of sensors include:

[0067] A depth camera is used to acquire the RGB image and the point cloud data. The RGB image represents an image containing three-dimensional morphological information of the detected object, and the point cloud data represents scene point cloud information containing the spatial relationship between the robotic arm and the detected object.

[0068] An ultrasonic probe, fixed to the robotic arm, is used to acquire ultrasonic images at the location of the robotic arm.

[0069] A text input device for receiving text commands input by a user.

[0070] A force sensor is used to collect the 6-dimensional force estimation data, which represents the magnitude of the force applied by the ultrasonic probe to the object being tested.

[0071] Specifically, a depth camera acquires RGB images and point cloud data at the current moment to obtain three-dimensional morphological information of the detected object's body (such as human pose information). This helps the system understand the spatial relationship between the robotic arm and the detected object, ensuring that the robotic arm can accurately locate and avoid obstacles. For example, a Microsoft Kinect V2 device can be used, which can acquire RGB images and point cloud data of the scene. By fixing an ultrasound probe to the robotic arm, the probe's position changes with the movement of the robotic arm to acquire ultrasound images from different positions and angles. This obtains real-time ultrasound image data, helping the system analyze and identify the structure and state of the target organ of the detected object, providing crucial information for subsequent operational decisions (generating ultrasound actions). A text command input device receives text commands from the user, allowing operators to input specific operational requirements (such as acquiring a cross-sectional image of organ A) or adjust parameters through the text command input device (such as a user interface), making the entire process more flexible and controllable. By using force sensors to detect the contact force between the robotic arm and the object being examined (such as a patient's skin) at the current moment (i.e., 6D force estimation data), it is ensured that the pressure applied by the robotic arm to the object being examined during ultrasound scanning is moderate, avoiding discomfort or damage to the patient.

[0072] like Figure 2 As shown, after acquiring ultrasound information from multiple sensors, a feature extraction module extracts features from the ultrasound information to obtain ultrasound features (RGB image features, point cloud data features, ultrasound image features, text command tokens, and force estimation features). The feature extraction module integrates ultrasound information from different sensors and, through complex algorithm analysis and processing, extracts ultrasound features crucial for operational decisions. For example, combining 3D morphological data provided by a depth camera with ultrasound images provided by an ultrasound probe can more accurately locate target organs, while data collected by a force sensor can help adjust the robotic arm's operating force, ensuring safety and comfort.

[0073] In one possible implementation, the feature extraction module is configured to:

[0074] The RGB image is scaled to a resolution of 224x224. A first visual feature extraction model is used to extract a 768-dimensional image feature vector from the RGB image. This extracted feature vector is then passed through a fully connected layer to obtain 4096-dimensional RGB image features. Specifically, the first visual feature extraction model can be the ViT-S (Visual Transformer) model. The ViT-S model is an advanced visual neural network capable of efficiently processing image data and extracting a 768-dimensional feature vector.

[0075] In the base coordinate system of the robotic arm, the encoder of the PCT model extracts 1024-dimensional 3D feature information from the point cloud data. The extracted 3D feature information is then transformed through a fully connected layer to obtain 4096-dimensional point cloud data features. Specifically, for different sensors, due to different installation positions, the coordinate systems (origin positions) of the acquired information are different. Coordinate system transformation converts the point cloud data acquired by the depth camera into data in the coordinate system of the robotic arm before subsequent feature extraction to obtain more accurate ultrasonic data. For example, calibration can be used to obtain the rotation angle and translation distance (x-direction, y-direction, and z-direction translation distance) or transformation matrix of the depth camera's coordinate system relative to the robotic arm's base coordinate system. The data can then be corrected based on the rotation angle and translation distance (or transformation matrix). Point cloud data can be processed by the encoder of the Point Cloud Transformer (PCT) model (e.g., ...). Figure 2 The point cloud Transformer module shown extracts a 1024-dimensional feature vector. The PCT model is a neural network specifically designed for processing point cloud data, capable of efficiently extracting key 3D information from point clouds.

[0076] The ultrasound image is scaled to a resolution of 224x224. Using a second visual feature extraction model, a 768-dimensional ultrasound image feature vector is extracted from the ultrasound image. The extracted ultrasound image feature vector is then passed through a fully connected layer to obtain a 4096-dimensional ultrasound image feature.

[0077] In one possible implementation, the first visual feature extraction model and the second visual feature extraction model are the same visual feature extraction model.

[0078] Specifically, the ultrasound images acquired by the ultrasound probe can be RGB images, which can share the same visual feature extraction model (such as the ViT-S model) as the RGB images acquired by the depth camera, and thus undergo the same feature extraction process to ensure the consistency and efficiency of feature extraction.

[0079] Using a large language model, the text commands are converted into text command tokens with a feature dimension of 1024. Specifically, the user-input text commands are processed by a large language model. The large language model can be the LLama 7BTokenizer model, an advanced natural language processing tool capable of converting text commands into feature vectors. In this process, the text commands are converted into tokens with a feature dimension of 1024 for integration and processing with other perceptual data.

[0080] The 6-dimensional force estimation data is transformed using a fully connected layer to obtain 4096-dimensional force estimation features. Specifically, the 6-dimensional force estimation data collected by the force sensor is used to monitor the contact force between the robotic arm and the object being scanned (e.g., human skin). Transforming the 6-dimensional force estimation data using a fully connected layer yields 4096-dimensional force estimation features, which provide crucial information about the contact force of the robotic arm. This helps ensure that the robotic arm applies appropriate pressure during scanning, avoiding discomfort or injury to the patient.

[0081] In this embodiment, after extracting multimodal ultrasound features, these features are combined into a sequence input multimodal embodied large model to predict the robotic arm's state features and translational / rotation features based on the input ultrasound features. The multimodal embodied large model is a comprehensive artificial intelligence model capable of processing and fusing multimodal data from multiple sensors to perform complex operational decisions and reasoning. Having learned from a large amount of training data, this model possesses powerful analytical and decision-making capabilities, enabling accurate and efficient judgments in complex operating environments. This multimodal perception and processing method significantly improves the system's intelligence and automation level, allowing the robotic arm to complete ultrasound image scanning tasks more accurately and efficiently.

[0082] Optionally, the multimodal embodied large model can be a feature fusion model in OpenVLA. The core architecture of this model is LLama 7B, a powerful large language model capable of processing and fusing data sequences from different sensors to generate robotic arm state features and translation / rotation features. The output of the multimodal embodied large model includes robotic arm state features and robotic arm translation / rotation features. The robotic arm state features indicate whether the robotic arm is in an abnormal state at the current moment. These features provide real-time guidance on whether the robotic arm's operation should continue or require adjustment. For example, when the system detects that the robotic arm has not reached the ideal position or an abnormal situation has occurred, it can instruct the robotic arm to make corresponding adjustments to ensure the accuracy and safety of the operation.

[0083] The translational and rotational features of a robotic arm represent the translational and rotational actions that the robotic arm needs to perform at the current moment (i.e., the target pose of the robotic arm's next action), including the robotic arm's specific position and orientation information in space. Further, the translational and rotational features can represent the translational and rotational information of the robotic arm along the x, y, and z axes; for example, it could be moving 1 cm forward along the x-axis and rotating 5 degrees clockwise along the y-axis. Generating the translational and rotational features of the robotic arm through model reasoning and calculation can guide the robotic arm to perform precise motion control, ensuring that the robotic arm moves and rotates along a predetermined trajectory to achieve high-quality ultrasound imaging scans.

[0084] In one possible implementation, the robotic arm state characteristics include at least one of the following:

[0085] The first robotic arm status feature indicates whether the robotic arm has reached the predetermined target position at the current moment. By outputting the first robotic arm status feature, it is determined whether the robotic arm has reached the ideal final position, ensuring that the ultrasound probe is accurately positioned on the area to be scanned. For example, if the user inputs the text command "acquire a cross-section of organ A", the output first robotic arm status feature indicates whether the current robotic arm has reached the ideal position, that is, whether the ultrasound image acquired at the current position is a clear cross-sectional image of organ A.

[0086] The second robotic arm status characteristic indicates whether the robotic arm has detached from the target surface at the current moment. This second robotic arm status characteristic can monitor the contact between the robotic arm and the patient's skin, ensuring that the robotic arm does not accidentally detach from the patient's surface during the scanning process, thus affecting the scanning results.

[0087] The third robotic arm status feature indicates whether there is any abnormality in the relative position of the robotic arm to the target at the current moment. By detecting abnormalities in the relative position between the robotic arm and the patient through the third robotic arm status feature, the operation of the robotic arm can be adjusted in a timely manner if an abnormality is detected, so as to avoid causing discomfort or harm to the patient.

[0088] In one possible implementation, the multimodal embodied large model further includes a storage module for storing historical ultrasound features; the step of outputting robotic arm state features and robotic arm translation and rotation features based on the ultrasound features includes:

[0089] Retrieve historical ultrasound features from multiple moments in the storage module;

[0090] An ultrasound feature sequence is generated based on the historical ultrasound features at the multiple time points and the currently input ultrasound features;

[0091] Based on the ultrasonic feature sequence, the state characteristics and translational / rotation characteristics of the robotic arm are determined;

[0092] The storage module is also used to store the currently input ultrasound features as historical ultrasound features.

[0093] In this embodiment, the multimodal embodied large model can make predictions based solely on the ultrasonic features input at the current moment to obtain the output results (robotic arm state features and robotic arm translation and rotation features), or it can combine the current input with historical information to generate an ultrasonic feature sequence arranged in chronological order. Specifically, the ultrasonic features input to the multimodal embodied large model each time are stored in the storage module as historical ultrasonic features at the corresponding moment. By obtaining the historical ultrasonic features from the previous k moments and sorting them with the ultrasonic features at the current moment in chronological order, an ultrasonic feature sequence is obtained. This allows the multimodal embodied large model to combine historical information (historical ultrasonic features from multiple moments) to obtain more accurate output results.

[0094] An action decoder is used to decode the state characteristics and translational / rotation characteristics of the robotic arm to generate ultrasonic motions, which guide the robotic arm to move according to the ultrasonic motions at the current moment.

[0095] Specifically, the prediction results generated by the multimodal embodied large model (robotic arm state features and robotic arm translation and rotation features) are decoded by the action decoder, which generates ultrasound motion (i.e., the decoded result). This ultrasound motion can be represented as a 9-dimensional vector. In this 9-dimensional vector, the first 3 dimensions represent the state of the robotic arm (corresponding to the robotic arm state features), including whether the robotic arm has reached the ideal position, whether it has detached from the human body surface, and whether there are any abnormalities in its position relative to the human body. The last 6 dimensions represent the translation and rotation information of the robotic arm on the x, y, and z axes (corresponding to the robotic arm translation and rotation features), which guide the specific actions of the robotic arm. This embodiment of the application, through this detailed and precise feature representation method, enables the model to generate highly reliable operating instructions, ensuring that the robotic arm performs accurate and safe ultrasound imaging scans in complex medical environments. Furthermore, based on the generated ultrasound motion, the robotic arm precisely moves and adjusts its position and posture, ensuring that the ultrasound probe can accurately align with the target organ, completing a high-quality ultrasound imaging scan. This process not only improves the accuracy and efficiency of the scan but also reduces the workload of the operator.

[0096] In this embodiment, after generating the ultrasound motion using the motion decoder, the robotic arm performs translational and rotational movements based on the ultrasound motion to reach the next position. It then automatically acquires ultrasound information at the next position, repeating the above process (feature extraction is performed on the acquired ultrasound information by the feature extraction module, and the extracted ultrasound features are input into the multimodal embodied large model for feature fusion and decision-making, ultimately generating the ultrasound motion for the next moment through the motion decoder). This process is repeated multiple times until the robotic arm reaches the ideal position (for example, based on the first robotic arm state feature output by the multimodal embodied large model, it is determined that the robotic arm has reached the ideal position at the current moment, and a clear cross-sectional image of the target organ can be obtained through the ultrasound probe, thus ending the ultrasound process). The ultrasound system based on the multimodal embodied large model constructed in this embodiment not only improves the accuracy and efficiency of ultrasound image scanning through this multi-sensor, multimodal perception and processing method, but also significantly improves the safety and reliability of medical operations, promoting the development of intelligent medical devices.

[0097] To maximize the accuracy of ultrasound system decision-making, the model used in the system needs to be pre-trained. In one possible implementation, the feature extraction module and the multimodal embodied large model are trained according to the following steps:

[0098] Step S1: Obtain training samples, which are ultrasonic information of the robotic arm's location obtained through the multiple sensors, and training sample labels; wherein, the ultrasonic information includes: sample RGB image, sample point cloud data, sample ultrasonic image, sample text command, and sample 6-dimensional force estimation data; the training sample label indicates the state of the robotic arm at the time of acquisition of the training sample, and the movement action performed by the robotic arm in the next moment.

[0099] Specifically, in this embodiment, the robotic arm is manually controlled to move along the optimal path, and multiple sensors (depth camera, ultrasonic probe, text command input device, force sensor) are controlled to acquire ultrasonic information at multiple moments to generate multiple training samples. Each training sample represents the ultrasonic information collected at one moment. The label corresponding to the training sample is the action to be performed by the robotic arm at the next moment during the manual control of the robotic arm operation, as well as the current state of the robotic arm (i.e., the corresponding robotic arm state characteristics and robotic arm translation and rotation characteristics).

[0100] Step S2: Input the training samples into the feature extraction module to extract the ultrasonic features of the samples. The ultrasonic features of the samples include: RGB image features of the samples, point cloud data features of the samples, ultrasonic image features of the samples, text command tokens of the samples, and force estimation features of the samples.

[0101] Specifically, the training samples are input into the feature extraction module for feature extraction. The sample RGB images are scaled to a resolution of 224x224. Using a first visual feature extraction model, a 768-dimensional image feature vector is extracted from the sample RGB images. The extracted image feature vector is then passed through a fully connected layer to obtain 4096-dimensional sample RGB image features. In the base coordinate system of the robotic arm, the encoder of the PCT model extracts 1024-dimensional three-dimensional feature information from the sample point cloud data. The extracted three-dimensional feature information is then transformed through a fully connected layer to obtain 4096-dimensional sample point cloud data features. The sample ultrasound images are scaled to a resolution of 224x224. Using a second visual feature extraction model, a 768-dimensional ultrasound image feature vector is extracted from the sample ultrasound images. The extracted ultrasound image feature vector is then passed through a fully connected layer to obtain 4096-dimensional sample ultrasound image features. Using a large language model, the text commands are converted into sample text command tokens with a feature dimension of 1024. The 6-dimensional force estimation data of the samples is then transformed through a fully connected layer to obtain 4096-dimensional sample force estimation features.

[0102] Step S3: Input the ultrasonic features of the sample into the multimodal embodied large model to obtain the output results, which include: sample robotic arm state features and sample robotic arm translation and rotation features.

[0103] Optionally, the ultrasound features of each input sample can be stored as historical sample ultrasound features by the storage module. These sample ultrasound features can be combined with historical sample ultrasound features from multiple previous time points in chronological order to form a sample ultrasound feature sequence. This allows the multi-module integrated large model to make decisions and obtain output results based on this sample ultrasound feature sequence.

[0104] Step S4: Calculate the loss function value based on the output result and the training sample labels.

[0105] Step S5: Based on the loss function value, update the parameters of the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, the large language model, and the multimodal embodied large model in the feature extraction module. The first visual feature extraction model and the second visual feature extraction model can be the same visual feature extraction model.

[0106] Step S6: Repeat the above steps until the preset number of training iterations or the loss function value converges, then end the training.

[0107] In this embodiment, through the above steps S1-S6, the model used in the feature extraction module (first visual feature extraction model, encoder of PCT model, second visual feature extraction model, large language model) and multimodal embodied large model are trained in tandem using training samples. This is beneficial for the trained multimodal embodied large model to better integrate the extracted multimodal ultrasonic features, thereby generating more accurate robotic arm state features and robotic arm translation and rotation features, and realizing precise control of the robotic arm.

[0108] On the one hand, this embodiment utilizes an information acquisition module to collect ultrasonic information through multiple sensors (ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text commands, and 6-dimensional force estimation data). This comprehensively leverages the advantages of various sensors, compensating for the shortcomings of a single sensor, ensuring the comprehensiveness of data acquisition, and thus improving the accuracy of the acquired ultrasonic information. This facilitates more accurate ultrasonic motion calculations in subsequent models. On the other hand, this embodiment utilizes a multimodal embodied large model, fusing multimodal feature information from different sensors (including RGB image features, point cloud data features, ultrasonic image features, text command tokens, and force estimation features) for comprehensive analysis and decision-making. It outputs robotic arm state features (whether the robotic arm is in an abnormal state at the current moment) and robotic arm translation and rotation features (the translational and rotational actions the robotic arm needs to perform at the current moment). Based on the output results, it guides the movement of the robotic arm, achieving intelligent control of the robotic arm. By integrating multimodal ultrasonic feature information, the ultrasonic imaging scanning process becomes more accurate, efficient, and safe, improving the quality and efficiency of ultrasonic scanning.

[0109] The second aspect of this application also provides an ultrasound method, executed by the ultrasound system based on a multimodal embodied large model described in the first aspect of this application, referring to... Figure 3 , Figure 3 A flowchart of an ultrasound method is shown, as follows: Figure 3 As shown, the method includes:

[0110] Step S101: Acquire ultrasonic information of the current position of the robotic arm through multiple sensors. The ultrasonic information includes: RGB image, point cloud data, ultrasonic image, text command, and 6-dimensional force estimation data.

[0111] Step S102: The ultrasound information is extracted using the feature extraction module to obtain ultrasound features, which include: RGB image features, point cloud data features, ultrasound image features, text command tokens, and force estimation features.

[0112] Step S103: Using the multimodal embodied large model, based on the ultrasonic features, output the robotic arm state features and robotic arm translation and rotation features; wherein, the robotic arm state features indicate whether the robotic arm is in an abnormal state at the current moment, and the robotic arm translation and rotation features indicate the translation and rotation actions that the robotic arm needs to perform at the current moment.

[0113] Step S104: The state features and translational / rotation features of the robotic arm are decoded using an action decoder to generate ultrasonic motions, which guide the robotic arm to move according to the ultrasonic motions at the current moment.

[0114] In one possible implementation, acquiring ultrasonic information about the current position of the robotic arm through multiple sensors includes:

[0115] The RGB image and the point cloud data are acquired using a depth camera. The RGB image represents an image containing the three-dimensional morphological information of the detected object, and the point cloud data represents scene point cloud information containing the spatial relationship between the robotic arm and the detected object.

[0116] An ultrasonic probe is used to acquire ultrasonic images of the location of the robotic arm; the ultrasonic probe is fixed to the robotic arm.

[0117] The text input device is used to receive the text commands input by the user.

[0118] The 6-dimensional force estimation data is acquired using a force sensor, and the 6-dimensional force estimation data represents the magnitude of the force applied by the ultrasonic probe to the object being tested.

[0119] In one possible implementation, the step of extracting features from the ultrasound information to obtain ultrasound features includes:

[0120] The RGB image is scaled to a resolution of 224x224. Using a first visual feature extraction model, a 768-dimensional image feature vector is extracted from the RGB image. The extracted image feature vector is then passed through a fully connected layer to obtain a 4096-dimensional RGB image feature.

[0121] In the base coordinate system of the robotic arm, the encoder of the PCT model extracts 1024-dimensional three-dimensional feature information from the point cloud data. The extracted three-dimensional feature information is then transformed through a fully connected layer to obtain 4096-dimensional point cloud data features.

[0122] The ultrasound image is scaled to a resolution of 224x224. Using a second visual feature extraction model, a 768-dimensional ultrasound image feature vector is extracted from the ultrasound image. The extracted ultrasound image feature vector is then passed through a fully connected layer to obtain a 4096-dimensional ultrasound image feature.

[0123] Using a large language model, the text instructions are converted into text instruction tokens with a feature dimension of 1024;

[0124] The 6-dimensional force estimation data is transformed by a fully connected layer to obtain the 4096-dimensional force estimation features.

[0125] In one possible implementation, the first visual feature extraction model and the second visual feature extraction model are the same visual feature extraction model.

[0126] In one possible implementation, the feature extraction module and the multimodal embodied large model are trained according to the following steps:

[0127] Acquire training samples, which are ultrasonic information of the robotic arm's location obtained through the multiple sensors, and training sample labels; wherein, the ultrasonic information of the samples includes: sample RGB image, sample point cloud data, sample ultrasonic image, sample text command, and sample 6-dimensional force estimation data; the training sample labels indicate the state of the robotic arm at the time of acquisition of the training samples, and the movement action performed by the robotic arm in the next moment;

[0128] The training samples are input into the feature extraction module to extract the ultrasonic features of the samples. The ultrasonic features of the samples include: RGB image features of the samples, point cloud data features of the samples, ultrasonic image features of the samples, text command tokens of the samples, and force estimation features of the samples.

[0129] The ultrasonic features of the sample are input into the multimodal embodied large model to obtain the output results, which include: sample robotic arm state features and sample robotic arm translation and rotation features.

[0130] Calculate the loss function value based on the output and the training sample labels;

[0131] Based on the loss function value, the parameters of the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, the large language model, and the multimodal embodied large model in the feature extraction module are updated.

[0132] Repeat the above steps until the preset number of training iterations or the loss function value converges, then end the training.

[0133] In one possible implementation, the robotic arm state characteristics include at least one of the following:

[0134] The first robotic arm state characteristic indicates whether the robotic arm has reached the predetermined target position at the current moment;

[0135] The second robotic arm state characteristic indicates whether the robotic arm has detached from the surface of the target object at the current moment;

[0136] The third robotic arm status characteristic indicates whether there is any abnormality in the relative position of the robotic arm with respect to the target at the current moment.

[0137] In one possible implementation, the step of outputting the robotic arm state characteristics and the robotic arm translational and rotational characteristics based on the ultrasonic characteristics includes:

[0138] Acquire historical ultrasound features at multiple time points;

[0139] An ultrasound feature sequence is generated based on the historical ultrasound features at the multiple time points and the currently input ultrasound features;

[0140] Based on the ultrasonic feature sequence, the state characteristics and translational / rotation characteristics of the robotic arm are determined;

[0141] The method further includes storing the currently input ultrasound features as historical ultrasound features.

[0142] This application also provides an electronic device, see embodiments thereof. Figure 4 , Figure 4 This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 4 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the ultrasound method disclosed in the embodiments of this application.

[0143] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the ultrasound method disclosed in this application.

[0144] This application also provides a computer program product that, when run on an electronic device, causes a processor to execute the steps of the ultrasonic method disclosed in this application.

[0145] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0146] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0151] The above provides a detailed description of the ultrasound system, method, and product based on a multimodal embodied large model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An ultrasound system based on a multimodal embodied large model, characterized in that, The system includes: The information acquisition module is used to acquire ultrasonic information of the current position of the robotic arm through multiple sensors. The ultrasonic information includes: RGB image, point cloud data, ultrasonic image, text command, and 6-dimensional force estimation data. The feature extraction module is used to extract features from the ultrasound information to obtain ultrasound features, which include: RGB image features, point cloud data features, ultrasound image features, text command tokens, and force estimation features. A multimodal embodied large model is used to output robotic arm state features and robotic arm translation and rotation features based on the ultrasonic features; wherein, the robotic arm state features indicate whether the robotic arm is in an abnormal state at the current moment, and the robotic arm translation and rotation features indicate the translation and rotation actions that the robotic arm needs to perform at the current moment. An action decoder is used to decode the state characteristics and translational and rotational characteristics of the robotic arm to generate ultrasonic motions, so as to guide the robotic arm to move according to the ultrasonic motions at the current moment. The multimodal embodied large model also includes a storage module for storing historical ultrasound features; the step of outputting robotic arm state features and robotic arm translation and rotation features based on the ultrasound features includes: Retrieve historical ultrasound features from multiple moments in the storage module; An ultrasound feature sequence is generated based on the historical ultrasound features at the multiple time points and the currently input ultrasound features; Based on the ultrasonic feature sequence, the state characteristics and translational / rotation characteristics of the robotic arm are determined; The storage module is also used to store the currently input ultrasound features as historical ultrasound features.

2. The ultrasound system based on a multimodal embodied large model according to claim 1, characterized in that, The plurality of sensors include: A depth camera is used to acquire the RGB image and the point cloud data. The RGB image represents an image containing three-dimensional morphological information of the detected object, and the point cloud data represents scene point cloud information containing the spatial relationship between the robotic arm and the detected object. An ultrasonic probe, which is fixed on the robotic arm, is used to acquire ultrasonic images at the location of the robotic arm; A text input device for receiving text commands input by a user; A force sensor is used to collect the 6-dimensional force estimation data, which represents the magnitude of the force applied by the ultrasonic probe to the object being tested.

3. The ultrasound system based on a multimodal embodied large model according to claim 1, characterized in that, The feature extraction module is used for: The RGB image is scaled to a resolution of 224x224. Using a first visual feature extraction model, a 768-dimensional image feature vector is extracted from the RGB image. The extracted image feature vector is then passed through a fully connected layer to obtain a 4096-dimensional RGB image feature. In the base coordinate system of the robotic arm, the encoder of the PCT model extracts 1024-dimensional three-dimensional feature information from the point cloud data. The extracted three-dimensional feature information is then transformed through a fully connected layer to obtain 4096-dimensional point cloud data features. The ultrasound image is scaled to a resolution of 224x224. Using a second visual feature extraction model, a 768-dimensional ultrasound image feature vector is extracted from the ultrasound image. The extracted ultrasound image feature vector is then passed through a fully connected layer to obtain a 4096-dimensional ultrasound image feature. Using a large language model, the text instructions are converted into text instruction tokens with a feature dimension of 1024; The 6-dimensional force estimation data is transformed by a fully connected layer to obtain the 4096-dimensional force estimation features.

4. The ultrasound system based on a multimodal embodied large model according to claim 3, characterized in that, The first visual feature extraction model and the second visual feature extraction model are the same visual feature extraction model.

5. The ultrasound system based on a multimodal embodied large model according to claim 3, characterized in that, The feature extraction module and the multimodal embodied large model are trained according to the following steps: Acquire training samples, which are ultrasonic information of the robotic arm's location obtained through the multiple sensors, and training sample labels; wherein, the ultrasonic information of the samples includes: sample RGB image, sample point cloud data, sample ultrasonic image, sample text command, and sample 6-dimensional force estimation data; the training sample labels indicate the state of the robotic arm at the time of acquisition of the training samples, and the movement action performed by the robotic arm in the next moment; The training samples are input into the feature extraction module to extract the ultrasonic features of the samples. The ultrasonic features of the samples include: RGB image features of the samples, point cloud data features of the samples, ultrasonic image features of the samples, text command tokens of the samples, and force estimation features of the samples. The ultrasonic features of the sample are input into the multimodal embodied large model to obtain the output results, which include: sample robotic arm state features and sample robotic arm translation and rotation features. Calculate the loss function value based on the output and the training sample labels; Based on the loss function value, the parameters of the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, the large language model, and the multimodal embodied large model in the feature extraction module are updated. Repeat the above steps until the preset number of training iterations or the loss function value converges, then end the training.

6. The ultrasound system based on a multimodal embodied large model according to claim 1, characterized in that, The state characteristics of the robotic arm include at least one of the following: The first robotic arm state characteristic indicates whether the robotic arm has reached the predetermined target position at the current moment; The second robotic arm state characteristic indicates whether the robotic arm has detached from the surface of the target object at the current moment; The third robotic arm status characteristic indicates whether there is any abnormality in the relative position of the robotic arm with respect to the target at the current moment.

7. An ultrasonic method, characterized in that, Performed by the ultrasound system based on a multimodal embodied large model according to any one of claims 1-6, the method comprises: The ultrasonic information of the current position of the robotic arm is obtained through multiple sensors. The ultrasonic information includes: RGB image, point cloud data, ultrasonic image, text command, and 6-dimensional force estimation data. The ultrasonic information is processed by a feature extraction module to obtain ultrasonic features, which include: RGB image features, point cloud data features, ultrasonic image features, text command tokens, and force estimation features. Using a multimodal embodied large model, based on the ultrasonic features, the robot arm state features and robot arm translation and rotation features are output; wherein, the robot arm state features indicate whether the robot arm is in an abnormal state at the current moment, and the robot arm translation and rotation features indicate the translation and rotation actions that the robot arm needs to perform at the current moment. The motion decoder decodes the state features and translational / rotation features of the robotic arm to generate ultrasonic motions, which guide the robotic arm to move according to the ultrasonic motions at the current moment.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the ultrasound method of claim 7.

9. A computer-readable storage medium, characterized in that, It stores a computer program / instruction that, when executed by a processor, implements the ultrasonic method of claim 7.

Images