Ultrasonic system, method and product based on multi-mode body model

Abstract

The invention provides an ultrasonic system, method and product based on a multi-mode body model, and relates to the technical field of ultrasonic acquisition, the system comprises an information acquisition module used for acquiring ultrasonic information of the position of a mechanical arm at the current moment through a plurality of sensors, the ultrasonic information comprises an RGB image, point cloud data, an ultrasonic image and a character instruction, 6-dimensional force estimation data; the feature extraction module is used for extracting features to obtain ultrasonic features; the multi-mode body model is used for outputting mechanical arm state features and mechanical arm translation and rotation features according to the ultrasonic features; the mechanical arm state feature represents whether the mechanical arm at the current moment is in an abnormal state or not, and the mechanical arm translation and rotation feature represents the translation action and rotation action needing to be executed by the mechanical arm at the current moment; and the action decoder is used for decoding the mechanical arm state feature and the mechanical arm translation and rotation feature to generate an ultrasonic action so as to guide the mechanical device at the current moment to move according to the ultrasonic action.

Description

Technical Field

[0001] The present application relates to the field of ultrasound acquisition technology, and in particular to an ultrasound system, method and product based on a multi-modal embodied large model. Background Art

[0002] Ultrasound imaging based on embodied robotic arms is an emerging and important task in the medical field. Its purpose is to replace handheld ultrasound scanning. Compared with traditional manual operations, automatic ultrasound scanning using robotic arms has higher accuracy and consistency, and can also reduce the operator's workload and improve work efficiency.

[0003] However, the structure of human organs is complex and diverse, and individual differences are significant (such as height, weight, age, gender, etc.), so the ultrasonic imaging scanning method based on the robotic arm often cannot accurately find the ideal cross-sectional position of the target organ in practical applications, and the success rate is low, making it difficult to cope with the complex and changeable clinical actual needs. Therefore, it is urgent to propose an ultrasonic system, method and product based on a multi-modal embodied large model to improve the reliability of ultrasonic scanning based on the embodied robotic arm. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide an ultrasound system, method and product based on a multi-modal embodied large model to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, an ultrasound system based on a multimodal embodied large model is provided, the system comprising: An information acquisition module is used to obtain ultrasonic information of the current position of the robotic arm through multiple sensors, wherein the ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text instructions, and 6-dimensional force estimation data; A feature extraction module, used to extract features from the ultrasound information to obtain ultrasound features, wherein the ultrasound features include: RGB image features, point cloud data features, ultrasound image features, text instruction tokens, and force estimation features; The multimodal embodied large model is used to output the robot arm state feature and the robot arm translation and rotation feature according to the ultrasonic feature; wherein the robot arm state feature indicates whether the robot arm is in an abnormal state at the current moment, and the robot arm translation and rotation feature indicates the translation action and rotation action that the robot arm needs to perform at the current moment; The motion decoder is used to decode the state characteristics of the robot arm and the translation and rotation characteristics of the robot arm to generate ultrasonic motion to guide the robot at the current moment to move according to the ultrasonic motion.

[0006] In a possible implementation, the plurality of sensors include: A depth camera, used to acquire the RGB image and the point cloud data, wherein the RGB image represents an image including three-dimensional morphological information of the detection object, and the point cloud data represents scene point cloud information including a spatial relationship between the robot arm and the detection object; An ultrasonic probe, which is fixed on the mechanical arm and is used to collect an ultrasonic image of the position where the mechanical arm is located; A text instruction input device, used for receiving the text instruction input by a user; The force sensor is used to collect the 6-dimensional force estimation data, where the 6-dimensional force estimation data represents the magnitude of the force applied by the ultrasonic probe to the detection object.

[0007] In a possible implementation, the feature extraction module is used to: Scaling the RGB image to a resolution of 224x224, extracting a 768-dimensional image feature vector from the RGB image using a first visual feature extraction model, and passing the extracted image feature vector through a fully connected layer to obtain a 4096-dimensional RGB image feature; In the base coordinate system of the robot arm, 1024-dimensional three-dimensional feature information is extracted from the point cloud data using the encoder of the PCT model, and the extracted three-dimensional feature information is converted through a fully connected layer to obtain 4096-dimensional point cloud data features; Scaling the ultrasound image to a resolution of 224x224, extracting a 768-dimensional ultrasound image feature vector from the ultrasound image using a second visual feature extraction model, and passing the extracted ultrasound image feature vector through a fully connected layer to obtain a 4096-dimensional ultrasound image feature; Using a large language model, converting the text instruction into the text instruction token with a feature dimension of 1024; The 6-dimensional force estimation data is converted through a fully connected layer to obtain the 4096-dimensional force estimation features.

[0008] In a possible implementation, the first visual feature extraction model and the second visual feature extraction model are the same visual feature extraction model.

[0009] In a possible implementation, the feature extraction module and the multimodal embodied large model are trained according to the following steps: Acquire training samples, wherein the training samples are sample ultrasonic information of the position of the manipulator obtained by the multiple sensors, and training sample labels; wherein the sample ultrasonic information includes: sample RGB images, sample point cloud data, sample ultrasonic images, sample text instructions, and sample 6-dimensional force estimation data; the training sample labels represent the state of the manipulator at the time of collecting the training samples, and the movement action performed by the manipulator at the next time; Input the training sample into the feature extraction module to extract the sample ultrasound features, wherein the sample ultrasound features include: sample RGB image features, sample point cloud data features, sample ultrasound image features, sample text instruction tokens, and sample force estimation features; Inputting the sample ultrasonic features into the multimodal embodied large model to obtain output results, wherein the output results include: sample mechanical arm state features and sample mechanical arm translation and rotation features; Calculating a loss function value according to the output result and the training sample label; According to the loss function value, updating parameters of the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, the large language model, and the multimodal embodied large model in the feature extraction module; Repeat the above steps until the preset number of training times is reached or the loss function value converges, and then the training ends.

[0010] In a possible implementation manner, the robot arm state feature includes at least one of the following: The first robot arm state feature indicates whether the robot arm has reached the predetermined target position at the current moment; The second robot arm state feature indicates whether the robot arm is detached from the target surface at the current moment; The third robot arm state feature indicates whether there is an abnormality in the relative position of the robot arm and the target to be measured at the current moment.

[0011] In a possible implementation, the multimodal embodied large model further includes a storage module, the storage module being used to store historical ultrasonic features; the outputting of the robot arm state features and the robot arm translation and rotation features according to the ultrasonic features comprises: Acquire historical ultrasonic features at multiple moments in the storage module; generating an ultrasonic feature sequence according to the historical ultrasonic features at the plurality of moments and the ultrasonic features currently input; Determining the state characteristics of the robotic arm and the translation and rotation characteristics of the robotic arm according to the ultrasonic characteristic sequence; The storage module is further used to store the currently input ultrasonic feature as a historical ultrasonic feature.

[0012] The second aspect of the embodiment of the present application further provides an ultrasound method, which is performed by the ultrasound system based on the multimodal embodied large model described in the first aspect of the embodiment of the present application, and the method includes: Acquire ultrasonic information of the current position of the robotic arm through multiple sensors, wherein the ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text instructions, and 6-dimensional force estimation data; The ultrasonic information is subjected to feature extraction to obtain ultrasonic features through a feature extraction module, wherein the ultrasonic features include: RGB image features, point cloud data features, ultrasonic image features, text instruction tokens, and force estimation features; Through the multimodal embodied large model, according to the ultrasonic features, the robot arm state features and the robot arm translation and rotation features are output; wherein the robot arm state features indicate whether the robot arm is in an abnormal state at the current moment, and the robot arm translation and rotation features indicate the translation and rotation actions that the robot arm needs to perform at the current moment; The robot state characteristics and the robot translation and rotation characteristics are decoded by a motion decoder to generate ultrasonic motions to guide the robot at the current moment to move according to the ultrasonic motions.

[0013] The third aspect of the embodiments of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the ultrasound method described in the second aspect of the embodiments of the present application.

[0014] The fourth aspect of the embodiments of the present application further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the ultrasonic method described in the second aspect of the embodiments of the present application are implemented.

[0015] The fifth aspect of the embodiments of the present application further provides a computer program product, which, when run on an electronic device, enables a processor to implement the steps of the ultrasound method described in the second aspect of the embodiments of the present application.

[0016] The embodiment of the present application provides an ultrasound system based on a multimodal embodied large model, the system comprising: an information acquisition module, used to obtain ultrasound information of the current position of a robotic arm through multiple sensors, the ultrasound information comprising: RGB images, point cloud data, ultrasound images, text instructions, and 6-dimensional force estimation data; a feature extraction module, used to extract features from the ultrasound information to obtain ultrasound features, the ultrasound features comprising: RGB image features, point cloud data features, ultrasound image features, text instruction tokens, and force estimation features; a multimodal embodied large model, used to output a robotic arm state feature and a robotic arm translation and rotation feature according to the ultrasound feature; wherein the robotic arm state feature indicates whether the robotic arm is in an abnormal state at the current moment, and the robotic arm translation and rotation feature indicates the translation and rotation actions that the robotic arm needs to perform at the current moment; an action decoder, used to decode the robotic arm state feature and the robotic arm translation and rotation feature, and generate an ultrasonic action to guide the robot at the current moment to move according to the ultrasonic action.

[0017] The specific beneficial effects are: On the one hand, the embodiment of the present application utilizes an information acquisition module to collect ultrasonic information through multiple sensors (ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text instructions, and 6-dimensional force estimation data), thereby comprehensively utilizing the advantages of various sensors, making up for the shortcomings of a single sensor, and ensuring the comprehensiveness of data acquisition, thereby improving the accuracy of the acquired ultrasonic information, which is conducive to subsequent model calculations to obtain more accurate ultrasonic actions.

[0018] On the other hand, the embodiment of the present application uses a multimodal embodied large model to fuse multimodal feature information from different sensors (including RGB image features, point cloud data features, ultrasonic image features, text command tokens, and force estimation features) to conduct comprehensive analysis and decision-making, output the robot arm state features (whether the robot arm is in an abnormal state at the current moment) and the robot arm translation and rotation features (the translation and rotation actions that the robot arm needs to perform at the current moment), and guide the movement of the robot arm according to the output results, thereby realizing intelligent control of the robot arm. By integrating multimodal ultrasonic feature information, the ultrasonic imaging scanning process is made more accurate, efficient, and safe, and the quality and efficiency of ultrasonic scanning are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 It is a structural schematic diagram of an ultrasound system based on a multi-modal embodied large model provided in an embodiment of the present application; Figure 2 It is a schematic diagram of a processing flow of an ultrasound system provided in an embodiment of the present application; Figure 3 is a flowchart of the steps of an ultrasonic method provided in an embodiment of the present application; Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0022] Ultrasound imaging based on embodied robotic arms is an emerging and important task in the medical field. Its purpose is to replace handheld ultrasound scanning. Compared with traditional manual operations, automatic ultrasound scanning using robotic arms has higher accuracy and consistency, and can also reduce the operator's workload and improve work efficiency.

[0023] In the related art, ultrasonic imaging scanning based on a robotic arm is usually completed through two main methods: rule-based and ultrasonic imaging-based visual neural network. The rule-based method relies on a pre-set operating procedure and uses a robotic arm to scan according to a predetermined trajectory and parameters. This method performs well in a simple and fixed operating environment, but due to the complex and diverse structure of human organs and significant individual differences (such as height, weight, age, gender, etc.), the rule-based method often cannot accurately find the ideal cross-sectional position of the target organ in practical applications, and the success rate is not high. In order to overcome this limitation, a visual neural network method based on ultrasonic imaging came into being. This method uses deep learning technology and a large amount of ultrasonic imaging data for training, so that the robotic arm can scan more intelligently. Although this method improves the accuracy and generalization ability of scanning to a certain extent, due to its single signal source, it still has a high failure rate when facing special patients (such as obese patients, disabled people, etc.).

[0024] In addition, in actual operation, not only ultrasound images are relied upon, but also a variety of sensory information is used to judge and adjust the scanning process. For example, while manually performing ultrasound, a real person will observe the patient's position and posture, analyze the ultrasound image in real time, feel the contact force between the probe and the skin, and determine the optimal probe position and posture based on the patient's verbal feedback. The comprehensive use of this multimodal information greatly improves the accuracy and effect of the scan. Therefore, how to enable the robotic arm to obtain and process information of multiple modalities in automated ultrasound scanning like a real person has become a key research direction. Although the automated medical ultrasound imaging scanning technology based on a robotic arm has taken shape, the current technology is still in the initial exploration stage, and targeted optimization measures have not yet been taken, making it difficult to cope with the complex and changing actual clinical needs.

[0025] In view of the above problems, the embodiments of the present application propose an ultrasound system, method and product based on a multi-modal embodied large model to improve the reliability of ultrasound scanning based on an embodied robotic arm. In conjunction with the accompanying drawings, the ultrasound system based on a multi-modal embodied large model provided by the embodiments of the present application is described in detail through some embodiments and their application scenarios.

[0026] In a first aspect, the present application provides an ultrasound system based on a multi-modal embodied large model, referring to Figure 1 , Figure 1 A schematic diagram of the structure of an ultrasound system based on a multi-modal embodied large model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the system comprises: The information acquisition module is used to obtain ultrasonic information of the current position of the robotic arm through multiple sensors. The ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text instructions, and 6-dimensional force estimation data.

[0027] The feature extraction module is used to extract features from the ultrasound information to obtain ultrasound features, wherein the ultrasound features include: RGB image features, point cloud data features, ultrasound image features, text instruction tokens, and force estimation features.

[0028] The multimodal embodied large model is used to output the robot arm state characteristics and the robot arm translation and rotation characteristics according to the ultrasonic characteristics; wherein the robot arm state characteristics indicate whether the robot arm is in an abnormal state at the current moment, and the robot arm translation and rotation characteristics indicate the translation and rotation movements that the robot arm needs to perform at the current moment.

[0029] The motion decoder is used to decode the state characteristics of the robot arm and the translation and rotation characteristics of the robot arm to generate ultrasonic motion to guide the robot at the current moment to move according to the ultrasonic motion.

[0030] In the embodiments of this application, refer to Figure 2 , Figure 2 A schematic diagram of a processing flow of an ultrasound system is shown. Figure 2 As shown in the figure, each time the robotic arm performs an ultrasonic acquisition action, it controls multiple sensors to collect data simultaneously, and the information acquisition module obtains the perception data (that is, the ultrasonic information of the current position of the robotic arm) through multiple sensors. During the application process, each time the robotic arm moves, it automatically performs an ultrasonic acquisition to obtain the ultrasonic image and related information of the corresponding position. The information acquisition module in this system comprehensively utilizes the advantages of various sensors to make up for the shortcomings of a single sensor, ensure the comprehensiveness of data acquisition, and thus improve the accuracy of the acquired ultrasonic information.

[0031] In a possible implementation, the plurality of sensors include: A depth camera is used to obtain the RGB image and the point cloud data, wherein the RGB image represents an image containing three-dimensional morphological information of the detection object, and the point cloud data represents scene point cloud information containing the spatial relationship between the robotic arm and the detection object.

[0032] An ultrasonic probe is fixed on the mechanical arm and is used to collect an ultrasonic image of the position where the mechanical arm is located.

[0033] The text instruction input device is used to receive the text instruction input by the user.

[0034] The force sensor is used to collect the 6-dimensional force estimation data, where the 6-dimensional force estimation data represents the magnitude of the force applied by the ultrasonic probe to the detection object.

[0035] Specifically, the RGB image and point cloud data at the current moment are collected by the depth camera, and the three-dimensional morphological information of the body of the detection object (such as human body posture information) is obtained from it, which helps the system understand the spatial relationship between the robotic arm and the detection object, and ensures that the robotic arm can accurately locate and avoid obstacles. Exemplarily, the depth camera can be a Microsoft Kinect V2 device, which can obtain RGB images and point cloud data in the scene. By fixing the ultrasound probe on the robotic arm, the ultrasound probe changes position with the movement of the robotic arm to obtain ultrasound images at different positions and angles. In this way, real-time ultrasound image data is obtained to help the system analyze and identify the structure and state of the target organ of the detection object, and provide a key basis for subsequent operation decisions (generating ultrasound actions). The text command input by the user is received through the text command input device, allowing the operator to input specific operation requirements (such as collecting a cross-sectional image of organ A) or adjust parameters to the system through the text command input device (such as a user interface), making the whole process more flexible and controllable. The force sensor detects the contact force (i.e., 6-dimensional force estimation data) between the robotic arm and the test object (such as the patient's skin) at the current moment to ensure that the pressure applied by the robotic arm to the test object during the ultrasound scan is moderate to avoid discomfort or injury to the patient.

[0036] like Figure 2 As shown in the figure, after acquiring the ultrasonic information collected by multiple sensors, the ultrasonic information is extracted through the feature extraction module to obtain ultrasonic features (RGB image features, point cloud data features, ultrasonic image features, text command tokens, and force estimation features). The feature extraction module integrates the ultrasonic information from different sensors, and extracts ultrasonic features that are critical to operational decisions through complex algorithm analysis and processing. For example, the combination of the three-dimensional morphological data provided by the depth camera and the ultrasonic image provided by the ultrasonic probe can more accurately locate the target organ, while the data collected by the force sensor can help adjust the operating force of the robotic arm to ensure safety and comfort.

[0037] In a possible implementation, the feature extraction module is used to: The RGB image is scaled to a resolution of 224x224, and a 768-dimensional image feature vector is extracted from the RGB image using a first visual feature extraction model. The extracted image feature vector is passed through a fully connected layer to obtain a 4096-dimensional RGB image feature. Specifically, the first visual feature extraction model may be a ViT-S (Visual Transformer) model, which is an advanced visual neural network that can efficiently process image data and extract a 768-dimensional feature vector.

[0038] In the base coordinate system of the robot, the encoder of the PCT model is used to extract 1024-dimensional three-dimensional feature information from the point cloud data, and the extracted three-dimensional feature information is transformed through a fully connected layer to obtain 4096-dimensional point cloud data features. Specifically, for different sensors, due to different installation positions, the coordinate systems to which the collected information belongs are different (the origin positions of the coordinate systems are different). Through coordinate system conversion, the point cloud data collected by the depth camera is converted into data in the coordinate system to which the robot belongs, and then subsequent feature extraction is performed to obtain more accurate ultrasonic data. Exemplarily, the rotation angle and translation distance (translation distance in the x-direction, translation distance in the y-direction, and translation distance in the z-direction) or transformation matrix of the coordinate system of the depth camera relative to the base coordinate system of the robot can be obtained through calibration, and then the data can be corrected according to the rotation angle and translation distance (or transformation matrix). The point cloud data can be processed by the encoder (Encoder) of the Point Cloud Transformer (PCT) model (such as Figure 2 The PCT model is a neural network specifically designed for processing point cloud data, which can efficiently extract key three-dimensional information from point clouds.

[0039] The ultrasound image is scaled to a resolution of 224x224, and a 768-dimensional ultrasound image feature vector is extracted from the ultrasound image using a second visual feature extraction model. The extracted ultrasound image feature vector is passed through a fully connected layer to obtain a 4096-dimensional ultrasound image feature.

[0040] In a possible implementation, the first visual feature extraction model and the second visual feature extraction model are the same visual feature extraction model.

[0041] Specifically, the ultrasound image captured by the ultrasound probe may be an RGB image, which may share the same visual feature extraction model (eg, ViT-S model) with the RGB image captured by the depth camera, thereby going through the same feature extraction process to ensure consistency and efficiency of feature extraction.

[0042] The text instruction is converted into the text instruction token with a feature dimension of 1024 by using a large language model. Specifically, the text instruction input by the user is processed by the large language model. The large language model can be an LLama 7BTokenizer model, which is an advanced natural language processing tool that can convert text instructions into feature vectors. In this process, the text instruction is converted into a token with a feature dimension of 1024 so as to be integrated and processed with other perception data.

[0043] The 6-dimensional force estimation data is converted through a fully connected layer to obtain a 4096-dimensional force estimation feature. Specifically, the 6-dimensional force estimation data collected by the force sensor is used to monitor the contact force between the robotic arm and the detection object (such as human skin). The 6-dimensional force estimation data is converted through a fully connected layer to obtain a 4096-dimensional force estimation feature. The force estimation feature provides important information about the contact force of the robotic arm, which helps to ensure that the robotic arm applies appropriate pressure during the scanning process to avoid discomfort or injury to the patient.

[0044] In this embodiment, after extracting the multimodal ultrasonic features, these ultrasonic features are combined into a sequence input into the multimodal embodied large model, so as to predict the state features of the robotic arm and the translation and rotation features of the robotic arm based on the input ultrasonic features. Among them, the multimodal embodied large model is a comprehensive artificial intelligence model that can process and fuse multimodal data from multiple sensors to make complex operational decisions and reasoning. After learning from a large amount of training data, the model has powerful analysis and decision-making capabilities and can make accurate and efficient judgments in complex operating environments. This multimodal perception and processing method greatly improves the intelligence and automation level of the system, enabling the robotic arm to complete ultrasonic imaging scanning tasks more accurately and efficiently.

[0045] Optionally, the multimodal embodied large model can be a feature fusion model in OpenVLA, the core architecture of which is LLama 7B, a powerful large language model that can process and fuse data sequences from different sensors to generate robot arm state features and translation and rotation features. Among them, the output results of the multimodal embodied large model include: robot arm state features and robot arm translation and rotation features. Among them, the robot arm state features indicate whether the robot arm is in an abnormal state at the current moment. The robot arm state features can be used to guide in real time whether the operation of the robot arm continues to move forward or needs to be adjusted. For example, when it is detected that the robot arm has not reached the ideal position or an abnormal situation occurs, the system can instruct the robot arm to make corresponding adjustments to ensure the accuracy and safety of the operation.

[0046] The robot arm translation and rotation feature indicates the translation and rotation action that the robot arm needs to perform at the current moment (i.e., the target position of the next action of the robot arm), and contains the specific position and posture information of the robot arm in space. Furthermore, the robot arm translation and rotation feature can be: indicating the translation and rotation information of the robot arm on the x, y, and z axes. For example, it can be 1 cm forward on the x axis and 5 degrees clockwise rotation on the y axis. The robot arm translation and rotation feature is generated through the reasoning and calculation of the model, which can guide the robot arm to perform precise motion control and ensure that the robot arm moves and rotates according to the predetermined trajectory to achieve high-quality ultrasound image scanning.

[0047] In a possible implementation manner, the robot arm state feature includes at least one of the following: The first robotic arm state feature indicates whether the robotic arm has reached the predetermined target position at the current moment. The output first robotic arm state feature is used to determine whether the robotic arm has reached the ideal final position to ensure that the ultrasound probe is accurately positioned at the part to be scanned. For example, if the text instruction input by the user is "collect the cross-section of organ A", the output first robotic arm state feature indicates whether the current robotic arm has reached the ideal position, that is, whether the ultrasound image collected at the current position is a clear cross-section image of organ A.

[0048] The second robotic arm state feature indicates whether the robotic arm is currently separated from the target surface to be measured. The second robotic arm state feature can be used to monitor the contact between the robotic arm and the patient's skin to ensure that the robotic arm does not accidentally separate from the patient's surface during the scanning process, thereby affecting the scanning results.

[0049] The third robotic arm state feature indicates whether there is an abnormality in the relative position of the robotic arm to the target at the current moment. The third robotic arm state feature is used to detect whether there is an abnormality in the relative position between the robotic arm and the patient. If an abnormality is detected, the operation of the robotic arm is adjusted in time to avoid discomfort or injury to the patient.

[0050] In a possible implementation, the multimodal embodied large model further includes a storage module, the storage module being used to store historical ultrasonic features; and outputting the robot arm state features and the robot arm translation and rotation features according to the ultrasonic features, comprising: Acquire historical ultrasonic features at multiple moments in the storage module; generating an ultrasonic feature sequence according to the historical ultrasonic features at the plurality of moments and the ultrasonic features currently input; Determining the state characteristics of the robotic arm and the translation and rotation characteristics of the robotic arm according to the ultrasonic characteristic sequence; The storage module is further used to store the currently input ultrasonic feature as a historical ultrasonic feature.

[0051] In this embodiment, the multimodal embodied large model can make predictions based only on the ultrasonic features of the current moment of the input to obtain the output results (mechanical arm state features and mechanical arm translation and rotation features), or it can combine the current input with historical information to generate an ultrasonic feature sequence composed in chronological order. Specifically, the ultrasonic features of each input to the multimodal embodied large model are stored in the storage module as the historical ultrasonic features of the corresponding moment, so as to obtain the historical ultrasonic features of the previous k moments and sort them with the ultrasonic features of the current moment in chronological order to obtain the ultrasonic feature sequence. In this way, the multi-module embodied large model can combine historical information (historical ultrasonic features at multiple moments) to obtain a more accurate output result.

[0052] The motion decoder is used to decode the state characteristics of the robot arm and the translation and rotation characteristics of the robot arm to generate ultrasonic motion to guide the robot at the current moment to move according to the ultrasonic motion.

[0053] Specifically, the prediction results (mechanical arm state features and mechanical arm translation and rotation features) generated by the multimodal embodied large model are decoded by the action decoder, and the action decoder generates an ultrasonic action (i.e., the decoding result), which can be represented as a 9-dimensional vector. In the 9-dimensional vector, the first 3 dimensions are used to represent the state of the robotic arm (corresponding to the mechanical arm state features), including whether the robotic arm has reached the ideal position, whether it has been separated from the human body surface, and whether there is an abnormality between the human body posture and the human body posture. The last 6 dimensions are used to represent the translation and rotation information of the robotic arm on the x, y, and z axes (corresponding to the mechanical arm translation and rotation features), which is used to guide the specific actions of the robotic arm. The embodiment of the present application uses this detailed and accurate feature representation method to enable the model to generate highly reliable operation instructions, ensuring that the robotic arm performs accurate and safe ultrasonic imaging scans in complex medical environments. Then, the robotic arm accurately moves and adjusts its position and posture according to the generated ultrasonic action, ensuring that the ultrasonic probe can accurately align with the target organ and complete high-quality ultrasonic imaging scans. This process not only improves the accuracy and efficiency of the scan, but also reduces the workload of the operator.

[0054] In this embodiment, after the ultrasonic action is generated by the action decoder, the robot arm performs translation and rotation according to the ultrasonic action to reach the next position, and then automatically collects ultrasonic information at the next position, thereby repeating the above process (extracting features from the collected ultrasonic information through the feature extraction module, inputting the extracted ultrasonic features into the multimodal embodied large model for feature fusion and decision-making, and finally generating the ultrasonic action at the next moment through the action decoder), and repeating this process multiple times until the robot arm reaches the ideal position (for example, according to the first robot arm state feature output by the multimodal embodied large model, it is judged that the robot arm at the current moment has reached the ideal position, and a clear cross-sectional image of the target organ can be obtained by taking a picture with an ultrasonic probe, and then the ultrasonic process is terminated. The ultrasonic system based on the multimodal embodied large model constructed in the embodiment of the present application not only improves the accuracy and efficiency of ultrasonic image scanning through this multi-sensor, multi-modal perception and processing method, but also significantly improves the safety and reliability of medical operations, and promotes the development of intelligent medical equipment.

[0055] In order to improve the accuracy of ultrasound system decision making as much as possible, the model used in the system needs to be trained in advance. In a possible implementation, the feature extraction module and the multimodal embodied large model are trained according to the following steps: Step S1, obtaining training samples, wherein the training samples are sample ultrasonic information of the position of the robotic arm obtained by the multiple sensors, and training sample labels; wherein the sample ultrasonic information includes: sample RGB images, sample point cloud data, sample ultrasonic images, sample text instructions, and sample 6-dimensional force estimation data; the training sample labels represent the state of the robotic arm at the time of collecting the training samples, and the moving action performed by the robotic arm at the next moment.

[0056] Specifically, in this embodiment, the robotic arm is manually controlled to move along the optimal path, and multiple sensors (depth camera, ultrasonic probe, text command input device, force sensor) are controlled to obtain ultrasonic information at multiple times, and multiple training samples are generated. Each training sample represents the ultrasonic information collected at one time, and the label corresponding to the training sample is the action performed by the robotic arm at the next moment during the manual control of the robotic arm, as well as the current state of the robotic arm (that is, the corresponding robotic arm state characteristics and robotic arm translation and rotation characteristics).

[0057] Step S2, inputting the training sample into the feature extraction module to perform feature extraction to obtain sample ultrasound features, wherein the sample ultrasound features include: sample RGB image features, sample point cloud data features, sample ultrasound image features, sample text instruction tokens, and sample force estimation features.

[0058] Specifically, the training sample is input into the feature extraction module for feature extraction, the sample RGB image is scaled to a resolution of 224x224, a 768-dimensional image feature vector is extracted from the sample RGB image using the first visual feature extraction model, the extracted image feature vector is passed through a fully connected layer, and a 4096-dimensional sample RGB image feature is obtained; in the base coordinate system of the robotic arm, a 1024-dimensional three-dimensional feature information is extracted from the sample point cloud data using the encoder of the PCT model, the extracted three-dimensional feature information is converted through a fully connected layer, and a 4096-dimensional sample point cloud data feature is obtained; the sample ultrasound image is scaled to a resolution of 224x224, a 768-dimensional ultrasound image feature vector is extracted from the sample ultrasound image using the second visual feature extraction model, the extracted ultrasound image feature vector is passed through a fully connected layer, and a 4096-dimensional sample ultrasound image feature is obtained; the text instruction is converted into a sample text instruction token with a feature dimension of 1024 using a large language model; the sample 6-dimensional force estimation data is converted through a fully connected layer to obtain a 4096-dimensional sample force estimation feature.

[0059] Step S3, inputting the sample ultrasonic features into the multimodal embodied large model to obtain output results, wherein the output results include: sample mechanical arm state features and sample mechanical arm translation and rotation features.

[0060] Optionally, the sample ultrasonic feature inputted each time can be stored as a historical sample ultrasonic feature by the storage module. The sample ultrasonic feature and the historical sample ultrasonic features at multiple moments before can be combined into a sample ultrasonic feature sequence in chronological order, so that the multi-module embodied large model can make a decision based on the sample ultrasonic feature sequence to obtain an output result.

[0061] Step S4, calculating the loss function value according to the output result and the training sample label.

[0062] Step S5: updating parameters of the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, the large language model, and the multimodal embodied large model in the feature extraction module according to the loss function value. The first visual feature extraction model and the second visual feature extraction model may be the same visual feature extraction model.

[0063] Step S6, repeat the above steps until the preset number of training times is reached or the loss function value converges, and then the training ends.

[0064] The embodiment of the present application uses the above steps S1-S6 to collaboratively train the models used in the feature extraction module (the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, and the large language model) and the multimodal embodied large model using training samples. This is beneficial for the trained multimodal embodied large model to better integrate the extracted multimodal ultrasonic features, thereby generating more accurate robot arm state features and robot arm translation and rotation features, thereby achieving precise control of the robot arm.

[0065] On the one hand, the embodiment of the present application utilizes the information acquisition module to collect ultrasonic information through multiple sensors (ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text instructions, 6-dimensional force estimation data), so as to comprehensively utilize the advantages of various sensors, make up for the shortcomings of a single sensor, ensure the comprehensiveness of data acquisition, thereby improving the accuracy of the acquired ultrasonic information, and facilitating the subsequent model calculation to obtain more accurate ultrasonic actions. On the other hand, the embodiment of the present application utilizes a multi-modal embodied large model to fuse multi-modal feature information from different sensors (including RGB image features, point cloud data features, ultrasonic image features, text instruction tokens, and force estimation features), conducts comprehensive analysis and decision-making, outputs the state features of the robotic arm (whether the robotic arm is in an abnormal state at the current moment) and the translation and rotation features of the robotic arm (the translation and rotation actions that the robotic arm needs to perform at the current moment), so as to guide the movement of the robotic arm according to the output results, and realize the intelligent control of the robotic arm. By integrating multi-modal ultrasonic feature information, the ultrasonic imaging scanning process is made more accurate, efficient, and safe, and the quality and efficiency of ultrasonic scanning are improved.

[0066] The second aspect of the embodiment of the present application further provides an ultrasound method, which is performed by the ultrasound system based on the multi-modal embodied large model described in the first aspect of the embodiment of the present application, referring to Figure 3 , Figure 3 A flow chart showing the steps of an ultrasound method is shown, such as Figure 3 As shown, the method includes: Step S101, obtaining ultrasonic information of the current position of the robot arm through multiple sensors, wherein the ultrasonic information includes: RGB image, point cloud data, ultrasonic image, text instruction, and 6-dimensional force estimation data; Step S102, extracting features from the ultrasound information to obtain ultrasound features through a feature extraction module, wherein the ultrasound features include: RGB image features, point cloud data features, ultrasound image features, text instruction tokens, and force estimation features; Step S103, outputting the robot arm state feature and the robot arm translation and rotation feature according to the ultrasonic feature through the multimodal embodied large model; wherein the robot arm state feature indicates whether the robot arm is in an abnormal state at the current moment, and the robot arm translation and rotation feature indicates the translation action and rotation action that the robot arm needs to perform at the current moment; Step S104: decoding the state characteristics of the robot arm and the translation and rotation characteristics of the robot arm through a motion decoder to generate ultrasonic motions to guide the robot at the current moment to move according to the ultrasonic motions.

[0067] In a possible implementation, the acquiring ultrasonic information of the position of the robot arm at the current moment through multiple sensors includes: Acquire the RGB image and the point cloud data using a depth camera, wherein the RGB image represents an image including three-dimensional morphological information of the detection object, and the point cloud data represents scene point cloud information including a spatial relationship between the robot arm and the detection object; An ultrasonic probe is used to collect an ultrasonic image of the position where the mechanical arm is located; the ultrasonic probe is fixed on the mechanical arm; Using a text instruction input device to receive the text instruction input by the user; The 6-dimensional force estimation data is collected by using a force sensor, and the 6-dimensional force estimation data represents the magnitude of the force applied by the ultrasonic probe on the detection object.

[0068] In a possible implementation, extracting features from the ultrasound information to obtain ultrasound features includes: Scaling the RGB image to a resolution of 224x224, extracting a 768-dimensional image feature vector from the RGB image using a first visual feature extraction model, and passing the extracted image feature vector through a fully connected layer to obtain a 4096-dimensional RGB image feature; In the base coordinate system of the robot arm, 1024-dimensional three-dimensional feature information is extracted from the point cloud data using the encoder of the PCT model, and the extracted three-dimensional feature information is converted through a fully connected layer to obtain 4096-dimensional point cloud data features; Scaling the ultrasound image to a resolution of 224x224, extracting a 768-dimensional ultrasound image feature vector from the ultrasound image using a second visual feature extraction model, and passing the extracted ultrasound image feature vector through a fully connected layer to obtain a 4096-dimensional ultrasound image feature; Using a large language model, converting the text instruction into the text instruction token with a feature dimension of 1024; The 6-dimensional force estimation data is converted through a fully connected layer to obtain the 4096-dimensional force estimation features.

[0069] In a possible implementation, the first visual feature extraction model and the second visual feature extraction model are the same visual feature extraction model.

[0070] In a possible implementation, the feature extraction module and the multimodal embodied large model are trained according to the following steps: Acquire training samples, wherein the training samples are sample ultrasonic information of the position of the manipulator obtained by the multiple sensors, and training sample labels; wherein the sample ultrasonic information includes: sample RGB images, sample point cloud data, sample ultrasonic images, sample text instructions, and sample 6-dimensional force estimation data; the training sample labels represent the state of the manipulator at the time of collecting the training samples, and the movement action performed by the manipulator at the next time; Input the training sample into the feature extraction module to extract the sample ultrasound features, wherein the sample ultrasound features include: sample RGB image features, sample point cloud data features, sample ultrasound image features, sample text instruction tokens, and sample force estimation features; Inputting the sample ultrasonic features into the multimodal embodied large model to obtain output results, wherein the output results include: sample mechanical arm state features and sample mechanical arm translation and rotation features; Calculating a loss function value according to the output result and the training sample label; According to the loss function value, updating parameters of the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, the large language model, and the multimodal embodied large model in the feature extraction module; Repeat the above steps until the preset number of training times is reached or the loss function value converges, and then the training ends.

[0071] In a possible implementation manner, the robot arm state feature includes at least one of the following: The first robot arm state feature indicates whether the robot arm has reached the predetermined target position at the current moment; The second robot arm state feature indicates whether the robot arm is detached from the target surface at the current moment; The third robot arm state feature indicates whether there is an abnormality in the relative position of the robot arm and the target to be measured at the current moment.

[0072] In a possible implementation, outputting a robot arm state feature and a robot arm translation and rotation feature according to the ultrasonic feature includes: Acquire historical ultrasonic features at multiple moments; generating an ultrasonic feature sequence according to the historical ultrasonic features at the plurality of moments and the ultrasonic features currently input; Determining the state characteristics of the robotic arm and the translation and rotation characteristics of the robotic arm according to the ultrasonic characteristic sequence; The method further includes: storing the currently input ultrasonic feature as a historical ultrasonic feature.

[0073] The present application also provides an electronic device, referring to Figure 4 , Figure 4 Schematic diagram of an electronic device proposed in an embodiment of the present application. Figure 4 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the ultrasonic method disclosed in the embodiment of the present application.

[0074] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the ultrasonic method disclosed in the embodiment of the present application are implemented.

[0075] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor implements the steps of the ultrasound method disclosed in the embodiment of the present application.

[0076] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0077] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0080] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0081] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0082] The above is a detailed introduction to the ultrasound system, method and product based on a multimodal embodied large model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An ultrasound system based on a multi-modal embodied large model, characterized in that: The system comprises: An information acquisition module is used to obtain ultrasonic information of the current position of the robotic arm through multiple sensors, wherein the ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text instructions, and 6-dimensional force estimation data; A feature extraction module, used to extract features from the ultrasound information to obtain ultrasound features, wherein the ultrasound features include: RGB image features, point cloud data features, ultrasound image features, text instruction tokens, and force estimation features; The multimodal embodied large model is used to output the robot arm state feature and the robot arm translation and rotation feature according to the ultrasonic feature; wherein the robot arm state feature indicates whether the robot arm is in an abnormal state at the current moment, and the robot arm translation and rotation feature indicates the translation action and rotation action that the robot arm needs to perform at the current moment; The motion decoder is used to decode the state characteristics of the robot arm and the translation and rotation characteristics of the robot arm to generate ultrasonic motion to guide the robot at the current moment to move according to the ultrasonic motion.

2. The ultrasound system based on a multimodal embodied large model according to claim 1, characterized in that: The plurality of sensors include: A depth camera, used to acquire the RGB image and the point cloud data, wherein the RGB image represents an image including three-dimensional morphological information of the detection object, and the point cloud data represents scene point cloud information including a spatial relationship between the robot arm and the detection object; An ultrasonic probe, which is fixed on the mechanical arm and is used to collect an ultrasonic image of the position where the mechanical arm is located; A text instruction input device, used for receiving the text instruction input by a user; The force sensor is used to collect the 6-dimensional force estimation data, where the 6-dimensional force estimation data represents the magnitude of the force applied by the ultrasonic probe to the detection object.

3. The ultrasound system based on a multimodal embodied large model according to claim 1, characterized in that: The feature extraction module is used to: Scaling the RGB image to a resolution of 224x224, extracting a 768-dimensional image feature vector from the RGB image using a first visual feature extraction model, and passing the extracted image feature vector through a fully connected layer to obtain a 4096-dimensional RGB image feature; In the base coordinate system of the robot arm, 1024-dimensional three-dimensional feature information is extracted from the point cloud data using the encoder of the PCT model, and the extracted three-dimensional feature information is converted through a fully connected layer to obtain 4096-dimensional point cloud data features; Scaling the ultrasound image to a resolution of 224x224, extracting a 768-dimensional ultrasound image feature vector from the ultrasound image using a second visual feature extraction model, and passing the extracted ultrasound image feature vector through a fully connected layer to obtain a 4096-dimensional ultrasound image feature; Using a large language model, converting the text instruction into the text instruction token with a feature dimension of 1024; The 6-dimensional force estimation data is converted through a fully connected layer to obtain the 4096-dimensional force estimation features.

4. The ultrasound system based on a multimodal embodied large model according to claim 3, characterized in that: The first visual feature extraction model and the second visual feature extraction model are the same visual feature extraction model.

5. The ultrasound system based on a multimodal embodied large model according to claim 3, characterized in that: The feature extraction module and the multimodal embodied large model are trained according to the following steps: Acquire training samples, wherein the training samples are sample ultrasonic information of the position of the manipulator obtained by the multiple sensors, and training sample labels; wherein the sample ultrasonic information includes: sample RGB images, sample point cloud data, sample ultrasonic images, sample text instructions, and sample 6-dimensional force estimation data; the training sample labels represent the state of the manipulator at the time of collecting the training samples, and the movement action performed by the manipulator at the next time; Input the training sample into the feature extraction module to extract the sample ultrasound features, wherein the sample ultrasound features include: sample RGB image features, sample point cloud data features, sample ultrasound image features, sample text instruction tokens, and sample force estimation features; Inputting the sample ultrasonic features into the multimodal embodied large model to obtain output results, wherein the output results include: sample mechanical arm state features and sample mechanical arm translation and rotation features; Calculating a loss function value according to the output result and the training sample label; According to the loss function value, updating parameters of the first visual feature extraction model, the encoder of the PCT model, the second visual feature extraction model, the large language model, and the multimodal embodied large model in the feature extraction module; Repeat the above steps until the preset number of training times is reached or the loss function value converges, and then the training ends.

6. The ultrasound system based on a multimodal embodied large model according to claim 1, characterized in that: The robot arm state characteristics include at least one of the following: The first robot arm state feature indicates whether the robot arm has reached the predetermined target position at the current moment; The second robot arm state feature indicates whether the robot arm is detached from the target surface at the current moment; The third robot arm state feature indicates whether there is an abnormality in the relative position of the robot arm and the target to be measured at the current moment.

7. The ultrasound system based on a multimodal embodied large model according to claim 1, characterized in that: The multimodal embodied large model also includes a storage module, which is used to store historical ultrasonic features; outputting the robot arm state features and the robot arm translation and rotation features according to the ultrasonic features includes: Acquire historical ultrasonic features at multiple moments in the storage module; generating an ultrasonic feature sequence according to the historical ultrasonic features at the plurality of moments and the ultrasonic features currently input; Determining the state characteristics of the robotic arm and the translation and rotation characteristics of the robotic arm according to the ultrasonic characteristic sequence; The storage module is further used to store the currently input ultrasonic feature as a historical ultrasonic feature.

8. An ultrasonic method, characterized in that The method is performed by the ultrasound system based on a multimodal embodied large model according to any one of claims 1 to 7, and comprises: Acquire ultrasonic information of the current position of the robotic arm through multiple sensors, wherein the ultrasonic information includes: RGB images, point cloud data, ultrasonic images, text instructions, and 6-dimensional force estimation data; The ultrasonic information is subjected to feature extraction to obtain ultrasonic features through a feature extraction module, wherein the ultrasonic features include: RGB image features, point cloud data features, ultrasonic image features, text instruction tokens, and force estimation features; Through the multimodal embodied large model, according to the ultrasonic features, the robot arm state features and the robot arm translation and rotation features are output; wherein the robot arm state features indicate whether the robot arm is in an abnormal state at the current moment, and the robot arm translation and rotation features indicate the translation and rotation actions that the robot arm needs to perform at the current moment; The robot state characteristics and the robot translation and rotation characteristics are decoded by a motion decoder to generate ultrasonic motions to guide the robot at the current moment to move according to the ultrasonic motions.

9. An electronic device, characterized in that: The apparatus comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the ultrasound method according to claim 8.

10. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the ultrasonic method of claim 8 is implemented.

Images