B-ultrasonic detection skill data acquisition system and method

Abstract

The application belongs to the field of automatic control, and discloses a B-ultrasonic detection skill data acquisition system and method, which adopts an acquisition system architecture including a remote control device, a mechanical arm, a force sensor and a data processing unit, and is remotely controlled by the remote control device to move the B-ultrasonic probe carried by the mechanical arm, so that the probe does not need to be directly held, and irrelevant targets can be avoided from appearing in the B-ultrasonic image acquisition site, thereby avoiding affecting the subsequent skill learning process. The force sensor arranged at the end effector of the mechanical arm acquires the contact force and contact torque in real time, and the data processing unit stores the contact force and contact torque in association with the mechanical arm movement information and the B-ultrasonic image, thereby solving the technical problems that the direct holding method cannot synchronously acquire the force sensation data and cannot convert the expert skill into structured data. The contact force between the probe and the patient can be accurately controlled, and the safety hidden danger caused by improper force in the handheld operation is avoided. Finally, high-quality structured data sets are provided for skill learning of robot autonomous detection.

Description

Technical Field

[0001] This invention belongs to the field of automatic control and relates to a B-ultrasound detection skill data acquisition system and method. Background Technology

[0002] Ultrasound examination uses ultrasound waves to transmit and receive echoes, generating real-time two-dimensional or three-dimensional images of internal organs in a non-invasive manner, thus aiding doctors in diagnosis. Traditional ultrasound examination methods are primarily manual; however, maintaining close contact between the ultrasound probe and the body requires extremely high physical strength and experience from medical personnel, and is also inefficient, exacerbating the growing contradiction with the increasing demand for such examinations.

[0003] With the development of electromechanical technology, some robot-based ultrasound detection devices have gradually emerged. However, the control modes of these devices mostly remain at the level of model-based programming control. This control method requires precise modeling of the work object in advance, and the adjustment of control parameters is relatively cumbersome. When the characteristics of the work object change, the control mode and parameter adjustment methods need to be adjusted extensively. At the same time, people who need ultrasound examinations have significant differences in height, body shape, and weight distribution, making it difficult to form a unified robot control strategy and parameter adjustment method, which seriously hinders the application of robot technology in automated ultrasound detection.

[0004] To achieve automated ultrasound detection, a unified closed-loop link for perception, decision-making, and control is needed. This link should automatically adjust the ultrasound probe's trajectory based on the specific characteristics of the patient and the ultrasound image, eliminating the need for frequent manual parameter adjustments when the patient's body shape or posture changes. To achieve this, it's first necessary to acquire skill data from medical professionals during ultrasound detection. Learning from this data will then endow the robot with autonomous detection capabilities. However, if medical personnel directly hold the ultrasound probe to collect data, only the probe's trajectory is recorded; the actual contact force / torque between the probe and the patient cannot be simultaneously captured. Furthermore, the expert's techniques cannot be transformed into structured data that the robot can directly learn and reproduce. Additionally, the handheld position exposes the medical personnel's body to the ultrasound image acquisition area, interfering with subsequent image-based scene understanding and skill learning. Moreover, the lack of force feedback and compliant control mechanisms makes it difficult to precisely control the contact force between the probe and the patient during acquisition, posing safety risks. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art, which makes it difficult to effectively collect B-ultrasound detection data, and to provide a cardiac B-ultrasound detection data acquisition system and method.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention provides a B-ultrasound detection skill data acquisition system, including a remote control device, a B-ultrasound machine, a robotic arm, a B-ultrasound probe, a force sensor, and a data processing unit; the remote control device is used to generate control commands for the robotic arm in response to control operations and send them to the robotic arm; the B-ultrasound probe and the force sensor are disposed on the end effector of the robotic arm, and the robotic arm is used to control the movement of the B-ultrasound probe and the force sensor in response to the control commands, and to acquire B-ultrasound detection signals through the B-ultrasound probe and send them to the B-ultrasound machine, and to acquire contact force and contact torque through the force sensor and send them to the data processing unit; the B-ultrasound machine is used to generate B-ultrasound images based on the B-ultrasound detection signals and send them to the data processing unit; the data processing unit is used to acquire the motion information of the robotic arm, and to correlate the motion information of the robotic arm, the B-ultrasound images, the contact force, and the contact torque into B-ultrasound detection skill data.

[0007] Optionally, it also includes a global depth camera and / or a local depth camera; the global depth camera is used to acquire full-scene depth images or full-scene depth images and full-scene color images during the ultrasound detection process and send them to the data processing unit; the local depth camera is set on the end effector of the robotic arm and is used to acquire local scene depth images or local scene depth images and local scene color images during the ultrasound detection process and send them to the data processing unit; the data processing unit is also used to associate the acquired images with ultrasound detection skill data; wherein, the acquired images are full-scene depth images, full-scene color images, local scene depth images, and local scene color images.

[0008] Optionally, the force sensor is a six-dimensional force sensor; before associating the motion information of the robotic arm, ultrasound images, contact force, and contact force rectangle into ultrasound detection data, the data processing unit further includes: processing the contact force and contact torque using the following formula to obtain the true contact force and true contact torque:

[0009]

[0010] in, The contact force and contact torque are collected by the force sensor as a six-dimensional column vector, including the force components in the x-direction, y-direction, z-direction, torque about the x-axis, torque about the y-axis, and torque about the z-axis. The actual contact force and actual contact torque are represented by a six-dimensional column vector. This is the matrix corresponding to the rotational portion of the robot arm's base coordinate system in the force sensor coordinate system; This is the intermediate matrix; The force sensor load is the gravity. and These represent the x-axis, y-axis, and z-axis positions of the center of gravity at the load end of the force sensor in the force sensor coordinate system, respectively. and These are the zero drift of the force component in the x-direction, the zero drift of the force component in the y-direction, the zero drift of the force component in the z-direction, the zero drift of the torque component around the x-axis, the zero drift of the torque component around the y-axis, and the zero drift of the torque component around the z-axis of the force sensor, respectively.

[0011] Optionally, the , , , , , , , , and The following method was used to obtain the force sensors for robotic arm control: the sensors were positioned in three configurations: x-axis pointing towards the negative z-axis of the robotic arm's base coordinate system; y-axis pointing towards the negative z-axis of the robotic arm's base coordinate system; and z-axis pointing towards the negative z-axis of the robotic arm's base coordinate system. Data was collected for each configuration. Force sensor readings, recorded as ;in, The first In the first case The force components in the x, y, and z directions, as well as the torque about the x-axis, y-axis, and z-axis, of the force sensor readings are obtained using the following formula. , , , , , , , , and :

[0012]

[0013]

[0014] Control the robotic arm to move from a preset starting posture to a preset ending posture using joint movements, and obtain... Force sensor readings are combined and optimized using the following formula. , , , , , , , , and :

[0015] in, Represents the magnitude of a vector. This represents a 3×3 matrix where all elements are 0. For the first Force sensor readings, For the first The matrix corresponding to the rotational portion of the robot arm base coordinate system represented by the force sensor coordinate system, based on the force sensor readings.

[0016] Optionally, the acquisition of the robotic arm's motion information includes: calculating the observed Cartesian translational velocity of the robotic arm using the following formula. :

[0017] in, and They are respectively t Time and t The position of the robotic arm tool coordinate system in the base coordinate system at time -1. This is the robotic arm status access cycle.

[0018] based on Calculate the rotation increment matrix of the robotic arm; where, For the increment of the rotation matrix, and They are respectively t Time and t The matrix representing the rotational portion of the end effector coordinate system in the robot arm's base coordinate system at time -1; calculate the system of linear equations. The general solution is obtained by normalization to obtain the unit axis of rotation. ;in, Let u be a 3×3 identity matrix, and u be the independent variable.

[0019] Construction and unit axis of rotation Three-dimensional column vectors that are not parallel ,pass Calculate the column vector obtained after rotation ,pass calculate On the axis of rotation Projection point on .

[0020] The following formula is used to calculate the winding rotation angle :

[0021] And based on the following formula Correct the direction:

[0022] Among them, symbols Represents the cross product of vectors, symbol This is the dot product of vectors.

[0023] The observed Cartesian rotational speed of the robotic arm is calculated using the following formula:

[0024] The observed joint space rotational angular velocity of the robotic arm is calculated using the following formula:

[0025] in, for t The rotational velocities of each axis in the observation space calculated at each moment. and They are respectively t Time and t- At time 1, observe the joint rotation angles of each axis in space.

[0026] The Kalman state transfer and observation function with impulse is constructed based on the following formula: , , ,

[0027] in, for t The velocity of the robotic arm in Cartesian space and joint space at any given moment. , and They are respectively t The Cartesian translational velocity component, the Cartesian rotational velocity component, and the joint space angular velocity component at time t. For correction factors, for t The momentum term at time t. for t The measured value of the velocity in Cartesian space at any given moment.

[0028] Based on the following equations, combining the state transfer equation and the observation equation, and through a recursive approach, the Cartesian space velocity and joint space velocity of the robotic arm after filtering and smoothing are obtained:

[0029] in, for tThe Cartesian space velocity and joint space velocity are obtained at each moment according to the state transfer equation; A is the state transition function transfer matrix; for t Cartesian space velocity and joint space velocity obtained by filtering at time -1; This refers to the robot's state access cycle; and As an intermediate variable; and These are the noise covariance matrices of the state transfer equation and the measurement equation, respectively. To integrate the covariance matrix.

[0030] The robot arm's end effector pose and joint angles are obtained, and the filtered and smoothed Cartesian space velocity and joint space velocity of the robot arm are correlated to obtain the robot arm's motion information.

[0031] Optionally, the generation of control commands for the robotic arm in response to the control operation includes: calculating the increment of the coordinate system of the remote control device's end effector using the following formula. :

[0032] in, This represents the coordinate system of the telescopic device's end effector in the telescopic device's reference coordinate system at the current moment. This represents the coordinate system of the end point of the telecontrol device at the previous moment in the reference coordinate system of the telecontrol device.

[0033] The desired incremental pose of the robotic arm's end effector coordinate system is calculated using the following formula. :

[0034] in, This represents the coordinate system of the current end effector of the robotic arm in the base coordinate system of the robotic arm.

[0035] Will The matrix corresponding to the rotated part is transformed into a 3D column vector. ,Will The matrix corresponding to the translation part is transformed into a 3D column vector. and by the following formula and Limiting the amplitude: ,

[0036] in, For rotation threshold, This is the translation threshold.

[0037] After the limit and As control commands for the robotic arm.

[0038] Optionally, the control command generated in response to the control operation includes: setting the translational speed of the end effector of the robotic arm in the robotic arm's base coordinate system. and rotational speed :

[0039] in, and These represent the translational and rotational velocities of the telescopic device's end-effector coordinate system relative to the telescopic device's reference system, respectively. and This refers to the speed multiplier.

[0040] For robotic arms that include speed control interfaces, and As control commands for the robotic arm.

[0041] For a robotic arm that only has a position control interface, calculate the position increment. and rotation increment And it serves as the control command for the robotic arm.

[0042] Optionally, the data processing unit is further configured to: convert the contact force collected by the force sensor from the force sensor coordinate system to the robot arm base coordinate system to obtain the contact force components in each direction under the robot arm base coordinate system; for each contact force component, when the absolute value of the current contact force component is less than a preset threshold, reset the current contact force component to zero; when the absolute value of the current contact force component is not less than the preset threshold, scale the current contact force component to obtain the scaled contact force component; send the scaled contact force component to the remote control device; the remote control device is further configured to repeatedly execute the following steps at a preset control cycle: determine whether the current cycle count value is an integer multiple of a preset multiple, if so, write the scaled contact force component into the force feedback drive kernel of the remote control device, otherwise write zero into the force feedback drive kernel of the remote control device.

[0043] Optionally, the data processing unit is further configured to: store ultrasound detection skill data using a preset skill data file organization structure; wherein the preset skill data file organization structure includes: a root directory storage area; at least one dataset storage area located within the root directory storage area, each dataset storage area corresponding to a dataset name; a training data storage area and a calibration data storage area located within each dataset storage area; at least one skill recording storage area located within the training data storage area, each skill recording storage area corresponding to a recording index; and a ontology data storage area, an instruction data storage area, and at least one image data storage area located within each skill recording storage area.

[0044] In a second aspect, the present invention provides a method for acquiring ultrasound detection skill data based on the aforementioned ultrasound detection skill data acquisition system, comprising: generating control commands for a robotic arm in response to a control operation via a remote control device and sending them to the robotic arm; controlling the movement of an ultrasound probe and a force sensor via the robotic arm in response to the control commands, acquiring ultrasound detection signals via the ultrasound probe and sending them to an ultrasound machine, and acquiring contact force and contact torque via the force sensor and sending them to a data processing unit; generating an ultrasound image based on the ultrasound detection signals via the ultrasound machine and sending it to the data processing unit; acquiring motion information of the robotic arm via the data processing unit, and correlating the motion information of the robotic arm, the ultrasound image, the contact force, and the contact torque to form ultrasound detection skill data.

[0045] Compared with the prior art, the present invention has the following beneficial effects: This invention relates to an ultrasound detection skills data acquisition system. The system architecture includes a remote control device, a robotic arm, force sensors, and a data processing unit. The robotic arm, carrying the ultrasound probe, is remotely controlled by the remote control device. Therefore, during skills data acquisition, the probe does not need to be held directly in the hand, preventing irrelevant objects from appearing at the ultrasound image acquisition site and avoiding interference with subsequent skills learning. Simultaneously, the force sensor on the robotic arm's end effector collects contact force and torque in real time. The data processing unit associates and stores this data with the robotic arm's motion information and ultrasound images, synchronously recording the probe trajectory, real force information, and the robot's state. This solves the technical problems of not being able to synchronously collect force data and convert expert techniques into structured data when using a direct handheld method. Furthermore, the remote control device provides a foundation for force feedback and compliant control during the acquisition process, enabling precise control of the contact force between the probe and the patient, avoiding safety hazards caused by improper force during handheld operation. This achieves complete, safe, and interference-free acquisition of ultrasound detection skills data, providing a high-quality structured dataset for the robot's autonomous detection skills learning. Attached Figure Description

[0046] Figure 1 This is a structural block diagram of a B-ultrasound detection skill data acquisition system according to an embodiment of the present invention.

[0047] Figure 2 This is a structural block diagram of a B-ultrasound detection skill data acquisition system according to another embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of the first placement of the force sensor according to an embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram illustrating the second placement of the force sensor according to an embodiment of the present invention.

[0050] Figure 5 This is a schematic diagram illustrating the placement of the force sensor in the third case according to an embodiment of the present invention.

[0051] Figure 6 This is a flowchart of the efficient following control method for a robotic arm according to an embodiment of the present invention.

[0052] Figure 7 This is a schematic diagram of the coordinate system of the remote control device according to an embodiment of the present invention.

[0053] Figure 8 This is a schematic diagram of the main control node process of the robotic arm in an embodiment of the present invention.

[0054] Figure 9 This is a schematic diagram of the remote control device driver process according to an embodiment of the present invention.

[0055] Figure 10 This is a schematic diagram of the skill data file organization structure according to an embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, a B-ultrasound detection skill data acquisition system is provided, which realizes skill data acquisition during the process of B-ultrasound detection by an operator controlling a robotic arm to carry a B-ultrasound probe and a force sensor probe set on the end effector through a remote control device with force feedback function.

[0059] Specifically, the ultrasound detection data acquisition system of the present invention includes a remote control device, an ultrasound machine, a robotic arm, an ultrasound probe, a force sensor, and a data processing unit. The remote control device generates control commands for the robotic arm in response to control operations and sends them to the robotic arm. The ultrasound probe and the force sensor are mounted on the end effector of the robotic arm. The robotic arm controls the movement of the ultrasound probe and the force sensor in response to control commands, and collects ultrasound detection signals through the ultrasound probe and sends them to the ultrasound machine, and collects contact force and contact torque through the force sensor and sends them to the data processing unit. The ultrasound machine generates ultrasound images based on the ultrasound detection signals and sends them to the data processing unit. The data processing unit collects the motion information of the robotic arm and associates the motion information of the robotic arm, the ultrasound images, the contact force, and the contact torque to form ultrasound detection data.

[0060] Explaining the principle of the ultrasound detection skill data acquisition system of the present invention, the end effector of the robotic arm is equipped with an ultrasound probe. The ultrasound probe is moved to a preset position by a remote control device. The image quality on the ultrasound machine is observed and the posture of the ultrasound probe is adjusted accordingly. During this process, the contact force and contact torque are measured by a force sensor, and the following data are saved at certain periodic intervals: the motion information of the robotic arm, the ultrasound image, the contact force and contact torque, forming an ultrasound detection skill dataset.

[0061] For example, the robotic arm and data processing unit are connected via a network cable, the ultrasound probe and ultrasound machine are connected via a data cable, the ultrasound machine and data processing unit are connected via a video cable, and the remote control device and data processing unit are connected via a USB cable to ensure stable data transmission. For example, a six-dimensional force sensor can be used, and the remote control device can be an Openhaptics remote control device.

[0062] This invention relates to an ultrasound detection skills data acquisition system. The system architecture includes a remote control device, a robotic arm, force sensors, and a data processing unit. The robotic arm, carrying the ultrasound probe, is remotely controlled by the remote control device. Therefore, during skills data acquisition, the probe does not need to be held directly in the hand, preventing irrelevant objects from appearing at the ultrasound image acquisition site and avoiding interference with subsequent skills learning. Simultaneously, the force sensor on the robotic arm's end effector collects contact force and torque in real time. The data processing unit associates and stores this data with the robotic arm's motion information and ultrasound images, synchronously recording the probe trajectory, real force information, and the robot's state. This solves the technical problems of not being able to synchronously collect force data and convert expert techniques into structured data when using a direct handheld method. Furthermore, the remote control device provides a foundation for force feedback and compliant control during the acquisition process, enabling precise control of the contact force between the probe and the patient, avoiding safety hazards caused by improper force during handheld operation. This achieves complete, safe, and interference-free acquisition of ultrasound detection skills data, providing a high-quality structured dataset for the robot's autonomous detection skills learning.

[0063] In one possible implementation, see Figure 2 The ultrasound detection data acquisition system further includes a global depth camera and / or a local depth camera; the global depth camera is used to acquire full-scene depth images or full-scene depth images and full-scene color images during the ultrasound detection process and send them to the data processing unit; the local depth camera is mounted on the end effector of the robotic arm and is used to acquire local scene depth images or local scene depth images and local scene color images during the ultrasound detection process and send them to the data processing unit; the data processing unit is also used to associate the acquired images with ultrasound detection data; wherein, the acquired images are full-scene depth images, full-scene color images, local scene depth images, and local scene color images.

[0064] Interpretively, by adding a global depth camera and / or a local depth camera to the ultrasound detection skill data acquisition system, and having the data processing unit associate and store the acquired depth and color images of the entire scene and / or local scene with the original ultrasound images, robotic arm motion information, contact force / torque, and other data, a complete skill representation is formed through spatiotemporal correlation between multi-view, multi-level visual information and robotic arm state and force information. This significantly improves the information dimension and richness of the acquired data, laying a solid data foundation for subsequent robot autonomous ultrasound detection skill learning based on the fusion of vision and force.

[0065] Specifically, the global depth camera can record the relative positions and overall environmental information between the operator, the robotic arm, and the patient, providing a global context for the robot to learn and understand the spatial layout and object relationships in the detection scene. The local depth camera moves with the end effector of the robotic arm and can acquire the fine geometric shape and texture information of the area where the ultrasound probe contacts the patient's body surface at close range, providing local perception data for learning the mapping relationship between ultrasound probe posture adjustment and body surface deformation.

[0066] In one possible implementation, before the data processing unit associates the motion information of the robotic arm, ultrasound images, contact force, and contact force rectangle into ultrasound detection skill data, it further includes: processing the contact force and contact torque using the following formula to obtain the true contact force and true contact torque:

[0067]

[0068] in, The contact force and contact torque are collected by the force sensor as a six-dimensional column vector, including the force components in the x-direction, y-direction, z-direction, torque about the x-axis, torque about the y-axis, and torque about the z-axis. The actual contact force and actual contact torque are represented by a six-dimensional column vector. This is the matrix corresponding to the rotational portion of the robot arm's base coordinate system in the force sensor coordinate system; This is the intermediate matrix; The force sensor load is the gravity. and These represent the x-axis, y-axis, and z-axis positions of the center of gravity at the load end of the force sensor in the force sensor coordinate system, respectively. and These are the zero drift of the force component in the x-direction, the zero drift of the force component in the y-direction, the zero drift of the force component in the z-direction, the zero drift of the torque component around the x-axis, the zero drift of the torque component around the y-axis, and the zero drift of the torque component around the z-axis of the force sensor, respectively.

[0069] Optionally, the , , , , , , , , and The following method was used to obtain the force sensors for robotic arm control: the sensors were positioned in three configurations: x-axis pointing towards the negative z-axis of the robotic arm's base coordinate system; y-axis pointing towards the negative z-axis of the robotic arm's base coordinate system; and z-axis pointing towards the negative z-axis of the robotic arm's base coordinate system. Data was collected for each configuration. Force sensor readings, recorded as ;in, The first In the first case The force components in the x-direction, y-direction, and z-direction of the force sensor readings, as well as the torque about the x-axis, torque about the y-axis, and torque about the z-axis.

[0070] The following formula is obtained , , , , , , , , and :

[0071]

[0072]

[0073] Control the robotic arm to move from a preset starting posture to a preset ending posture using joint movements, and obtain... Force sensor readings are combined and optimized using the following formula. , , , , , , , , and :

[0074] in, Represents the magnitude of a vector. This represents a 3×3 matrix where all elements are 0. For the first Force sensor readings, For the first The matrix corresponding to the rotational portion of the robot arm base coordinate system represented by the force sensor coordinate system, based on the force sensor readings.

[0075] Interpretive data from force sensors includes not only the actual contact force / torque in all directions but also components generated by the load and the sensor's own weight in a free state. In practical applications, irrelevant components need to be removed to obtain the true contact force / torque. However, force sensors are highly nonlinear; the magnitudes of irrelevant components vary depending on the sensor's orientation, making calibration difficult. Surveys of various force sensors reveal that, in a free state, the contact force / torque processed by the robot system deviates significantly from zero, exhibiting considerable differences across different regions. To address this, this implementation proposes an efficient calibration method. Experimental verification shows that, in a free state, the force / torque readings obtained after removing irrelevant components are close to zero, and exhibit high consistency across regions. For example, after removing external forces, noise can be filtered out using a low-pass filter.

[0076] Specifically, the purpose of force sensor calibration is to calibrate the load gravity, the load center of gravity position, and the zero drift of the 6-dimensional force sensor in all directions. This implementation proposes a high-precision force sensor calibration strategy that combines special postures with optimization algorithms. First, calibration parameters are initially obtained by collecting force / torque readings in three special postures. Second, force / torque readings are collected in multiple sets of continuous postures, and an optimization function is constructed based on the calibration parameters obtained in the special postures. High-precision force sensor parameter values ​​are obtained through optimization calculations of this function.

[0077] Step 1: Record the gravity at the load end of the force sensor. The position of the load's center of gravity in the force sensor coordinate system is denoted as... Zero drift notation of a six-dimensional force sensor , respectively, are the zero drift of the force component in the x direction, the zero drift of the force component in the y direction, the zero drift of the force component in the z direction, the zero drift of the torque component about the x-axis, the zero drift of the torque component about the y-axis, and the zero drift of the torque component about the z-axis.

[0078] Step 2, see Figure 3 , Figure 4 and Figure 5 In a free state where the load at the end effector of the robotic arm is not subjected to external forces, the robotic arm control force sensor is positioned according to three special cases: the x-axis is oriented towards the negative z-axis of the base coordinate system, the y-axis is oriented towards the negative z-axis of the base coordinate system, and the z-axis is oriented towards the negative z-axis of the base coordinate system. Data is collected for each of these three cases. Force sensor readings. The readings for the three cases are recorded as follows: , and

[0079] in, , This indicates that the x-axis of the force sensor is along the negative z-axis of the base coordinate system; This indicates that the y-axis of the force sensor is along the negative z-axis of the base coordinate system; This indicates that the z-axis of the force sensor is along the negative z-axis of the base coordinate system.

[0080] Step 3: Calculate the sensor zero drift value based on the following formula:

[0081] Step 4: Calculate the load gravity based on the following formula:

[0082] Step 5: Calculate the load centroid based on the following formula:

[0083] Thus, the load gravity, the position of the load center of gravity, and the zero drift values ​​of the force sensor in each direction have been initially obtained.

[0084] Step Six: Select two significantly different robotic arm postures. Control the robotic arm to slowly move from the initial posture to the final posture using joint motion. Acquire force sensor readings in real time and store the force sensor values ​​and the rotation matrix of the robotic arm's base coordinate system in the force sensor's coordinate system (generally, during installation, the directions of the robotic arm's tool coordinate system are aligned with the directions of the force sensor). Assume a total of... Group, Group i Force sensor readings are recorded as The rotational portion of the corresponding robotic arm base coordinate system represented in the force sensor coordinate system is denoted as... Construct the optimization objective function as shown in the following equation:

[0085] For example, the objective function can be quickly solved using optimization tools such as OpenCV or Ceres Solver, thus obtaining high-precision parameters of the force sensor. Based on the calibration, assume the force sensor reading is... Let R be the matrix corresponding to the rotational portion of the robotic arm's base coordinate system in the force sensor coordinate system. Then, based on the following formula, the representation of the actual contact force / torque in the force sensor coordinate system can be obtained:

[0086] In one possible implementation, the acquisition of motion information from the robotic arm includes: The Cartesian translational velocity of the robotic arm, as observed, is calculated using the following formula. :

[0087] in, and They are respectively t Time and t The position of the robotic arm tool coordinate system in the base coordinate system at time -1. This is the robotic arm status access cycle.

[0088] based on Calculate the rotation increment matrix of the robotic arm; where, For the increment of the rotation matrix, and They are respectively t Time and t The matrix representing the rotational portion of the end effector coordinate system in the robot arm's base coordinate system at time -1; calculate the system of linear equations. The general solution is obtained by normalization to obtain the unit axis of rotation. ;in, Let u be a 3×3 identity matrix, and u be the independent variable.

[0089] Construction and unit axis of rotation Three-dimensional column vectors that are not parallel ,pass Calculate the column vector obtained after rotation ,pass calculate On the axis of rotation Projection point on .

[0090] The following formula is used to calculate the winding rotation angle :

[0091] And based on the following formula Correct the direction:

[0092] Among them, symbols Represents the cross product of vectors, symbol This is the dot product of vectors.

[0093] The observed Cartesian rotational speed of the robotic arm is calculated using the following formula:

[0094] The observed joint space rotational angular velocity of the robotic arm is calculated using the following formula:

[0095] in, for tThe rotational velocities of each axis in the observation space calculated at each moment. and They are respectively t Time and t- At time 1, observe the joint rotation angles of each axis in space.

[0096] The Kalman state transfer and observation function with impulse is constructed based on the following formula: , , ,

[0097] in, for t The velocity of the robotic arm in Cartesian space and joint space at any given moment. , and They are respectively t The Cartesian translational velocity component, the Cartesian rotational velocity component, and the joint space angular velocity component at time t. For correction factors, for t The momentum term at time t. for t The measured value of the velocity in Cartesian space at any given moment.

[0098] Based on the following equations, combining the state transfer equation and the observation equation, and through a recursive approach, the Cartesian space velocity and joint space velocity of the robotic arm after filtering and smoothing are obtained:

[0099] in, t Take a positive integer; for t The Cartesian space velocity and joint space velocity are obtained at each moment according to the state transfer equation; A is the state transition function transfer matrix; for t Cartesian space velocity and joint space velocity obtained by filtering at time -1; This refers to the robot's state access cycle; and As an intermediate variable; and These are the noise covariance matrices of the state transfer equation and the measurement equation, respectively. To integrate the covariance matrix.

[0100] The robot arm's end effector pose and joint angles are obtained, and the filtered and smoothed Cartesian space velocity and joint space velocity of the robot arm are correlated to obtain the robot arm's motion information.

[0101] For interactive operations and situations involving dynamically changing objects, robotic arms need the ability to correct their planned paths in real time. Clearly, traditional linear or circular motion is insufficient. Currently, various robotic arms, including JAKA and UR, provide servo motion interfaces that can accept motion increment commands in Cartesian and joint space in real time, facilitating real-time planning of robotic arm motion. The velocity information in Cartesian and joint space is crucial for control smoothness and is widely used in admittance / impedance control, force / position hybrid control, and other processes. However, most robotic arms only provide end-effector Cartesian and joint space position information, lacking velocity information. Therefore, this embodiment presents a method for calculating and filtering the general Cartesian and joint space velocities of a robotic arm.

[0102] Specifically, in this embodiment, the steps for calculating the Cartesian space velocity and joint space velocity of the robotic arm based on the Kalman filter method with impulse are as follows: Step 1: Calculate the Cartesian translation velocity obtained from observation:

[0103] Step 2: Calculate the observed rotational velocity in Cartesian space. Based on Calculate the rotation increment matrix of the robotic arm, where, For the increment of the rotation matrix, and Let be the matrices representing the rotational portions of the tool coordinate system in the robot arm base coordinate system at times t and t-1, respectively; calculate the general solution of the following system of linear equations and normalize it to obtain the unit axis of rotation. The calculated unit axis of rotation is denoted as Selecting three-dimensional column vectors (make sure With unit rotation axis (They are not parallel relationships), and through Calculate the column vector obtained after rotation At the same time, through calculate On the axis of rotation Projection point on Furthermore, the following formula is used to calculate the value about the unit axis. rotation angle :

[0104] Based on this, and using the following formula... Correct the direction:

[0105] Thus, we obtained the agreement with Equivalent Lie algebra form Based on this, the observed rotational velocity in Cartesian space is calculated using the following formula:

[0106] Step 3: Calculate the observed joint space rotational angular velocity:

[0107] Step 4: Construct the Kalman state transfer and observation function with impulse based on the following formula: , ,

[0108] in, for t The robot arm's velocity in Cartesian space and joint space at any given time is a 12-dimensional column vector (taking 6 rotational joints as an example). This is a correction factor (usually taken as 0.5). for t The momentum term at time.

[0109] Step 5: Based on the following equations, combining the state transfer equation and the observation equation, and through recursion, obtain the filtered and smoothed Cartesian space and joint space velocities:

[0110] Where A is the state transition function transfer matrix, which is a 12×12 identity matrix; , These are the noise covariance matrices for the state transfer equation and the measurement equation, respectively. Both are 12×12 diagonal matrices, and the diagonal elements are set to values ​​related to the noise distribution. In this embodiment, they are set to 0.0001 and 0.001, respectively. To integrate the covariance matrix, the initial value is a 12×12 identity matrix.

[0111] In the ultrasound detection data acquisition system of this invention, the control of the robotic arm is based on a remote control device. Therefore, for intuitive reasons, directly mapping the end effector position of the remote control device to the end effector of the robotic arm is the most straightforward approach. However, there are significant differences between the configurations of the remote control device and the robotic arm, making a direct one-to-one correspondence difficult.

[0112] Regarding this issue, see Figure 6 This invention designs and implements two efficient tracking control methods for robotic arms: Method 1: By establishing a correspondence between the end-effector pose increment of the telecontrol device and the pose increment of the robotic arm tool coordinate system, the motion synchronization problem between heterogeneous devices can be solved. In practice, the end-effector pose increment is directly assigned to the robotic arm tool coordinate system pose increment, supplemented by a threshold limit, thus enabling efficient teleoperation of the robotic arm. This method is a typical position control mode.

[0113] Method Two: After establishing the correspondence between the coordinate systems of the remote control device and the robotic arm, for the remote control device, the translational and rotational velocities of the end effector coordinate system relative to the device reference system are calculated based on the aforementioned "robotic arm Cartesian space velocity solution method." Signal filtering and communication delay compensation are then performed based on the aforementioned "Kalman filtering method with impulse" to obtain the translational and rotational velocities of the end effector coordinate system relative to the device reference system. Based on this, threshold limits are applied to the translational and rotational velocities using velocity and acceleration thresholds. Furthermore, the velocities are transformed based on a multiplier of force and torque thresholds to improve safety and maneuverability, and this transformation is used as the desired safe velocity. The desired velocity can then be directly sent; this is a typical speed control mode. For robotic arms that only have a position control interface, the average velocity is calculated based on the desired velocity and the actual velocity. Combining this with the control cycle yields the pose increment, which can then be sent in real time.

[0114] Both control methods require establishing a correspondence between the movement of the telescopic device and the movement of the robotic arm. Extensive testing of the Open Happics telescopic device in ROS2 revealed high accuracy in the rotation of its six joints, closely matching actual conditions. However, speed and end-effector pose accuracy were poor, and inconsistencies between the state feedback and the actual situation occurred. To address this issue, this invention, after measuring the parameters of each link in the telescopic device, created a corresponding reference system for each joint and established a highly recognizable tool coordinate system at the device's tip, facilitating the operator's quick correspondence between the telescopic device's end-effector coordinate system and the robotic arm's end-effector tool coordinate system. Based on this, the representation of the telescopic device's end-effector coordinate system in the telescopic device's reference reference system was derived and calculated.

[0115] For example, when constructing a highly recognizable end-effector reference system at the end of the remote control device, the origin of the end-effector reference system established in this invention is located at the tip of the pen, the positive z-axis points from the center of the pen to the tip, the negative y-axis points towards the pen button, and the x-axis is determined by the right-hand rule. Before actual remote control, the remote control device needs to be properly positioned, generally ensuring that the x, y, and z axes of the device are basically consistent with the three axes of the robotic arm's base coordinate system. Furthermore, the pen needs to be moved to ensure that the orientation of the end-effector reference system constructed on it in the global reference system of the remote control device is basically consistent with the orientation of the robotic arm's end-effector coordinate system in the base coordinate system. After the aforementioned conditions are met, the pen tip up button on the remote control device is pressed. The remote control process begins after the program detects an odd number of button presses; the remote control process ends after the program detects an even number of button presses.

[0116] See Figure 7 The coordinate system of the created telescopic device is shown, with the following meanings: {Base} of Openhaptics represents the telescopic device reference frame, ix, iy, iz ( ) represent the positive x-axis, positive y-axis, and positive z-axis of the coordinate system constructed at the center of the i-th actual rotational joint of the remote control device, respectively. Figure 7 The states of shafts 1, 2, 3, and 4 given in the document are consistent with the state where the actual rotation angle of the corresponding joints of the equipment is 0. Figure 7 The zero-point position of the central axis 5 reference system is obtained only when the joint rotation angles of axes 1, 2, 3, and 4 are 0°, the pen at the end of the device is parallel to the xoy plane of the remote control device reference system, and the pen tip points away from the remote control device. Measurements show that at the zero-point reference position of axis 5, the joint rotation angle of axis 5 is -2.3 radians. For ease of identification, the coordinate system constructed at axis 6 has its origin at the pen tip position, the z-axis pointing from the axis of the remote control device's pen tip, the button on the pen at the end of the device pointing in the negative y-axis direction, and the x-axis direction determined by the right-hand rule. Its positive z-axis rotation direction is opposite to the actual positive rotation direction of the joint. The directions of each axis of the created remote control device reference system are consistent with the directions of each axis in the zero-point state of the reference system constructed at axis 1, and the origin is located in the negative z-axis direction of axis 1, at a distance of h (h can be any value). l1, l2, and l3 are the measured lengths of the three connecting rods.

[0117] For solving the kinematics of remote-controlled devices, since the inverse kinematics part is not used, this invention focuses on the forward kinematics part of the kinematics solution. The steps are as follows: Step 1: Record the angles of each joint. , , , , , .

[0118] Step 2: The coordinate system bound to joint 1 is represented in the remote control device reference frame as follows: And it is determined by the following formula: .

[0119] Step 3: Calculate the representation of the coordinate system bound to joint 2 in the coordinate system bound to joint 1, denoted as . And it is determined by the following formula: .

[0120] Step 4: Calculate the representation of the coordinate system bound to joint 3 in the coordinate system bound to joint 2, denoted as . And it is determined by the following formula: .

[0121] Step 5: Calculate the representation of the coordinate system bound to joint 4 in the coordinate system bound to joint 3, denoted as . And it is determined by the following formula: .

[0122] Step 6: Calculate the representation of the coordinate system bound to joint 5 in the coordinate system bound to joint 4, denoted as... And it is determined by the following formula: .

[0123] Step 7: Calculate the representation of the coordinate system bound to joint 6 in the coordinate system bound to joint 5, denoted as . And it is determined by the following formula: .

[0124] Step 8: Based on the following formula, obtain the representation of the remote control device's end-effector coordinate system in the device reference frame, denoted as . And it is determined by the following formula: Thus, the kinematic solution of the remote control device is completed.

[0125] In one possible implementation, the control command for generating the robotic arm in response to a control operation includes: calculating the increment of the end-effector coordinate system of the remote control device using the following formula. :

[0126] in, This represents the coordinate system of the telescopic device's end effector in the telescopic device's reference coordinate system at the current moment. This represents the coordinate system of the end point of the telecontrol device at the previous moment in the reference coordinate system of the telecontrol device.

[0127] The desired incremental pose of the robotic arm's end effector coordinate system is calculated using the following formula. :

[0128] in, This represents the coordinate system of the current end effector of the robotic arm in the base coordinate system of the robotic arm.

[0129] Will The matrix corresponding to the rotated part is transformed into a 3D column vector. ,Will The matrix corresponding to the translation part is transformed into a 3D column vector. and by the following formula and Limiting the amplitude: ,

[0130] in, For rotation threshold, This is the translation threshold.

[0131] After the limit and As control commands for the robotic arm.

[0132] Explanatory, this embodiment employs master-slave following based on end-effector pose increment, with the specific steps as follows: Step 1: Access the rotation angles of each joint of the remote control device at the current moment, and calculate the representation of the end-effector coordinate system of the remote control device in the reference coordinate system of the remote control device based on the above information, denoted as . .

[0133] Step 2: Record the representation of the telescopic device's end-effector coordinate system in the telescopic device's reference coordinate system at the previous moment as follows: The increment of the coordinate system at the end of the remote control device is calculated using the following formula: .

[0134] Step 3: Calculate the desired incremental pose of the robotic arm tool coordinate system using the following formula: ,in, This represents the incremental pose of the robotic arm's end effector coordinate system. This represents the coordinate system of the current robotic arm end effector in the base coordinate system.

[0135] Step 4: Using tools such as the Eigen library, the incremental matrix is... The matrix corresponding to the rotated part is transformed into a 3D column vector. (Li algebra or rpy angle), the translation part of the increment matrix is ​​denoted as a 3D column vector. .

[0136] Step 5: Set the rotation threshold to The translation threshold is set to The following ensures that the magnitudes of the rotation and translation vectors obtained in step four are less than a threshold: ,

[0137] Based on this, the rotation increment and translation increment can be directly sent to the robot controller.

[0138] In one possible implementation, generating control commands for the robotic arm in response to a control operation includes: setting the translational speed of the robotic arm's end effector in the robotic arm's base coordinate system. and rotational speed :

[0139] in, and These represent the translational and rotational velocities of the telescopic device's end-effector coordinate system relative to the telescopic device's reference system, respectively. and This refers to the speed multiplier.

[0140] For robotic arms that include speed control interfaces, and As control commands for the robotic arm; for robotic arms that only have a position control interface, calculate the position increment. and rotation increment And it serves as the control command for the robotic arm.

[0141] Explanatory purposes, this implementation adopts a master-slave following method based on end-effector speed control, and the specific steps are as follows: Step 1: For the remote-controlled device, based on the aforementioned method for solving the Cartesian space velocity of the robotic arm, the translational and rotational velocity vectors of the end-effector coordinate system relative to the device's reference system can be calculated. A Kalman filtering process is then performed according to the relevant content, and a threshold is applied based on the aforementioned amplitude limiting processing. Let the translational velocity be... The rotational speed is Based on this, the translational and rotational velocities of the robotic arm's end effector in the base coordinate system are set according to the following formula: ; in, , The rules for the speed multiplier are set as follows: ,

[0142] in, , These represent the magnitudes of the actual contact force vector and torque vector, respectively. Their function is to strictly limit the movement of the robotic arm end when contact occurs, ensuring safety during data acquisition.

[0143] Step 2: For robotic arms that include a speed control interface, issue the command directly. , That's all; for cases that only include a position control interface: 1. Position increment: issue 2. Rotation Increment: The rotation increment is obtained through... For cases where the rotation part of the control interface is in Lie algebra form, the calculation is performed by directly sending the data. For cases involving rpy angles, Lie algebras are converted to rpy angles using matrix libraries such as Eigen and then distributed.

[0144] In one possible implementation, the data processing unit is further configured to: convert the contact force collected by the force sensor from the force sensor coordinate system to the robot arm base coordinate system to obtain the contact force components in each direction under the robot arm base coordinate system; for each contact force component, when the absolute value of the current contact force component is less than a preset threshold, reset the current contact force component to zero; when the absolute value of the current contact force component is not less than the preset threshold, scale the current contact force component to obtain the scaled contact force component; send the scaled contact force component to the remote control device; the remote control device is further configured to repeatedly execute the following steps at a preset control cycle: determine whether the current cycle count value is an integer multiple of a preset multiple, if so, write the scaled contact force component into the force feedback drive kernel of the remote control device, otherwise write zero into the force feedback drive kernel of the remote control device.

[0145] Explanatoryly, in the actual remote operation process, the force and torque measured by the force sensor in this embodiment are scaled and a transmission mode is designed to achieve the vibration effect at the remote operation device end. By modulating the vibration frequency proportional to the contact force / torque, the data collector is alerted, thereby protecting the patient's personal safety.

[0146] See Figure 8 This illustrates the workflow of the robotic arm's master control node. A key operation is calculating the actual contact forces in three directions and transforming them into the robotic arm's base coordinate system. The transformation uses... That's all ( This represents the force sensor coordinate system in the robot arm's base coordinate system. (This refers to the actual contact force after removing external forces). The contact forces f1, f2, and f3 in the three directions are reset to 0 when they are below a threshold to avoid fluctuations. When they are above the threshold, they are scaled to a range acceptable to the remote control device or preferred by the operator. Based on this, the robotic arm master node sends the scaled contact forces through messages in ROS2.

[0147] See Figure 9This demonstrates the flow of the remote control device's driver. The key operation is the `hdSetDoublev` function, which sends force parameters in three directions to the device driver kernel to achieve a damping effect. In the original program, the externally transmitted contact force was directly set in the `hdSetDoublev` function, leading to some abnormal situations. In this implementation, the force set in the interface function is the actual externally transmitted contact force parameter when the count value is a multiple of 30; otherwise, it is set to 0. This produces a vibration-like effect and effectively prevents significant reverse movement of the device end in abnormal situations.

[0148] In one possible implementation, the data processing unit is further configured to: store ultrasound detection skill data using a preset skill data file organization structure. The preset skill data file organization structure includes: a root directory storage area; at least one dataset storage area within the root directory storage area, each dataset storage area corresponding to a dataset name; a training data storage area and a calibration data storage area within each dataset storage area; at least one skill recording storage area within the training data storage area, each skill recording storage area corresponding to a recording index; and a ontology data storage area, an instruction data storage area, and at least one image data storage area within each skill recording storage area.

[0149] For example, the skill data collected in this embodiment includes the following: 1. Global and local RGBD depth images: These data enable scene understanding and lay a solid foundation for robot motion inference. 2. Ultrasound images: These allow for a closed-loop mapping between ultrasound images and robot motion during the probe-to-human contact phase, facilitating high-quality ultrasound image acquisition. 3. Contact force / torque between the ultrasound probe and the patient: High-quality ultrasound images require high contact force / torque between the probe and the patient. Collecting contact state data serves as a basis for high-efficiency control and prevents excessive contact force from harming the patient's safety. 4. Robotic arm body data: This mainly includes the robotic arm end-effector pose, joint angles, Cartesian space velocity, and joint space velocity.

[0150] For example, see Figure 10 The skill data file organization structure in this embodiment is shown.

[0151] The `dataset` folder is automatically created in the program's root directory. This folder contains multiple subfolders, each representing a different dataset, allowing for expansion. Each dataset subfolder contains two subfolders: `train` and `calib`. The `train` subfolder contains multiple subfolders named `path` (each containing a recorded complete skill session), with `×` representing the index of the number of skill recordings. The `path` subfolder contains five subfolders: `high_freq_data`, `robot_command`, `cam_{id_l}`, `cam_{id_g}`, and `cam_{id_b}`. The `high_freq_data` folder contains the `high_freq_data.npy` file, which stores all collected robot body data and actual contact force / torque data, including the robot arm end-effector pose (translation and RPI angles), joint rotation angles, TCP Cartesian space velocity, joint space velocity, six-dimensional actual contact force / torque, and timestamps. The `robot_command` folder contains all control commands sent to the robotic arm controller. Each command includes a command type (0 / 1: 0 represents incremental end-effector pose control, 1 represents incremental end-effector velocity control), 6D incremental control parameters, and a timestamp. `{id_l}`, `{id_g}`, and `{id_b}` represent the local depth camera ID, global depth camera ID, and ultrasound machine ID, respectively. The `cam_{id_l}` folder contains two subfolders: `color` and `depth`, which store aligned color and depth images captured by the local camera, respectively (all images in the `color` and `depth` folders are named with the acquisition timestamp and have a `.png` extension). The `cam_{id_g}` folder contains data acquired by the global depth camera; its internal folder contents and image arrangement are consistent with the `cam_{id_l}` folder. The `cam_{id_b}` folder contains all acquired ultrasound images, named with the acquisition timestamp and have a `.png` extension. The calib folder contains the key_para.yaml file, which stores the local camera ID, global camera ID, ultrasound machine ID, local camera intrinsic parameter matrix and hand-eye calibration matrix, global camera intrinsic parameter matrix and global pose transformation matrix, image acquisition frequency, high-frequency data acquisition frequency, and parameters required for robotic arm control.

[0152] The following are method embodiments of the present invention, which can be implemented based on the device embodiments of the present invention. For details not disclosed in the method embodiments, please refer to the device embodiments of the present invention.

[0153] In another embodiment of the present invention, a method for acquiring ultrasound detection skills data is provided, which can be used to implement the ultrasound detection skills data acquisition system described above. Specifically, the method includes the following steps: generating control commands for a robotic arm in response to a control operation via a remote control device and sending them to the robotic arm; controlling the movement of the ultrasound probe and force sensor via the robotic arm in response to the control commands, acquiring ultrasound detection signals via the ultrasound probe and sending them to the ultrasound machine, and acquiring contact force and contact torque via the force sensor and sending them to a data processing unit; generating ultrasound images via the ultrasound machine based on the ultrasound detection signals and sending them to the data processing unit; acquiring the motion information of the robotic arm via the data processing unit, and correlating the motion information of the robotic arm, the ultrasound images, the contact force, and the contact torque to form ultrasound detection skills data.

[0154] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A B-ultrasound detection data acquisition system, characterized in that, It includes remote control equipment, ultrasound machine, robotic arm, ultrasound probe, force sensor and data processing unit; The remote control device is used to generate control commands for the robotic arm in response to control operations and send them to the robotic arm; The ultrasound probe and force sensor are mounted on the end effector of the robotic arm. The robotic arm is used to control the movement of the ultrasound probe and force sensor in response to control commands, and to collect ultrasound detection signals through the ultrasound probe and send them to the ultrasound machine, and to collect contact force and contact torque through the force sensor and send them to the data processing unit. The ultrasound machine is used to generate ultrasound images based on ultrasound detection signals and send them to the data processing unit. The data processing unit is used to collect motion information of the robotic arm, as well as correlate the motion information of the robotic arm, ultrasound images, contact force, and contact force rectangles into ultrasound detection data.

2. The ultrasound detection data acquisition system according to claim 1, characterized in that, It also includes global depth cameras and / or local depth cameras; The global depth camera is used to acquire full-scene depth images or full-scene depth images and full-scene color images during the ultrasound detection process and send them to the data processing unit; A local depth camera is mounted on the end effector of the robotic arm to acquire local scene depth images or local scene depth images and local scene color images during the ultrasound detection process and send them to the data processing unit. The data processing unit is also used to associate the acquired images with ultrasound detection data; the acquired images include full-scene depth images, full-scene color images, partial-scene depth images, and partial-scene color images.

3. The ultrasound detection data acquisition system according to claim 1, characterized in that, The force sensor is a six-dimensional force sensor; before the data processing unit associates the motion information of the robotic arm, ultrasound images, contact force, and contact force rectangles with ultrasound detection data, it also includes: The true contact force and true contact torque are obtained by processing the contact force and contact torque using the following formula: in, The contact force and contact torque are collected by the force sensor as a six-dimensional column vector, including the force components in the x-direction, y-direction, z-direction, torque about the x-axis, torque about the y-axis, and torque about the z-axis. The actual contact force and actual contact torque are represented by a six-dimensional column vector. This is the matrix corresponding to the rotational portion of the robot arm's base coordinate system in the force sensor coordinate system; This is the intermediate matrix; The force sensor load is the gravity. and These represent the x-axis, y-axis, and z-axis positions of the center of gravity at the load end of the force sensor in the force sensor coordinate system, respectively. and These are the zero drift of the force component in the x-direction, the zero drift of the force component in the y-direction, the zero drift of the force component in the z-direction, the zero drift of the torque component around the x-axis, the zero drift of the torque component around the y-axis, and the zero drift of the torque component around the z-axis of the force sensor, respectively.

4. The ultrasound detection data acquisition system according to claim 3, characterized in that, The , , , , , , , , and It is obtained in the following way: Based on the robotic arm control force sensor, it was placed in three cases: x-axis pointing towards the negative z-axis of the robotic arm base coordinate system, y-axis pointing towards the negative z-axis of the robotic arm base coordinate system, and z-axis pointing towards the negative z-axis of the robotic arm base coordinate system, and data was collected in each case. Force sensor readings, recorded as ;in, The first In the first case The force components in the x-direction, y-direction, and z-direction of the force sensor readings, as well as the torque about the x-axis, torque about the y-axis, and torque about the z-axis; The following formula is obtained , , , , , , , , and : Control the robotic arm to move from a preset starting posture to a preset ending posture using joint movements, and obtain... Force sensor readings, and optimized using the following formula , , , , , , , , and : in, Represents the magnitude of a vector. This represents a 3×3 matrix where all elements are 0. For the first Force sensor readings, For the first The matrix corresponding to the rotational portion of the robot arm base coordinate system represented by the force sensor coordinate system, based on the force sensor readings.

5. The ultrasound detection data acquisition system according to claim 1, characterized in that, The motion information of the robotic arm being collected includes: The Cartesian translational velocity of the robotic arm, as observed, is calculated using the following formula. : in, and They are respectively t Time and t The position of the robotic arm tool coordinate system in the base coordinate system at time -1. This is the robotic arm status access cycle; based on Calculate the rotation increment matrix of the robotic arm; where, For the increment of the rotation matrix, and They are respectively t Time and t The matrix representing the rotational portion of the end effector coordinate system in the robot arm's base coordinate system at time -1; calculate the system of linear equations. The general solution is obtained by normalization to obtain the unit axis of rotation. ;in, It is a 3×3 identity matrix, and u is the independent variable; Construction and unit axis of rotation Three-dimensional column vectors that are not parallel ,pass Calculate the column vector obtained after rotation ,pass calculate On the axis of rotation Projection point on ; The following formula is used to calculate the winding rotation angle : And based on the following formula Correct the direction: Among them, symbols Represents the cross product of vectors, symbol Dot product of vectors; The observed Cartesian rotational speed of the robotic arm is calculated using the following formula: The observed joint space rotational angular velocity of the robotic arm is calculated using the following formula: in, for t The rotational velocities of each axis in the observation space calculated at each moment and They are respectively t Time and t- Observe the joint rotation angles of each axis in space at time 1; The Kalman state transfer and observation function with impulse is constructed based on the following formula: ， ， ， in, for t The velocity of the robotic arm in Cartesian space and joint space at any given moment. , and They are respectively t The Cartesian translational velocity component, the Cartesian rotational velocity component, and the joint space angular velocity component at time t. For correction factors, for t The momentum term at time t. for t The measured value of the velocity in Cartesian space at any given moment; Based on the following equations, combining the state transfer equation and the observation equation, and through a recursive approach, the Cartesian space velocity and joint space velocity of the robotic arm after filtering and smoothing are obtained: in, for t The Cartesian space velocity and joint space velocity are obtained at each moment according to the state transfer equation; A is the state transition function transfer matrix. for t Cartesian space velocity and joint space velocity obtained by filtering at time -1; This refers to the robot's state access cycle; and As an intermediate variable; and These are the noise covariance matrices of the state transfer equation and the measurement equation, respectively. To integrate the covariance matrix; The robot arm's end effector pose and joint angles are obtained, and the filtered and smoothed Cartesian space velocity and joint space velocity of the robot arm are correlated to obtain the robot arm's motion information.

6. The ultrasound detection data acquisition system according to claim 1, characterized in that, The control commands for generating the robotic arm in response to the control operation include: The increment of the coordinate system at the end of the remote control device is calculated using the following formula. : in, This represents the coordinate system of the telescopic device's end effector in the telescopic device's reference coordinate system at the current moment. This represents the coordinate system of the telescopic device's end effector in the telescopic device's reference coordinate system at the previous moment; The desired incremental pose of the robotic arm's end effector coordinate system is calculated using the following formula. : in, This represents the coordinate system of the current end effector of the robotic arm in the base coordinate system of the robotic arm; Will The matrix corresponding to the rotated part is transformed into a 3D column vector. ,Will The matrix corresponding to the translation part is transformed into a 3D column vector. and by the following formula and Limiting the amplitude: ， in, For rotation threshold, The translation threshold; After the limit and As control commands for the robotic arm.

7. The ultrasound detection data acquisition system according to claim 1, characterized in that, The control commands for generating the robotic arm in response to the control operation include: Set the translation speed of the end effector of the robotic arm in the robotic arm's base coordinate system. and rotational speed : in, and These represent the translational and rotational velocities of the telescopic device's end-effector coordinate system relative to the telescopic device's reference system, respectively. and This refers to the speed multiplier. For robotic arms that include speed control interfaces, and As control commands for the robotic arm; For a robotic arm that only has a position control interface, calculate the position increment. and rotation increment And it serves as the control command for the robotic arm.

8. The ultrasound detection data acquisition system according to claim 1, characterized in that, The data processing unit is also used to: transform the contact force collected by the force sensor from the force sensor coordinate system to the robot arm base coordinate system to obtain the contact force components in each direction in the robot arm base coordinate system; for each contact force component, when the absolute value of the current contact force component is less than a preset threshold, reset the current contact force component to zero. When the absolute value of the current contact force component is not less than a preset threshold, the current contact force component is numerically scaled to obtain the scaled contact force component. The scaled contact force component is sent to the remote control device; The remote control device is also used to perform the following steps in a preset control cycle: determine whether the current cycle count value is an integer multiple of the preset multiple; if so, write the scaled contact force component into the force feedback drive kernel of the remote control device; otherwise, write zero into the force feedback drive kernel of the remote control device.

9. The ultrasound detection data acquisition system according to claim 1, characterized in that, The data processing unit is also used to: store ultrasound detection skill data using a preset skill data file organization structure; The preset skill data file organization structure includes: A root directory storage area; at least one dataset storage area located within the root directory storage area, each dataset storage area corresponding to a dataset name; a training data storage area and a calibration data storage area located within each dataset storage area; at least one skill recording storage area located within the training data storage area, each skill recording storage area corresponding to a recording index; an ontology data storage area, an instruction data storage area, and at least one image data storage area located within each skill recording storage area.

10. A method for acquiring ultrasound detection data based on the ultrasound detection data acquisition system according to any one of claims 1 to 9, characterized in that, include: The remote control device generates control commands for the robotic arm in response to control operations and sends them to the robotic arm. The robotic arm responds to control commands to control the movement of the ultrasound probe and force sensor, and collects ultrasound detection signals through the ultrasound probe and sends them to the ultrasound machine, and collects contact force and contact torque through the force sensor and sends them to the data processing unit. The ultrasound machine generates ultrasound images based on the ultrasound detection signals and sends them to the data processing unit. The data processing unit collects motion information of the robotic arm, as well as associated motion information of the robotic arm, ultrasound images, contact force, and contact force rectangles to form ultrasound detection data.

Images