Handheld ultrasonic acquisition equipment and multi-modal data processing method and system thereof
By integrating adjustment components and multimodal data processing methods into handheld ultrasound acquisition equipment, the problems of equipment versatility and data adaptability are solved, and efficient acquisition and processing of multimodal data are achieved to adapt to the ultrasound scanning needs of different scenarios.
Patent Information
- Application Number
- CN202510710327.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-09-09
AI Technical Summary
Existing handheld ultrasound acquisition equipment cannot adapt to ultrasound probes of different sizes, and cannot collect multiple data at the same time. It has poor versatility and cannot adapt to the needs of different scenarios.
A handheld ultrasound acquisition device was designed, which included an adjustment component to adapt to ultrasound probes of different sizes, and integrated an image acquisition component and a force sensor. Multimodal data was processed through Kalman filtering and timestamp alignment to achieve multimodal data acquisition and processing.
It improves the versatility of handheld ultrasound acquisition equipment, ensures data quality and time consistency, supports the collection and analysis of multiple data, adapts to complex scanning scenarios, and improves diagnostic efficiency.
Smart Images
Figure CN120605043A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of ultrasonic acquisition equipment, and in particular to a handheld ultrasonic acquisition equipment and a multimodal data processing method and system thereof. Background Art
[0002] Ultrasound scanning is a noninvasive medical diagnostic technique that uses high-frequency sound waves (ultrasound waves) to image the internal structures and organs of the human body. The development of ultrasound scanning strategies involves multiple modalities of information (such as ultrasound images, ergonomics, and anatomical structures). Ultrasound physicians need to undergo extensive theoretical study and clinical practice to learn ultrasound techniques, resulting in a long transition from training to medical practice. Currently, many research teams, both domestically and internationally, are conducting research on robot-assisted ultrasound scanning to alleviate the personal scanning burden on ultrasound physicians and the training burden on medical institutions, thereby improving diagnostic efficiency.
[0003] Similar to the learning path of ultrasound physicians, ultrasound robots need to first learn a large number of expert examples of multimodal techniques (such as force, ultrasound images, position information, etc.) to obtain anthropomorphic ultrasound scanning techniques. The current teaching / technique acquisition method is mostly implemented by robot dragging teaching. However, because the robot is too bulky, it is difficult to achieve more precise operation and cannot reflect the scanning techniques of ultrasound physicians during ultrasound scanning. Therefore, compared with robot dragging teaching, bare hands are lighter and more flexible, more sensitive to force changes, and can complete a variety of complex posture changes, which can better cover different scanning scenarios and scanning needs.
[0004] Currently, there are a variety of handheld ultrasonic acquisition devices on the market, which realize the position detection and positioning of the scanning target by calculating the three-dimensional coordinates of the object to be measured. However, the current handheld ultrasonic acquisition devices are usually adapted to a certain ultrasonic probe and cannot adapt to different ultrasonic probe sizes. They are also unable to collect and process multiple data, and have poor versatility and cannot adapt to different scenarios. Summary of the Invention
[0005] The purpose of the present invention is to address the problem of poor versatility of the above-mentioned handheld ultrasonic acquisition device and to propose a handheld ultrasonic acquisition device and a multimodal data processing method and system thereof.
[0006] On the one hand, the present invention adopts the following technical solutions:
[0007] A handheld ultrasonic acquisition device comprises a handheld outer shell and a fixed inner shell arranged in the handheld outer shell;
[0008] The fixed inner shell includes a fixed shell and a sliding shell that cooperates with the fixed shell to form an accommodating cavity. An adjustment component is provided between the sliding shell and the fixed shell. The adjustment component is used to adjust the relative position of the fixed shell and the sliding shell to adjust the size of the accommodating cavity.
[0009] An elastic buffer is provided between the handheld outer shell and the fixed inner shell;
[0010] An image acquisition component with a built-in IMU is installed on the outer side of the handheld housing;
[0011] A force sensor is provided on the inner top of the handheld housing.
[0012] Optionally, the force sensor is coaxial with the handheld outer shell, the fixed inner shell, and the ultrasonic probe installed in the accommodating cavity, and there is a gap between the force sensor and the fixed inner shell.
[0013] Optionally, the adjustment assembly includes a slide rail and a slider, the slide rail is provided on the sliding housing, the slider is provided on the fixed housing, and the slider is in sliding engagement with the slide rail.
[0014] In the second aspect, the present invention adopts the following technical solutions:
[0015] A multimodal data processing method, comprising:
[0016] Acquiring multimodal data, the multimodal data including ultrasound image data, camera image data, force data, and IMU position information data;
[0017] performing Kalman filtering on the multimodal data respectively to obtain preprocessed data;
[0018] respectively extracting timestamps of the preprocessed data;
[0019] The obtained timestamps are timestamp aligned to obtain a training data set.
[0020] Optionally, performing timestamp alignment on the obtained timestamps to obtain a training data set includes:
[0021] Arranging the timestamps of each multimodal data in ascending time order to form an ordered sequence corresponding to each multimodal data;
[0022] Selecting the lowest acquisition frequency in the multimodal data as a reference frequency, and generating a target time series with equal intervals based on the reference frequency;
[0023] The target time series is used as a resampling frequency, and each of the multimodal data is resampled in each of the ordered sequences to obtain the training data set.
[0024] Optionally, the resampling of each of the multimodal data in each of the ordered sequences using the target time series as a resampling frequency includes:
[0025] respectively comparing the relationship between the acquisition frequency of each of the multimodal data and the reference frequency;
[0026] If the acquisition frequency of the multimodal data is high frequency data relative to the reference frequency, resampling is performed using a linear interpolation method;
[0027] If the acquisition frequency of the multimodal data is intermediate frequency data relative to the reference frequency, resampling is performed using a downsampling mean method;
[0028] If the acquisition frequency of the multimodal data is the same frequency data or low frequency data relative to the reference frequency, the original data or the most recent valid value is used.
[0029] In the third aspect, the present invention adopts the following technical solutions:
[0030] A multimodal data processing system, comprising:
[0031] An acquisition module is used to acquire multimodal data, wherein the multimodal data includes ultrasound image data, camera image data, force data, and IMU position information data;
[0032] A Kalman filter module, configured to perform Kalman filtering on the multimodal data to obtain preprocessed data;
[0033] A timestamp extraction module, used for respectively extracting the timestamps of the preprocessed data;
[0034] The calculation module is used to perform timestamp alignment on the obtained timestamps to obtain a training data set.
[0035] Optionally, the calculation module includes:
[0036] an arranging submodule, configured to arrange the timestamps of the multimodal data in ascending time order to form an ordered sequence corresponding to each multimodal data;
[0037] A reference selection submodule is used to select the lowest acquisition frequency in the multimodal data as a reference frequency, and generate a target time series with equal intervals based on the reference frequency;
[0038] The data generation submodule is used to resample each of the multimodal data in each of the ordered sequences using the target time series as a resampling frequency to obtain the training data set.
[0039] The beneficial effects achieved by the present invention are:
[0040] 1. The size of the accommodating cavity is adjusted by adjusting the component, and the image acquisition component and force sensor are integrated into the handheld ultrasonic acquisition device to achieve multimodal data acquisition, thereby improving the versatility of the handheld ultrasonic acquisition device.
[0041] 2. By performing Kalman filtering on multimodal data (ultrasound image data, camera image data, force data, and IMU position information data) separately, noise can be effectively removed and data quality can be improved.
[0042] 3. Arrange the timestamps of each multimodal data in ascending time order to form an ordered sequence, select the lowest acquisition frequency as the reference frequency to generate the target time series, and use linear interpolation, downsampling mean, or use original data / repeated latest valid value to perform resampling processing according to different acquisition frequencies to achieve multimodal data timestamp alignment, ensure data consistency in the time dimension, facilitate subsequent analysis and processing, and help improve the accuracy of data analysis and model training.
[0043] 4. The system consists of an acquisition module, a Kalman filter module, a timestamp extraction module and a calculation module. Each module has a clear function and cooperates with each other, which facilitates the maintenance, upgrade and expansion of the system.
[0044] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and illustration and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 This is a schematic diagram of the overall structure of a handheld ultrasound acquisition device according to an embodiment of the present application;
[0046] Figure 2 This is a structural diagram of a handheld ultrasonic acquisition device according to an embodiment of the present application;
[0047] Figure 3 This is a flowchart of a multimodal data processing method according to an embodiment of the present application;
[0048] Figure 4 This is one of the flow charts of a multimodal data processing method according to another embodiment of the present application;
[0049] Figure 5 This is a second flowchart of a multimodal data processing method according to another embodiment of the present application;
[0050] Figure 6 This is a third flowchart of a multimodal data processing method according to another embodiment of the present application;
[0051] Figure 7This is a system diagram of a multimodal data processing system according to another embodiment of the present application.
[0052] Figure numerals: 1, force sensor; 2, handheld outer shell; 3, fixed inner shell; 31, fixed shell; 32, sliding shell; 4, slide rail; 41, slider; 5, ultrasonic probe; 6, image acquisition component. DETAILED DESCRIPTION
[0053] The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. In addition, the drawings of the present invention are only for simple schematic illustrations and are not depicted according to actual dimensions. It is stated in advance. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention.
[0054] Ultrasound scanning techniques are complex, and developing scanning strategies involves multiple modalities (such as ultrasound images, ergonomics, and anatomical structures). Ultrasound physicians need to undergo extensive theoretical study and clinical practice to learn these techniques, resulting in a long transition from training to medical practice. Currently, many research teams, both domestically and internationally, are conducting research on robot-assisted ultrasound scanning to alleviate the personal scanning burden on ultrasound physicians and the training burden on medical institutions, thereby improving diagnostic efficiency.
[0055] Similar to the learning path of ultrasound physicians, ultrasound robots need to learn a large number of expert examples of multimodal techniques (such as force, ultrasound images, position information, etc.) in advance to obtain anthropomorphic ultrasound scanning techniques. The current teaching / technique acquisition method is mostly implemented by robot dragging teaching. However, because the robot is too bulky, it is not only unable to capture relatively small force changes when dragging, but also unable to achieve more complex scanning postures, and cannot reflect the scanning techniques of ultrasound physicians when performing ultrasound scanning. Compared with robot dragging teaching, bare hands are lighter and more flexible, and are more sensitive to capturing force changes. They can also complete a variety of complex posture changes and can better cover different scanning scenarios and scanning needs. Therefore, this application is based on barehand teaching to realize the collection, filtering and alignment of multimodal information (ultrasound images, force data, camera images, position information, etc.), and finally the processed data is made into a multimodal scanning technique data set to facilitate the subsequent robot to learn ultrasound scanning technique strategies and realize the robot's anthropomorphic scanning.
[0056] Reference Figure 1 and Figure 2Embodiment 1: This embodiment provides a handheld ultrasonic acquisition device, including a handheld outer shell 2 and a fixed inner shell 3 arranged in the handheld outer shell 2.
[0057] In this embodiment, the handheld housing 2 is produced by 3D printing. In other embodiments of the present application, the handheld housing 2 may also be produced by known techniques such as injection molding.
[0058] An image acquisition component 6 equipped with an IMU (Inertial Measurement Unit) is mounted on the outer shell of the handheld housing 2. In this embodiment, the image acquisition component 6 is a Go Pro camera with a built-in IMU. In other embodiments of the present application, other existing devices capable of image acquisition may also be used, which will not be described in detail here.
[0059] An image acquisition module with an IMU is used, and the IMU can adapt to the angle change of the ultrasonic probe 5 during force detection to ensure the accuracy of the collected data.
[0060] The fixed inner housing 3 includes a fixed housing 31 and a sliding housing 32 that cooperates with the fixed housing 31 to form a receiving cavity for accommodating the ultrasound probe 5. An adjustment assembly is provided between the fixed housing 31 and the sliding housing 32 to adjust the relative position of the fixed housing 31 and the sliding housing 32 to adjust the size of the receiving cavity.
[0061] In this embodiment, the adjustment components are a slide rail 4 and a slider 41. The slide rail 4 is arranged on the sliding shell 32, and the slider 41 is arranged on the fixed shell 31. The slider 41 slides with the slide rail 4 so that the fixed shell 31 and the sliding shell 32 can move relative to each other along the sliding direction of the slide rail 4, thereby realizing the adjustment of the accommodating cavity to adapt to ultrasound probes 5 of different sizes.
[0062] In other embodiments of the present application, the adjustment component can be an elastic member (such as a tension spring), a stop block and a stop slot, etc., which will not be described in detail here. The size of the accommodating cavity can be adjusted by the adjustment component, and the image acquisition component 6 and force sensor 1 are integrated into the handheld ultrasonic acquisition device to achieve multimodal data acquisition, thereby improving the versatility of the handheld ultrasonic acquisition device.
[0063] An elastic buffer is provided between the handheld outer shell 2 and the fixed inner shell 3. In this embodiment, the elastic buffer is an elastic buffer layer (such as a silicone layer). The gap between the handheld outer shell 2 and the fixed inner shell 3 is filled inward with an elastic buffer layer with good deformation ability. While maintaining the stability of force signal transmission, it provides certain support for the fixed inner shell 3 to ensure that the fixed inner shell 3 can maintain a relatively stable fixed state of the handheld outer shell 2 after being deformed when using the ultrasonic probe 5.
[0064] A force sensor 1 is provided at the inner top of the handheld housing 2 .
[0065] Optionally, in this embodiment, the force sensor 1 is coaxial with the handheld outer shell 2, the fixed inner shell 3 and the ultrasonic probe 5 installed in the accommodating cavity, and the axial force is accurately measured. The force sensor 1 only detects the axial force, which improves the real-time performance of the detection and can provide more accurate and timely feedback of force information.
[0066] Optionally, there is a gap between the force sensor 1 and the fixed inner shell 3 to ensure that when the ultrasonic probe 5 is in the standby state, the force sensor 1 has no force detection display, thereby increasing the accuracy of force detection.
[0067] In another embodiment of the present application, refer to Figure 3 and Figure 4 , discloses a multimodal data processing method, which is applied to a handheld ultrasonic acquisition device as described above. As shown in the figure, the multimodal data processing method includes:
[0068] S1: Acquire multimodal data, where the multimodal data includes ultrasound image data, camera image data, force data, and IMU position information data;
[0069] S2: performing Kalman filtering on the multimodal data to obtain preprocessed data;
[0070] S3: extracting the timestamps of the pre-processed data respectively;
[0071] S4: performing timestamp alignment on the obtained timestamps to obtain a training data set.
[0072] In this embodiment, the ultrasonic image data is collected by the ultrasonic probe 5, the camera image data is captured and collected by the image acquisition component 6, the force data is collected by the force sensor 1, and the IMU position information data is collected by the IMU built into the image acquisition component 6.
[0073] In this embodiment, multimodal data refers to any one of the ultrasonic image data, camera image data, force data, and IMU position information data, but not all of the data. Preprocessed data refers to the data after Kalman filtering. In practice, multimodal data refers to the data that is processed. For example, if camera image data is processed, multimodal data refers to camera data. In this case, preprocessed data refers to the camera data after Kalman filtering.
[0074] Taking into account the acquisition techniques and context, this embodiment can approximately assume that the ultrasound images, camera images, force data, and IMU position information collected during the ultrasound scan process meet the following characteristics: ① The current state is only related to the previous state; ② The model and system both satisfy a linear relationship; and ③ The introduced noise conforms to a Gaussian distribution. Based on these characteristics, and considering the multi-sensor fusion and system dynamics of the multimodal data collected by the system, a Kalman filter method is used to perform preliminary filtering on the collected multimodal data.
[0075] The Kalman filter is an efficient recursive filter used to estimate the state of a dynamic system from a series of incomplete and noisy measurements.
[0076] For the collected data, the core idea of using Kalman filtering is as follows:
[0077] Equation of state (prediction stage)
[0078] x k =Ax k-1 +Bu k +w k (1)
[0079] Formula (1) describes the transition of the system state from time k-1 to time k, where A is the state transition matrix, Bu k is the control input, w k is the process noise (obeying the normal distribution N(0,Q)).
[0080] Observation equation (update phase)
[0081] z k =Hx k +v k (2)
[0082] Formula (2) describes the observation value z k With the system state x k The relationship between H and v is the observation matrix. k is the observation noise (obeying the normal distribution N(0,Q)).
[0083] Recursive process
[0084] The recursive process can be divided into two stages: prediction and update.
[0085] For the prediction stage, there are:
[0086]
[0087] P k|k-1 =AP k-1 A T+Q (4)
[0088] This embodiment is based on the state of the previous moment and covariance P k-1 , calculate the prior estimate of the current moment (Formula (3)) and the prior covariance (Formula (4)).
[0089] And for the update phase, there are:
[0090] K k =P k|k-1 H T (HP k|k-1 H T +R) -1 (5)
[0091]
[0092] P k =(IK k H)P k|k-1 (7)
[0093] This embodiment uses the current observation value z k Correct the prediction result and calculate the Kalman gain (Formula (5)), thereby obtaining the posterior estimate (Formula (6)) and the posterior covariance (Formula (7)).
[0094] In this embodiment, the timestamp of the data is extracted using the following method:
[0095] For ultrasound image data, if the output is a DICOM format file, the timestamp is stored in the metadata tag. Use a DICOM parsing library (such as Python's pydicom) to extract the timestamp. If the ultrasound acquisition device provides an API or SDK, you can obtain real-time data and timestamps through the interface by calling the GetFrame or GetImage interface of the device SDK to obtain the timestamp at the same time. If the ultrasound device transmits data via a custom protocol, the timestamp is embedded in the packet header or frame header. Parse the packet header to extract the timestamp field. In other embodiments of this application, other existing methods can also be used for extraction, which will not be detailed here.
[0096] For camera image data, if the timestamp is provided by the SDK or driver, use the camera SDK (such as OpenCV, FlyCapture) to obtain the frame timestamp; if the data is transmitted through RTSP or a custom protocol, the timestamp is embedded in the frame header or metadata, parse the data packet header, and extract the timestamp field. In other embodiments of this application, other existing methods can also be used for extraction, which will not be described here.
[0097] Force data, the force sensor 1 usually transmits data through an analog signal or a digital protocol (such as CAN, RS-232), and the timestamp may be added by the acquisition system. If the sensor directly outputs the timestamp, the data packet header is parsed; if the sensor has no timestamp, the timestamp is added by the acquisition system (such as a data acquisition card). In other embodiments of the present application, other existing methods can also be used for extraction, which will not be elaborated here.
[0098] IMU position information data. IMU devices typically transmit data via ROS topics or custom protocols, with the timestamp embedded in the message header. If using ROS, the timestamp is stored in the message header; if using a custom protocol, the data packet header is parsed. Some IMU devices provide SDKs, which can be used to obtain data and timestamps through interfaces. Calling the GetData interface of the device SDK also obtains the timestamp. In other embodiments of this application, other existing methods can also be used for extraction, which will not be detailed here.
[0099] In this embodiment, several timestamps are aligned by methods such as downsampling, upsampling, timestamp alignment based on dynamic time warping (DTW), timestamp alignment based on event triggering, timestamp alignment based on machine learning, and timestamp alignment based on timestamp conversion.
[0100] After aligning the timestamps of the multimodal data, a training dataset can be generated for model training. This application solution ensures that the collected data is consistent and relevant in the time dimension, improving the versatility of the product and providing more comprehensive and accurate data support for subsequent robot learning of ultrasound scanning techniques.
[0101] Reference Figure 5 , optionally, in another embodiment of the present application, S4 includes:
[0102] S41: Arranging the timestamps of the multimodal data in ascending time order to form an ordered sequence corresponding to the multimodal data;
[0103] S42: selecting the lowest acquisition frequency in the multimodal data as a reference frequency, and generating a target time series with equal intervals based on the reference frequency;
[0104] S43: Using the target time series as a resampling frequency, resampling the multimodal data in each ordered sequence to obtain the training data set.
[0105] Because the collected multimodal data have differences in frequency and source, this embodiment uses a timestamp alignment method to further align the data to ensure that the collected data is consistent and relevant in the time dimension, thereby providing a reliable time benchmark for subsequent data set production and robot learning.
[0106] Timestamp alignment is the process of converting multi-source data collected asynchronously at different frequencies into a unified time series. In this example, the core steps for collecting four types of data, including ultrasound images (20 Hz), camera image data (30 Hz), force data (100 Hz), and IMU position data (1000 Hz), are as follows:
[0107] 1. Timestamp collection and sorting (preprocessing)
[0108] In this step, this embodiment adds precise timestamps to the multi-source data and arranges them in ascending time order.
[0109] First, in this embodiment, the ultrasound image (I U ,20Hz), camera image (I C ,30Hz), force data (F,100Hz), and IMU position (P,1000Hz) are taken as examples, and the acquisition timestamps are recorded respectively: (20 per second), (30 per second), (100 per second), (1000 per second); then, arrange the sensor data in ascending order according to the timestamp to form an ordered sequence (such as ).
[0110] 2. Target time series generation (determining alignment targets)
[0111] In this step, this embodiment selects the lowest frequency (i.e., U, 20 Hz) as the benchmark to generate the target time series T with equal intervals. ref .
[0112] First, this embodiment determines the reference frequency f ref =20Hz, then its period ΔT = 1 / f ref =50ms; then, generate the reference time point: T ref ={t0, t0+ΔT, t0+2ΔT, ..., t0+NΔT} (where t0 is the starting time, N is the total number of points (e.g., 20 per second, for a total of 10 seconds, then N=200), and the reference period ΔT=1 / f ref is the time interval basis for subsequent resampling).
[0113] 3. Resampling Mapping (Core Processing)
[0114] In this embodiment, depending on the data, resampling can be performed in the following ways:
[0115] (1) Linear Interpolation
[0116] Estimate the value at an unknown time point using the linear relationship between known data points.
[0117] Implementation steps:
[0118] Determine the target timeline: select a unified timeline (such as the timeline with the highest sampling rate or a custom timeline); find the nearest neighbor data points: for each time point on the target timeline, find the two nearest data points in the original data; calculate the interpolation value: calculate the value at the target time point based on the linear relationship.
[0119] (2) Spline Interpolation
[0120] Use piecewise polynomials (splines) to fit the data points to ensure smoothness and continuity of the data.
[0121] Implementation steps:
[0122] Determine the target time axis; use a spline interpolation function, such as cubic spline interpolation, to fit the original data; and calculate the value at the target time point.
[0123] The above examples are just two methods selected in this embodiment. In this embodiment, a suitable resampling mapping method can be selected according to the actual data type. In other embodiments of this application, other existing methods can be used.
[0124] Reference Figure 6 , optionally, in another embodiment of the present application, S43 includes:
[0125] S431: comparing the relationship between the acquisition frequency of each multimodal data and the reference frequency respectively;
[0126] S432: If the acquisition frequency of the multimodal data is high frequency data relative to the reference frequency, resampling is performed using a linear interpolation method;
[0127] S433: If the acquisition frequency of the multimodal data is intermediate frequency data relative to the reference frequency, resampling is performed using a downsampling mean method;
[0128] S434: If the acquisition frequency of the multimodal data is the same frequency data or low frequency data relative to the reference frequency, use the original data or repeat the most recent valid value.
[0129] In this step, this embodiment selects three methods, linear interpolation, downsampling mean, and synchronization maintenance, to process the four types of data collected based on the relationship between the sensor data frequency and the reference frequency. It should be noted that the frequency of the multimodal data relative to the reference frequency is high frequency, medium frequency, same frequency, or low frequency, which is only a relative comparison result. Those skilled in the art can preset a threshold or difference based on the frequency of the actual data to be collected. This embodiment is only an example and is not considered to be a limitation of this application. This embodiment uses the reference frequency f ref =20HzExample:
[0130] ① High-frequency data → reference frequency (P, 1000Hz), using linear interpolation.
[0131] For high-frequency data, this embodiment will be at the reference time point t k ∈T ref Generate interpolated values.
[0132] First, for each reference time t k , find the adjacent original timestamp t i ≤t k ≤t j (t i , t j ∈T P ); secondly, apply the linear interpolation formula:
[0133]
[0134] where x i 、x j is the IMU at t i , t j The position value at the moment.
[0135] ②Intermediate frequency data → reference frequency (F, 100Hz), using the downsampling averaging method.
[0136] For the intermediate frequency data, this embodiment divides the window into windows according to the reference period and calculates the mean of the data within the window.
[0137] First, define each reference period ΔT = 50ms as a window, including M = f source ·ΔT=100×0.05=5, 5 original data points (such as arrive ); Then, use the mean downsampling formula:
[0138]
[0139] Among them F i For data in The measured value at a moment.
[0140] ③Low frequency / same frequency data→reference frequency (I U , 20Hz; I C , 30Hz), synchronization maintained.
[0141] In this step, for data with the same frequency or low frequency, this embodiment directly uses the original data or repeats the most recent valid value.
[0142] If the data frequency is low or equal to the reference frequency, I U , taking 20Hz as an example, match directly by timestamp: U k =U i .
[0143] After completing the above process, asynchronous, multi-frequency ultrasound image data, camera image data, force data, and IMU position information data can be converted into a standardized data set with strict time alignment and uniform frequency, laying the foundation for subsequent robot learning of ultrasound manipulation strategies and realizing robot anthropomorphic scanning.
[0144] Reference Figure 7 ,Example 3: This embodiment includes all the contents of Example 1 and Example 2, and provides a multimodal data processing system.
[0145] An acquisition module is used to acquire multimodal data, including ultrasound image data, camera image data, force data, and IMU position information data;
[0146] Kalman filter module, used to perform Kalman filtering on multimodal data to obtain preprocessed data;
[0147] A timestamp extraction module, used to extract the timestamps of preprocessed data respectively;
[0148] The calculation module is used to align the obtained timestamps to obtain a training data set.
[0149] Optionally, in another embodiment of the present application,
[0150] The calculation module includes:
[0151] The arrangement submodule is used to arrange the timestamps of each multimodal data in ascending time order to form an ordered sequence corresponding to each multimodal data;
[0152] The benchmark selection submodule is used to select the lowest acquisition frequency in the multimodal data as the benchmark frequency, and generate the target time series with equal intervals based on the benchmark frequency;
[0153] The data generation submodule is used to resample the multimodal data in each ordered sequence with the target time series as the resampling frequency to obtain the training data set.
[0154] Those skilled in the art will clearly understand that for the sake of convenience and conciseness of description, the division of the above-mentioned functional units and modules is only used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device described in this application is divided into different functional units or modules to complete all or part of the functions described above.
Claims
1. A handheld ultrasonic acquisition device, characterized in that: It comprises a handheld outer shell (2) and a fixed inner shell (3) arranged in the handheld outer shell (2); The fixed inner shell (3) comprises a fixed shell (31) and a sliding shell (32) that cooperates with the fixed shell (31) to form an accommodating cavity. An adjustment component is provided between the sliding shell (32) and the fixed shell (31). The adjustment component is used to adjust the relative position of the fixed shell (31) and the sliding shell (32) to adjust the size of the accommodating cavity. An elastic buffer is provided between the handheld outer shell (2) and the fixed inner shell (3); An image acquisition component (6) with a built-in IMU is installed on the outer side of the handheld housing (2); A force sensor (1) is provided at the inner top end of the handheld housing (2).
2. The handheld ultrasonic acquisition device according to claim 1, wherein: The force sensor (1) is coaxial with the handheld outer shell (2), the fixed inner shell (3), and the ultrasonic probe (5) installed in the accommodating cavity, and there is a gap between the force sensor (1) and the fixed inner shell (3).
3. The handheld ultrasonic acquisition device according to claim 1, wherein: The adjustment assembly comprises a slide rail (4) and a slider (41); the slide rail (4) is arranged on the sliding housing (32); the slider (41) is arranged on the fixed housing (31); and the slider (41) and the slide rail (4) are in sliding cooperation.
4. A multimodal data processing method, characterized in that: include, Acquiring multimodal data, the multimodal data including ultrasound image data, camera image data, force data, and IMU position information data; performing Kalman filtering on the multimodal data respectively to obtain preprocessed data; respectively extracting timestamps of the preprocessed data; The obtained timestamps are timestamp aligned to obtain a training data set.
5. The multimodal data processing method according to claim 4, wherein: The step of performing timestamp alignment on the obtained timestamps to obtain a training data set includes: Arranging the timestamps of each multimodal data in ascending time order to form an ordered sequence corresponding to each multimodal data; Selecting the lowest acquisition frequency in the multimodal data as a reference frequency, and generating a target time series with equal intervals based on the reference frequency; The target time series is used as a resampling frequency, and each of the multimodal data is resampled in each of the ordered sequences to obtain the training data set.
6. The multimodal data processing method according to claim 5, wherein: The resampling process is performed on each of the multimodal data in each of the ordered sequences using the target time series as the resampling frequency, including: respectively comparing the relationship between the acquisition frequency of each of the multimodal data and the reference frequency; If the acquisition frequency of the multimodal data is high frequency data relative to the reference frequency, resampling is performed using a linear interpolation method; If the acquisition frequency of the multimodal data is intermediate frequency data relative to the reference frequency, resampling is performed using a downsampling mean method; If the acquisition frequency of the multimodal data is the same frequency data or low frequency data relative to the reference frequency, the original data or the most recent valid value is used.
7. A multimodal data processing system, characterized in that: include: An acquisition module is used to acquire multimodal data, wherein the multimodal data includes ultrasound image data, camera image data, force data, and IMU position information data; A Kalman filter module, configured to perform Kalman filtering on the multimodal data to obtain preprocessed data; A timestamp extraction module, used for respectively extracting the timestamps of the preprocessed data; The calculation module is used to perform timestamp alignment on the obtained timestamps to obtain a training data set.
8. The multimodal data processing system according to claim 7, wherein: The calculation module includes: an arranging submodule, configured to arrange the timestamps of the multimodal data in ascending time order to form an ordered sequence corresponding to each multimodal data; A reference selection submodule is used to select the lowest acquisition frequency in the multimodal data as a reference frequency, and generate a target time series with equal intervals based on the reference frequency; The data generation submodule is used to resample each of the multimodal data in each of the ordered sequences using the target time series as a resampling frequency to obtain the training data set.