Ultrasonic key frame positioning method and device

Abstract

The invention discloses an ultrasonic key frame positioning method and device, and belongs to the field of medical instruments. An ultrasonic probe is rigidly fixed to the tail end of a mechanical arm; the mechanical arm drives the ultrasonic probe to scan, and a coordinate set in the moving process of the mechanical arm and an ultrasonic image obtained by the ultrasonic probe are obtained; time matching is conducted on the coordinate set and the ultrasonic images, and data pairs of one-to-one correspondence between the mechanical arm coordinates and the ultrasonic images are obtained; the method comprises the following steps: preprocessing an ultrasonic image, detecting key points of the preprocessed ultrasonic image, screening out the ultrasonic image containing the key points, extracting features with diagnostic significance in the ultrasonic image, performing similarity measurement on the features with diagnostic significance by adopting a key frame matching model, and determining the similarity of the features with diagnostic significance according to a measurement result. Key frames in the ultrasonic image are matched, and frames with similar features are found out to serve as the key frames; the mechanical arm coordinate corresponding to the key frame is found out through the data pair of the mechanical arm coordinate and the ultrasonic image, and through the steps, the detection precision is high, and the time is short.

Description

Technical Field

[0001] The present invention relates to the field of medical devices, and particularly to an ultrasonic key frame positioning method and device. Background Art

[0002] Ultrasonic imaging uses an ultrasonic beam to scan, and obtains a sectional image by differentiating reflected signals with different time delays and attenuations. When there are obvious differences in acoustic impedance and attenuation between various organs and tissues, the ultrasonic signal will have a large change, and the image can generate bright pixel points, thereby obtaining an obvious boundary contour.

[0003] When using two-dimensional ultrasound to locate the target position, the position and angle of the ultrasound probe are often repeatedly manually adjusted only based on the experience of clinicians, and the intercepted two-dimensional ultrasound images of the target are subjective and not necessarily the best observation angles and positions. During the process of locating the best observation angles and positions, it is often necessary to analyze the observed images, resulting in low detection accuracy and long time. Summary of the Invention

[0004] In order to overcome the deficiencies of the prior art, one of the purposes of the present invention is to provide an ultrasonic key frame positioning method with high detection accuracy and short time.

[0005] In order to overcome the deficiencies of the prior art, the second purpose of the present invention is to provide an ultrasonic key frame positioning device with high detection accuracy and short time.

[0006] One of the purposes of the present invention is achieved by adopting the following technical solutions:

[0007] An ultrasonic key frame positioning method includes the following steps:

[0008] Equipment setup: Rigidly fix the ultrasonic probe at the end of the robotic arm, the ultrasonic probe is communicatively connected to the ultrasonic instrument, and the ultrasonic instrument and the robotic arm are communicatively connected to the digital acquisition and analysis system;

[0009] Robotic arm movement: The robotic arm moves according to a preset trajectory, and the ultrasonic probe scans to obtain a coordinate set during the movement of the robotic arm and the ultrasonic images obtained by the ultrasonic probe;

[0010] Data matching: Perform time matching on the coordinate set and the ultrasonic images to obtain data pairs in which the robotic arm coordinates and the ultrasonic images correspond one by one;

[0011] Key frame determination: Preprocess the ultrasound image, detect the key points of the preprocessed ultrasound image, screen out the ultrasound images containing key points, extract the diagnostically significant features in the ultrasound image, use the key frame matching model to measure the similarity of the diagnostically significant features, and based on the measurement results, match the key frames in the ultrasound image to find the frames with similar features as the key frames;

[0012] Key frame positioning: Through the data pair of the robotic arm coordinates and the ultrasound image, find the robotic arm coordinates corresponding to the key frame.

[0013] Further, in the key frame determination step, the ultrasound images containing key points are screened out specifically by using the YOLOv8 key point detection algorithm.

[0014] Further, in the key frame determination step, the extraction of the diagnostically significant features in the ultrasound image is specifically as follows: Train the radiomics feature extraction model: Collect the ultrasound videos of clinical scans to find key frames, label the corresponding high-quality key frame labels, extract the radiomics features of the ultrasound images, splice them into multi-dimensional features, establish a relationship model with the key frame labels through the random forest model, and screen out the top several of them as the diagnostically significant features according to the feature importance of the random forest. Input the ultrasound images containing key points into the radiomics feature extraction model to extract the diagnostically significant features in the ultrasound image.

[0015] Further, in the key frame determination step, when using the key frame matching model to measure the similarity of the diagnostically significant features, the Euclidean distance or cosine similarity is used to calculate the similarity between the features.

[0016] Further, when using the cosine similarity to calculate the similarity between the features, the top several of the key frames in the collected ultrasound videos are used as the diagnostically significant features for feature averaging, and the cosine similarity between the diagnostically significant features of the ultrasound images obtained by the ultrasound probe and the average features is calculated to obtain the similarity score.

[0017] Further, in the key frame determination step, the preprocessing of the ultrasound image is specifically as follows: Perform histogram equalization, Gaussian filtering, and image size resampling on the ultrasound image.

[0018] Further, in the robotic arm movement step, the coordinates of the robotic arm are (x, y, z, θx, θy, θz), where x, y, and z are the coordinates in the X direction, Y direction, and Z direction respectively, θx is the angle in the X direction, θy is the angle in the Y direction, and θz is the angle in the Z direction.

[0019] Further, in the step of moving the robotic arm, the specific scanning of the ultrasonic probe is as follows: The time for a complete scan is T. According to the overall distance S of the scanning movement trajectory, the trajectory is disassembled into N points, and the scanning time for each point is T / N. According to the ultrasonic sampling interval of t and the rotation speed w, it satisfies wt <= T / N.

[0020] The second object of the present invention is achieved by the following technical solution:

[0021] An ultrasonic key frame positioning device for implementing any one of the above ultrasonic key frame positioning methods, including

[0022] A robotic arm that moves according to a preset trajectory;

[0023] An ultrasonic probe that is rigidly fixed to the end of the robotic arm and performs scanning;

[0024] An ultrasonic instrument that is communicatively connected to the ultrasonic probe and is used to form ultrasonic images;

[0025] A digital acquisition and analysis system that is communicatively connected to the robotic arm and the ultrasonic instrument. The digital acquisition and analysis system obtains the coordinate set during the movement of the robotic arm and the ultrasonic images obtained by the ultrasonic probe, and analyzes the coordinate set and the ultrasonic images to obtain the robotic arm coordinates corresponding to the key frames.

[0026] Compared with the prior art, in the ultrasonic key frame positioning method of the present invention, the ultrasonic probe is rigidly fixed to the end of the robotic arm, the ultrasonic probe is communicatively connected to the ultrasonic instrument, and the ultrasonic instrument and the robotic arm are communicatively connected to the digital acquisition and analysis system; the robotic arm moves according to a preset trajectory, the ultrasonic probe performs scanning, and the coordinate set during the movement of the robotic arm and the ultrasonic images obtained by the ultrasonic probe are acquired; the coordinate set and the ultrasonic images are time-matched to obtain the data pairs in one-to-one correspondence between the robotic arm coordinates and the ultrasonic images; the ultrasonic images are preprocessed, the key points of the preprocessed ultrasonic images are detected, the ultrasonic images containing key points are screened out and the diagnostically significant features in the ultrasonic images are extracted, a key frame matching model is used to perform similarity measurement on the diagnostically significant features, and according to the measurement results, the key frames in the ultrasonic images are matched to find the frames with similar features as the key frames; through the data pairs of the robotic arm coordinates and the ultrasonic images, the robotic arm coordinates corresponding to the key frames are found. Through the above steps, the ultrasonic detection has high accuracy and short time. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of the ultrasonic key frame positioning method of the present invention;

[0028] Figure 2This is a schematic diagram of the ultrasound key frame positioning device of the present invention. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0031] Please refer to Figure 1 and Figure 2 , a method for positioning ultrasound key frames of the present invention includes the following steps:

[0032] Equipment setup: Rigidly fix the ultrasound probe at the end of the robotic arm. The ultrasound probe is communicatively connected to the ultrasound instrument, and the ultrasound instrument and the robotic arm are communicatively connected to the digital acquisition and analysis system;

[0033] Robotic arm movement: The robotic arm moves according to a preset trajectory, and the ultrasound probe scans to obtain a coordinate set during the movement of the robotic arm and the ultrasound images obtained by the ultrasound probe;

[0034] Data matching: Perform time matching on the coordinate set and the ultrasound images to obtain data pairs in which the robotic arm coordinates and the ultrasound images correspond one by one;

[0035] Key frame determination: Preprocess the ultrasound images, detect the key points of the preprocessed ultrasound images, screen out the ultrasound images containing key points and extract the diagnostically significant features in the ultrasound images. Use a key frame matching model to perform similarity measurement on the diagnostically significant features. According to the measurement results, match the key frames in the ultrasound images and find the frames with similar features as the key frames;

[0036] Key frame positioning: Through the data pairs of the robotic arm coordinates and the ultrasound images, find the robotic arm coordinates corresponding to the key frames.

[0037] Specifically, in the equipment setup step, the robotic arm is a six-degree-of-freedom robotic arm.

[0038] The specific steps for the robotic arm to move are as follows: Since the area to be scanned is based on the human anatomical structure, for different application scenarios, a rectangular area to be scanned can be selected, and an S-shaped motion trajectory is set in this area. By controlling the rotation angle and speed of the probe, scans in different directions and depths can be achieved, and the size of the area to be scanned is determined to ensure the integrity of the image information. Adjust the rotation speed of the probe according to the inspection requirements to balance the image quality and inspection time. Specifically, the time for a complete scan is T. According to the overall distance S of the scanning motion trajectory, the trajectory is disassembled into N points, and the scanning time for each point is T / N. According to the ultrasonic sampling interval of t and the rotation speed of w, it satisfies wt <= T / N. Select an appropriate rotation angle to cover different directions of the target area. Perform rotational scanning along the set path to obtain more comprehensive image information. The ultrasonic sampling interval is t1, and N1 sample images are collected during the process.

[0039] In the steps of the robotic arm moving, the coordinates of the robotic arm are (x, y, z, θx, θy, θz), where x, y, and z are the coordinates in the X direction, Y direction, and Z direction respectively, θx is the angle in the X direction, θy is the angle in the Y direction, and θz is the angle in the Z direction. The sampling interval of the robotic arm position is t2, and N2 sample coordinate points are collected during the process. The N2 samples form a coordinate set.

[0040] The specific steps for data matching are as follows: The ultrasonic probe and the robotic arm are synchronized at the initial time. The ultrasonic sampling interval is t1, and N1 sample images are collected during the process. The sampling interval of the robotic arm position is t2, and N2 sample coordinate points are collected during the process. Each sample has a timestamp accurate to ms, and the data closest to the timestamp is found. The ultrasonic images are time-matched with the coordinate set to form data pairs. This is to facilitate the subsequent automatic positioning of the robotic arm to the corresponding scanning position based on the key frame images in the ultrasonic images.

[0041] The key frame determination steps are specifically as follows: Preprocess the ultrasound image: perform histogram equalization on the ultrasound image to enhance contrast, Gaussian filtering for denoising, and image size resampling. The resampling values are 960*720, and the interpolation method is the nearest neighbor interpolation method. Use a pre-trained key point detection algorithm to detect stable and significant feature points in the preprocessed ultrasound image. In this embodiment, the key point detection algorithm uses a key point detection algorithm based on YOLOv8. The point sets detected in Yolov8 have weights. For example, the weight is 75%. There is a 75% probability that the following is the detected object. Select those with higher weights (generally considered to be greater than 85%). Screen the ultrasound images in the key point detection algorithm that can detect stable and significant points. For these images, enter the ultrasound feature extraction step. The ultrasound feature extraction uses an imaging genomics feature extraction model, and the imaging genomics feature extraction model is first pre-trained. The pre-training process is as follows: Collect clinical scans of ultrasound videos for finding key frames, annotate the corresponding high-quality key frame labels, extract the imaging genomics features (including edges, textures, shapes) of the ultrasound images, splice them into n-dimensional features, and establish a relationship model with the key frame labels (0-1) through a random forest model. Through the feature importance of the random forest, screen the top 20 of them as features with diagnostic significance. Perform similarity measurement on the extracted features, and use an appropriate distance measurement method (such as Euclidean distance, cosine similarity, etc.) to calculate the similarity between the features. According to the similarity measurement results, match the key frames in the ultrasound image to find frames with similar features. In this embodiment, cosine similarity is used to calculate the similarity between the features. Specifically: Extract the top 20 high-importance features of all key frame images in the ultrasound video through the imaging genomics feature extraction model, and average the features of all key frame images. Also input the new ultrasound image collected by the ultrasound probe into the imaging genomics feature extraction model to obtain 20 feature values, and calculate the cosine similarity with the average feature to obtain a similarity score.

[0042] The ultrasound key frame positioning method of this application first performs rough screening on the images of the region through the key point integrity using a key point recognition method (such as YOLOv8), and quickly screens the remaining partial region segments. Then, in the narrowed region segments, perform fine key frame determination. Use a deep learning object detection algorithm based on region recommendation to calculate the key point integrity, select the optimal key frame from these finely screened frames, and trace back the coordinates of the robotic arm in reverse according to the optimal key frame, so that the robotic arm automatically returns to the specified position, reducing the time for the user to spend looking for key frames, improving the ultrasound detection accuracy and shortening the time.

[0043] Please continue to refer to Figure 2 , this application also discloses an ultrasound key frame positioning device for implementing the above-mentioned ultrasound key frame positioning method, including

[0044] The robotic arm moves along a preset trajectory;

[0045] The ultrasonic probe is rigidly fixed at the end of the robotic arm, and the ultrasonic probe performs scanning;

[0046] The ultrasonic diagnostic apparatus is communicatively connected to the ultrasonic probe, and the ultrasonic diagnostic apparatus is used to form ultrasonic images;

[0047] The digital acquisition and analysis system is communicatively connected to the robotic arm and the ultrasonic diagnostic apparatus. The digital acquisition and analysis system obtains the coordinate set during the movement of the robotic arm and the ultrasonic images obtained by the ultrasonic probe, and analyzes the coordinate set and the ultrasonic images to obtain the robotic arm coordinates corresponding to the key frames.

[0048] Compared with the prior art, in the ultrasonic key frame positioning method of the present invention, the ultrasonic probe is rigidly fixed at the end of the robotic arm, the ultrasonic probe is communicatively connected to the ultrasonic diagnostic apparatus, and the ultrasonic diagnostic apparatus and the robotic arm are communicatively connected to the digital acquisition and analysis system; the robotic arm moves along a preset trajectory, the ultrasonic probe performs scanning, and the coordinate set during the movement of the robotic arm and the ultrasonic images obtained by the ultrasonic probe are obtained; the coordinate set and the ultrasonic images are time-matched to obtain data pairs in which the robotic arm coordinates and the ultrasonic images correspond one by one; the ultrasonic images are preprocessed, the key points of the preprocessed ultrasonic images are detected, the ultrasonic images containing the key points are screened out and the diagnostically significant features in the ultrasonic images are extracted, a key frame matching model is used to perform similarity measurement on the diagnostically significant features, and according to the measurement result, the key frames in the ultrasonic images are matched to find the frames with similar features as the key frames; through the data pairs of the robotic arm coordinates and the ultrasonic images, the robotic arm coordinates corresponding to the key frames are found. Through the above steps, the ultrasonic detection has high accuracy and short time.

[0049] The above embodiments only represent several implementation manners of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made. These are all equivalent modifications and evolutions based on the essential technology of the present invention to the above embodiments, and these all belong to the protection scope of the present invention.

Claims

1. An ultrasonic key frame positioning method, characterized in that, Including the following steps: Equipment setup: Rigidly fix the ultrasonic probe at the end of the robotic arm. The ultrasonic probe is communicatively connected to the ultrasonic instrument, and the ultrasonic instrument and the robotic arm are communicatively connected to the digital acquisition and analysis system; Robotic arm movement: The robotic arm moves according to a preset trajectory, and the ultrasonic probe scans to obtain the coordinate set during the movement of the robotic arm and the ultrasonic images obtained by the ultrasonic probe; Data matching: Perform time matching on the coordinate set and the ultrasonic images to obtain data pairs in which the robotic arm coordinates and the ultrasonic images correspond one by one; Key frame determination: Preprocess the ultrasonic images, detect the key points of the preprocessed ultrasonic images, screen out the ultrasonic images containing key points and extract the diagnostically significant features in the ultrasonic images. Use a key frame matching model to measure the similarity of the diagnostically significant features. According to the measurement results, match the key frames in the ultrasonic images to find the frames with similar features as the key frames; Key frame localization: Through the data pairs of the robotic arm coordinates and the ultrasonic images, find the robotic arm coordinates corresponding to the key frames.

2. The ultrasound key frame positioning method according to claim 1, characterized in that: In the key frame determination step, the ultrasonic images containing key points are screened out specifically by using the YOLOv8 key point detection algorithm.

3. The ultrasonic key frame positioning method according to claim 1, characterized in that: In the key frame determination step, the extraction of the diagnostically significant features in the ultrasonic images is specifically as follows: Train an imaging genomics feature extraction model: Collect ultrasonic videos of clinical scans to find key frames, annotate the corresponding high-quality key frame labels, extract the imaging genomics features of the ultrasonic images, splice them into multi-dimensional features, establish a relationship model with the key frame labels through a random forest model, and screen out the top several of them as diagnostically significant features according to the feature importance of the random forest. Input the ultrasonic images containing key points into the imaging genomics feature extraction model to extract the diagnostically significant features in the ultrasonic images.

4. The ultrasonic key frame positioning method according to claim 3, characterized in that: In the key frame determination step, when using a key frame matching model to measure the similarity of the diagnostically significant features, the Euclidean distance or cosine similarity is used to calculate the similarity between the features.

5. The ultrasonic key frame positioning method according to claim 4, wherein: When using cosine similarity to calculate the similarity between the features, the top several of the key frames in the collected ultrasonic videos are averaged as diagnostically significant features, and the cosine similarity is calculated between the diagnostically significant features of the ultrasonic images obtained by the ultrasonic probe and the average features to obtain a similarity score.

6. The ultrasonic key frame positioning method according to claim 1, characterized in that: In the key frame determination step, the preprocessing of the ultrasonic images is specifically as follows: Perform histogram equalization, Gaussian filtering, and image size resampling on the ultrasonic images.

7. The ultrasonic key frame positioning method according to claim 1, wherein: In the robotic arm movement step, the coordinates of the robotic arm are (x, y, z, θx, θy, θz), where x, y, and z are the X-direction coordinate, Y-direction coordinate, and Z-direction coordinate respectively, θx is the X-direction angle, θy is the Y-direction angle, and θz is the Z-direction angle.

8. The ultrasonic key frame positioning method according to claim 1, wherein: In the robotic arm movement step, the scanning of the ultrasonic probe is specifically as follows: The time for a complete scan is T. According to the overall distance S of the scanning movement trajectory, the trajectory is decomposed into N points, and the scanning time for each point is T / N. According to the ultrasonic sampling interval t and the rotation speed w, it satisfies wt <= T / N.

9. An ultrasonic key frame positioning device for implementing the ultrasonic key frame positioning method according to any one of claims 1-8, characterized in that: Including A robotic arm that moves along a preset trajectory; An ultrasound probe rigidly fixed to the end of the robotic arm, and the ultrasound probe performs scanning; An ultrasound device communicatively connected to the ultrasound probe, and the ultrasound device is used to form an ultrasound image; A digital acquisition and analysis system communicatively connected to the robotic arm and the ultrasound device. The digital acquisition and analysis system obtains a coordinate set during the movement of the robotic arm and the ultrasound image obtained by the ultrasound probe, and analyzes the coordinate set and the ultrasound image to obtain the robotic arm coordinates corresponding to the key frames.

Images