Respiratory Motion Plane Coincidence System and Method Based on Ultrasound Images

Abstract

A respiratory motion plane alignment system and method based on ultrasound images is disclosed. The system acquires three-dimensional ultrasound images of respiratory organs at at least two different respiratory phases, calculates the motion direction of the respiratory organs based on the three-dimensional ultrasound images, controls and adjusts an ultrasound direction adjustment mechanism to align the scanning plane of the ultrasound probe with the motion direction of the respiratory organs, acquires a three-dimensional model image of the respiratory organs and multiple frames of real-time two-dimensional ultrasound images, performs registration between the multiple frames of real-time two-dimensional ultrasound images of the respiratory organs, and / or registers the three-dimensional model image and the multiple frames of real-time two-dimensional ultrasound images, acquires three-dimensional position data of a target point during respiratory motion, determines the tracking path of the ultrasound probe based on the three-dimensional position data of the target point during respiratory motion, and controls the ultrasound probe to move along the tracking path, thereby achieving long-term stable observation or tracking of target points in the respiratory organs.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and specifically to a respiratory motion plane overlap system and method based on ultrasound images. Background Technology

[0002] In the ultrasound diagnosis and treatment of organs, it is often required to maintain stable observation or tracking of the target tissue area for a long period of time. Since most ultrasound equipment can only acquire two-dimensional images in the scanning plane, when an organ undergoes respiratory motion in three-dimensional space, the target area may move out of the ultrasound scanning plane, making long-term stable observation or tracking impossible. Furthermore, the individual differences in organs and their movements reduce the stability and operability of two-dimensional ultrasound applications in monitoring or treating respiratory organs.

[0003] Taking high-energy focused ultrasound (HIFU) therapy as an example, HIFU is a non-invasive surgical technique that focuses high-energy ultrasound waves at a focal point to ablate tissue using thermal or mechanical effects. During treatment, it is often monitored in real-time using only a single linear or convex array probe to obtain two-dimensional images. However, for organs that move with respiration, such as the kidney, its movement during free breathing is an up-and-down motion along the psoas major muscle, with a maximum displacement of nearly 4 cm, far exceeding the focal size of HIFU. Therefore, without pre-aligning the scanning plane, it is impossible to continuously observe the target tissue, and it is also impossible to design subsequent tracking algorithms based on the continuously observed target position. Therefore, aligning the ultrasound transducer with the kidney's movement direction and the scanning plane of the ultrasound image, ensuring that the target area within the kidney remains within the ultrasound image for an extended period, is a necessary step for subsequent precise treatment of the target tissue.

[0004] Currently, existing technologies rely solely on ultrasound images for motion prediction, which is subject to significant interference, causing the prediction results to deviate considerably from reality. In addition, some methods obtain real-time target location information through two-dimensional and three-dimensional ultrasound images and incorporate surface motion detectors for position correction. However, none of these methods consider changing the ultrasound scanning plane to increase the duration of scanning the target area and obtain more input information for users to monitor or plan their next steps in motion. Summary of the Invention

[0005] The main technical problem solved by this invention is how to maintain stable observation or tracking of target points in the respiratory motor organs over a long period of time.

[0006] According to a first aspect, one embodiment provides a respiratory motion plane overlap system based on ultrasound images, comprising:

[0007] A front-end scanning mechanism includes an ultrasound probe and an ultrasound probe controller. The ultrasound probe is used to acquire two-dimensional ultrasound images of the scanning plane in the respiratory and motor organs. The ultrasound probe controller is used to control the movement of the ultrasound probe. During the movement, the scanning plane of the ultrasound probe is always parallel to the side plane of the front-end scanning mechanism. The y-axis of the front-end scanning mechanism is always parallel to the scanning plane of the ultrasound probe during the movement, and the y-axis of the front-end scanning mechanism is a pre-defined direction. The ultrasound probe has at least one degree of freedom relative to the front-end scanning mechanism.

[0008] robotic arm;

[0009] An ultrasonic direction adjustment mechanism is connected between the front-end scanning mechanism and the robotic arm, and is used to adjust the angle of the front-end scanning mechanism in space.

[0010] Human-computer interaction device, used to allow users to select target points in the respiratory motor organs that need to be tracked;

[0011] Control processor; wherein:

[0012] The respiratory motion plane coincidence system has a preparation phase, a planning phase, and a tracking phase;

[0013] During the preparation phase, the control processor acquires three-dimensional ultrasound images of the respiratory motor organ at at least two different respiratory phases, and calculates the direction of motion of the respiratory motor organ based on the three-dimensional ultrasound images; the respiratory phase is the phase corresponding to different respiratory moments within a respiratory cycle; the control processor controls and adjusts the angle of the ultrasound direction adjustment mechanism according to the direction of motion of the respiratory motor organ, so that the scanning plane of the ultrasound probe is consistent with the direction of motion of the respiratory motor organ.

[0014] During the planning phase, the control processor acquires a three-dimensional model image of the respiratory musculoskeletal organ, the three-dimensional model image containing the target point selected by the user and the position data of the target point in the three-dimensional model image; the control processor acquires multiple frames of real-time two-dimensional ultrasound images of the respiratory musculoskeletal organ through the ultrasound probe, wherein the first frame of the multiple frames of real-time two-dimensional ultrasound images contains the target point selected by the user and the position data of the target point in the first frame of real-time two-dimensional ultrasound images; the control processor performs registration between the multiple frames of real-time two-dimensional ultrasound images of the respiratory musculoskeletal organ, and / or registers the three-dimensional model image and the multiple frames of real-time two-dimensional ultrasound images to obtain the three-dimensional position data of the target point in respiratory motion;

[0015] During the tracking phase, the control processor determines the tracking path of the ultrasound probe based on the three-dimensional position data of the target point during respiratory motion, and uses the ultrasound probe controller in the front-end scanning mechanism to control the ultrasound probe to move along the tracking path.

[0016] According to a second aspect, one embodiment provides a method for aligning respiratory motion planes based on ultrasound images, comprising:

[0017] Preparation phase steps: Acquire three-dimensional ultrasound images of the respiratory motor organ at at least two different respiratory phases; calculate the direction of motion of the respiratory motor organ based on the three-dimensional ultrasound images; the respiratory phase is the phase corresponding to different respiratory moments within a respiratory cycle; control and adjust the angle of the ultrasound direction adjustment mechanism according to the direction of motion of the respiratory motor organ so that the scanning plane of the ultrasound probe is consistent with the direction of motion of the respiratory motor organ.

[0018] Planning phase steps: Acquire a 3D model image of the respiratory musculoskeletal organ, the 3D model image containing the target point selected by the user in the respiratory musculoskeletal organ and the position data of the target point in the 3D model image; acquire multiple frames of real-time 2D ultrasound images of the respiratory musculoskeletal organ using an ultrasound probe, wherein the first frame of the multiple frames of real-time 2D ultrasound images contains the target point selected by the user and the position data of the target point in the first frame of real-time 2D ultrasound images; register the multiple frames of real-time 2D ultrasound images of the respiratory musculoskeletal organ, and / or register the 3D model image and the multiple frames of real-time 2D ultrasound images to obtain the 3D position data of the target point in respiratory motion;

[0019] Tracking phase steps: Based on the three-dimensional position data of the target point during respiratory motion, determine the tracking path of the ultrasound probe, and use the ultrasound probe controller in the front-end scanning mechanism to control the ultrasound probe to move according to the tracking path.

[0020] According to the above embodiments, the respiratory motion plane coincidence system and method based on ultrasound images acquires three-dimensional ultrasound images of the respiratory motion organ at at least two different respiratory phases, calculates the motion direction of the respiratory motion organ based on the three-dimensional ultrasound images, controls the robotic arm and ultrasound direction adjustment mechanism to make the scanning plane of the ultrasound probe consistent with the motion direction of the respiratory motion organ, acquires a three-dimensional model image of the respiratory motion organ and multiple frames of real-time two-dimensional ultrasound images, performs registration between the multiple frames of real-time two-dimensional ultrasound images of the respiratory motion organ and / or registers the three-dimensional model image and the multiple frames of real-time two-dimensional ultrasound images, acquires the three-dimensional position data of the target point in the respiratory motion, determines the tracking path of the ultrasound probe based on the three-dimensional position data of the target point in the respiratory motion, and uses the ultrasound probe controller in the front-end scanning mechanism to control the ultrasound probe to move according to the tracking path, so as to achieve long-term stable observation or tracking of the target point in the respiratory motion organ. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of a respiratory motion plane overlap system based on ultrasound images, according to one embodiment.

[0022] Figure 2 This illustrates the spatial relationship between the front-end scanning mechanism and the ultrasonic probe in one embodiment.

[0023] Figure 3 A schematic diagram showing the displacement and direction of the respiratory organs under two different respiratory phases;

[0024] Figure 4 A schematic diagram of an angle measuring device for an ultrasonic direction adjustment mechanism according to one embodiment;

[0025] Figure 5 This is a schematic diagram illustrating the definition of the ultrasound image coordinate system and the three-dimensional model image coordinate system in one embodiment.

[0026] Figure 6 This is a schematic diagram illustrating trajectory planning for target control point tracking of target points in one embodiment.

[0027] Figure 7 This document presents an embodiment of a respiratory motion plane overlap method based on ultrasound images, along with a tracking flowchart. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0029] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0030] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0031] Respiratory musculoskeletal organs are organs that undergo periodic displacement movements within the body during respiration.

[0032] In this embodiment of the invention, by obtaining three-dimensional ultrasound images of the respiratory musculoskeletal system at multiple different respiratory phases, the direction of respiratory motion is obtained. The direction of the ultrasound probe is then adjusted so that the scanning plane of the ultrasound probe coincides with the motion plane of the target point in the respiratory musculoskeletal system. The position of the target point is obtained in the new spatial location. By fusing the registration of real-time two-dimensional ultrasound images and three-dimensional model images, a kinematic model of the target point is obtained. This model is then used to plan the motion trajectory of the target point within the scanning plane of the ultrasound probe, ensuring real-time tracking of the target point.

[0033] Please refer to Figure 1 , Figure 1This is a schematic diagram of a respiratory motion tracking system based on ultrasound imaging, according to one embodiment. The system includes a front-end scanning mechanism 101, an ultrasound probe 102, an ultrasound direction adjustment mechanism 103, a robotic arm 104, a control processor 105, and a human-machine interface 106. The ultrasound probe 102 is attached to the front-end scanning mechanism 101. The control processor 105 controls the front-end scanning mechanism 101 to control the ultrasound probe to perform rotational scanning. The ultrasound probe 102 has at least one degree of freedom relative to the front-end scanning mechanism 101. Simultaneously, the y-axis of the front-end scanning mechanism 101 is always parallel to the scanning plane of the ultrasound probe 102 during movement. The y-axis of the front-end scanning mechanism 101 is a pre-set direction in the ultrasound direction adjustment mechanism, which can be adjusted as needed. The ultrasound direction adjustment mechanism 103 connects the front-end scanning mechanism 101 and the robotic arm 104, and its direction is controlled by the control processor 105. The robotic arm 104 can be a 3-DOF robotic arm, and its movement is controlled by the control processor 105. In addition, the control processor 105 is electrically connected to the ultrasound probe 102 and the human-machine interface device 106. The ultrasound probe 102 can acquire two-dimensional ultrasound images within its imaging range. In this embodiment, the front-end scanning mechanism 101 and the ultrasound direction adjustment mechanism 103 are both located at the end of the robotic arm 104. The position of the front-end scanning mechanism 101 is controlled by the movement of the robotic arm 104, while the angle direction of the front-end scanning mechanism 101 can be controlled by the ultrasound direction adjustment mechanism 103. Furthermore, the two-dimensional ultrasound images acquired by the ultrasound probe 102 are transmitted to the control processor 105 for processing. The respiratory motion plane coincidence system provided in this embodiment also includes a memory 107, which is signal-connected to the control processor 105 and used to store the data information that the control processor 105 needs to process.

[0034] In some embodiments, the respiratory motion plane coincidence system provided by the present invention realizes the motion trajectory of the target point in the scanning plane of the ultrasound probe and ensures the real-time tracking of the target point, which includes three stages: preparation stage, planning stage and tracking stage. These three stages are described separately below.

[0035] During the preparation phase, the control processor 105 acquires three-dimensional ultrasound images of the respiratory motor organs at at least two different respiratory phases, and calculates the direction of motion of the respiratory motor organs based on the three-dimensional ultrasound images. Here, the respiratory phase refers to the phase corresponding to different respiratory moments within a respiratory cycle.

[0036] In some embodiments, acquiring three-dimensional ultrasound images of the respiratory motor organs at at least two different respiratory phases, depending on whether the patient is able to pause breathing, includes:

[0037] When a patient is able to stop breathing, external equipment such as a ventilator or anesthesia machine is used to induce breathing cessation. After breathing is stopped, a normal-speed three-dimensional ultrasound scan of the respiratory motor organs is performed.

[0038] In cases where the patient is unable to stop breathing, a rapid three-dimensional ultrasound scan of the respiratory motor organs is performed directly.

[0039] In some embodiments, three-dimensional ultrasound images of the respiratory motor organs at at least two different respiratory phases can be obtained by:

[0040] When the respiratory motor organs are in multiple different respiratory phases, the ultrasound probe controller rotates the ultrasound probe 102 to acquire multiple two-dimensional ultrasound images acquired by the ultrasound probe 102 at each different respiratory phase. The multiple two-dimensional ultrasound images corresponding to each respiratory phase are fused to obtain a three-dimensional ultrasound image corresponding to each respiratory phase, thereby obtaining three-dimensional ultrasound images of the respiratory motor organs at at least two different respiratory phases. That is, the three-dimensional ultrasound image corresponding to each respiratory phase is obtained by fusing multiple two-dimensional ultrasound images corresponding to that respiratory phase.

[0041] During multiple different respiratory phases, the ultrasound probe 102 in the moving front-end scanning mechanism 101 rotates around the y-axis of its own coordinate system {U}, where the ultrasound probe 102's own coordinate system {U} includes the x-axis, y-axis, and z-axis, and the front-end scanning mechanism 101's own coordinate system {R} includes the x′-axis, y′-axis, and z′-axis, as shown below. Figure 2 As shown, the y-axis of the ultrasound probe 102 is always parallel to the y′ axis of the front-end scanning mechanism 101, and multiple two-dimensional ultrasound images acquired by the ultrasound probe 102 are obtained respectively; the multiple two-dimensional ultrasound images are fused to obtain a three-dimensional ultrasound image corresponding to each respiratory phase. It should be noted that... Figure 2 The coordinate system {U} of the ultrasound probe and the coordinate system {R} of the front-end scanning mechanism 101 shown are examples. They can also be defined in other ways and can be adjusted according to specific circumstances. They will not be described in detail here.

[0042] It should be noted that during the preparation phase, the scanning plane of the ultrasound probe 102 is the initial scanning plane, which is not aligned with the direction of movement of the respiratory motor organs. Therefore, it is necessary to determine the direction of movement of the respiratory motor organs in order to adjust the scanning plane of the ultrasound probe so that the scanning plane of the ultrasound probe is consistent with the direction of movement of the respiratory motor organs.

[0043] In some embodiments, the direction of movement of the respiratory motor organs is obtained through the following methods:

[0044] The motion direction vector of the respiratory musculoskeletal system and the angle between the motion direction vector and the sagittal plane of the patient are calculated based on the three-dimensional ultrasound image.

[0045] Since the respiratory motor organs move to different positions at different respiratory phases, the center point of the three-dimensional ultrasound image corresponding to different respiratory phases will also be displaced accordingly. The spatial displacement vector of the respiratory motor organs in the corresponding two respiratory phases can be obtained in this way.

[0046] The calculation of the motion direction vector of the respiratory motor organ and the angle between the motion direction vector and the sagittal plane of the patient, based on the three-dimensional ultrasound image, includes:

[0047] The center points of all three-dimensional ultrasound images are obtained. The center points of multiple three-dimensional ultrasound images are paired up. Based on the difference in position of the center points of the two three-dimensional ultrasound images in each pair in three-dimensional space, the spatial displacement vector of the respiratory musculoskeletal organ in the corresponding two respiratory phases is determined to obtain at least one spatial displacement vector. The displacement vectors with more than one displacement vector are averaged to obtain the motion direction vector of the respiratory musculoskeletal organ. Then, based on the motion direction vector of the respiratory musculoskeletal organ, the angle between the motion direction vector of the respiratory musculoskeletal organ and the sagittal plane of the patient is determined.

[0048] By performing the above operation on the center point of every two 3D ultrasound images, the direction of movement of the respiratory motor organs can be obtained, such as... Figure 3 As shown.

[0049] In this embodiment, the final motion direction of the respiratory motor organs in the world coordinate system {E} is obtained. Then, it is provided to the ultrasonic direction adjustment mechanism 103 for angle adjustment around the z-axis. Based on the final direction of motion... The control processor 105 adjusts the angle of the ultrasound direction adjustment mechanism 103 so that the y-axis of the coordinate system {R} of the front-end scanning mechanism 101 is aligned with the final direction of motion. Parallel, meaning the y-axis of the ultrasonic probe 102 is parallel to the final direction of motion. parallel.

[0050] In this embodiment, the ultrasonic direction adjustment mechanism includes an angle measuring device, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of an angle measuring device for an ultrasound direction adjustment mechanism according to one embodiment. The mechanism involves controlling and adjusting the ultrasound direction adjustment mechanism based on the direction of movement of the respiratory motor organs, so that the scanning plane of the ultrasound probe aligns with the direction of movement of the respiratory motor organs.

[0051] Based on the direction vector of the respiratory motor organs and the angle between the direction vector and the patient's sagittal plane, the angle of the ultrasound direction adjustment mechanism is controlled and adjusted so that the angle displayed by the angle measuring device is consistent with the angle, so that the scanning plane of the ultrasound probe is consistent with the direction of the respiratory motor organs.

[0052] During the planning phase, in order to further move to the tissue plane of the target point for tracking, the control processor 105 rotates the ultrasound probe 102 in the front-end scanning mechanism 101 to the plane of the target point.

[0053] In one embodiment, the method for obtaining the target point includes:

[0054] If the patient is able to stop breathing, the patient's breathing is stopped again, and a three-dimensional ultrasound scan of the respiratory motor organs is performed to obtain a new three-dimensional ultrasound image, which the user can select target points in through the human-computer interaction device 106.

[0055] In cases where the patient is unable to breathe, the known target point locations in the 3D model image are used as the selected target points.

[0056] To obtain the kinematic model of the target point, it is necessary to obtain the three-dimensional position data of the target point during respiratory motion through ultrasound image registration.

[0057] First, a three-dimensional model image of the respiratory musculoskeletal system needs to be acquired. This three-dimensional model image contains the target point selected by the user and its position data within the image. In one embodiment, the three-dimensional model image can be a pre-acquired magnetic resonance (MRI) image.

[0058] Then, using an ultrasound probe whose scanning plane has been aligned with the direction of movement of the respiratory organs, continuous monitoring of the respiratory organs is performed, resulting in multiple frames of real-time two-dimensional ultrasound images. When the first frame of the real-time two-dimensional ultrasound image is acquired, the user needs to select a target point within it. The first frame of the multi-frame real-time two-dimensional ultrasound image contains the target point selected by the user and its position data within the first frame.

[0059] In one embodiment, the location of the target point includes: first real-time three-dimensional location data and second real-time three-dimensional location data. The first real-time three-dimensional location data is obtained by registering a first frame of real-time two-dimensional ultrasound image with other frames of real-time ultrasound images excluding the first frame. The second real-time three-dimensional location data is obtained by registering a three-dimensional model image with other frames of real-time ultrasound images excluding the first frame. The coordinates are defined as follows: Figure 5 As shown.

[0060] In another embodiment, the location of the target point may include only the first real-time three-dimensional location data. That is, the three-dimensional location data obtained by registering the first frame of real-time two-dimensional ultrasound image with other frames of real-time ultrasound images besides the first frame is used as the three-dimensional location data of the target point in respiratory motion.

[0061] In another embodiment, the location of the target point may include only the second real-time three-dimensional location data. That is, the three-dimensional location data obtained by configuring the three-dimensional model image with real-time ultrasound images of other frames besides the first frame is used as the three-dimensional location data of the target point in respiratory motion.

[0062] In this embodiment, the first real-time three-dimensional position data of the target point is obtained in the following manner:

[0063] Registration is performed between multiple frames of real-time two-dimensional ultrasound images to obtain the positional relationship of pixels between the first frame and other frames of real-time two-dimensional ultrasound images (excluding the first frame). Based on the positional relationship of pixels between the first frame and other frames of real-time two-dimensional ultrasound images (excluding the first frame) and the three-dimensional position data of the target point in the first frame of real-time two-dimensional ultrasound images, the first position data of the target point in the other frames of real-time two-dimensional ultrasound images is determined. Then, based on the positional relationship of the other frames of real-time two-dimensional ultrasound images (excluding the first frame) in the world coordinate system and the first position data of the target point in the other frames of real-time two-dimensional ultrasound images, the first three-dimensional position data of the target point in the world coordinate system is obtained, thus obtaining the first real-time three-dimensional position data of the target point.

[0064] The positional relationship of pixels between the real-time two-dimensional ultrasound images of all frames except the first frame and the first real-time two-dimensional ultrasound image is obtained according to the following formula:

[0065]

[0066] Among them, [I 1,x I 1,y 0] T The first frame of the real-time ultrasound image contains the location data of the target point in I1, [I k,x I k,y 0] T For the k-th frame of real-time two-dimensional ultrasound image I k The location data of the target point, where k is a natural number greater than 1, J 1,k This is the registration matrix.

[0067] In this embodiment, the second real-time three-dimensional position data of the target point is obtained in the following manner:

[0068] The three-dimensional model image and other real-time two-dimensional ultrasound images (excluding the first frame) are registered to obtain the positional relationship of pixels between the three-dimensional model image and other real-time two-dimensional ultrasound images (excluding the first frame). Based on the positional relationship of the three-dimensional model image relative to other real-time two-dimensional ultrasound images (excluding the first frame) and the three-dimensional position data of the target point in the three-dimensional model image, the second position data of the target point in other real-time two-dimensional ultrasound images (excluding the first frame) is determined. Then, based on the positional relationship of other real-time two-dimensional ultrasound images in the world coordinate system and the second position data of the target point in other real-time two-dimensional ultrasound images, the second three-dimensional position data of the target point in the world coordinate system is obtained, thus obtaining the second real-time three-dimensional position data of the target point.

[0069] The positional relationship of pixels between the three-dimensional model image and the other frames of real-time two-dimensional ultrasound images is obtained according to the following formula:

[0070]

[0071] Among them, [M x M y M z ] T For the position data of the target point in the 3D model image M, [I k,x I k,y 0] T For the k-th frame of real-time two-dimensional ultrasound image I k The location data of the target point, where k is a natural number greater than 1, J 2,k This is the registration matrix.

[0072] In some embodiments, the registration method between the first frame of real-time two-dimensional ultrasound image and other frames of real-time ultrasound images can be any one of the following: clustering matching of feature points and feature regions, and fusion of region correlation algorithm or pyramid algorithm, or a fusion algorithm of these.

[0073] In some embodiments, the registration method between the three-dimensional model image and the real-time two-dimensional ultrasound images of other frames besides the first frame can be any one of the following: clustering matching of feature points and feature regions, and fusion of region correlation algorithm or pyramid algorithm, or a fusion algorithm of these.

[0074] In this embodiment, the method of simultaneously fusing first real-time 3D position data and second real-time 3D position data using Kalman filtering to predict the kinematic model of the target point includes:

[0075] Based on the different update frequencies and reliability of the first and second real-time 3D position data, a Kalman filter update iteration method is designed, and the final 3D position data of the target point and the kinematic model of the target point are obtained by fusion.

[0076] During the tracking phase, the control processor 105 determines the tracking path of the ultrasound probe based on the kinematic model of the target point in respiratory motion, and controls the robotic arm to move the ultrasound probe along the tracking path. Specifically, a point in the ultrasound image is selected as the target control point, and the ultrasound probe is moved by the ultrasound probe controller in the front-end scanning mechanism so that the target control point coincides with the target point of the respiratory organ.

[0077] Because the kinematic model obtained during the planning phase is lagging due to the computational time required for image registration and Kalman filtering, it is necessary to appropriately select the iteration method of the kinematic model in the Kalman filter so that the predicted position of the target point in the kinematic model is after the actual time. Let the position of the target point obtained at time t in the world coordinate system {E} be... It is a lagging target point position, while the target control point in the control processor 105, continuing the previous motion plan, may have moved to a new position of t+Δt, and is set as... E g t+Δt Therefore, in order to ensure that the target point remains within a fixed position range on the scanning plane, trajectory calculation needs to predict the state after t+Δt.

[0078] In some embodiments, the control processor 105 plans the movement of the front-end scanning mechanism 101 and the robotic arm 103 to control the ultrasound probe 102, including:

[0079] If the time difference for state updates is set to h, then the predicted motion state of the target point should be: And this makes nh > Δt. Therefore, as Figure 6 At time t, the kinematic model of the target point can be used to obtain the kinematics from the target point. arrive Traj t (t), then using the current target control point position E g t+Δt The motion trajectory of the robotic arm was fitted using traj. f (t) makes the end of the robotic arm's motion trajectory tangent to the predicted motion trajectory, and controls the front-end scanning mechanism 101 and the robotic arm 103 to make the target control point coincide with the target point, and updates the control in real time.

[0080] In one embodiment, the target control point can be selected as the center point in the ultrasound scanning plane to ensure the scanning clarity and good tracking effect of the target point in the ultrasound scanning image.

[0081] Please refer to Figure 7The present invention also provides a method for coinciding respiratory motion planes based on ultrasound images. This method is applied to a control processor and includes steps 201 to 206, wherein steps 201 and 202 are preparation stage steps, steps 203, 204 and 205 are planning stage steps, and step 206 is a tracking stage step, which will be described in detail below.

[0082] Step 201: Acquire three-dimensional ultrasound images of the respiratory motor organs at at least two different respiratory phases, and calculate the direction of movement of the respiratory motor organs based on the three-dimensional ultrasound images; wherein, the respiratory phase is the phase corresponding to different respiratory moments within a respiratory cycle.

[0083] Step 202: Based on the direction of movement of the respiratory motor organs, control and adjust the angle of the ultrasound direction adjustment mechanism so that the scanning plane of the ultrasound probe is consistent with the direction of movement of the respiratory motor organs.

[0084] Step 203: Obtain a three-dimensional model image of the respiratory musculoskeletal system, wherein the three-dimensional model image contains the target point in the respiratory musculoskeletal system selected by the user and the position data of the target point in the three-dimensional model image.

[0085] Step 204: Acquire multiple real-time two-dimensional ultrasound images of the respiratory and motor organs using an ultrasound probe. The first real-time two-dimensional ultrasound image in the multiple real-time two-dimensional ultrasound images contains the target point selected by the user and the position data of the target point in the first real-time two-dimensional ultrasound image.

[0086] Step 205: Register multiple frames of real-time two-dimensional ultrasound images of respiratory motor organs, and / or register three-dimensional model images with multiple frames of real-time two-dimensional ultrasound images to obtain three-dimensional position data of the target point in respiratory motion.

[0087] Step 206: Based on the three-dimensional position data of the target point during respiratory motion, determine the tracking path of the ultrasound probe, and use the ultrasound probe controller in the front-end scanning mechanism to control the ultrasound probe to move along the tracking path.

[0088] It should be noted that the specific implementation of the method steps in the above embodiment has been described in detail in the above embodiment, and will not be repeated here.

[0089] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0090] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. An ultrasound image-based respiratory motion plane co-registration system, characterized by, The application relates to a respiratory motion organ tracking system, comprising: a front-end scanning mechanism, comprising an ultrasonic probe and an ultrasonic probe controller, the ultrasonic probe being used to collect two-dimensional ultrasonic images of a scanning plane in a respiratory motion organ; the ultrasonic probe controller being used to control the ultrasonic probe movement, the scanning plane of the ultrasonic probe being parallel to the lateral plane of the front-end scanning mechanism during the movement; wherein the y-axis of the front-end scanning mechanism is always parallel to the scanning plane of the ultrasonic probe during the movement, the y-axis of the front-end scanning mechanism being a preset direction in the front-end scanning mechanism; wherein the ultrasonic probe has at least one degree of freedom relative to the front-end scanning mechanism; a mechanical arm; an ultrasonic direction adjusting mechanism, connected between the front-end scanning mechanism and the mechanical arm, and used to adjust the angle of the front-end scanning mechanism in the spatial position; a man-machine interaction device, used for a user to select a target point in the respiratory motion organ which needs to be tracked; a control processor; wherein: the respiratory motion plane coincidence system has a preparation stage, a planning stage and a tracking stage; in the preparation stage, the control processor acquires three-dimensional ultrasonic images of the respiratory motion organ at least at two different respiratory phases, calculates the motion direction of the respiratory motion organ according to the three-dimensional ultrasonic images; the respiratory phase is a phase corresponding to different breathing moments in a breathing cycle; the control processor controls the angle of the ultrasonic direction adjusting mechanism according to the motion direction of the respiratory motion organ, so that the scanning plane of the ultrasonic probe is consistent with the motion direction of the respiratory motion organ; in the planning stage, the control processor acquires a three-dimensional model image of the respiratory motion organ, the three-dimensional model image containing the target point selected by the user and the position data of the target point in the three-dimensional model image; the control processor acquires multiple frames of real-time two-dimensional ultrasonic images of the respiratory motion organ through the ultrasonic probe, wherein the first frame of real-time two-dimensional ultrasonic images in the multiple frames of real-time two-dimensional ultrasonic images contains the target point selected by the user and the position data of the target point in the first frame of real-time two-dimensional ultrasonic images; the control processor registers the multiple frames of real-time two-dimensional ultrasonic images of the respiratory motion organ and / or registers the three-dimensional model image and the multiple frames of real-time two-dimensional ultrasonic images, and acquires three-dimensional position data of the target point in the respiratory motion; in the tracking stage, the control processor determines the tracking path of the ultrasonic probe according to the three-dimensional position data of the target point in the respiratory motion, and controls the ultrasonic probe to move according to the tracking path by using the ultrasonic probe controller in the front-end scanning mechanism.

2. The respiratory motion plane coincidence system of claim 1, wherein, the acquisition of the three-dimensional ultrasonic images of the respiratory motion organ at least at two different respiratory phases comprises: controlling the ultrasonic probe controller to rotate the ultrasonic probe to scan when the respiratory motion organ is at multiple different respiratory phases, and acquiring multiple two-dimensional ultrasonic images collected by the ultrasonic probe at each different respiratory phase; The multiple two-dimensional ultrasound images corresponding to each respiratory phase are fused to obtain a three-dimensional ultrasound image corresponding to each respiratory phase, so as to obtain the three-dimensional ultrasound images of the respiratory motion organ at at least two different respiratory phases.

3. The respiratory motion plane coincidence system of claim 1, wherein, The calculating the motion direction of the respiratory motion organ according to the three-dimensional ultrasound image comprises: The calculating the motion direction of the respiratory motion organ according to the three-dimensional ultrasound image comprises:

4. The respiratory motion piaiin re-coiinence system of claim 3, wherein, The calculating the motion direction of the respiratory motion organ according to the three-dimensional ultrasound image comprises: The center points of all the three-dimensional ultrasound images are obtained, the center points of the multiple three-dimensional ultrasound images are paired two by two, the spatial displacement vectors of the respiratory motion organ at the corresponding two respiratory phases are determined according to the difference between the positions of the center points of the two three-dimensional ultrasound images in each pair in the three-dimensional space, so as to obtain at least one spatial displacement vector; The spatial displacement vectors exceeding one are averaged to obtain the motion direction vector of the respiratory motion organ, and the angle between the motion direction vector of the respiratory motion organ and the sagittal plane of the patient is determined based on the motion direction vector of the respiratory motion organ.

5. The respiratory motion plane coincidence system of claim 3, wherein, The ultrasound direction adjusting mechanism comprises an angle measuring device, and the controlling the ultrasound direction adjusting mechanism to adjust the scanning plane of the ultrasound probe to be consistent with the motion direction of the respiratory motion organ according to the motion direction of the respiratory motion organ comprises: The angle of the ultrasound direction adjusting mechanism is controlled to be consistent with the angle between the motion direction vector of the respiratory motion organ and the sagittal plane of the patient according to the motion direction vector of the respiratory motion organ and the angle between the motion direction vector of the respiratory motion organ and the sagittal plane of the patient, so that the angle displayed by the angle measuring device is consistent with the angle, and the scanning plane of the ultrasound probe is consistent with the motion direction of the respiratory motion organ.

6. The respiratory motion plane coincidence system of claim 1, wherein, The obtaining the three-dimensional ultrasound images of the respiratory motion organ at at least two different respiratory phases comprises: In the case that the patient can pause breathing, the breathing of the patient is paused, and the respiratory motion organ is scanned by three-dimensional ultrasound at a normal speed; In the case that the patient cannot pause breathing, the respiratory motion organ is scanned by three-dimensional ultrasound at a high speed.

7. The respiratory motion plane coincidence system of claim 1, wherein, After the controlling the mechanical arm and the ultrasound direction adjusting mechanism to make the scanning plane of the ultrasound probe consistent with the motion direction of the respiratory motion organ, the method further comprises: In the case that the patient can pause breathing, the breathing of the patient is paused again, the respiratory motion organ is scanned by three-dimensional ultrasound to obtain a new three-dimensional ultrasound image, and the user selects a target point in the new three-dimensional ultrasound image through the human-computer interaction device; In the case that the patient cannot pause breathing, the position of a known target point in the three-dimensional model image is selected as the target point.

8. The respiratory motion plane coincidence system of claim 1, wherein, The controlling the processor to register the multiple frames of real-time two-dimensional ultrasound images of the respiratory motion organ and / or to register the three-dimensional model image and the multiple frames of real-time two-dimensional ultrasound images to obtain the three-dimensional position data of the target point in the respiratory motion comprises: The multiple frames of real-time two-dimensional ultrasound images are registered to obtain a position relationship between pixel points of the other frames of real-time two-dimensional ultrasound images and the first frame of real-time two-dimensional ultrasound image; According to the position relationship between pixel points of the other frames of real-time two-dimensional ultrasound images and the first frame of real-time two-dimensional ultrasound image and the three-dimensional position data of the target point in the first frame of real-time two-dimensional ultrasound image, the first position data of the target point in the other frames of real-time two-dimensional ultrasound images is determined; and then according to the position relationship of the other frames of real-time two-dimensional ultrasound images in the world coordinate system and the first position data of the target point in the other frames of real-time two-dimensional ultrasound images, the first three-dimensional position data of the target point in the world coordinate system is obtained, so as to obtain the first real-time three-dimensional position data of the target point; The three-dimensional model image and the other frames of real-time two-dimensional ultrasound images are registered to obtain a position relationship between pixel points of the three-dimensional model image and the other frames of real-time two-dimensional ultrasound images; According to the image position relationship of the three-dimensional model image relative to the other frames of real-time two-dimensional ultrasound images and the three-dimensional position data of the target point in the three-dimensional model image, the second position data of the target point in the other frames of real-time two-dimensional ultrasound images is determined; and then according to the position relationship of the other frames of real-time two-dimensional ultrasound images in the world coordinate system and the second position data of the target point in the other frames of real-time two-dimensional ultrasound images, the second three-dimensional position data of the target point in the world coordinate system is obtained, so as to obtain the second real-time three-dimensional position data of the target point; The first real-time three-dimensional position data and the second real-time three-dimensional position data of the target point are fused to obtain the three-dimensional position data of the target point in the respiratory motion.

9. The respiratory motion plane coincidence system of claim 1, wherein, The control processor registers the multiple frames of real-time two-dimensional ultrasound images of the respiratory motion organ, and / or the control processor registers the three-dimensional model image and the multiple frames of real-time two-dimensional ultrasound images, and obtains the three-dimensional position data of the target point in the respiratory motion, including: The multiple frames of real-time two-dimensional ultrasound images are registered to obtain a position relationship between pixel points of the other frames of real-time two-dimensional ultrasound images and the first frame of real-time two-dimensional ultrasound image; According to the position relationship between pixel points of the other frames of real-time two-dimensional ultrasound images and the first frame of real-time two-dimensional ultrasound image and the three-dimensional position data of the target point in the first frame of real-time two-dimensional ultrasound image, the first position data of the target point in the other frames of real-time two-dimensional ultrasound images is determined; and then according to the position relationship of the other frames of real-time two-dimensional ultrasound images in the world coordinate system and the first position data of the target point in the other frames of real-time two-dimensional ultrasound images, the first three-dimensional position data of the target point in the world coordinate system is obtained, so as to obtain the three-dimensional position data of the target point in the respiratory motion.

10. The respiratory motion plane coincidence system of claim 1, wherein, The control processor registers the multiple frames of real-time two-dimensional ultrasound images of the respiratory motion organ, and / or the control processor registers the three-dimensional model image and the multiple frames of real-time two-dimensional ultrasound images, and obtains the three-dimensional position data of the target point in the respiratory motion, including: registering the three-dimensional model image and other frames of real-time two-dimensional ultrasound images to obtain a position relationship of pixel points between the three-dimensional model image and the other frames of real-time two-dimensional ultrasound images; determining second position data of the target point in the other frames of real-time two-dimensional ultrasound images according to the image position relationship of the three-dimensional model image relative to the other frames of real-time two-dimensional ultrasound images and the three-dimensional position data of the target point in the three-dimensional model image, and obtaining second three-dimensional position data of the target point in the world coordinate system according to the position relationship of the other frames of real-time two-dimensional ultrasound images in the world coordinate system and the second position data of the target point in the other frames of real-time two-dimensional ultrasound images, to obtain the three-dimensional position data of the target point in the respiratory motion.

11. The respiratory motion plane coincidence system of claim 8, wherein, fusing the first real-time three-dimensional position data and the second real-time three-dimensional position data of the target point includes: fusing the first real-time three-dimensional position data and the second real-time three-dimensional position data of the target point by using a Kalman filtering algorithm, and predicting a kinematic model of the target point.

12. A method of respiratory motion plane co-registration based on ultrasound images, characterized in that, The respiratory motion plane coincidence system and method can be applied to the respiratory motion plane coincidence system and method of any one of claims 1 to 11, and the respiratory motion plane coincidence method includes: a preparation stage step of obtaining three-dimensional ultrasound images of a respiratory motion organ at at least two different respiratory phases, calculating a motion direction of the respiratory motion organ according to the three-dimensional ultrasound images, and controlling an angle of the ultrasound direction adjusting mechanism to be adjusted according to the motion direction of the respiratory motion organ, so that a scanning plane of the ultrasound probe is consistent with the motion direction of the respiratory motion organ; a planning stage step of obtaining a three-dimensional model image of the respiratory motion organ, the three-dimensional model image containing a target point in the respiratory motion organ selected by a user and position data of the target point in the three-dimensional model image, obtaining multiple frames of real-time two-dimensional ultrasound images of the respiratory motion organ by using the ultrasound probe, wherein a first frame of real-time two-dimensional ultrasound image in the multiple frames of real-time two-dimensional ultrasound images contains the target point selected by the user and position data of the target point in the first frame of real-time two-dimensional ultrasound image, and registering the multiple frames of real-time two-dimensional ultrasound images of the respiratory motion organ and / or registering the three-dimensional model image and the multiple frames of real-time two-dimensional ultrasound images to obtain three-dimensional position data of the target point in the respiratory motion; a tracking stage step of determining a tracking path of the ultrasound probe according to the three-dimensional position data of the target point in the respiratory motion, and controlling the ultrasound probe to move according to the tracking path by using an ultrasound probe controller in the front-end scanning mechanism.

Images