Multi-modal image registration method and system for intelligent acupuncture robot

Abstract

The invention discloses a multi-modal image registration method and system for an intelligent acupuncture robot. The system comprises a depth camera, an ultrasonic system, an optical tracker, a central processing unit and an auxiliary module. The ultrasonic system comprises an ultrasonic imager, an ultrasonic probe and a mechanical arm; the ultrasonic probe is mounted at the tail end of the mechanical arm and is connected with the ultrasonic imager; the central processing unit is respectively connected with the depth camera, the ultrasonic imager and the optical tracker; the auxiliary module is connected between a mechanical arm and an acupuncture needle of the intelligent acupuncture robot and comprises at least four light reflecting balls and a support used for supporting the light reflecting balls. The positions of the reflective balls are fixed, and the reflective balls are not collinear; the acupuncture needle is fixedly connected with the support. According to the intelligent acupuncture robot, the auxiliary module of at least four light reflecting balls is additionally arranged between the mechanical arm and the acupuncture needle of the intelligent acupuncture robot, the positions of the light reflecting balls in the auxiliary module are fixed, the light reflecting balls are not collinear, and multi-mode image registration can be rapidly carried out in an auxiliary mode.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal image registration, and in particular to a multimodal image registration method and system for an intelligent acupuncture robot. Background Art

[0002] The existing intelligent acupuncture robot system is equipped with a variety of image acquisition devices such as ultrasound system (including ultrasound imager, ultrasound probe, and robotic arm), depth camera, and optical tracker. In order to more accurately locate the tip of the acupuncture needle, multimodal image target registration is required. However, during the acupuncture operation of the intelligent acupuncture robot, the multimodal images acquired by these devices are taken from different shooting angles and from different imaging devices (different formats), which makes it difficult to match the image targets, making it difficult to provide the intelligent acupuncture robot with the acupoint structure and needle insertion planning required for acupuncture. Therefore, it is particularly important to study the multimodal image registration method and system of ultrasound, depth camera, and optical tracker to register images of different angles and formats. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a multimodal image registration method for an intelligent acupuncture robot.

[0004] To achieve the above purpose, the technical solution provided by the present invention is:

[0005] A multimodal image registration method for an intelligent acupuncture robot, comprising:

[0006] An auxiliary module with at least four reflective balls is installed between the mechanical arm of the intelligent acupuncture robot and the acupuncture needle. The positions of the reflective balls in the auxiliary module are fixed, and the reflective balls are not collinear.

[0007] Depth cameras, ultrasound systems, and optical trackers capture images of acupuncture needles from different angles;

[0008] Collect image data obtained by the depth camera and identify the pinpoint point to obtain its point set;

[0009] Collect the needle tip point set obtained by the optical tracker;

[0010] Perform rigid registration on the pinpoint point set obtained by the depth camera and the pinpoint point set obtained by the optical tracker to obtain the optimal rotation matrix and translation vector;

[0011] Use the optimal rotation matrix and translation vector to register the needle tip point set obtained by the optical tracker to the depth camera coordinate system;

[0012] Collecting ultrasound image data obtained by the ultrasound imager, and obtaining the position and posture of the two-dimensional section of the ultrasound image in three-dimensional space according to the coordinates and posture of the ultrasound probe at the end of the robotic arm of the ultrasound system;

[0013] Extract the features of the ultrasound image and identify the needle tip point, then combine the position and posture of the two-dimensional section of the ultrasound image in three-dimensional space to obtain its multiple positions in three-dimensional space to form a needle tip point set under the ultrasound image;

[0014] The pinpoint point set of the ultrasound image is rigidly registered with the pinpoint point set obtained by the depth camera, thereby completing the three-modal registration of the ultrasound system, optical tracker and depth camera.

[0015] Furthermore, the data of the depth camera is collected, and the pinpoint point is identified to obtain its point set, including:

[0016] The reflective balls in the auxiliary module are identified by Hough transform or YOL0 image recognition algorithm, and the coordinates and posture of the needle tip are calculated according to the position of each reflective ball in space, so as to obtain the point set under the depth camera image. The specific process is as follows:

[0017] The image coordinates (u) of any reflective ball P on the left and right camera imaging planes in the depth camera are identified by Hough transform or YOL0 image recognition algorithm. 1 , v 1 ) and (u 2 , v 2 ), according to the imaging model, we get:

[0018]

[0019] Among them, M 1 , M 2 are the internal parameters of the left and right cameras in the depth camera respectively; M r is the depth camera external parameter; x w ,y w 、z w is the three-dimensional coordinate of the reflective ball P; They are all matrix elements obtained by multiplying the left camera's intrinsic and extrinsic parameters; They are all matrix elements obtained by multiplying the intrinsic and extrinsic parameters of the right camera;

[0020] Eliminate z c1 、z c2 ,have to:

[0021] P=(A T A) -1 A T b

[0022] in,

[0023]

[0024] According to the above formula, the coordinates of the four reflective balls in space are obtained, namely P 1 , P 2 , P 3 , P 4 , determine the spatial coordinates C of the needle tip according to the four-point coordinates, and the calculation formula is as follows:

[0025]

[0026] Where T is the offset between the needle tip and the center.

[0027] Furthermore, the optical tracker determines the position of the needle tip point through the reflective ball, thereby obtaining the needle tip point set under the optical tracker.

[0028] Furthermore, by solving the rigid registration optimization problem, the pinpoint point set obtained by the depth camera and the pinpoint point set obtained by the optical tracker are rigidly registered;

[0029] The rigid registration optimization problem is as follows:

[0030]

[0031] Among them, R is the rotation matrix, t is the translation vector, x i and y(x i ) are the corresponding vertices in the pinpoint point set X obtained by the optical tracker and the pinpoint point set Y obtained by the depth camera.

[0032] Furthermore, the process of solving the rigid registration optimization problem includes:

[0033] A1. For each vertex x in the pinpoint point set X obtained by the optical tracker, find the nearest vertex y(x) in the pinpoint point set Y obtained by the depth camera as the corresponding point;

[0034] A2. Singular value decomposition to find the optimal transformation:

[0035] First subtract the centroid from each:

[0036]

[0037] is the mean value of x, is the mean value of y(x);

[0038] make Then the singular value decomposition W = UDV T , then the optimal transformation is:

[0039]

[0040] R * is the optimal rotation matrix, t * is the optimal translation vector, U is the left singular vector, V is the right singular vector, and D is the singular value;

[0041] A3. The optimal rotation matrix R * and the optimal translation vector t * In the needle point set X obtained by the optical tracker, if the error is greater than the set threshold at this time, iteration is performed again until the number of iterations reaches the set threshold or the error is less than the set threshold.

[0042] Furthermore, the process of collecting ultrasonic image data obtained by the ultrasonic imager and obtaining the position and posture of the two-dimensional section of the ultrasonic image in the three-dimensional space according to the coordinates and posture of the ultrasonic probe at the end of the mechanical arm of the ultrasonic system includes:

[0043] Get the ultrasound probe position:

[0044]

[0045] in, is the transformation matrix between the ultrasonic probe coordinate system and the acupuncture robot arm end coordinate system, which is obtained by the calibration method of the tool coordinate system; T probe is the ultrasound probe position, R E With t E are the rotation matrix and translation vector of the end position of the robotic arm of the ultrasound system respectively;

[0046] Obtain the position T of the two-dimensional section of the ultrasound image in three-dimensional space ultra :

[0047] T ultra =T z T probe

[0048] Among them, T z is the bias conversion matrix determined by the ultrasonic detection range.

[0049] Furthermore, the needle tip point in the ultrasound image is identified by Hough transform or YOL0 image recognition algorithm, and then combined with the position and posture of the two-dimensional section of the ultrasound image in three-dimensional space, multiple positions and postures in three-dimensional space are obtained to form a needle tip point set under the ultrasound image.

[0050] Furthermore, the needle tip point set of the ultrasound image and the needle tip point set obtained by the depth camera are rigidly registered based on an iterative closest point algorithm.

[0051] To achieve the above object, the present invention further provides a multimodal image registration system for an intelligent acupuncture robot, which is used to implement the above multimodal image registration method for an intelligent acupuncture robot, and includes a depth camera, an ultrasonic system, an optical tracker, a central processing unit and an auxiliary module;

[0052] The ultrasound system includes an ultrasound imager, an ultrasound probe, and a robotic arm;

[0053] The ultrasonic probe is installed at the end of the mechanical arm and connected to the ultrasonic imager;

[0054] The central processor is connected to the depth camera, the ultrasonic imager, and the optical tracker respectively;

[0055] The auxiliary module is connected between the mechanical arm of the intelligent acupuncture robot and the acupuncture needle, and includes at least four reflective balls and a bracket for supporting the reflective balls;

[0056] The positions of the reflective balls are fixed, and the reflective balls are not collinear;

[0057] The acupuncture needle is fixedly connected to the bracket.

[0058] Compared with the prior art, the principles and advantages of this technical solution are as follows:

[0059] In this technical solution, an auxiliary module with at least four reflective balls is installed between the mechanical arm and the acupuncture needle of the intelligent acupuncture robot. The positions of the reflective balls in the auxiliary module are fixed, and the reflective balls are not collinear, which can assist in rapid multimodal image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 Schematic diagram of a multimodal image registration system for an intelligent acupuncture robot (A is the acupuncture target tissue);

[0062] Figure 2 A schematic diagram of the connection between the acupuncture needle and the auxiliary module;

[0063] Figure 3 This is a principle flow chart of a multimodal image registration method for intelligent acupuncture robots;

[0064] Figure 4Schematic diagram of needle tip recognition in ultrasound images (B is the recognized needle tip of the acupuncture needle).

[0065] Reference numerals:

[0066] 1-1: Ultrasound imager, 1-2: Depth camera, 1-3: Optical tracker; 1-4: Ultrasound probe; 1-5: Reflective ball, 1-6: Bracket, 1-7: Acupuncture needle, 1-8: Robotic arm (ultrasound system), 1-9: Central processing unit. DETAILED DESCRIPTION

[0067] The present invention will be further described below in conjunction with specific embodiments:

[0068] like Figure 1 As shown, a multimodal image registration system for an intelligent acupuncture robot described in this embodiment includes a depth camera 1-2, an ultrasound system, an optical tracker 1-3, a central processor 1-9 and an auxiliary module.

[0069] Among them, the ultrasound system includes an ultrasound imager 1-1, an ultrasound probe 1-4, and a robotic arm 1-8; the ultrasound probe 1-4 is installed at the end of the robotic arm 1-8 and connected to the ultrasound imager 1-1; the central processor 1-9 is respectively connected to the depth camera 1-2, the ultrasound imager 1-1, and the optical tracker 1-3; the auxiliary module is connected between the robotic arm 1-8 and the acupuncture needle 1-7 of the intelligent acupuncture robot, and it includes four reflective balls 1-5 and a bracket 1-6 for supporting the reflective balls 1-5; the position of each reflective ball 1-5 is fixed, and each reflective ball 1-5 is not collinear; the acupuncture needle 1-7 is fixedly connected to the bottom of the bracket 1-6, such as Figure 2 shown.

[0070] like Figure 3 As shown, the working principle of this embodiment is as follows:

[0071] S1. Install an auxiliary module between the mechanical arm 1-8 and the acupuncture needle 1-7 of the intelligent acupuncture robot;

[0072] S2, depth camera 1-2, ultrasound system, optical tracker 1-3 collect images of acupuncture needles 1-7 from different angles;

[0073] S3, the central processor 1-9 collects the image data obtained by the depth camera 1-2, and identifies the pinpoint point to obtain its point set;

[0074] The specific process of this step is as follows:

[0075] The image coordinates (u) of any reflective ball P on the left and right camera imaging planes in the depth camera are identified by Hough transform or YOL0 image recognition algorithm. 1 , v 1 ) and (u2 , v 2 ), according to the imaging model, we get:

[0076]

[0077] Among them, M 1 , M 2 are the internal parameters of the left and right cameras in depth camera 1-2; M r is the external parameter of depth camera 1-2; x w ,y w 、z w is the three-dimensional coordinate of the reflective ball P; They are all matrix elements obtained by multiplying the left camera's intrinsic and extrinsic parameters; They are all matrix elements obtained by multiplying the intrinsic and extrinsic parameters of the right camera;

[0078] Eliminate z c1 、z c2 ,have to:

[0079] P=(A T A) -1 A T b

[0080] in,

[0081]

[0082] According to the above formula, the coordinates of the four reflective balls in space are obtained, namely P 1 , P 2 , P 3 , P 4 , determine the spatial coordinates C of the needle tip according to the four-point coordinates, and the calculation formula is as follows:

[0083]

[0084] Where T is the offset between the needle tip and the center.

[0085] S4. The optical tracker 1-3 determines the position of the needle tip point through the reflective ball 1-5, and the central processor 1-9 obtains the needle tip point set under the optical tracker 1-3 from the optical tracker 1-3.

[0086] S5. The central processor 1-9 performs rigid registration on the pinpoint point set obtained by the depth camera 1-2 and the pinpoint point set obtained by the optical tracker 1-3 by solving the rigid registration optimization problem, and obtains the optimal rotation matrix and translation vector;

[0087] The rigid registration optimization problem is as follows:

[0088]

[0089] Among them, R is the rotation matrix, t is the translation vector, x i and y(x i ) are the corresponding vertices in the pinpoint point set X obtained by the optical tracker 1-3 and the pinpoint point set Y obtained by the depth camera 1-2.

[0090] The process of solving the rigid registration optimization problem includes:

[0091] A1. For each vertex x in the pinpoint point set X obtained by the optical tracker 1-3, find the nearest vertex y(x) in the pinpoint point set Y obtained by the depth camera 1-2 as the corresponding point;

[0092] A2. Singular value decomposition to find the optimal transformation:

[0093] First subtract the centroid from each:

[0094]

[0095] is the mean value of x, is the mean value of y(x);

[0096] make Then the singular value decomposition W = UDV T , then the optimal transformation is:

[0097]

[0098] R * is the optimal rotation matrix, t * is the optimal translation vector, U is the left singular vector, V is the right singular vector, and D is the singular value;

[0099] A3, CPU 1-9 converts the optimal rotation matrix R * and the optimal translation vector t * In the needle point set X obtained by acting on the optical tracker 1-3, if the error is greater than the set threshold at this time, iteration is performed again until the number of iterations reaches the set threshold or the error is less than the set threshold.

[0100] S6, the central processor 1-9 uses the optimal rotation matrix and translation vector to register the needle tip point set obtained by the optical tracker 1-3 to the coordinate system of the depth camera 1-2;

[0101] S7, the central processor 1-9 collects the ultrasound image data obtained by the ultrasound imager 1-1, and obtains the position and posture of the two-dimensional section of the ultrasound image in the three-dimensional space according to the coordinates and posture of the ultrasound probe 1-4 at the end of the mechanical arm 1-8 of the ultrasound system;

[0102] The specific process of this step includes:

[0103] Get ultrasound probe 1-4 postures:

[0104]

[0105] in, is the transformation matrix between the coordinate system of the ultrasonic probe 1-4 and the end coordinate system of the acupuncture robot arm 1-8, which is obtained by the calibration method of the tool coordinate system; T probe is the ultrasound probe position 1-4, R E With t E are the rotation matrix and translation vector of the end positions of the robotic arms 1-8 of the ultrasound system respectively;

[0106] Obtain the position T of the two-dimensional section of the ultrasound image in three-dimensional space ultra :

[0107] T ultra =T z T probe

[0108] Among them, T z is the bias conversion matrix determined by the ultrasonic detection range.

[0109] S8, the central processor 1-9 extracts the features of the ultrasound image and identifies the needle tip point in the ultrasound image through Hough transform or YOL0 image recognition algorithm, and then combines the posture of the two-dimensional section of the ultrasound image in the three-dimensional space to obtain its multiple postures in the three-dimensional space to form a needle tip point set under the ultrasound image.

[0110] S9. The central processor 1-9 performs rigid registration of the pinpoint point set of the ultrasound image with the pinpoint point set obtained by the depth camera 1-2 based on an iterative closest point algorithm, thereby completing the three-modal registration of the ultrasound system, the optical tracker 1-3 and the depth camera 1-2.

[0111] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal image registration method for an intelligent acupuncture robot, characterized in that: include: An auxiliary module with at least four reflective balls is installed between the mechanical arm of the intelligent acupuncture robot and the acupuncture needle. The positions of the reflective balls in the auxiliary module are fixed, and the reflective balls are not collinear. Depth cameras, ultrasound systems, and optical trackers capture images of acupuncture needles from different angles; Collect image data obtained by the depth camera and identify the pinpoint point to obtain its point set; Collect the needle tip point set obtained by the optical tracker; Perform rigid registration on the pinpoint point set obtained by the depth camera and the pinpoint point set obtained by the optical tracker to obtain the optimal rotation matrix and translation vector; Use the optimal rotation matrix and translation vector to register the needle tip point set obtained by the optical tracker to the depth camera coordinate system; Collecting ultrasound image data obtained by the ultrasound imager, and obtaining the position and posture of the two-dimensional section of the ultrasound image in three-dimensional space according to the coordinates and posture of the ultrasound probe at the end of the robotic arm of the ultrasound system; Extract the features of the ultrasound image and identify the needle tip point, then combine the position and posture of the two-dimensional section of the ultrasound image in three-dimensional space to obtain its multiple positions in three-dimensional space to form a needle tip point set under the ultrasound image; The pinpoint point set of the ultrasound image is rigidly registered with the pinpoint point set obtained by the depth camera, thereby completing the three-modal registration of the ultrasound system, optical tracker and depth camera.

2. The multimodal image registration method for an intelligent acupuncture robot according to claim 1, characterized in that: Collect data from the depth camera and identify the pinpoint to obtain its point set, including: The reflective balls in the auxiliary module are identified by Hough transform or YOLO image recognition algorithm, and the coordinates and posture of the needle tip are calculated according to the position of each reflective ball in space, so as to obtain the point set under the depth camera image. The specific process is as follows: The image coordinates (u1, v1) and (u2, v2) of any reflective ball P on the left and right camera imaging planes in the depth camera are identified by Hough transform or YOLO image recognition algorithm. According to the imaging model, we get: Among them, M1 and M2 are the internal parameters of the left and right cameras in the depth camera respectively; M r is the depth camera external parameter; x w ,y w 、z w is the three-dimensional coordinate of the reflective ball P; They are all matrix elements obtained by multiplying the left camera's intrinsic and extrinsic parameters; They are all matrix elements obtained by multiplying the intrinsic and extrinsic parameters of the right camera; Eliminate z c1 、z c2 ,have to: P=(A T A) -1 A T b in, According to the above formula, the coordinates of the four reflective balls in space are obtained, namely P1, P2, P3, and P4. The spatial coordinate C of the needle tip is determined according to the coordinates of the four points. The calculation formula is as follows: Where T is the offset between the needle tip and the center.

3. The multimodal image registration method for an intelligent acupuncture robot according to claim 1, characterized in that: The optical tracker determines the position of the needle tip through the reflective ball, thereby obtaining the needle tip point set under the optical tracker.

4. The multimodal image registration method for an intelligent acupuncture robot according to claim 1, characterized in that: By solving the rigid registration optimization problem, the pinpoint point set obtained by the depth camera and the pinpoint point set obtained by the optical tracker are rigidly registered; The rigid registration optimization problem is as follows: Among them, R is the rotation matrix, t is the translation vector, x i and y(x i ) are the corresponding vertices in the pinpoint point set X obtained by the optical tracker and the pinpoint point set Y obtained by the depth camera.

5. The multimodal image registration method for an intelligent acupuncture robot according to claim 1, characterized in that: The process of solving the rigid registration optimization problem includes: A1. For each vertex x in the pinpoint point set X obtained by the optical tracker, find the nearest vertex y(x) in the pinpoint point set Y obtained by the depth camera as the corresponding point; A2. Singular value decomposition to find the optimal transformation: First subtract the centroid from each: is the mean value of x, is the mean value of y(x); make Then the singular value decomposition W = UDV T , then the optimal transformation is: R * is the optimal rotation matrix, t * is the optimal translation vector, U is the left singular vector, V is the right singular vector, and D is the singular value; A3. The optimal rotation matrix R * and the optimal translation vector t * In the needle point set X obtained by the optical tracker, if the error is greater than the set threshold at this time, iteration is performed again until the number of iterations reaches the set threshold or the error is less than the set threshold.

6. The multimodal image registration method for an intelligent acupuncture robot according to claim 1, characterized in that: The process of collecting ultrasound image data obtained by an ultrasound imager and obtaining the position and posture of a two-dimensional section of the ultrasound image in three-dimensional space according to the coordinates and posture of the ultrasound probe at the end of the mechanical arm of the ultrasound system includes: Get the ultrasound probe position: in, is the transformation matrix between the ultrasonic probe coordinate system and the acupuncture robot arm end coordinate system, which is obtained by the calibration method of the tool coordinate system; T probe is the ultrasound probe position, R E With t E are the rotation matrix and translation vector of the end position of the robotic arm of the ultrasound system respectively; Obtain the position T of the two-dimensional section of the ultrasound image in three-dimensional space ultra : T ultra =T z T probe Among them, T z is the bias conversion matrix determined by the ultrasonic detection range.

7. The multimodal image registration method for an intelligent acupuncture robot according to claim 1, characterized in that: The needle tip point in the ultrasound image is identified by Hough transform or YOLO image recognition algorithm, and then combined with the position and posture of the two-dimensional section of the ultrasound image in three-dimensional space, its multiple positions in three-dimensional space are obtained to form the needle tip point set under the ultrasound image.

8. The multimodal image registration method for an intelligent acupuncture robot according to claim 1, characterized in that: The needle tip point set of the ultrasound image and the needle tip point set obtained by the depth camera are rigidly registered based on the iterative closest point algorithm.

9. A multimodal image registration system for an intelligent acupuncture robot, comprising a depth camera, an ultrasound system, and an optical tracker; the ultrasound system comprises an ultrasound imager, an ultrasound probe, and a mechanical arm; the ultrasound probe is mounted at the end of the mechanical arm and connected to the ultrasound imager, characterized in that: A multimodal image registration method for an intelligent acupuncture robot for implementing any one of claims 1 to 8, which also includes a central processing unit and an auxiliary module; The central processor is connected to the depth camera, the ultrasonic imager, and the optical tracker respectively; The auxiliary module is connected between the mechanical arm of the intelligent acupuncture robot and the acupuncture needle, and includes at least four reflective balls and a bracket for supporting the reflective balls; The positions of the reflective balls are fixed, and the reflective balls are not collinear; The acupuncture needle is fixedly connected to the bracket.

Images