Ultrasonic probe posture positioning system and method for medical remote ultrasonic scanning robot

Abstract

The present invention discloses an ultrasonic probe posture positioning system and method for a medical remote ultrasonic scanning robot. The system includes an ultrasonic probe, a depth camera, an image preprocessing module, a scanning target positioning module, a coordinate conversion module, a posture positioning module, a posture correction module, a force feedback correction module, a robotic arm, and a matching fixture. The present invention acquires an image containing a target point, locates the ultrasonic scanning target area using a target positioning method based on the fusion of visual information and depth information, then corrects the probe posture using a multi-scale compensation method based on time and space. Finally, the probe posture is further optimized in combination with a force feedback sensor, thereby achieving ultrasonic probe posture positioning for the medical remote ultrasonic scanning robot. This system can achieve real-time, accurate, and convenient posture positioning using low-cost sensors, greatly improving positioning accuracy and expanding the autonomy of the medical ultrasonic scanning robot.

Description

Technical Field

[0001] The present invention belongs to the field of robotics and relates to a method for positioning an ultrasonic probe, and in particular to a method and system for positioning an ultrasonic probe of a medical remote ultrasonic scanning robot. Background Art

[0002] In medical diagnosis and testing, ultrasound imaging offers advantages over other imaging modalities such as CT, MRI, and X-rays, including low cost, non-destructiveness, high real-time performance, and ease of use. The development of ultrasound scanning robots has significantly alleviated the shortage of medical resources, reduced workload for physicians, and effectively reduced the risk of indirect illness. However, the accuracy of ultrasound probe positioning directly impacts imaging quality. Currently, commonly used ultrasound probe positioning methods include visual positioning, point cloud positioning, and electromagnetic positioning. Existing technologies have several drawbacks.

[0003] CN112336374B discloses a method for precisely positioning an ultrasonic probe using a binocular camera. With the aid of coded markers, the binocular camera accurately locates the probe's posture. However, this method can only determine the relative position of the probe and requires coding assistance, resulting in significant limitations. CN113940699A discloses a method for self-positioning an ultrasonic probe, utilizing reflective particle lines arranged within a designed ultrasonic coupling gasket. This method is costly and poorly suited for medical ultrasound scanning. Therefore, there is an urgent need for a simple, fast, accurate, and real-time method for positioning an ultrasonic probe's posture to serve ultrasonic scanning robots. Summary of the Invention

[0004] In order to solve the problem that a medical ultrasonic scanning robot has a large positioning error of an ultrasonic probe before scanning, resulting in poor ultrasonic imaging effect, the present invention provides an ultrasonic probe positioning system and method for a medical remote ultrasonic scanning robot.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] An ultrasound probe posture positioning system for a medical remote ultrasound scanning robot includes an ultrasound probe, a depth camera, an image preprocessing module, a scanning target positioning module, a coordinate conversion module, a posture positioning module, a posture correction module, a force feedback correction module, a robotic arm, and a matching fixture, wherein:

[0007] The ultrasonic probe is used to collect ultrasonic image information;

[0008] The depth camera is used to capture an image of an area containing a scan target point and obtain depth information of each pixel on the image;

[0009] The image preprocessing module is used to perform related preprocessing operations such as quality detection, size unification, and contrast enhancement on the images collected by the depth camera;

[0010] The scanning target positioning module is used to automatically identify and locate the scanning target point according to manual settings, output the two-dimensional coordinates of the scanning target point, and calculate the first coordinates of the landing point in combination with the depth information;

[0011] The coordinate conversion module is used to convert the three-dimensional coordinates of the landing point in the depth camera coordinate system into the robotic arm base coordinate system to obtain a second coordinate as an input for driving the robotic arm movement;

[0012] The posture positioning module is used to obtain the three-dimensional posture information of the ultrasonic scanning probe at the second coordinate point, that is, the three angles between the probe and the coordinate axis of the robot arm base;

[0013] The posture correction module is used to correct the three-dimensional posture information of the ultrasound probe based on multi-scale compensation, thereby reducing positioning errors;

[0014] The force feedback correction module is used to combine the force feedback information of the end-effector of the manipulator and further optimize the position of the ultrasound probe through a compensation strategy to ensure that higher quality ultrasound images can be obtained;

[0015] The matching fixture is used to fix the depth camera and the ultrasonic probe at the end of the robotic arm.

[0016] A method for positioning the ultrasound probe of a medical remote ultrasound scanning robot using the above system comprises the following steps:

[0017] Step 1: Use a depth camera installed at a fixed position on the robotic arm to capture an image of the patient's area to be scanned, and calibrate the color channel and depth channel of the depth camera;

[0018] Step 2: Input the image captured by the depth camera into the image preprocessing module to change the image size, enhance the contrast, and perform quality inspection;

[0019] Step 3: Input the image processed by the image preprocessing module into the scanning target positioning module to obtain the two-dimensional coordinates P0 (x, y) of the landing coordinate point;

[0020] Step 4: Combine the depth data value d of the landing coordinate point and map the landing coordinate point to the three-dimensional coordinate in the camera coordinate system. This coordinate is called the first coordinate P1.

[0021] Step 5: Determine the second coordinate P2 of the first coordinate in the robotic arm base coordinate system through the coordinate conversion module;

[0022] Step 6: Input the second coordinate into the posture positioning module to obtain the three-dimensional posture angle of the ultrasound probe, and then combine it with the second coordinate to finally output the three-dimensional posture P of the ultrasound probe. prode ;

[0023] Step 7: Input the three-dimensional pose of the ultrasonic probe of the ultrasonic scanning robot into the pose correction module to obtain the compensated three-dimensional pose P';

[0024] Step 8: After the probe contacts the human skin, the force feedback correction module is used to correct the three-dimensional position of the ultrasound probe again, and the final three-dimensional position P of the ultrasound probe is output.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] The present invention captures an image containing a target point, locates the ultrasound scanning target area using a target positioning method based on the fusion of visual and depth information, obtains a plane normal vector near the landing coordinate point as the ultrasound probe's posture, and then corrects the probe's posture using a multi-scale compensation method based on time and space. Finally, the probe's posture is further optimized by combining a force feedback sensor. This achieves ultrasound probe posture positioning for a medical remote ultrasound scanning robot. Using low-cost sensors, this method achieves real-time, accurate, and convenient posture positioning, greatly improving positioning accuracy and expanding the autonomy of the medical ultrasound scanning robot. This provides a solid foundation for enabling automatic ultrasound scanning robots to perform high-quality ultrasound scanning inspections while ensuring patient and system safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the method for positioning the ultrasound probe of a medical remote ultrasound scanning robot in an embodiment of the present invention:

[0028] Figure 2 This is the installation method of the ultrasound probe posture positioning system of the medical remote ultrasound scanning robot in the embodiment of the present invention;

[0029] Figure 3 Schematic diagram of an ultrasound probe posture positioning system of a medical remote ultrasound scanning robot in an embodiment of the present invention;

[0030] Figure 4 1 is a schematic diagram of solving the plane normal vector of the ultrasound probe of the medical remote ultrasound scanning robot in an embodiment of the present invention;

[0031] Figure 5 2 is a schematic diagram of a multi-scale compensation method in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0033] The present invention provides an ultrasonic probe posture positioning system for a medical remote ultrasonic scanning robot, such as Figure 3 As shown, the system includes an ultrasonic probe, a depth camera, an image preprocessing module, a scanning target positioning module, a coordinate conversion module, a posture positioning module, a posture correction module, a force feedback correction module, a robotic arm and a matching fixture, wherein:

[0034] The ultrasonic probe is used to collect ultrasonic image information;

[0035] The depth camera is used to capture an image of an area containing a scan target point and obtain depth information of each pixel on the image;

[0036] The image preprocessing module is used to perform related preprocessing operations such as quality detection, size unification, and contrast enhancement on the images collected by the depth camera;

[0037] The scanning target positioning module is used to automatically identify and locate the scanning target point according to manual settings, output the two-dimensional coordinates of the scanning target point, and calculate the first coordinates of the landing point in combination with the depth information;

[0038] The coordinate conversion module is used to convert the three-dimensional landing point coordinates in the depth camera coordinate system into the robotic arm base coordinate system to obtain a second coordinate as an input for driving the robotic arm movement;

[0039] The posture positioning module is used to obtain the three-dimensional posture information of the ultrasonic scanning probe at the second coordinate point, that is, the three angles between the probe and the coordinate axis of the robot arm base;

[0040] The posture correction module is used to correct the three-dimensional posture information of the ultrasound probe based on multi-scale compensation, thereby reducing positioning errors;

[0041] The force feedback correction module is used to combine the force feedback information of the end-effector of the manipulator and further optimize the position of the ultrasound probe through a compensation strategy to ensure that higher quality ultrasound images can be obtained;

[0042] The matching fixture is used to fix the depth camera and the ultrasonic probe at the end of the robotic arm.

[0043] The present invention also provides a method for positioning the ultrasound probe of a medical remote ultrasound scanning robot using the above system, the method comprising the following steps:

[0044] Step 1: Use a depth camera installed at a fixed position on the robotic arm to capture an image of the patient's area to be scanned, and calibrate the color channel and depth channel of the depth camera.

[0045] Step 2: Input the image captured by the depth camera into the image preprocessing module to change the image size, enhance the contrast, and perform quality inspection.

[0046] Step 3: Input the image processed by the image preprocessing module into the scanning target positioning module, and use the target detection algorithm based on Yolov4 to locate the coordinates of the landing point in the scanning area in real time to obtain the two-dimensional coordinates P0 (x, y) of the landing coordinate point.

[0047] Step 4: Combine the depth data value d of the landing coordinate point and map the landing coordinate point to the three-dimensional coordinate in the camera coordinate system. This coordinate is called the first coordinate P1. The calculation formula for the first coordinate P1 is shown in formula (1):

[0048]

[0049] Where f represents the focal length of the infrared camera of the depth camera.

[0050] Step 5: Determine the second coordinate P2 of the first coordinate in the coordinate system of the robot base through the coordinate conversion module. According to the rotation matrix from the camera coordinate system to the ultrasound probe coordinate system And the rotation transformation matrix from the ultrasound probe coordinate system to the robotic arm base coordinate system To obtain the second coordinate P2, the calculation formula is as shown below:

[0051]

[0052] Where: rotation matrix The rotation matrix is ​​determined by the position of the camera mounted on the robotic arm. Determined by the size of the robotic arm.

[0053] Step 6: Input the second coordinate into the posture positioning module to obtain the three-dimensional posture angle of the ultrasound probe, and then combine it with the second coordinate to finally output the three-dimensional posture P of the ultrasound probe. prode The three-dimensional pose of the ultrasound probe consists of the probe's landing coordinates P2 and attitude angle. The attitude angle is determined by the normal vector to the human chest skin surface near the landing point. The normal vector is determined by the plane formed by three non-collinear points in the neighborhood near the landing point. The angle between the normal vector and the coordinate axis of the robotic arm base constitutes the attitude angle of the ultrasound probe.

[0054] Step seven: Input the three-dimensional posture of the ultrasonic probe of the ultrasonic scanning robot into the posture correction module to obtain the compensated three-dimensional posture P'. The function of the posture correction module is to eliminate the positioning error caused by the small-scale movement of the patient's body, so as to improve the positioning accuracy as much as possible. The method adopted is a three-dimensional probe posture positioning algorithm based on multi-scale compensation, which compensates the three-dimensional probe posture from two dimensions of space and time. Spatial compensation is to use four pixel points with a distance of Δ near the second coordinate point to obtain the normal vectors of the four pixel points, move the starting points of these normal vectors to the landing point P2, sum and normalize them to obtain the compensated normal vector. In terms of time, the normal vectors after spatial compensation for five consecutive sampling times Δt are summed and normalized to obtain the final normal vector. The calculation formula is shown in formula (3):

[0055]

[0056] Where α represents the normal vector, Represents the normal vector of the target landing point, and the normal vectors of other symbols are the normal vectors involved in compensation.

[0057] Step 8: After the probe contacts the human skin, the force feedback correction module is used to correct the three-dimensional position of the ultrasound probe again, and the final three-dimensional position P of the ultrasound probe is output, so that high-quality ultrasound images can be obtained while ensuring the safety of the human body and the robot system. The force feedback correction module uses the expected value of the pressure felt by the end effector of the robotic arm to correct the three-dimensional position of the ultrasound probe, and uses the compensation strategy of formula (4) to further optimize the three-dimensional position of the ultrasound probe:

[0058]

[0059] where k p is the proportionality coefficient, k d is the differential coefficient, pressure F end Feedback comes from the pressure sensor at the end of the robot arm. Finally, the component of the compensation amount L in the robot arm base coordinate system is calculated to obtain the final 3D pose.

[0060] Example:

[0061] like Figure 1 As shown, this embodiment performs scanning target positioning of the lung ultrasound automatic scanning robot according to the following steps:

[0062] Step 1: Use a depth camera installed at a fixed position on the robotic arm to capture an image of the patient's area to be scanned. In this embodiment, an ultrasonic coupling agent needs to be applied to the patient's area to be scanned in advance, and the color channel and depth channel of the depth camera are calibrated so that they are in the same coordinate system.

[0063] Step 2: Input the collected image into the image preprocessing module. In this embodiment, the image size is converted to 512×512 after the image preprocessing module, and the blurred image is removed and the contrast of the retained image is improved.

[0064] Step 3: Input the processed image into the scanning target positioning module, which can calculate the two-dimensional coordinates P0(x, y) of the landing point. In this embodiment, the method for determining the coordinates of the landing point is to identify and locate the target based on the pre-set scanning target using the target positioning method based on the deep convolutional neural network YoLov5.

[0065] Step 4: Combine the depth data value d of the landing coordinate point and map the landing coordinate point to the three-dimensional coordinate in the camera coordinate system. This coordinate is called the first coordinate P1. The calculation formula for obtaining the first coordinate is shown in formula (1).

[0066] Step 5: Determine the second coordinate P2 of the first coordinate in the robot base coordinate system through the coordinate conversion module. The calculation formula is shown in formula (2).

[0067] Step 6: Input the second coordinate into the posture positioning module to obtain the three-dimensional posture angle of the ultrasound probe, and then combine it with the second coordinate to finally output the three-dimensional posture P of the ultrasound probe. prode In this embodiment, the normal vector solution diagram is as follows Figure 4 shown.

[0068] Step 7: Input the three-dimensional posture of the ultrasonic probe of the ultrasonic scanning robot into the posture correction module to obtain the compensated three-dimensional posture P'. In this embodiment, the multi-scale compensated three-dimensional probe posture positioning algorithm is as follows: Figure 5 shown.

[0069] Step 8. After the probe contacts the human skin, the force feedback correction module is used to correct the three-dimensional posture of the ultrasound probe again. The compensation strategy of formula (4) is used to further optimize the three-dimensional posture of the ultrasound probe. Finally, the component of the compensation amount L in the coordinate system of the robot base is calculated to obtain the final three-dimensional posture P of the ultrasound probe, thereby obtaining high-quality ultrasound images while ensuring the safety of the human body and the robot system.

[0070] Taking a medical remote ultrasound scanning robot scanning a patient's lungs as an example, five characteristic points on the patient's chest are typically scanned to obtain an ultrasound image. When using the ultrasound probe posture positioning method of this embodiment, the three-dimensional posture positioning error of the ultrasound probe is shown in Table 1, where the distance error is the Euclidean distance between the probe landing point and the actual target point, and the angle error is the absolute value of the difference between the probe posture angle and the actual posture angle. The average distance error is approximately 6.38 mm, and the average angle error is approximately 0.15 rad, which is within the error range of lung ultrasound scanning and can provide high-precision positioning for subsequent ultrasound image acquisition.

[0071] Table 1

[0072]

Claims

1. An ultrasonic probe posture positioning system for a medical remote ultrasonic scanning robot, characterized in that The system includes an ultrasonic probe, a depth camera, an image preprocessing module, a scanning target positioning module, a coordinate conversion module, a posture positioning module, a posture correction module, a force feedback correction module, a robotic arm and a matching fixture, wherein: The ultrasonic probe is used to collect ultrasonic image information; The depth camera is used to capture an image of an area containing a scan target point and obtain depth information of each pixel on the image; The image preprocessing module is used to perform quality detection, size unification, and contrast enhancement related preprocessing operations on the images collected by the depth camera; The scanning target positioning module is used to automatically identify and locate the scanning target point according to manual settings, output the two-dimensional coordinates of the scanning target point, and calculate the first coordinates of the landing point in combination with the depth information; The coordinate conversion module is used to convert the three-dimensional coordinates of the landing point in the depth camera coordinate system into the robotic arm base coordinate system to obtain a second coordinate as an input for driving the robotic arm movement; The posture positioning module is used to obtain three-dimensional posture information of the ultrasonic scanning probe at the second coordinate point; The posture correction module is used to correct the three-dimensional posture information of the ultrasound probe based on multi-scale compensation, thereby reducing the positioning error; the compensation in space is to use the second coordinate point near the distance of Calculate the normal vectors of the four pixel points and move the starting points of these normal vectors to the landing point. In terms of time, the normal vector after compensation is obtained by summing and normalizing. In terms of time, the normal vector after compensation is obtained by summing and normalizing. The normal vectors after spatial compensation are summed and normalized to obtain the final normal vector ; The force feedback correction module is used to combine the force feedback information of the end effector of the manipulator and further optimize the position of the ultrasound probe through the designed control strategy to ensure that higher quality ultrasound images can be obtained; The matching fixture is used to fix the depth camera and the ultrasonic probe at the end of the robotic arm.

2. The ultrasonic probe posture positioning system of the medical remote ultrasonic scanning robot according to claim 1 is characterized in that The compensation strategy is as follows: in is the compensation amount, is the proportionality coefficient, is the differential coefficient, pressure Feedback comes from a pressure sensor at the end of the robotic arm.

3. A method for positioning the ultrasonic probe of a medical remote ultrasonic scanning robot using the system according to any one of claims 1-2, characterized in that The method comprises the following steps: Step 1: Use a depth camera installed at a fixed position on the robotic arm to capture an image of the patient's area to be scanned, and calibrate the color channel and depth channel of the depth camera; Step 2: Input the image captured by the depth camera into the image preprocessing module to change the image size, enhance the contrast, and perform quality inspection; Step 3: Input the image processed by the image preprocessing module into the scanning target positioning module to obtain the two-dimensional coordinates of the landing coordinate point ; Step 4: Combine the depth data value of the landing coordinate point , map the landing coordinate point to the three-dimensional coordinate in the camera coordinate system, and call this coordinate the first coordinate , the first coordinate The calculation formula is as follows: in Indicates the focal length of the depth camera's infrared camera; Step 5: Use the coordinate conversion module to determine the second coordinate of the first coordinate in the coordinate system of the robot base , the second coordinate The calculation formula is as follows: in is the rotation matrix from the camera coordinate system to the ultrasound probe coordinate system; is the rotation transformation matrix from the ultrasound probe coordinate system to the robotic arm base coordinate system; Step 6: Input the second coordinate into the posture positioning module to obtain the three-dimensional posture angle of the ultrasound probe, and then combine it with the second coordinate to finally output the three-dimensional posture of the ultrasound probe. ; Step 7: Input the 3D pose of the ultrasonic probe of the ultrasonic scanning robot into the pose correction module to obtain the compensated 3D pose , a 3D probe attitude positioning algorithm based on multi-scale compensation is used to compensate the 3D probe attitude from two dimensions: space and time, where: The spatial compensation is to use the distance near the second coordinate point Calculate the normal vectors of the four pixel points and move the starting points of these normal vectors to the landing point. Above, the compensated normal vector is obtained by summing and normalizing; In terms of time, for five consecutive sampling times The normal vectors after spatial compensation are summed and normalized to obtain the final normal vector ; Step 8: After the probe contacts the human skin, the force feedback correction module is used to correct the three-dimensional position of the ultrasound probe again, and the final three-dimensional position of the ultrasound probe is output. .

4. The method for positioning the ultrasonic probe of a medical remote ultrasonic scanning robot according to claim 3, characterized in that In step 6, the three-dimensional position of the ultrasound probe includes the landing coordinates of the probe And attitude angle, where: the attitude angle is determined by the normal vector of the human chest skin surface near the landing point, the normal vector is determined by the plane formed by three non-collinear points in the neighborhood near the point, and the angle between the normal vector and the coordinate axis of the robotic arm base constitutes the attitude angle of the ultrasound probe.

5. The method for positioning the ultrasonic probe of a medical remote ultrasonic scanning robot according to claim 3, characterized in that described The calculation formula is as follows: in represents the normal vector, The normal vector representing the target landing point.

Images