Ultrasonic scanning robot scanning target positioning system and method

Abstract

The present invention discloses a system and method for positioning a target for an ultrasonic scanning robot. The target positioning system includes a six-degree-of-freedom robotic arm, a depth camera, an image preprocessing module, a target positioning module, and a patient position movement determination module. The present invention uses a depth camera to capture color images and corresponding depth information of ultrasonic scanning target points, and uses the D-CBAM-HRNet human body key point detection network to achieve two-dimensional positioning of the scanning key points. This system then combines depth information with coordinate system conversion, and utilizes a position movement determination module to effectively avoid the problem of large positioning errors caused by large-scale patient movement. This system achieves high-precision positioning of the ultrasonic scanning robot's scanning target with good robustness. The entire positioning system can achieve real-time, accurate, and convenient scanning target positioning using low-cost sensors, greatly improving positioning accuracy and expanding the autonomy of the ultrasonic scanning robot.

Description

Technical Field

[0001] The present invention belongs to the field of robotics and relates to a system and method for positioning a scanning target of an ultrasonic scanning robot, and more particularly to a system and method for positioning a scanning target of an ultrasonic scanning robot based on a convolutional neural network that fuses vision and depth information. Background Art

[0002] In recent years, ultrasound imaging has played an increasingly important role in the medical field. Compared to computed tomography, magnetic resonance imaging, and X-ray imaging, ultrasound imaging is a safe, convenient, real-time, non-invasive, and low-cost imaging method. Therefore, it has been widely used in orthopedics, cardiology, respiratory medicine, and other fields, becoming a vital tool for doctors in diagnosis and treatment. Ultrasound scanning relies to a certain extent on the doctor's diagnostic experience, which can easily lead to misdiagnosis. Therefore, using ultrasound scanning robots to replace doctors in scanning tasks can not only improve scanning accuracy but also alleviate the shortage of medical resources. Target point localization for human ultrasound scanning is a crucial step in the automated scanning of ultrasound robots. The accuracy of target point localization significantly affects the ultrasound imaging quality and further scanning path planning. Due to individual patient variability and body movement during the scanning process, accurate target point localization is difficult. Currently, target point localization mainly involves laser point cloud processing and visual processing. Graumann et al. (Christoph G, Bernhard F, Christoph H, et al, “Robotic Ultrasound Trajectory Planning for Volume of Interest Coverage,” in IEEE International Conference on Robotics and Automation (ICRA), Stockholm, SWEDEN, 2016, pp. 736-741.) used a depth camera to generate a three-dimensional point cloud of the target and used an expert system to locate the scanning target points. This process required a high-performance depth camera to construct a precise three-dimensional point cloud of the patient's body surface. Furthermore, human intervention was required to further determine the scanning target points, which not only increased hardware costs but also reduced the degree of automation of the entire robotic system. Huang et al. (HuangQ H, Lan JL, Li XL, “Robotic Arm Based Automatic Ultrasound Scanning for Three-Dimensional Imaging,” IEEE Transactions on Industrial Informatics, vol. 15, no. 2, pp. 1173-1182, Feb. 2019.) proposed a solution that uses the color channel of a depth camera for threshold segmentation to identify targets. At the same time, the depth channel is used to construct a three-dimensional point cloud model of the human skin surface to obtain the three-dimensional coordinates of the scanned target points. This method requires manual marking of the scanning points before scanning and has strict requirements on lighting conditions.The current mainstream scanning target point positioning methods cannot effectively solve the problems of low positioning accuracy and low degree of automation, and cannot solve the problem of large positioning errors caused by patient movement during the scanning process. Summary of the Invention

[0003] In order to solve the problems of low target positioning accuracy, low degree of automation and poor robustness of ultrasonic scanning robots, the present invention provides an ultrasonic scanning robot target positioning system and method, which takes into account both positioning accuracy and real-time performance while ensuring the use of low-cost hardware.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] An ultrasonic scanning robot scanning target positioning system includes a six-degree-of-freedom robotic arm, a depth camera, an image preprocessing module, a target positioning module, and a patient position movement determination module, wherein:

[0006] The six-degree-of-freedom robotic arm is used to clamp the ultrasonic probe and carry a depth camera to perform ultrasonic scanning tasks;

[0007] The depth camera is used to obtain a patient image containing a scan target point and simultaneously obtain depth information of corresponding pixels in the image;

[0008] The image preprocessing module is used to perform relevant preprocessing operations on the color image and depth information from the depth camera;

[0009] The target positioning module is used to store the trained D-CBAM-HRNet human key point detection network, and perform three-dimensional positioning and coordinate system conversion operations on the color image preprocessed by the image preprocessing module and the corresponding depth information to detect and locate the target point of the patient's ultrasound scan;

[0010] The patient position movement determination module is used to determine whether the patient has moved in a large range during the scanning process, and then perform a repositioning operation to effectively prevent the phenomenon of large positioning errors.

[0011] A method for performing scanning target positioning by an ultrasonic scanning robot using the scanning target positioning system comprises the following steps:

[0012] Step S1: using a depth camera to collect a color image and depth information of a patient's body surface scan area, and preprocessing the color image and depth information through an image preprocessing module;

[0013] Step S2: Input the color image and the corresponding depth information pre-processed in step S1 into the target positioning module to complete the three-dimensional precise positioning of the scanned target point. The specific method is as follows:

[0014] Step S21: Merge the RGB image preprocessed in step S1 and the corresponding normalized depth data into four-channel data as the fusion input of the D-CBAM-HRNet human key point detection network;

[0015] Step S22: The fused input is first downsampled through two convolutional layers in the HRNet-W32 backbone network, and the number of channels of the feature layer is adjusted by repeatedly stacking bottleneck modules, while the size of the feature layer remains unchanged;

[0016] Step S23: The transition structure divides the feature map into four branches with different resolutions by downsampling. The stage structure adds the feature layers of different branches by upsampling and downsampling, and then obtains the fused output through the ReLU activation function.

[0017] Step S24: Add a CBAM module to each branch at different scales to process the channel attention mechanism and spatial attention mechanism of each feature layer at different scales respectively, so as to improve the ability to extract effective information of the feature layer;

[0018] Step S25: Upsample the feature layers of different branches to obtain a fusion output, and then pass it through the convolution layer to finally obtain the heat map of the key points of different parts, completing the two-dimensional scanning target point positioning;

[0019] Step S26: After completing the two-dimensional scanning target point positioning, the two-dimensional pixel coordinates of the ultrasonic scanning target point are obtained. This coordinate is called the first coordinate P1 = (x1, y1). Then, combined with the corresponding depth information, the three-dimensional coordinates of the first coordinate in the depth camera coordinate system are obtained using the following formula. This coordinate is called the second coordinate P2 = (x2, y2, z2):

[0020]

[0021]

[0022] z2=d

[0023] Among them, d is the depth data corresponding to the first coordinate P1, ω c and h c are the width and height of the image respectively, and f is the focal length of the depth camera;

[0024] Step S27: Convert the second coordinate in the camera coordinate system to the third coordinate P3 in the robot base coordinate system through coordinate transformation. The conversion formula is as follows:

[0025]

[0026] in, is the transformation matrix from the camera coordinate system to the robot end effector coordinate system, is the transformation matrix from the robot end effector coordinate system to the robot base coordinate system;

[0027] Step S3: Use the depth camera to detect the position of the human body. If the depth camera detects that the movement of the human body exceeds the set threshold during the automatic ultrasound scanning process, the patient position movement determination module will determine that the patient has moved over a large range, terminate the automatic scanning, and re-perform the scanning target point positioning of steps S1-S2, effectively preventing the problem of large positioning errors caused by large-scale movement of the patient.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] The present invention uses a depth camera to capture color images and corresponding depth information of ultrasound scanning target points. It then uses the D-CBAM-HRNet human key point detection network to achieve two-dimensional positioning of scanning key points. This, combined with depth information and coordinate system transformation, and a position movement determination module, effectively avoids the problem of large positioning errors caused by large-scale patient movement. This achieves high-precision positioning of the ultrasound scanning robot's scanning target with good robustness. The entire positioning system can achieve real-time, accurate, and convenient scanning target positioning while using low-cost sensors, greatly improving positioning accuracy and expanding the autonomy of the ultrasound scanning robot. The performance indicators of the D-CBAM-HRNet human key point detection network are OKS = 89.16%, mIOU = 95.22%, AP = 86.88%, and AR = 86.62%. The target point positioning error of the ultrasound scanning robot using this scanning target positioning system is 14.94±0.31mm. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is the structural block diagram of the ultrasonic scanning robot scanning and positioning system;

[0031] Figure 2 It is a physical picture of the ultrasonic scanning robot scanning and positioning system and a schematic diagram of the relevant coordinate system positions;

[0032] Figure 3 is a flow chart of a scanning and positioning method of an ultrasonic scanning robot;

[0033] Figure 4 This is a schematic diagram of the D-CBAM-HRNet human key point detection network architecture, (a) is the overall architecture of the network, and (b) is a schematic diagram of the convolutional block attention module (CBAM) module. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0035] The present invention provides a target positioning system for an ultrasonic scanning robot. Figure 1 and Figure 2 As shown, the system mainly consists of a six-degree-of-freedom robotic arm, a depth camera, an image preprocessing module, a target positioning module, and a patient position movement determination module, wherein:

[0036] The six-degree-of-freedom robotic arm is used to clamp the ultrasonic probe and carry a depth camera to perform ultrasonic scanning tasks;

[0037] The depth camera is used to obtain patient images containing the scan target points and can also obtain depth information of corresponding pixels in the image. It is the main sensor used for positioning.

[0038] The image preprocessing module is used to perform relevant preprocessing operations on the color image and depth information from the depth camera, including: size unification, color image contrast enhancement, depth information filtering, depth information normalization and other operations;

[0039] The target positioning module is used to store the trained D-CBAM-HRNet human key point detection network, and perform operations such as scanning target point three-dimensional positioning and coordinate system conversion on the color image preprocessed by the image preprocessing module and the corresponding depth information;

[0040] The patient position movement determination module includes an external camera and a position determination algorithm, which is used to determine whether the patient has moved over a large range during the scanning process, and then perform a repositioning operation to effectively prevent large positioning errors;

[0041] The position determination algorithm uses an existing Yolo-based target detection algorithm to locate the patient's position, thereby calculating the position movement distance.

[0042] The present invention also provides a method for performing scanning target positioning of an ultrasonic scanning robot using the scanning target positioning system. Figure 3 As shown, the method includes the following steps:

[0043] Step S1: Use the depth camera fixed at the end of the robotic arm to collect color images and depth information of the patient's body surface scanning area, and preprocess the color images and depth information through the image preprocessing module, including: size unification, contrast enhancement, color image depth information alignment, depth information normalization and other operations.

[0044] Step S2: Input the processed color image and the corresponding depth information into the target positioning module, and complete the three-dimensional precise positioning of the scanning target point in the target positioning module. The target positioning module uses a human key point detection network based on a deep convolutional neural network to perform two-dimensional detection and positioning of the patient's ultrasound scanning target points. Among them, the framework of the proposed deep convolutional neural network model is a human key point detection network with a color-depth dual-channel input with an attention mechanism, named D-CBAM-HRNet, which is used to process the color image and depth information obtained by the depth camera in real time to detect and locate the target points of the patient's ultrasound scan. Since when the human body remains relatively still, the relative positions between the key points of the human body have a certain correlation in three-dimensional space and are not easily disturbed by environmental factors such as light. Therefore, the normalized depth data can also be used as feature information for human key point detection.

[0045] like Figure 4 As shown in the figure, the D-CBAM-HRNet consists of input feature fusion, convolution layer, multi-scale transition structure and Stage structure, CBAM attention mechanism module, feature map output structure and other parts, among which: the input of D-CBAM-HRNet is four-channel data consisting of RGB three-channel color image and depth single-channel information, and 4x downsampling is completed through two convolution layers with a convolution kernel size of 3*3 and a stride of 2, and the number of channels is further adjusted by stacking Bottlencek structure; next, through three multi-scale transition structures and Stage structures with different resolutions, the features of original resolution and downsampled 4 times, 8 times and 16 times are extracted respectively, and the CBAM attention mechanism module is added after the feature maps of four different resolutions to adjust the parameters in both channel and space. Finally, the heat map and pixel coordinate information of key points are output through the feature map output structure.

[0046] The D-CBAM-HRNet proposed in this invention is a network with HRNet-W32 as the backbone. On the one hand, the input of the network is changed from single RGB image data to dual-modal data of RGB image data and depth data, which enriches the network's extraction of key point features and enhances the network's robustness to the influence of illumination. On the other hand, by adding the CBAM module at a specific position of HRNet-W32, the network can adaptively focus on the feature information of the image in terms of channels and space. Although the proposed method increases the complexity of the network to a certain extent, it improves the performance of the network in detecting key points of the human body and improves the accuracy of two-dimensional target point detection. The structure of the D-CBAM-HRNet human key point detection network can be divided into the following parts:

[0047] First, the RGB image and the corresponding normalized depth data are merged into four-channel data (R, G, B, and D) as the fusion input of the D-CBAM-HRNet network.

[0048] Secondly, the fused input is first downsampled through two convolutional layers in the HRNet-W32 backbone network, and the number of channels in the feature layer is adjusted by repeatedly stacking bottleneck modules, while the size of the feature layer remains unchanged.

[0049] Next, transition structures and Stage structures with three scales (original resolution, 4x downsampling, 8x downsampling, and 16x downsampling) are used. The transition structure divides the feature map into four branches with different resolutions through downsampling, while the Stage structure is the most important part of the backbone network. It adds the feature layers of different branches through upsampling and downsampling, and then obtains the fused output through the ReLU activation function.

[0050] Then, a CBAM module is added to each branch at different scales to process the channel attention mechanism and spatial attention mechanism of each feature layer at different scales respectively, so as to improve the ability to extract effective information of the feature layer.

[0051] Finally, the feature layers of different branches are upsampled to obtain the fused output, which is then passed through the convolution layer to finally obtain the heat maps of the key points of different parts.

[0052] During the actual ultrasound scanning process, the structure of the last convolutional layer can be adjusted according to different scanning parts so that the number of output key points meets the scanning needs. While meeting the personalized needs of clinical ultrasound scanning, it greatly reduces the number of network parameters and improves real-time performance.

[0053] After completing the two-dimensional scanning target point positioning, the two-dimensional pixel coordinates of the ultrasonic scanning target point are obtained, which is called the first coordinate P1 = (x1, y1). Then, combined with the corresponding depth information, the three-dimensional coordinates of the first coordinate in the depth camera coordinate system are obtained using formula (1), which is called the second coordinate P2 = (x2, y2, z2):

[0054]

[0055] Among them, d is the depth data corresponding to the first coordinate P1, ω c and h c are the width and height of the image respectively, and f is the focal length of the depth camera.

[0056] Furthermore, through coordinate transformation, the second coordinate in the camera coordinate system is converted to the third coordinate P3 in the robot base coordinate system. It is necessary to obtain the transformation matrix from the camera coordinate system to the robot end effector coordinate system in advance. Since the depth camera is fixed on the robotic arm, the two are rigidly connected, and the transformation matrix can be determined by the relative position of the installation Transformation matrix from the robot arm end effector coordinate system to the robot arm base coordinate system The transformation matrix can be calculated through the kinematic equation of the robot arm The transformation formula is shown in formula (2):

[0057]

[0058] Step S3: In order to solve the problem of positioning error caused by large-scale movement of the patient during the scanning process, a patient position movement determination module is added. The depth camera is used to detect the position of the human body. If the depth camera detects that the human body movement exceeds the set threshold (the human body movement distance is greater than or equal to 3 cm) during the automatic ultrasound scanning process, the patient position movement determination module will determine that the patient has moved over a large range, terminate the automatic scanning, and re-position the scanning target point, effectively preventing the problem of large positioning error caused by large-scale movement of the patient.

[0059] Example:

[0060] Taking scanning a human arm as an example, the method specifically includes the following steps:

[0061] Step S1: Use the depth camera fixed at the end of the robotic arm to collect the color image and depth information of the patient's arm scanning area, and preprocess the color image and depth information through the image preprocessing module to unify the size to 512×512, align the color image depth information, and normalize the depth information.

[0062] Step S2: Input the preprocessed color image and the corresponding depth information into the target positioning module. First, the color image and the corresponding depth information are fused into a four-channel data and input into the trained D-CBAM-HRNet human key point detection network to output the two-dimensional pixel coordinates of the three expected scanning target points on the patient's arm, i.e., the first coordinates.

[0063] Step S3: Combine the corresponding depth information and use formula (1) to obtain the three-dimensional coordinates of the first coordinate in the depth camera coordinate system, that is, the second coordinate.

[0064] Step S4: According to formula (2), the second coordinate of the target point is mapped to the coordinate system of the robot base to obtain the third coordinate, thereby completing the positioning of the ultrasonic scanning target point.

[0065] Step S5: During the positioning process, if the patient's body is determined by the position movement determination module to have moved over a large range within a period of time, the system will re-perform the workflow of steps S1 to S4, ultimately achieving accurate positioning of the ultrasound scanning target point.

[0066] This example demonstrates the ability of an ultrasonic scanning robot to locate a target on a human arm under both normal and abnormal lighting conditions. The positioning results for three target points on the human arm are shown in Table 1. These results demonstrate that the proposed ultrasonic scanning robot positioning method can achieve real-time, accurate, and convenient target positioning using low-cost sensors. This significantly improves positioning accuracy, expands the autonomy of the ultrasonic scanning robot, and effectively enhances the robustness of the positioning method.

[0067] Table 1 Experimental results of human arm ultrasonic scanning robot scanning target positioning under normal and abnormal lighting conditions

[0068]

Claims

1. An ultrasonic scanning robot scanning target positioning system, characterized in that The target positioning system includes a six-degree-of-freedom robotic arm, a depth camera, an image preprocessing module, a target positioning module, and a patient position movement determination module, wherein: The six-degree-of-freedom robotic arm is used to clamp the ultrasonic probe and carry a depth camera to perform ultrasonic scanning tasks; The depth camera is used to obtain a patient image containing a scan target point and simultaneously obtain depth information of corresponding pixels in the image; The image preprocessing module is used to perform relevant preprocessing operations on the color image and depth information from the depth camera; The target positioning module is used to store the trained D-CBAM-HRNet human key point detection network, and perform three-dimensional positioning and coordinate system conversion operations on the color image preprocessed by the image preprocessing module and the corresponding depth information to detect and locate the target point of the patient's ultrasound scan; The D-CBAM-HRNet human key point detection network includes a convolutional layer, a multi-scale transition structure, a Stage structure, a CBAM attention mechanism module, and a feature map output structure. The input of D-CBAM-HRNet is a four-channel data consisting of RGB three-channel color image and depth single-channel information, which is obtained by two convolution kernels with a size of 3.

3. A convolutional layer with a stride of 2 performs 4x downsampling, and the number of channels is further adjusted by stacking the Bottlencek structure. Next, a multi-scale transition structure and a stage structure with three different resolutions are used to extract features at the original resolution and those downsampled by 4x, 8x, and 16x, respectively. The CBAM attention mechanism module is added to the feature maps of the four different resolutions to adjust parameters in both channel and spatial terms. Finally, the feature map output structure outputs the heat map and pixel coordinate information of the key points. The patient position movement determination module is used to determine whether the patient has moved in a large range during the scanning process, and then perform a repositioning operation to effectively prevent the phenomenon of large positioning errors.

2. The ultrasonic scanning robot scanning target positioning system according to claim 1, characterized in that The preprocessing operations include: size unification, color image contrast enhancement, depth information filtering, and depth information normalization operations.

3. A method for positioning a target by an ultrasonic scanning robot using the target positioning system according to any one of claims 1 to 2, characterized in that The method comprises the following steps: Step S1: using a depth camera to collect a color image and depth information of a patient's body surface scan area, and preprocessing the color image and depth information through an image preprocessing module; Step S2: input the color image pre-processed in step S1 and the corresponding depth information into the target positioning module to complete the three-dimensional precise positioning of the scanned target point; Step S3: Use the depth camera to detect the position of the human body. If the depth camera detects that the movement of the human body exceeds the set threshold during the automatic ultrasound scanning process, the patient position movement determination module will determine that the patient has moved over a large range, terminate the automatic scanning, and re-perform the scanning target point positioning of steps S1-S2, effectively preventing the problem of large positioning errors caused by large-scale movement of the patient.

4. The method for scanning target positioning by an ultrasonic scanning robot according to claim 3, characterized in that The specific method of step S2 is as follows: Step S21: Merge the RGB image preprocessed in step S1 and the corresponding normalized depth data into four-channel data as the fusion input of the D-CBAM-HRNet human key point detection network; Step S22: The fused input is first downsampled through two convolutional layers in the HRNet-W32 backbone network, and the number of channels of the feature layer is adjusted by repeatedly stacking bottleneck modules, while the size of the feature layer remains unchanged; Step S23: The transition structure divides the feature map into four branches with different resolutions by downsampling. The stage structure adds the feature layers of different branches by upsampling and downsampling, and then obtains the fused output through the ReLU activation function. Step S24: Add a CBAM attention mechanism module to each branch at different scales to process the channel attention mechanism and spatial attention mechanism of each feature layer at different scales respectively, so as to improve the ability to extract effective information of the feature layer; Step S25: Upsample the feature layers of different branches to obtain a fusion output, and then pass it through the convolution layer to finally obtain the heat map of the key points of different parts, completing the two-dimensional scanning target point positioning; Step S26: After completing the two-dimensional scanning target point positioning, the two-dimensional pixel coordinates of the ultrasonic scanning target point are obtained, which are called the first coordinates. , and then combined with the corresponding depth information, the following formula is used to obtain the three-dimensional coordinates of the first coordinate in the depth camera coordinate system, which is called the second coordinate : in, The first coordinate The corresponding depth data, and are the width and height of the image, is the focal length of the depth camera; Step S27: Convert the second coordinate in the camera coordinate system to the third coordinate in the robot base coordinate system through coordinate transformation. , the conversion formula is as follows: in, is the transformation matrix from the camera coordinate system to the robot end effector coordinate system, is the transformation matrix from the robot end effector coordinate system to the robot base coordinate system.

5. The method for scanning target positioning by an ultrasonic scanning robot according to claim 3, characterized in that In step S3, exceeding the set threshold means that the human body moves a distance greater than or equal to 3 cm.

Images