Route planning method of kidney ultrasonic scanning robot

Abstract

The invention discloses a path planning method for a kidney ultrasonic scanning robot, and belongs to the field of intelligent medical treatment. The invention aims to solve the problems of poor automation, low efficiency and resource waste in the existing automatic renal region scanning. A depth camera is adopted to shoot a color image and a depth image when a to-be-detected object lies on a plane prostrate; calculating a three-dimensional coordinate of each point cloud point of the to-be-detected object in a world coordinate system, drawing a rectangular frame on the color image, and segmenting the to-be-detected object and the point cloud of the near plane area from the three-dimensional point cloud model; removing the plane region by adopting a random sampling consensus algorithm, and obtaining a three-dimensional point cloud model of the to-be-detected object; calculating a symmetric center line of the back surface of the to-be-detected object, and adjusting the three-dimensional point cloud model to an expected pose by using the symmetric center line; acquiring a waist line and a ridge line of the to-be-detected object by adopting a slicing method; according to the human body structure and the waist line and the ridge line of the to-be-detected object, two-side kidney ultrasonic scanning paths are obtained. The method is suitable for kidney ultrasonic scanning path planning.

Description

Technical Field

[0001] The invention belongs to the field of intelligent medical treatment. Background Art

[0002] In the field of medical imaging, ultrasound examination is widely used in clinical practice as a non-invasive, real-time and cost-effective diagnostic tool. However, traditional manual ultrasound scanning relies on the operator's skills and experience, and its results may be affected by subjective factors. In addition, manual scanning may not achieve ideal image quality and diagnostic effects for some difficult-to-reach or complex anatomical structures. In addition, long-term manual operation may also cause operator fatigue and occupational injuries.

[0003] In recent years, with the development of robotics and artificial intelligence, ultrasound scanning robots have gradually become a research hotspot, aiming to improve the accuracy, reliability and efficiency of ultrasound examinations while reducing the workload of operators. Despite this, current ultrasound robot systems still face many challenges, especially in terms of autonomy and intelligent decision-making, and there is still a certain gap from comprehensive clinical application. The main technical bottlenecks include accurate path planning, effective force feedback control, precise probe posture adjustment, and image-guided servo control.

[0004] Path planning is the first step in the operation of ultrasound robots and is the key to achieving automated ultrasound scanning. However, individual differences in human anatomical structures and positional uncertainty make path planning difficult. Existing path planning methods often have the following limitations:

[0005] Some algorithms adopt a coverage scanning strategy, which can ensure full coverage of the scanning area, but is inefficient and consumes a lot of time and resources.

[0006] Other methods rely on the operator to draw the scanning path through an external device (such as a mouse), which not only increases the complexity of the operation but also limits the autonomy of the system.

[0007] Some ultrasound robots require the operator to manually position the probe to the scanning area. This semi-automatic mode reduces the overall automation of the system. Summary of the invention

[0008] The present invention aims to solve the problems of poor automation, low efficiency and waste of resources in the existing automatic renal area scanning, and now provides a path planning method for a renal ultrasound scanning robot.

[0009] The path planning method of a kidney ultrasound scanning robot described in the present invention comprises:

[0010] Step 1: Use a depth camera to capture a color image and a depth image of the object to be detected when it lies prone on a plane; use the color image and the depth image, combined with the focal length of the camera and the position of the camera's principal point, to calculate the three-dimensional coordinates of each point cloud point of the object to be detected in the world coordinate system, and then obtain a three-dimensional point cloud model of the object to be detected when it lies prone on the plane;

[0011] Step 2: Draw a rectangular frame on the color image, and use the rectangular frame to segment the point cloud of the object to be detected and its adjacent plane area from the three-dimensional point cloud model in step 1; obtain a three-dimensional point cloud model that only includes the object to be detected and its adjacent plane area;

[0012] Step 3: Using a random sampling consensus algorithm to remove the plane area in the three-dimensional point cloud model described in step 2, to obtain a three-dimensional point cloud model of the object to be detected;

[0013] Step 4: Use a directional bounding box to define the contour range of the three-dimensional point cloud model of the object to be detected, use the defined contour range to calculate the symmetry center line of the back of the object to be detected, and use the symmetry center line to adjust the three-dimensional point cloud model of the object to be detected to the desired posture;

[0014] Step 5: Use the slicing method to cut the three-dimensional point cloud model of the object to be detected in the desired posture state to obtain the waistline and spine line of the object to be detected;

[0015] Step six, search for the point cloud coordinates of the intersection area of ​​the spine line and the waist line, take the point with the largest Y value in the point cloud coordinates in the intersection area as the starting point, set a local interception window with a length of m according to the human body structure, intercept a section of the spine line, and use the intersection of the intercepted spine line and the local interception window as the vertex. Use the plane projection method to rotate the intercepted spine line 45 degrees to the left and right respectively, and use the rotated spine line as the ultrasound scanning path of the kidneys on both sides, and m is 8cm-15cm.

[0016] Furthermore, in the present invention, step three also includes a step of using a region growing algorithm to remove noise and outliers in the three-dimensional point cloud model of the object to be detected.

[0017] Furthermore, in the present invention, in step 2, the specific method for obtaining a three-dimensional point cloud model containing only the object to be detected and its adjacent plane area is:

[0018] Create a binary mask layer with the same size as the color image but all zeros, set the mask value in the rectangular box to 255, use the rectangular box to select the object to be inspected and its adjacent plane area, and obtain a three-dimensional point cloud model of the object to be inspected and its adjacent plane area;

[0019] The rectangular frame is:

[0020] PROI ={pp=f(I,D,M,(x,y)),M(x,y)=255}

[0021] Where P ROI represents the point in the rectangular box, f represents the inverse projection function of the camera model, which depends on the internal parameters and pixel coordinates of the camera, M(x, y) represents a binary mask of the same size as the color image but all zeros, I represents the color image of the object to be detected lying prone on the plane, D represents the depth image of the object to be detected lying prone on the plane, and p represents a point in the three-dimensional point cloud model.

[0022] Furthermore, in the present invention, in step 3, the method for obtaining the three-dimensional point cloud model of the object to be detected is:

[0023] Using the formula:

[0024] n T x+d=0

[0025] The plane area in the three-dimensional point cloud model described in step 2 is removed, wherein n is the normal vector of the plane, x is the coordinate of the plane point in the three-dimensional space, and d is a constant term.

[0026] Furthermore, in the present invention, in step 4, the specific method for extracting the back three-dimensional point cloud model of the object to be detected is:

[0027] First, the curvature values ​​of the points in the point cloud model are sorted, and the point with the smallest curvature value is selected as the initial seed point. According to the preset parameters and the growth criterion, the neighboring points with similar characteristics are merged into the same area until all the points that meet the conditions are assigned to the corresponding areas to form multiple independent clusters. The independent cluster containing the most point cloud points is selected as the back 3D point cloud model of the object to be detected.

[0028] The preset parameters include: the minimum number of point cloud points in independent clusters, the maximum number of point cloud points in independent clusters, the number of neighborhood search points, the smoothing threshold and the curvature threshold;

[0029] The specific method of merging neighboring points with similar characteristics into the same area is:

[0030] Calculate the normal vector between the seed point P0 and any point Q in its neighborhood and The angle θ between them is determined to determine whether the angle θ satisfies condition 1:

[0031]

[0032] If so, then point Q is in the same direction as point P0, · represents the vector dot product, and are the modulus of the vector, θthreshold Indicates the angle threshold;

[0033] At the same time, determine the curvature k of the neighborhood point Q Q The curvature k of the seed point P0 Q Whether condition 2 is met:

[0034] |k Q -k P |≤Δk threshold

[0035] If so, point Q and point P0 are included in the same growth region, and all points in the neighborhood of seed point P0 are traversed in turn. All points in the neighborhood that meet conditions 1 and 2 are placed in the same growth region to form an independent cluster. All clusters are traversed, and the cluster with the most point cloud points is used as the back 3D point cloud model of the object to be detected, where Δk threshold is the curvature difference threshold.

[0036] Furthermore, in the present invention, in step 4, an oriented bounding box (OBB) is used to define the contour range of the denoised three-dimensional point cloud model of the back of the object to be detected, and the symmetric midline of the back of the object to be detected is calculated using the defined contour range:

[0037] Calculate the center point C of the bounding box obb And the centroid C2 of the line segment defined by the vertices of the oriented bounding box, calculate the reference line of the midline of the human back model:

[0038] C bb =(center x ,center y ,center z )

[0039]

[0040] P(t)=(1-t)·C2+t·C1,0≤t≤1

[0041] Where N is the number of points that make up the line segment, P i is a point on the line segment, i represents the i-th point on the line segment, center x ,center y ,center z They represent the coordinates of the center point of the bounding box in the three axis directions, P(t) represents the median line, and t is restricted to the interval [0,1] to ensure that P(t) is on the straight line from C2 to C1.

[0042] Furthermore, in the present invention, in step 4, the method of adjusting the three-dimensional point cloud model of the object to be detected to the desired posture by using the symmetry midline is:

[0043] Align the symmetrical midline direction of the back of the object to be detected in the bounding box with the coordinate axis of the world coordinate system, and calculate the cosine value of the angle θ between it and the y-axis of the camera coordinate system by the dot product formula;

[0044]

[0045] in, is the y-axis unit vector, is the vector of the human body reference center line; after obtaining the angle θ through the inverse cosine function, the rotation matrix R is constructed to rotate the point cloud data around the y-axis;

[0046]

[0047] The rotation matrix R is cross-multiplied with the three-dimensional point cloud model, and the three-dimensional point cloud model of the back of the object to be detected is adjusted to be aligned with the camera coordinate axis, so as to adjust the position and posture of the three-dimensional point cloud model of the back of the object to be detected.

[0048] Furthermore, in the present invention, in step 5, the slicing method is used to cut the three-dimensional point cloud model of the back of the object to be detected in the desired posture, and the method for obtaining the waistline and spine line of the back of the target object is:

[0049] First, the slicing method is used to slice the back 3D point cloud model at equal intervals along the direction perpendicular to the spine to obtain multiple slices, and the average value of each slice along the Z-axis direction of the camera coordinate system is calculated, and the point cloud data of the slice with the highest average value is used as the waistline point cloud data;

[0050] Then, the slices within the head and hip range are removed according to the human body proportions to obtain a three-dimensional point cloud model of the back of the object to be detected;

[0051] Finally, the slicing method is used to cut multiple slices of the back three-dimensional point cloud model at equal intervals along the spine direction, and the average value of each slice along the Z-axis direction of the camera coordinate system is calculated, and the point cloud data of the slice with the highest average value is used as the point cloud data of the spine line.

[0052] Furthermore, in the present invention, the formula for calculating the average value of each slice along the Z-axis direction of the camera coordinate system is:

[0053]

[0054] Among them, z P is the coordinate of the pth slice on the Z axis, and |S i | is slice S i The number of midpoints, p represents the pth slice.

[0055] Furthermore, in the present invention, the method for removing the slices within the head and buttocks is:

[0056] According to the human body proportion, set a starting index s and an ending index e to remove the slices within the head and hip range.

[0057]

[0058] Where n is the number of slices:

[0059]

[0060] Among them, y min Indicates the minimum value of the slice along the Z axis of the camera coordinate system, y max It represents the maximum value of the slice along the Z axis of the camera coordinate system, and Δy represents the width of the slice.

[0061] Furthermore, in the present invention, step six also includes: processing the ultrasound scanning paths of the kidneys on both sides using point cloud voxels to obtain scanning path trajectory lines.

[0062] The present invention makes full use of human anatomical knowledge, especially the spinal line, the thinnest waist area and the costovertebral angle, to provide clear directional guidance for renal ultrasound scanning. This not only simplifies the path planning process, so that the location of the kidney can be quickly determined in two-dimensional ultrasound, but also avoids the problem of requiring additional markings or relying on specific image quality in other studies. Secondly, the present invention uses advanced point cloud processing technology to accurately detect the key feature lines of the human body. The process is fully automated, which not only reduces the difficulty of operation, but also greatly reduces the possibility of manual intervention, making the entire system more reliable. In addition, since the point cloud data can better reflect the true morphology of the human body surface, even in the face of individuals of different body shapes, it can initially adapt to their body shape characteristics. Compared with research that relies on expensive equipment or other complex technologies, this method is cheaper and easier to promote and popularize. Finally, no matter where the object to be inspected is, the system can compensate for the posture difference by adjustment to ensure the consistency and accuracy of the scanning path. Compared with those methods that perform well only under fixed conditions, our solution has stronger adaptability and robustness in the face of clinical environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a flow chart of the method of the present invention;

[0064] Figure 2 A schematic diagram of the process of segmenting the 3D point cloud model of the back of a human body;

[0065] Figure 3 It is a schematic diagram of the correction process when the posture of the human body model is skewed;

[0066] Figure 4 It is a schematic diagram of the feature line detection process of the 3D point cloud model of the back of the human body;

[0067] Figure 5 It is a schematic diagram of the ultrasound scanning path planning process and results;

[0068] Figure 6 This is a schematic diagram of the path planning scenario and results of a real human body;

[0069] Figure 7 Comparison of renal cross-sectional ultrasound images obtained from 30 sets of scanning experiments. DETAILED DESCRIPTION

[0070] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0071] Specific implementation method 1: Referring to FIG. Figures 1 to 5 Specifically describing this embodiment, the method described in this embodiment includes:

[0072] Step 1: Use a depth camera to capture a color image and a depth image of the object to be detected when it lies prone on a plane; use the color image and the depth image, combined with the focal length of the camera and the position of the camera's principal point, to calculate the three-dimensional coordinates of each point cloud point of the object to be detected in the world coordinate system, and then obtain a three-dimensional point cloud model of the object to be detected when it lies prone on the plane;

[0073] Step 2: Draw a rectangular frame on the color image, and use the rectangular frame to segment the point cloud of the object to be detected and its adjacent plane area from the three-dimensional point cloud model in step 1; obtain a three-dimensional point cloud model that only includes the object to be detected and its adjacent plane area;

[0074] Step 3: Using a random sampling consensus algorithm to remove the plane area in the three-dimensional point cloud model described in step 2, to obtain a three-dimensional point cloud model of the object to be detected;

[0075] Step 4: Use a directional bounding box to define the contour range of the three-dimensional point cloud model of the object to be detected, use the defined contour range to calculate the symmetry center line of the back of the object to be detected, and use the symmetry center line to adjust the three-dimensional point cloud model of the object to be detected to the desired posture;

[0076] Step 5: Use the slicing method to cut the three-dimensional point cloud model of the object to be detected in the desired posture state to obtain the waistline and spine line of the object to be detected;

[0077] Step six, search for the point cloud coordinates of the intersection area of ​​the spine line and the waist line, take the point with the largest Y value in the point cloud coordinates in the intersection area as the starting point, set a local interception window with a length of m according to the human body structure, intercept a section of the spine line, and use the intersection of the intercepted spine line and the local interception window as the vertex. Use the plane projection method to rotate the intercepted spine line 45 degrees to the left and right respectively, and use the rotated spine line as the ultrasound scanning path of the kidneys on both sides, and m is 8cm-15cm.

[0078] Furthermore, in the present invention, step three also includes a step of using a region growing algorithm to remove noise and outliers in the three-dimensional point cloud model of the object to be detected.

[0079] Furthermore, in the present invention, in step 2, the specific method for obtaining a three-dimensional point cloud model containing only the object to be detected and its adjacent plane area is:

[0080] Create a binary mask layer with the same size as the color image but all zeros, set the mask value in the rectangular box to 255, use the rectangular box to select the object to be inspected and its adjacent plane area, and obtain a three-dimensional point cloud model of the object to be inspected and its adjacent plane area;

[0081] The rectangular frame is:

[0082] P ROI ={pp=f(I,D,M,(x,y)),M(x,y)=255}

[0083] Where P ROI represents the point in the rectangular box, f represents the inverse projection function of the camera model, which depends on the internal parameters and pixel coordinates of the camera, M(x, y) represents a binary mask of the same size as the color image but all zeros, I represents the color image of the object to be detected lying prone on the plane, D represents the depth image of the object to be detected lying prone on the plane, and p represents a point in the three-dimensional point cloud model.

[0084] Furthermore, in the present invention, in step 3, the method for obtaining the three-dimensional point cloud model of the object to be detected is:

[0085] Using the formula:

[0086] n T x+d=0

[0087] The plane area in the three-dimensional point cloud model described in step 2 is removed, wherein n is the normal vector of the plane, x is the coordinate of the plane point in the three-dimensional space, and d is a constant term.

[0088] Furthermore, in the present invention, in step 4, the specific method for extracting the back three-dimensional point cloud model of the object to be detected is:

[0089] First, the curvature values ​​of the points in the point cloud model are sorted, and the point with the smallest curvature value is selected as the initial seed point. According to the preset parameters and the growth criterion, the neighboring points with similar characteristics are merged into the same area until all the points that meet the conditions are assigned to the corresponding areas to form multiple independent clusters. The independent cluster containing the most point cloud points is selected as the back 3D point cloud model of the object to be detected.

[0090] The preset parameters include: the minimum number of point cloud points in independent clusters, the maximum number of point cloud points in independent clusters, the number of neighborhood search points, the smoothing threshold and the curvature threshold;

[0091] The specific method of merging neighboring points with similar characteristics into the same area is:

[0092] Calculate the normal vector between the seed point P0 and any point Q in its neighborhood and The angle θ between them is determined to determine whether the angle θ satisfies condition 1:

[0093]

[0094] If so, then point Q is in the same direction as point P0, · represents the vector dot product, and are the modulus of the vector, θ threshold Indicates the angle threshold;

[0095] At the same time, determine the curvature k of the neighborhood point Q Q The curvature k of the seed point P0 Q Whether condition 2 is met:

[0096] |k Q -k P |≤Δk threshold

[0097] If so, point Q and point P0 are included in the same growth region, and all points in the neighborhood of seed point P0 are traversed in turn. All points in the neighborhood that meet conditions 1 and 2 are placed in the same growth region to form an independent cluster. All clusters are traversed, and the cluster with the most point cloud points is used as the back 3D point cloud model of the object to be detected, where Δk threshold is the curvature difference threshold.

[0098] Furthermore, in the present invention, in step 4, an oriented bounding box (OBB) is used to define the contour range of the denoised three-dimensional point cloud model of the back of the object to be detected, and the symmetric midline of the back of the object to be detected is calculated using the defined contour range:

[0099] Calculate the center point C of the bounding box obbAnd the centroid C2 of the line segment defined by the vertices of the oriented bounding box, calculate the reference line of the midline of the human back model:

[0100] C bb =(center x ,center y ,center z )

[0101]

[0102] P(t)=(1-t)·C2+t·C1,0≤t≤1

[0103] Where N is the number of points that make up the line segment, P i is a point on the line segment, i represents the i-th point on the line segment, center x ,center y ,center z They represent the coordinates of the center point of the bounding box in the three axis directions, P(t) represents the median line, and t is restricted to the interval [0,1] to ensure that P(t) is on the straight line from C2 to C1.

[0104] Furthermore, in the present invention, in step 4, the method of adjusting the three-dimensional point cloud model of the object to be detected to the desired posture by using the symmetry midline is:

[0105] Align the symmetrical midline direction of the back of the object to be detected in the bounding box with the coordinate axis of the world coordinate system, and calculate the cosine value of the angle θ between it and the y-axis of the camera coordinate system by the dot product formula;

[0106]

[0107] in, is the y-axis unit vector, is the vector of the human body reference center line; after obtaining the angle θ through the inverse cosine function, the rotation matrix R is constructed to rotate the point cloud data around the y-axis;

[0108]

[0109] The rotation matrix R is cross-multiplied with the three-dimensional point cloud model, and the three-dimensional point cloud model of the back of the object to be detected is adjusted to be aligned with the camera coordinate axis, so as to adjust the position and posture of the three-dimensional point cloud model of the back of the object to be detected.

[0110] Furthermore, in the present invention, in step 5, the slicing method is used to cut the three-dimensional point cloud model of the back of the object to be detected in the desired posture, and the method for obtaining the waistline and spine line of the back of the target object is:

[0111] First, the slicing method is used to slice the back 3D point cloud model at equal intervals along the direction perpendicular to the spine to obtain multiple slices, and the average value of each slice along the Z-axis direction of the camera coordinate system is calculated, and the point cloud data of the slice with the highest average value is used as the waistline point cloud data;

[0112] Then, the slices within the head and hip range are removed according to the human body proportions to obtain a three-dimensional point cloud model of the back of the object to be detected;

[0113] Finally, the slicing method is used to cut multiple slices of the back three-dimensional point cloud model at equal intervals along the spine direction, and the average value of each slice along the Z-axis direction of the camera coordinate system is calculated, and the point cloud data of the slice with the highest average value is used as the point cloud data of the spine line.

[0114] Furthermore, in the present invention, the formula for calculating the average value of each slice along the Z-axis direction of the camera coordinate system is:

[0115]

[0116] Among them, z P is the coordinate of the pth slice on the Z axis, and |S i | is slice S i The number of midpoints, p represents the pth slice.

[0117] Furthermore, in the present invention, the method for removing the slices within the head and buttocks is:

[0118] According to the human body proportion, set a starting index s and an ending index e to remove the slices within the head and hip range.

[0119]

[0120] Where n is the number of slices:

[0121]

[0122] Among them, y min Indicates the minimum value of the slice along the Z axis of the camera coordinate system, y max It represents the maximum value of the slice along the Z axis of the camera coordinate system, and Δy represents the width of the slice.

[0123] Furthermore, in the present invention, in step 6, the method for acquiring the ultrasound scanning paths of the kidneys on both sides is:

[0124] According to the point cloud data of the waistline and the point cloud data of the spine line, the point cloud data of the intersection area of ​​the waistline and the spine line are extracted, and the point with the largest Y value is found in the point cloud data of the intersection area. The point with the largest Y value is used as the starting point, and the spine line is intercepted using a interception window with a length of n to obtain a spine segment. The interception point on the spine is used as the vertex, and the spine segment is rotated 45 degrees to the left and right respectively using a plane projection method. The rotated spine segment is used as the point cloud of the ultrasound scanning path of the kidneys on both sides; the ultrasound scanning path of the kidneys on both sides is obtained by downsampling the point cloud voxels.

[0125] The present invention can automatically identify key features of the human body surface, such as the spine line and the thinnest waist area, and can accurately locate the kidney on the body surface according to the anatomical features of the kidney, such as the costovertebral angle. Compared with the traditional method based on template matching or simple geometric model, this strategy can more accurately adapt to individual differences and improve the personalization of the scanning path. Using the three-dimensional point cloud data obtained by the depth camera, a fine model of the human body surface can be constructed, thereby achieving more accurate path planning. Compared with the traditional method that only relies on two-dimensional images, the introduction of three-dimensional data makes path planning more intuitive and accurate, reducing the possibility of misidentification. At the same time, a fully automatic process from point cloud data preprocessing to path planning is realized, which greatly improves the autonomous operation ability of the ultrasonic robot. Compared with the system that requires manual intervention or semi-automatic path planning, the present invention significantly reduces the workload of the operator and improves the overall intelligence level of the system. The present invention can also guide the ultrasound probe to quickly locate the kidney area through anatomical positioning and intelligent path planning, avoiding unnecessary scanning range and improving scanning efficiency. Compared with traditional coverage scanning or manual operation, this method greatly shortens the scanning time and reduces the time to find the kidney area.

[0126] Reference Figure 6 and Figure 7 The present embodiment is described in detail. The present embodiment is specifically as follows:

[0127] The scanning path of the robot is autonomously planned using a path planning method based on anatomical positioning, such as Figure 1-Figure 5 shown.

[0128] include:

[0129] 1. Use a depth camera to capture images of the prone position of the object to be detected, and obtain a color image and a depth image of the object to be detected in the prone position; use the color image and the depth image to obtain a three-dimensional point cloud model of the object to be detected in the prone position;

[0130] 2. Use a binary mask layer to preliminarily screen the object to be detected in the three-dimensional point cloud model, and remove the environmental area in the point cloud model; then use a random sampling consensus algorithm to perform a second screening on the bed plane area in the preliminarily screened three-dimensional point cloud model to obtain a separate three-dimensional point cloud model of the object to be detected;

[0131] Segmentation of the human body Figure 1 ). Define a rectangular ROI based on the two-dimensional color image I and the corresponding depth image D to segment the object to be inspected and the bed, and remove irrelevant scenes. Create a binary mask M with the same size as the original image but all zeros, draw a rectangular ROI on the color image, and set the mask value in the ROI to 255, indicating that the area is selected. Use the mask M to combine with the depth image D and the color image I, and perform three-dimensional reconstruction through the camera model. Combined with the focal length and principal point position of the camera, calculate the three-dimensional coordinates of the point in the world coordinate system. The points in the object to be inspected and the bed are obtained by

[0132] P ROI ={pp=f(I,D,M,(x,y)),M(x,y)=255}, where f represents the inverse projection function of the camera model, which depends on the internal parameters of the camera and the pixel coordinates.

[0133] From the obtained ROI point cloud, use a plane model n T x+d=0 fits the bed plane point cloud to remove it, n is the normal vector of the plane, x is the coordinate of the point in three-dimensional space, and d is a constant term.

[0134] 3. Use the region growing algorithm to segment the remaining human back to remove noise and outliers, and obtain the 3D point cloud model of the object to be detected after removing noise;

[0135] 4. Perform independent clustering to segment the 3D point cloud model of the object to be detected;

[0136] First, sort the points of the point cloud model according to their curvature values, select the point with the smallest curvature value (the smoothest area) as the initial seed point, and gradually merge neighboring points with similar characteristics into the same area according to preset parameters (such as the minimum number of points, the maximum number of points, the number of neighborhood search points, the smoothness threshold and the curvature threshold, etc.). This process is repeated until all eligible points are assigned to the corresponding areas, forming multiple independent clusters. The growth criterion as the key to regional segmentation is mainly based on two points. One is the consistency of the normal direction.

[0137] Calculate the normal vector between the seed point P0 and any point Q in its neighborhood and The angle θ between them is used to determine whether the angle θ satisfies condition 1 (normal direction consistency):

[0138]

[0139] If so, then point Q is in the same direction as point P0, · represents the vector dot product, and are the modulus of the vector, θ threshold Indicates the angle threshold;

[0140] At the same time, determine the curvature k of the neighborhood point Q Q The curvature k of the seed point P0 Q Whether condition 2 (similarity of curvature) is met:

[0141] |k Q -k P |≤Δk threshold

[0142] If so, point Q and point P0 are included in the same growth region, and all points in the neighborhood of seed point P0 are traversed in turn. All points in the neighborhood that meet conditions 1 and 2 are placed in the same growth region to form independent clusters. All clusters are traversed, and the cluster with the most point cloud points is used as the back 3D point cloud model of the object to be detected, where Δk threshold is the curvature difference threshold

[0143] 5. Correct human posture ( Figure 2 ), in a real ultrasound diagnostic environment, patients have different body shapes and lie in different positions on the examination bed. The variability of the patient's initial position and posture will have a great impact on the subsequent path planning, so it is necessary to first realize adaptive adjustment of these individual position differences.

[0144] An oriented bounding box (OBB) is used to define the contour range of the three-dimensional point cloud model of the back of the object to be detected, and a symmetric midline of the back of the object to be detected is calculated using the defined contour range, and the three-dimensional point cloud model of the back of the object to be detected is adjusted to a desired posture using the midline;

[0145] Specifically, the center position of the OBB can be regarded as the point where the symmetric midline of the human body must pass through, and the center point C of the bounding box is calculated. bb =(center x ,center y ,center z ), and the centroid of the line segment defined by the OBB vertices, and then with C obb Construct a reference line for the midline of the human back model. This midline represents the lying direction of the human body and also provides a clear reference for the symmetry axis for path planning, ensuring that the positioning operation can be performed along the natural axis of the human body.

[0146] C bb =(center x ,centery ,center z )

[0147] Where N is the number of points that make up the line segment, P i is a point on the line segment.

[0148] Then, the orientation of the human body in the bounding box (i.e., the direction pointed by the center line of the human body) is aligned with the coordinate axis, and the human body model is corrected to the desired posture. The cosine value of the angle between it and the y-axis of the camera coordinate system is calculated using the dot product formula.

[0149]

[0150] in, is the y-axis unit vector, and is the vector of the human body reference center line. After the angle θ is obtained by the inverse cosine function, the rotation matrix R can be constructed to rotate the point cloud data around the y-axis.

[0151]

[0152] Rotate P by constructing the rotation matrix R ds , OBB and center line, can accurately adjust the human body point cloud to align with the camera coordinate axis, and the human body point cloud changes from distorted to vertical, ensuring the consistency of subsequent operations. Through the above steps, no matter what degree of tilt or offset the patient initially presents, the algorithm can effectively correct its posture to the set standard state. Figure 3 As shown in the figure, the model is tilted and is restored to a state strictly parallel to the coordinate axis through algorithm processing, which is equivalent to simulating the correction process from a non-standard posture to an ideal detection posture.

[0153] Find θ, use θ to find the rotation matrix, and use the rotation matrix (cross-multiply with the 3D point cloud model) to adjust the pose.

[0154] 6. Feature line detection. Feature curves that are instructive for kidney ultrasound scanning are identified from the preprocessed human point cloud model, including the spine and waist lines. A micro-dimensional segmentation analysis method is used to refine and identify potential scanning path feature lines as reference curves for locating the kidneys.

[0155] 7. Use the slicing method to cut multiple slices of the back 3D point cloud model at equal intervals along the direction perpendicular to the spine, calculate the point cloud data of the slice with the highest average value along the Z axis direction of the camera coordinate system, and use the point cloud data of the slice as the waistline point cloud data, such as Figure 4 As shown;

[0156] 8. Indexing is performed on multiple slices of the back 3D point cloud model cut at equal intervals along the direction perpendicular to the spine, and a starting index s and an ending index e are set according to the proportion of the human body to remove the slices within the head and buttocks to obtain the back 3D point cloud model;

[0157] 9. Use the slicing method to cut multiple slices of the back three-dimensional point cloud model at equal intervals along the spine direction, calculate the point cloud data of the slice with the highest average value along the Z-axis direction of the camera coordinate system, and use the point cloud data of the slice as the spine line point cloud data;

[0158] First, uniformly slice the cross section of the 3D point cloud data of the back of the human body (in the Y-axis direction of the camera coordinate system) to simplify the complex 3D space problem into a series of 2D plane problems. The slice width Δy is set to maintain sufficient resolution while avoiding excessive computational complexity. The number of slices is:

[0159]

[0160] Each slice S i Represents the point cloud within a specific height range [h i,min ,h i,max ] is a cross section. Set a start index s and an end index e to remove the slices within the head and hip range, focus on the lowest waistline slice of the back of the human body and reduce unnecessary data analysis burden. The removal ratio is determined according to the human body proportion:

[0161]

[0162] For the retained intermediate slices, calculate the average point cloud height of each slice on the Z axis (vertical height direction)

[0163]

[0164] Among them, z P is the coordinate of the pth slice on the Z axis, and |S i | is slice S i The number of midpoints, p represents the pth slice. By finding the slice index corresponding to the maximum average height, mark the slice R where the waist is the lowest, because the slice below is usually the hip bone, which can be excluded according to the anatomical position of the kidney.

[0165] Similarly, the spine line is found in this way. The previously selected middle slices are merged into an overall point cloud. The same slice width is set to repeat the micro-dimensional segmentation process to maintain the continuity and consistency of the analysis. The edge point clouds on both sides are removed to reduce the impact, and the average height of the remaining slices is calculated. The slice with the lowest height points to the most concave area on the back, which is the spine line J.

[0166] Scanning path planning. After the human back is initially divided using the spine line and the lowest waist line, in order to allow the scanning path to pass through the surface of the kidney area, the path is designed to be along one side of the costovertebral angle according to the general anatomical structure of the human kidney, and the path passes through the kidney area.

[0167] 10. First, extract the respective point coordinates R from the previously identified spine line point cloud and the lowest waist line point cloud P , J P Direct comparison is performed to create the intersection of two point clouds, search for the point cloud coordinates of the intersection area of ​​the spine line point cloud and the lowest waist line point cloud, take the maximum point Y of the point cloud in the intersection area as the starting point, set a local interception window with a length of m according to the human body structure, intercept a section of the spine line, and use the intersection of the intercepted spine line and the local interception window as the vertex. Use the plane projection method to rotate 45 degrees to the left and right respectively, and use the rotated spine line as the ultrasound scanning path for the kidneys on both sides.

[0168] I={p|p∈R P ∩J P}

[0169] A local interception window is set based on the maximum Y value coordinate point, and a specific vertical area around the junction of the spine and waistline is selected, representing an edge of the rib angle. The range setting is corrected by redistributing the minimum and maximum values ​​to ensure the rationality of the selected range. This process can be expressed as:

[0170] S={p∈J P :y low ≤p y ≤y up}

[0171] Then, according to the angle of the costovertebral angle, the other side of the costovertebral angle is found between 45° and 60°. Since most of the kidney is within the angle range of the costovertebral angle, the initial value set by the algorithm is 45°, and the angle is reduced later according to the scanning situation. The previously determined spinal edge point cloud is fitted with a three-dimensional straight line, and the least squares method is used to solve the straight line parameters, which are projected onto the XOY plane (a uniform dot matrix is ​​created on the XOY plane to assist visual display and subsequent spatial operations), and the straight line is rotated to generate the costovertebral angle. The rotated straight line is projected onto the human body model point cloud, and a set of point cloud data located on or close to the projection plane is screened out by calculating the shortest distance from the point to the projection plane, which is the other side of the costovertebral angle.

[0172] Finally, the filtered point cloud is merged with the previously defined spine line point cloud to form a new composite point cloud, which is the scanning path of the ultrasonic robot. Since the robot end motion is point-to-point, in order to make the robot scan along this point cloud path, it is only necessary to pass a series of key points to the robot. Therefore, the extracted point cloud path must be downsampled to obtain the path points (such as Figure 5 shown).

[0173] When the robot executed the scanning path, a renal ultrasound scan was performed on a healthy adult using the ultrasound robot to autonomously plan the path. The scan was repeated 30 times to quantify the stability and success rate of renal ultrasound image acquisition. The healthy adult was in a prone position to simulate the ultrasound examination process in a clinical environment. The robot system captured the three-dimensional point cloud data of the healthy adult's waist and back through a depth camera, and then performed point cloud preprocessing, human posture correction, feature line detection and feature area recognition steps for path planning.

[0174] In each set of scans, the robot system operates according to a standardized process, starting with the automatic execution of point cloud data acquisition and preprocessing processes, including noise reduction, filtering, and point cloud segmentation, automatic detection of the spine line and the thinnest waist area, combined with the recognition of the costovertebral angle, and planning a scanning path that can reach the left kidney area (such as Figure 6 The robot autonomously controls the ultrasound probe to move along the set trajectory according to the planned path, while collecting ultrasound images in real time. During the whole process, the movement speed and strength of the robot system are finely adjusted to ensure the stability and safety of the operation and avoid any discomfort to the healthy adult. The healthy adult has an emergency stop button in his hand and can stop the process if he feels any discomfort.

[0175] The ultrasound image feedback during the ultrasound scanning process was recorded as a video to accurately record the time when the kidney image first appeared, reflecting the speed of the system to identify and locate the kidney. At the same time, the total time of the entire scanning process was recorded to evaluate the operating efficiency of the robotic system.

[0176] Clear kidney slice images were successfully obtained in 28 of the 30 experiments. The kidneys did not appear in the 14th and 29th scans marked by red boxes. Figure 7As shown. The scanning results show that the robot system has a success rate of 93.33% in obtaining kidney slice images in 30 repeated experiments, which proves the high stability and accuracy of the path planning method described in the present invention. The average time for the first appearance of the kidney image is about 4 seconds and 22 minutes, indicating that the system has the ability to quickly locate the target area. The average time taken for the entire scanning process is 17.371 seconds, which reflects the high efficiency of the robot operation. The success of the robot ultrasound scanning not only verifies the feasibility of the path planning strategy, but also highlights its significant advantages in improving the efficiency and accuracy of ultrasound scanning.

[0177] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A path planning method for a kidney ultrasound scanning robot, characterized in that: include: Step 1: Use a depth camera to capture a color image and a depth image of the object to be detected when it lies prone on a plane; use the color image and the depth image, combined with the focal length of the camera and the position of the camera's principal point, to calculate the three-dimensional coordinates of each point cloud point of the object to be detected in the world coordinate system, and then obtain a three-dimensional point cloud model of the object to be detected when it lies prone on the plane; Step 2: Draw a rectangular frame on the color image, and use the rectangular frame to segment the point cloud of the object to be detected and its adjacent plane area from the three-dimensional point cloud model in step 1; obtain a three-dimensional point cloud model that only includes the object to be detected and its adjacent plane area; Step 3: Using a random sampling consensus algorithm to remove the plane area in the three-dimensional point cloud model described in step 2, to obtain a three-dimensional point cloud model of the object to be detected; Step 4: Use a directional bounding box to define the contour range of the three-dimensional point cloud model of the object to be detected, use the defined contour range to calculate the symmetry center line of the back of the object to be detected, and use the symmetry center line to adjust the three-dimensional point cloud model of the object to be detected to the desired posture; Step 5: Use the slicing method to cut the three-dimensional point cloud model of the object to be detected in the desired posture state to obtain the waistline and spine line of the object to be detected; Step six, search for the point cloud coordinates of the intersection area of ​​the spine line and the waist line, take the point with the largest Y value in the point cloud coordinates in the intersection area as the starting point, set a local interception window with a length of m according to the human body structure, intercept a section of the spine line, and use the intersection of the intercepted spine line and the local interception window as the vertex. Use the plane projection method to rotate the intercepted spine line 45 degrees to the left and right respectively, and use the rotated spine line as the ultrasound scanning path of the kidneys on both sides, and m is 8cm-15cm.

2. The path planning method of a kidney ultrasound scanning robot according to claim 1, characterized in that: Step three also includes the step of using a region growing algorithm to remove noise and outliers in the three-dimensional point cloud model of the object to be detected.

3. The path planning method of a kidney ultrasound scanning robot according to claim 2, characterized in that: In step 2, the specific method for obtaining a three-dimensional point cloud model containing only the object to be detected and its adjacent plane area is: Create a binary mask layer with the same size as the color image and all elements are zero. The mask value in the rectangular box is set to 255. The rectangular box is used to select the object to be detected and its adjacent plane area to obtain a three-dimensional point cloud model of the object to be detected and its adjacent plane area. The rectangular frame is: P ROI ={pp=f(I,D,M,(x,y)),M(x,y)=255} Where P ROI represents the point in the rectangular box, f represents the inverse projection function of the camera model, M(x, y) represents a binary mask of the same size as the color image but all zeros, I represents the color image of the object to be detected lying prone on the plane, D represents the depth image of the object to be detected lying prone on the plane, and p represents a point in the three-dimensional point cloud model.

4. The path planning method of a kidney ultrasound scanning robot according to claim 3, characterized in that: In step 3, the method for obtaining the 3D point cloud model of the object to be detected is: Using the formula: n T x+d=0 The plane area in the three-dimensional point cloud model described in step 2 is removed, wherein n is the normal vector of the plane, x is the coordinate of the plane point in the three-dimensional space, and d is a constant term.

5. The path planning method of a kidney ultrasound scanning robot according to claim 4, characterized in that: In step 3, the specific method of obtaining the three-dimensional point cloud model of the object to be detected is: First, the curvature values ​​of the points in the point cloud model are sorted, and the point with the smallest curvature value is selected as the initial seed point. According to the preset parameters and the growth criterion, the neighboring points with similar characteristics are merged into the same area until all the point cloud points are assigned to the corresponding areas to form multiple independent clusters. Select the independent cluster containing the most point cloud points as the 3D point cloud model of the object to be detected; The preset parameters include: the minimum number of point cloud points in independent clusters, the maximum number of point cloud points in independent clusters, the number of neighborhood search points, the smoothing threshold and the curvature threshold; The specific method of merging neighboring points with similar characteristics into the same area is: Calculate the normal vector between the seed point P0 and any point Q in its neighborhood and The angle θ between them is determined to determine whether the angle θ satisfies condition 1: If so, then point Q is in the same direction as point P0, · represents the vector dot product, and are the modulus of the vector, θ threshold Indicates the angle threshold; At the same time, determine the curvature k of the neighborhood point Q Q The curvature k of the seed point P0 Q Whether condition 2 is met: |k Q -k P |≤Δk threshold If so, point Q and point P0 are included in the same growth region, and all points in the neighborhood of seed point P0 are traversed in turn. All points in the neighborhood that meet conditions 1 and 2 are placed in the same growth region, where Δk threshold is the curvature difference threshold.

6. The path planning method of a kidney ultrasound scanning robot according to claim 5, characterized in that: In step 4, a directional bounding box is used to define the contour range of the denoised three-dimensional point cloud model of the back of the object to be detected, and the symmetric midline of the back of the object to be detected is calculated using the defined contour range: Calculate the center point C of the bounding box obb And the centroid C2 of the line segment defined by the vertices of the oriented bounding box, calculate the midline of the human back model: C bb =(center x ,center y ,center z ) P(t)=(1-t)·C2+t·C1,0≤t≤1 Where N is the number of points that make up the line segment, P i is a point on the line segment, i represents the i-th point on the line segment, center x ,center y ,center z They represent the coordinates of the center point of the bounding box in the three axis directions, P(t) represents the median line, and t is restricted to the interval [0,1] to ensure that P(t) is on the straight line from C2 to C1.

7. The path planning method of a kidney ultrasound scanning robot according to claim 6, characterized in that: In step 4, the method of adjusting the three-dimensional point cloud model of the object to be detected to the desired posture by using the symmetry center line is: Align the symmetric midline direction of the back of the object to be detected in the bounding box with the coordinate axis of the world coordinate system, and calculate the cosine value of the angle θ between the symmetric midline of the back of the object to be detected and the y-axis of the camera coordinate system by using the dot product formula; in, is the y-axis unit vector, is the vector of the human body reference center line; after obtaining the angle θ through the inverse cosine function, the rotation matrix R is constructed to rotate the point cloud data around the y-axis; The rotation matrix R is cross-multiplied with the three-dimensional point cloud model, and the three-dimensional point cloud model of the back of the object to be detected is adjusted to be aligned with the camera coordinate axis, so as to adjust the position and posture of the three-dimensional point cloud model of the back of the object to be detected.

8. The path planning method of a kidney ultrasound scanning robot according to claim 7, characterized in that: In step 5, the slicing method is used to cut the three-dimensional point cloud model of the back of the object to be detected in the desired posture, and the method for obtaining the waistline and spine line of the back of the target object is: First, the slicing method is used to cut the three-dimensional point cloud model of the back of the object to be detected at equal intervals along the direction perpendicular to the spine, and multiple slices are obtained. The average value of each slice along the Z-axis direction of the camera coordinate system is calculated, and the point cloud data of the slice with the highest average value is used as the waistline point cloud data; Then, the slices within the head and hip range are removed according to the human body proportions to obtain a three-dimensional point cloud model of the back of the object to be detected; Finally, a slicing method is used to cut multiple slices of the back three-dimensional point cloud model at equal intervals along the spine direction, and the average value of each slice along the Z-axis direction of the camera coordinate system is calculated, and the point cloud data of the slice with the highest average value is used as the point cloud data of the spine line.

9. The path planning method of a kidney ultrasound scanning robot according to claim 8, characterized in that: The formula for calculating the average value of each slice along the Z axis of the camera coordinate system is: Among them, z P is the coordinate of the pth slice on the Z axis, and |S i | is slice S i The number of midpoints, p represents the pth slice.

10. The path planning method of a kidney ultrasound scanning robot according to claim 9, characterized in that: Step six also includes: processing the ultrasound scanning paths of the kidneys on both sides using point cloud voxels to obtain scanning path trajectory lines.

Images