Ultrasonic robot gall bladder automatic scanning method and device

Through the ultrasonic robot combining the segmentation network and the robotic arm, the segmentation and blocking technology are used to determine the occlusion area and control the swing of the robotic arm, which solves the problem of poor gallbladder imaging quality caused by rib occlusion, and achieves complete scanning and high quality of the gallbladder Imaging.

CN118948334AActive Publication Date: 2024-11-15武汉库柏特科技股份有限公司
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411264486.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-11-15
Estimated Expiration
2044-09-10

AI Technical Summary

Technical Problem

During the ultrasound imaging of the gallbladder, due to the obstruction of the rib, it is difficult for the prior art to effectively avoid the obstruction to ensure a complete scanning of the gallbladder.

Method used

Ultrasonic robots are used to combine segmentation networks and robotic arms to determine the occlusion area through ultrasonic image segmentation, elliptical trajectory search and blocking processing, and control the swing of the robotic arm according to the positional relationship between the occlusion area and the gallbladder profile to avoid the occlusion to achieve the gallbladder. Automatic scanning.

Benefits of technology

Effectively avoid rib obstruction, ensure that the ultrasound probe can scan the complete contour of the gallbladder and improve the quality of gallbladder imaging.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118948334A_ABST
    Figure CN118948334A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of ultrasonic robots, and discloses an automatic gall bladder scanning method and device for an ultrasonic robot, and the method comprises the steps: recording the position of a gall bladder contour in an abdomen scanning process, finally calculating to obtain an optimal gall bladder position, then controlling a probe to reach the optimal gall bladder position, and then carrying out the ellipse search; when the gallbladder is searched, judging whether the outline of the gallbladder in the ultrasonic image is shielded by a shielding object or not; and if the gallbladder contour is shielded, acquiring a shielding area, shielded by a shielding object, of the gallbladder contour in the ultrasonic image, controlling the swinging direction of the mechanical arm according to the position relation between the shielding area and the gallbladder contour, and automatically scanning the gallbladder. The swing direction of the mechanical arm is controlled on the basis of the position relation between the shielding area and the gallbladder contour, so that shielding objects are avoided, and the ultrasonic probe can scan the complete gallbladder.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] The gallbladder is a pear-shaped sac located behind the liver under the right ribs. During the automatic gallbladder scan, the ultrasound robot usually scans the liver in the abdomen first, and then calculates and records the contour information of the gallbladder during the scan. After the abdominal liver scan is completed, the gallbladder is returned to the optimal position and scanned from multiple angles. However, during the gallbladder scan, it may be blocked by the ribs, resulting in poor gallbladder ultrasound imaging. Summary of the invention

[0003] In order to overcome the problem that the gallbladder may be blocked by ribs during gallbladder scanning, resulting in poor gallbladder ultrasound imaging quality, the present invention provides an ultrasonic robot gallbladder automatic scanning method and device.

[0004] In a first aspect, in order to solve the above technical problems, the present invention provides an automatic gallbladder scanning method using an ultrasonic robot, comprising:

[0005] Scanning the abdomen of a human body according to a preset path, acquiring an ultrasound image through the ultrasound probe during the scanning process, segmenting the gallbladder contour in the ultrasound image through a segmentation network, and calculating the position and posture of the gallbladder segment formed by the continuous gallbladder contour in the world coordinate system;

[0006] After the human abdomen is scanned, the target gallbladder segment is calculated according to the weight, the ultrasound probe is controlled to reach the position corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is found;

[0007] After the gallbladder is found, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block where an occluder exists;

[0008] Based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, determining whether the gallbladder contour in the ultrasound image is blocked by an occluder in each target block;

[0009] If the gallbladder contour is blocked, the blocked area of ​​the gallbladder contour in the ultrasound image is obtained, and according to the positional relationship between the blocked area and the gallbladder contour, the swing direction of the robot arm is controlled to automatically scan the gallbladder.

[0010] In a second aspect, the present invention provides an ultrasonic robot gallbladder automatic scanning device, comprising: an ultrasonic robot and a terminal device, wherein the ultrasonic robot comprises a mechanical arm and an ultrasonic probe, the ultrasonic probe is arranged at the end of the mechanical arm, and the terminal device is respectively connected to the mechanical arm and the ultrasonic probe, and the terminal device is specifically used for:

[0011] Scanning the abdomen of a human body according to a preset path, acquiring an ultrasound image through the ultrasound probe during the scanning process, segmenting the gallbladder contour in the ultrasound image through a segmentation network, and calculating the position and posture of the gallbladder segment formed by the continuous gallbladder contour in the world coordinate system;

[0012] After the human abdomen is scanned, the target gallbladder segment is calculated according to the weight, the ultrasound probe is controlled to reach the position corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is found;

[0013] After the gallbladder is found, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block where an occluder exists;

[0014] Based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, determining whether the gallbladder contour in the ultrasound image is blocked by an occluder in each target block;

[0015] If the gallbladder contour is blocked, the blocked area of ​​the gallbladder contour in the ultrasound image is obtained, and according to the positional relationship between the blocked area and the gallbladder contour, the swing direction of the robot arm is controlled to automatically scan the gallbladder.

[0016] The beneficial effects of the present invention are as follows: when scanning the gallbladder, the ultrasound image is divided into blocks, and the target block is determined based on the pixel value of each block, that is, the positional relationship between the gallbladder contour and each target block in the ultrasound image can be used to determine whether the gallbladder contour is blocked by the occlusion in each target block. If there is occlusion, the swing direction of the robot arm is controlled according to the positional relationship between the occlusion area and the gallbladder contour, and the gallbladder is automatically scanned. The present application controls the swing direction of the robot arm based on the positional relationship between the occlusion area and the gallbladder contour, thereby avoiding the occlusion and enabling the ultrasound probe to scan the complete gallbladder. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0018] Figure 1 A schematic diagram of a flow chart of an automatic gallbladder scanning method using an ultrasonic robot according to an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of an ultrasound image being blocked by an occluder;

[0020] Figure 3 Schematic diagram of the established plane coordinate system o-xy;

[0021] Figure 4 is a schematic diagram of the first boundary coordinate point on the plane coordinate system;

[0022] Figure 5 is a schematic diagram of the first angle on the plane coordinate system;

[0023] Figure 6 It is a schematic diagram of the angles corresponding to the left and right boundaries in the ultrasound image on the plane coordinate system;

[0024] Figure 7 It is a schematic diagram of the blocks in the ultrasound image;

[0025] Figure 8 is a schematic diagram of the second angle on the plane coordinate system;

[0026] Fig. 9 Schematic diagram of the swing direction of the ultrasonic probe;

[0027] Fig.10 This is a schematic diagram showing that the current scanning position is above the ribs;

[0028] Fig.11 This is a schematic diagram showing that the current scanning position is below the ribs;

[0029] Fig.12 The figure is a schematic structural diagram of an ultrasonic robot automatic gallbladder scanning device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following examples are provided to further explain and supplement the present invention and do not constitute any limitation to the present invention.

[0031] The following describes an ultrasonic robot gallbladder automatic scanning method and device according to an embodiment of the present invention in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, the present invention provides an automatic gallbladder scanning method using an ultrasonic robot, comprising:

[0033] S1. Scan the human abdomen along a preset path, acquire an ultrasound image through the ultrasound probe during the scanning process, segment the gallbladder contour in the ultrasound image through a segmentation network, and calculate the position of the gallbladder segment composed of the continuous gallbladder contour in the world coordinate system.

[0034] S2. After the human abdomen is scanned, the target gallbladder segment is calculated according to the weight, the ultrasound probe is controlled to reach the position corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is found.

[0035] S3. After the gallbladder is found, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block where an occluder exists.

[0036] S4. Based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, determine whether the gallbladder contour in the ultrasound image is blocked by an occluder in each target block.

[0037] S5. If the gallbladder contour is blocked, the blocked area of ​​the gallbladder contour in the ultrasound image is obtained, and according to the positional relationship between the blocked area and the gallbladder contour, the swing direction of the robot arm is controlled, and the gallbladder is automatically scanned.

[0038] In this embodiment, when scanning the gallbladder, the ultrasound image is divided into blocks, and the target block is determined based on the pixel value of each block. The positional relationship between the gallbladder contour and each target block in the ultrasound image can be used to determine whether the gallbladder contour is blocked by the obstructions in each target block. If there is an obstruction, the swing direction of the robot arm is controlled according to the positional relationship between the obstruction area and the gallbladder contour, and the gallbladder is automatically scanned. The present application controls the swing direction of the robot arm based on the positional relationship between the obstruction area and the gallbladder contour, thereby avoiding obstructions and enabling the ultrasound probe to scan the complete gallbladder.

[0039] The segmentation model in this embodiment is a common segmentation model, such as a U-Net segmentation network.

[0040] In this embodiment, the position of the gallbladder in the ultrasound image is transformed into the world coordinate system, and the method for calculating the position of the gallbladder in the world coordinate system is as follows:

[0041] After the gallbladder contour is obtained, an ellipse fitting is performed on the gallbladder contour to obtain the ellipse contour, and the center point of the gallbladder contour (c x , c y ), the center point is converted to the physical space coordinates (P y , P z ):

[0042] P y =c x ×(W / validW)

[0043] P z =c y ×(W / validW);

[0044] Wherein, W is the physical distance corresponding to the width of the ultrasound image, and validW is the pixel distance corresponding to the width of the ultrasound image.

[0045] Get the coordinates P of the gallbladder in the tool coordinate system at the end of the robot arm TCP =(0,P y , P z ).

[0046] P TCPConvert to the world coordinate system and get the position P of the gallbladder in the world coordinate system B , the formula is as follows:

[0047] P B =R c P TCP +P c

[0048] Among them, R c represents the posture of the end of the robot based on the world coordinate system, P c The position of the end of the robot arm based on the world coordinate system.

[0049] In this embodiment, in order to quickly search out the gallbladder, the search trajectory of the probe at the end of the manipulator is designed according to the ellipse equation, i.e., the ellipse trajectory search. The gallbladder is affected by breathing, and the position change is mainly up and down (along the y-axis direction of the world coordinate system). Therefore, the focus of the ellipse trajectory is designed on the y-axis of the world coordinate system. The ellipse trajectory can be:

[0050]

[0051] Among them, a and b are two parameters of the elliptic equation, and the specific calculation is as follows:

[0052] The current position of the ultrasound probe P cur Take it as the center point and establish the coordinate system o. In order to ensure that the search range is large enough and take into account the search efficiency, the two foci of the ellipse (0, m) and (0, -m) are set at a distance of half the probe, and the intersection points of the ellipse on the x-axis, namely (m, 0) and (-m, 0), are set at a distance of half the probe, where m represents half the width of the probe, and the reference value is 0.02m.

[0053] According to the above description, the parameter values ​​of the elliptical trajectory equation a and b can be calculated, and then the end probe of the robotic arm can be controlled to search for the elliptical trajectory in the xy plane of the world coordinate system, as follows:

[0054] The position of the probe at the end of the robotic arm at the next moment is: (Δx, Δy), where Δy is the distance from the ultrasonic probe to P. cur The distance to start moving along the y-axis of the world coordinate system (fixed step size can be used for movement, and the recommended moving step size is 0.001m). Δx represents the distance of the end probe of the manipulator in the x-axis direction of the world coordinate system relative to P calculated based on the above ellipse equation and Δy. cur The displacement amount.

[0055] In addition, it is important to note that when the robotic arm is searching along an elliptical trajectory, if the gallbladder cannot be found after searching for a circle, or the number of gallbladders that exist does not reach a certain threshold, the gallbladder scan will not be performed, and the gallbladder scan phase will be entered instead.

[0056] Optionally, after the gallbladder is found, the ultrasound image is divided into blocks, and a target block is determined based on a pixel value of each block, including:

[0057] Establishing a plane coordinate system along the intersection point of the ultrasound image boundary, and acquiring first boundary coordinate points on the left and right sides of the ultrasound image based on the plane coordinate system;

[0058] Taking the y-axis of the plane coordinate system as the center line, determining the first angles from the left and right boundaries of the ultrasound image to the y-axis respectively according to the boundary coordinate points and the inverse tangent trigonometric function;

[0059] Dividing the ultrasound image into blocks based on a first angle corresponding to a left and right boundary in the ultrasound image;

[0060] Add the image pixels of each block and determine the sum result;

[0061] The block whose sum result is less than the pixel sum threshold is taken as the target block.

[0062] like Figure 2 As shown in the figure, the marked area is the obstruction, which is usually caused by the ultrasound probe not fitting well or the ribs blocking it. Therefore, it is necessary to judge the obstruction and the area where the obstruction is located, as follows:

[0063] like Figure 3 -As shown in FIG. 1 , a plane coordinate system o-xy is established along the intersection point o of the ultrasound image boundary, wherein the origin of the coordinate system is (x0, y0).

[0064] like Figure 4 As shown in the figure, the first boundary coordinate points on the left and right sides of the ultrasound image are obtained by the coordinate system o-xy, respectively (x l ,y l )、(x r ,y r ).

[0065] like Figure 5 As shown, taking the y-axis of the plane coordinate system as the center line, the positive and negative tangent trigonometric functions are established according to the first boundary coordinate point and the y-axis, and the first angles of the left and right boundaries in the ultrasound image to the y-axis are calculated as θ l ,θ r .

[0066] like Figure 6 As shown, according to θ l ,θ r The angle between the left and right boundaries of the ultrasound image can be obtained as θ = |θ r -θ l |θ l <0.

[0067] In order to determine the specific location of the gallbladder obstruction, the ultrasound image is divided into fan-shaped blocks, and the angle value τ of each block is:

[0068]

[0069] Wherein, n is the number of blocks, which can be set according to actual conditions. Preferably, in this embodiment, n can be 8, that is, the ultrasound image is divided into 8 blocks. Figure 7 shown.

[0070] Calculate the sum of the pixel values ​​of each block, denoted as S j (j=1, 2, ..., n, represents the number of blocks from the left to the right of the image). Set the pixel sum threshold (recommended value is 50000). If S j If S is greater than the pixel sum threshold, it means that block j is not blocked. j If it is less than the pixel sum threshold, there is occlusion in block j, and the block is taken as the target block.

[0071] Optionally, based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, determining whether the gallbladder contour in the ultrasound image is blocked by an occluder in each target block includes:

[0072] Obtaining the second boundary coordinate points of the left and right boundaries of the gallbladder contour in the plane coordinate system through the segmentation network;

[0073] Determine a second angle between the left and right edges of the gallbladder contour and the y-axis according to the second boundary coordinate point;

[0074] Taking the center line of the ultrasound image as the origin, obtain the angle range of each target block;

[0075] The second angle is compared with the angle range of each target block. If the second angle is within the angle range of a target block, it is determined that the gallbladder contour is blocked by the target block.

[0076] In this embodiment, the second boundary coordinate points of the left and right boundaries of the gallbladder contour obtained by the segmentation network in the plane coordinate system are (x cl ,y cl )、(x cr ,y cr ).

[0077] like Figure 8 As shown in the figure, the second angles of the left and right edges of the gallbladder contour to the y-axis (the coordinates of the origin of the plane coordinate system are (x0, y0)) are θ cl ,θ cr , where θ cl ,θ cr The calculation formula is as follows:

[0078]

[0079] Taking the center line of the ultrasound image as the origin, the angle range of the jth target block is:

[0080]

[0081] where θ jmin represents the minimum value of the angle range of the jth target block; θ jmax represents the maximum value of the angle range of the jth target block. If θ cl ≥θ jmin And θ cl ≤θ jmax , or θ cr ≥θ jmin And θ cr ≤θ jmax , it means that the gallbladder contour in the ultrasound image is blocked at the jth target block.

[0082] Optionally, controlling the swing direction of the robotic arm according to the positional relationship between the shielded area and the gallbladder contour includes:

[0083] If the blocked area is located on the right side of the gallbladder outline, the robot arm is controlled to swing in the first preset direction; if the blocked area is located on the left side of the gallbladder outline, the robot arm is controlled to swing in the second preset direction, as follows:

[0084]

[0085] Where dir represents the swing direction. If θ cr ≥θ jmin And θ cr ≤θ jmax , it means that the occluded area is on the right side of the gallbladder contour, and the swing direction is dir=1, that is, the first preset direction; if θ cl ≥θ jmin And θ cl ≤θ jmax , it means that the blocked area is on the left side of the gallbladder contour, and the swing direction is dir=-1, which is the second preset direction.

[0086] Optionally, controlling the swing direction of the robotic arm according to the positional relationship between the shielded area and the gallbladder contour further includes:

[0087] Controlling the mechanical arm to swing and scan, performing a first total number of occlusion judgments on the gallbladder contour to determine whether there are occlusions on the gallbladder contour;

[0088] If it is determined within the first total number of times that there is an obstruction on the gallbladder contour, the current scanning position of the robot arm is obtained;

[0089] If the current scanning position is above the ribs, the robot arm is controlled to swing around the y-axis of the tool coordinate system at the end of the robot arm, and the gallbladder is automatically scanned;

[0090] If the current scanning position is below the ribs, the robotic arm is controlled to swing around the x-axis of the tool coordinate system at the end of the robotic arm, and the gallbladder is automatically scanned.

[0091] like Fig. 9 As shown, by controlling the robotic arm (ultrasound probe) to rotate around the x-axis or y-axis of the tool coordinate system at the end of the robotic arm, the completeness of the gallbladder contour scan can be improved to a certain extent.

[0092] In this embodiment, the first total number of times can be set according to actual conditions.

[0093] Optionally, if the current scanning position is above the ribs, the robotic arm is controlled to swing around the y-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder, including:

[0094] If the current scanning position is above the ribs, controlling the robotic arm to rotate around the y-axis of the tool coordinate system at the end of the robotic arm for a second total number of times to automatically scan the gallbladder;

[0095] Before each rotation, the robot arm calculates the posture angle vector required for the current rotation as (0, Δr yi , 0);

[0096] Δr yi The calculation formula is as follows:

[0097] Δr yi =dir*r ystep

[0098] Among them, r ystep Indicates the angular step of the current rotation of the manipulator around the y-axis of the tool coordinate system of the manipulator end. If the occluded area is on the right side of the gallbladder contour, dir represents the first preset direction. If the occluded area is on the left side of the gallbladder contour, dir represents the second preset direction. Δr yi Indicates the angle that the robot arm needs to swing around the y-axis of the end tool coordinate system for the current rotation.

[0099] like Fig.10 As shown in FIG. 1 , if the current scanning position is above the ribs, the robot arm is controlled to swing around the y-axis of the tool coordinate system at the end of the robot arm to avoid the ribs for gallbladder scanning, and the posture angle vector of the robot arm during each swing is (0, Δr yi , 0).

[0100] Among them, r ystep It can be set according to the actual situation. Preferably, in this embodiment, rystep =0.008rad.

[0101] In this embodiment, it is necessary to control the rotation range of the robot arm around the y-axis of the tool coordinate system at the end of the robot arm, that is, the maximum search range around the y-axis is r ymax , r ymax It is set according to the actual situation. Preferably, in this embodiment, ymax =0.8rad, then according to r ymax and r ystep The ratio between them determines the total number of rotations, i.e., the second total number, to ensure that the rotation range of the robot arm around the y-axis is controlled within r ymax Within.

[0102] Optionally, if the current scanning position is above the ribs, controlling the robotic arm to rotate around the y-axis of the tool coordinate system at the end of the robotic arm a second total number of times to automatically scan the gallbladder, further comprising:

[0103] Each time the robot rotates, the current robot arm is controlled to translate along the x-axis of the tool coordinate system at the end of the robot arm, where the translation vector is (Δp fxtcpi , 0, Δp ztcpi );

[0104] Δp fxtcpi The calculation formula is as follows:

[0105]

[0106] Among them, Δp fxtcpi Indicates the distance moved along the x-axis of the tool coordinate system at the end of the robot due to lateral force, Δp ztcpi Indicates the displacement of the robot arm along the x-axis of the tool coordinate system at the end of the robot arm. x0 represents the preset force threshold, f xi It represents the force along the x-axis of the tool coordinate system at the end of the robot arm each time the robot arm rotates, and δ represents the preset proportional coefficient.

[0107] In this embodiment, by adding a translation vector during each rotation, the ultrasound probe is automatically inserted into the intercostal space, so that the ultrasound probe at the end of the robotic arm can scan the gallbladder in a flexible position, thereby improving the integrity of the gallbladder scan.

[0108] In this embodiment, x0 , δ are set according to actual conditions. Preferably, in this embodiment, f x0 =10N,δ=0.002.

[0109] In this embodiment, the robot arm's posture angle vector is (0, Δr yi , 0), the translation vector is (Δp fxtcpi , 0, Δpztcpi ) to achieve smooth control of the position and posture of the ultrasonic probe at the end of the robotic arm.

[0110] Optionally, if the current scanning position is below the ribs, the robotic arm is controlled to swing around the x-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder, including:

[0111] If the current scanning position is below the ribs, controlling the robotic arm to rotate a third total number of times around the x-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder;

[0112] Among them, before each rotation, the posture angle vector that the robot arm needs to move in the current rotation is calculated as (Δr xi , 0, 0);

[0113] Δr xi The calculation formula is as follows:

[0114] Δr xi =dir*r xstep

[0115] Among them, r xstep Indicates the angular step of the current rotation of the manipulator around the x-axis of the tool coordinate system of the manipulator end. If the occluder is on the right side of the gallbladder contour, dir represents the first preset direction. If the occluder is on the left side of the gallbladder contour, dir represents the second preset direction. Δr xi Indicates the angle that the robot arm needs to swing around the x-axis of the end tool coordinate system for the current rotation.

[0116] In this embodiment, Fig.11 As shown, if the current scanning position is under the ribs, in order to ensure that the probe can avoid the ribs, the robotic arm is controlled to swing around the x-axis of the tool coordinate system at the end of the robotic arm.

[0117] Among them, r xstep It can be set according to the actual situation. Preferably, in this embodiment, r xstep =0.008rad.

[0118] In this embodiment, it is necessary to control the rotation range of the robot arm around the x-axis of the tool coordinate system at the end of the robot arm, that is, the maximum search range around the x-axis is r xmax , r xmax It is set according to the actual situation. Preferably, in this embodiment, xmax =1.0rad, then according to r xmax and r xstep The ratio between them determines the total number of rotations, i.e., the third total number, to ensure that the range of rotation of the robot arm around the x-axis is controlled within r xmax Within.

[0119] Optionally, the human abdomen is scanned according to a preset path, an ultrasound image is acquired by the ultrasound probe during the scanning process, and the gallbladder contour in the ultrasound image is segmented by a segmentation network, and the position of the gallbladder segment constituted by the continuous gallbladder contour in the world coordinate system is calculated, including:

[0120] The gallbladder contour in the liver area of ​​the abdomen is segmented by a segmentation network, and the aspect ratio of the gallbladder contour corresponding to each section is calculated;

[0121] If the aspect ratio corresponding to the current section meets the preset range, the gallbladder section contour information corresponding to the current section is stored in the corresponding container; wherein the gallbladder section contour information includes the gallbladder area, aspect ratio, position vector of the ultrasound probe and posture matrix of the ultrasound probe corresponding to the current section;

[0122] The continuous sections are regarded as a gallbladder segment, and all gallbladder segments in the container are traversed to determine the gallbladder segment information corresponding to each gallbladder segment; wherein the gallbladder segment information includes the length of the current gallbladder segment, the target area, the position vector of the ultrasound probe corresponding to the target area, the posture matrix of the ultrasound probe corresponding to the target area, and the number of sections corresponding to the current gallbladder segment, and the target area is the maximum value of the gallbladder areas corresponding to the sections in the current gallbladder segment;

[0123] Acquire the maximum value of the gallbladder segment length among all the gallbladder segments as a first reference value, and the maximum value of the gallbladder segment target area among all the gallbladder segments as a second reference value;

[0124] Calculate the length ratio of each gallbladder segment to the first reference value, and the area ratio of the target area to the second reference value;

[0125] According to the length proportion and area proportion corresponding to each gallbladder segment, the weighted score of each gallbladder segment is calculated by weighted score, and the gallbladder segment corresponding to the maximum weighted score is taken as the target gallbladder segment;

[0126] The position vector and posture matrix corresponding to the target gallbladder segment are used as the position and posture corresponding to the target gallbladder segment.

[0127] In this embodiment, the ultrasonic robot usually scans the liver first, and then records and analyzes the morphology of the gallbladder during the scanning process. After scanning the liver, the robot returns to the optimal position of the gallbladder to scan the gallbladder. When scanning the liver area, ultrasonic image sections are obtained one by one from the ultrasonic probe. In addition, continuous sections refer to the first time a section with a gallbladder length-to-width ratio that meets a preset range is obtained, until a section with a gallbladder length-to-width ratio that meets the preset range is not scanned for multiple consecutive times. The above-mentioned continuously scanned gallbladder sections with a length-to-width ratio that meets the preset range are regarded as continuous sections, that is, a gallbladder segment.

[0128] The length l and width w of the gallbladder contour after fitting are obtained through the U-Net segmentation network, so as to calculate the length-to-width ratio lwRatio=l / w (w≠0) of the gallbladder contour corresponding to each section.

[0129] Establish a container partList to store the information of each gallbladder segment. partList[i] represents the gallbladder information corresponding to the i-th gallbladder segment in the container. If the gallbladder aspect ratio meets the preset range, the gallbladder contour information is stored in partList[i], where the preset range is [r0,1), that is, the value of lwRatio falls within the range [r0,1), then the gallbladder area, aspect ratio, position vector of the ultrasound probe and the posture matrix of the ultrasound probe corresponding to the section are stored in partList[i], and partList[i] = {…, (area, lwRatio, P c ,R c )}, area represents gallbladder area, P c , R c Represents the position vector and attitude matrix of the ultrasound probe respectively. If no gallbladder data is stored in partList[i] for k consecutive times, it means that the current continuous gallbladder segment has been scanned. At this time, i is increased by 1, k is a fixed threshold, and the recommended value is 5.

[0130] In this embodiment, r0 is a fixed threshold value, which is set according to actual conditions. Preferably, in this embodiment, r0=0.6.

[0131] After scanning all liver regions according to the above method, all partList[i] are traversed to calculate the gallbladder segment information corresponding to each gallbladder segment and store it in the container partStats[i] corresponding to the gallbladder segment. partStats[i] represents the gallbladder segment information corresponding to the i-th gallbladder segment. i ,P cmaxi ,R cmaxi ,L i}, where maxArea i represents the maximum gallbladder area corresponding to each section in partList[i] corresponding to the i-th gallbladder segment, i.e., the target area. cmaxi , R cmaxi They represent the position vector and posture matrix of the ultrasound probe corresponding to the target area in partList[i] corresponding to the i-th gallbladder segment, respectively. i Represents the length corresponding to the i-th gallbladder segment.

[0132] At this time, since there are many gallbladder segments, it is necessary to find the most suitable gallbladder posture from each gallbladder segment as the search starting position of the ultrasound probe. Therefore, by traversing partStats[i], the maximum length of the gallbladder segment among all gallbladder segments can be obtained as the first reference value L max , and each maxArea i The maximum target area in is used as the second reference value allMaxArea.

[0133] Calculate the length ratio of each gallbladder segment r Li =L i / L max , and area ratio r ai =maxArea i / allMaxArea,r Li represents the length ratio of the i-th gallbladder segment, r ai Represents the area proportion of the i-th gallbladder segment.

[0134] Calculate the weighted score of the i-th gallbladder segment as follows:

[0135] score=αr Li +βr ai

[0136] α+β=1

[0137] Wherein, score represents a weighted score, α and β are preset weight coefficient values, which are set according to actual conditions. Preferably, in this embodiment, α=0.6 and β=0.4.

[0138] According to the above weighted score calculation method, the target gallbladder segment with the highest score, partStats[i best ], and from partStats[i best ] to obtain P cmaxi , R cmaxi As the position vector and posture matrix of the probe at the end of the robotic arm, that is, the position and posture corresponding to the target gallbladder.

[0139] In this embodiment, the terminal device may be a computer, which will not be described in detail here.

[0140] like Fig.12 As shown, an ultrasonic robot gallbladder automatic scanning device includes: an ultrasonic robot and a terminal device, the ultrasonic robot includes a mechanical arm and an ultrasonic probe, the ultrasonic probe is arranged at the end of the mechanical arm, the terminal device is connected to the mechanical arm and the ultrasonic probe respectively, and the terminal device is specifically used for:

[0141] Scanning the abdomen of a human body according to a preset path, acquiring an ultrasound image through the ultrasound probe during the scanning process, segmenting the gallbladder contour in the ultrasound image through a segmentation network, and calculating the position and posture of the gallbladder segment in a world coordinate system;

[0142] After the human abdomen is scanned, the target gallbladder segment is calculated according to the weight, the ultrasound probe is controlled to reach the position corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is found;

[0143] After the gallbladder is found, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block where an occluder exists;

[0144] Based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, determining whether the gallbladder contour in the ultrasound image is blocked by an occluder in each target block;

[0145] If the gallbladder contour is blocked, the blocked area of ​​the gallbladder contour in the ultrasound image is obtained, and according to the positional relationship between the blocked area and the gallbladder contour, the swing direction of the robot arm is controlled to automatically scan the gallbladder.

[0146] Optionally, the terminal module is specifically used for:

[0147] Establishing a plane coordinate system along the intersection point of the ultrasound image boundary, and acquiring first boundary coordinate points on the left and right sides of the ultrasound image based on the plane coordinate system;

[0148] Taking the y-axis of the plane coordinate system as the center line, determining the first angles from the left and right boundaries in the ultrasound image to the y-axis respectively according to the first boundary coordinate point and the inverse tangent trigonometric function;

[0149] Dividing the ultrasound image into blocks based on a first angle corresponding to a left and right boundary in the ultrasound image;

[0150] Add the image pixels of each block and determine the sum result;

[0151] The block whose sum result is less than the pixel sum threshold is taken as the target block.

[0152] Optionally, the terminal module is specifically used for:

[0153] Obtaining the second boundary coordinate points of the left and right boundaries of the gallbladder contour in the plane coordinate system through the segmentation network;

[0154] Determine a second angle between the left and right edges of the gallbladder contour and the y-axis according to the second boundary coordinate point;

[0155] Taking the center line of the ultrasound image as the origin, obtain the angle range of each target block;

[0156] The second angle is compared with the angle range of each target block. If the second angle is within the angle range of a target block, it is determined that the gallbladder contour is blocked by the target block.

[0157] Optionally, the terminal module is specifically used to include:

[0158] If the shielding area is located on the right side of the gallbladder outline, the robotic arm is controlled to swing in a first preset direction; if the shielding area is located on the left side of the gallbladder outline, the robotic arm is controlled to swing in a second preset direction.

[0159] Optionally, the terminal module is also used to:

[0160] Controlling the mechanical arm to swing and scan, performing a first total number of occlusion judgments on the gallbladder contour to determine whether there are occlusions on the gallbladder contour;

[0161] If it is determined within the first total number of times that there is an obstruction on the gallbladder contour, the current scanning position of the robot arm is obtained;

[0162] If the current scanning position is above the ribs, the robot arm is controlled to swing around the y-axis of the tool coordinate system at the end of the robot arm, and the gallbladder is automatically scanned;

[0163] If the current scanning position is below the ribs, the robotic arm is controlled to swing around the x-axis of the tool coordinate system at the end of the robotic arm, and the gallbladder is automatically scanned.

[0164] Optionally, the terminal module is specifically used for:

[0165] If the current scanning position is above the ribs, controlling the robotic arm to rotate around the y-axis of the tool coordinate system at the end of the robotic arm for a second total number of times to automatically scan the gallbladder;

[0166] Before each rotation, the robot arm calculates the posture angle vector required for the current rotation as (0, Δr yi , 0);

[0167] Δr yi The calculation formula is as follows:

[0168] Δr yi =dir*r ystep

[0169] Among them, r ystep Indicates the angular step of the current rotation of the manipulator around the y-axis of the tool coordinate system of the manipulator end. If the occluded area is on the right side of the gallbladder contour, dir represents the first preset direction. If the occluded area is on the left side of the gallbladder contour, dir represents the second preset direction. Δr yi Indicates the angle that the robot arm needs to swing around the y-axis of the end tool coordinate system for the current rotation.

[0170] Optionally, the terminal module is also used to:

[0171] Each time the robot rotates, the current robot arm is controlled to translate toward the x-axis direction of the tool coordinate system at the end of the robot arm, where the translation vector is (Δp fxtcpi , 0, Δp ztcpi );

[0172] Δp fxtcpi The calculation formula is as follows:

[0173]

[0174] Among them, Δp fxtcpi Indicates the distance moved along the x-axis of the tool coordinate system at the end of the robot due to lateral force, Δp ztcpi Indicates the displacement of the robot arm along the x-axis of the tool coordinate system at the end of the robot arm. x0 represents the preset force threshold, f xi It represents the force of the robot arm along the x-axis of the tool coordinate system at the end of the robot arm, and δ represents the preset proportional coefficient.

[0175] Optionally, the terminal module is specifically used for:

[0176] If the current scanning position is below the ribs, controlling the robotic arm to rotate a third total number of times around the x-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder;

[0177] Among them, before each rotation, the posture angle vector that the robot arm needs to move in the current rotation is calculated as (Δr xi , 0, 0);

[0178] Δr xi The calculation formula is as follows:

[0179] Δr xi =dir*r xstep

[0180] Among them, r xstep Indicates the angular step of the current rotation of the manipulator around the x-axis of the tool coordinate system of the manipulator end. If the occluded area is on the right side of the gallbladder contour, dir represents the first preset direction. If the occluded area is on the left side of the gallbladder contour, dir represents the second preset direction. Δr xi Indicates the angle that the robot arm needs to swing around the x-axis of the end tool coordinate system for the current rotation.

[0181] Optionally, the terminal module is specifically used for:

[0182] The gallbladder contour in the liver area of ​​the abdomen is segmented by a segmentation network, and the aspect ratio of the gallbladder contour corresponding to each section is calculated;

[0183] If the aspect ratio corresponding to the current section meets the preset range, the gallbladder section contour information corresponding to the current section is stored in the corresponding container; wherein the gallbladder section contour information includes the gallbladder area, aspect ratio, position vector of the ultrasound probe and posture matrix of the ultrasound probe corresponding to the current section;

[0184] The continuous sections are regarded as a gallbladder segment, and all gallbladder segments in the container are traversed to determine the gallbladder segment information corresponding to each gallbladder segment; wherein the gallbladder segment information includes the length of the current gallbladder segment, the target area, the position vector of the ultrasound probe corresponding to the target area, the posture matrix of the ultrasound probe corresponding to the target area, and the number of sections corresponding to the current gallbladder segment, and the target area is the maximum value of the gallbladder areas corresponding to the sections in the current gallbladder segment;

[0185] Acquire the maximum value of the gallbladder segment length among all the gallbladder segments as a first reference value, and the maximum value of the gallbladder segment target area among all the gallbladder segments as a second reference value;

[0186] Calculate the length ratio of each gallbladder segment to the first reference value, and the area ratio of the target area to the second reference value;

[0187] According to the length proportion and area proportion corresponding to each gallbladder segment, the weighted score of each gallbladder segment is calculated by weighted score, and the gallbladder segment corresponding to the maximum weighted score is taken as the target gallbladder segment;

[0188] The position vector and posture matrix corresponding to the target gallbladder segment are used as the position and posture corresponding to the target gallbladder segment.

[0189] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code. Computer-readable storage media can be, for example, but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above.

[0190] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0191] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An automatic gallbladder scanning method using an ultrasonic robot, characterized in that: include: Scanning the abdomen of a human body according to a preset path, acquiring an ultrasound image through the ultrasound probe during the scanning process, segmenting the gallbladder contour in the ultrasound image through a segmentation network, and calculating the position and posture of the gallbladder segment formed by the continuous gallbladder contour in the world coordinate system; After the human abdomen is scanned, the target gallbladder segment is calculated according to the weight, the ultrasound probe is controlled to reach the position corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is found; After the gallbladder is found, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block where an occluder exists; Based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, determining whether the gallbladder contour in the ultrasound image is blocked by an occluder in each target block; If the gallbladder contour is blocked, the blocked area of ​​the gallbladder contour in the ultrasound image is obtained, and according to the positional relationship between the blocked area and the gallbladder contour, the swing direction of the robot arm is controlled to automatically scan the gallbladder.

2. The method according to claim 1, characterized in that After the gallbladder is found, the ultrasound image is divided into blocks, and the target block is determined based on the pixel value of each block, including: Establishing a plane coordinate system along the intersection point of the ultrasound image boundary, and acquiring first boundary coordinate points on the left and right sides of the ultrasound image based on the plane coordinate system; Taking the y-axis of the plane coordinate system as the center line, determining first angles from the left and right boundaries of the ultrasound image to the y-axis respectively according to the first boundary coordinate point and the inverse tangent trigonometric function; Dividing the ultrasound image into blocks based on a first angle corresponding to a left and right boundary in the ultrasound image; Adding the image pixels of each of the blocks to determine a sum result; The block corresponding to which the sum result is smaller than the pixel sum threshold is taken as the target block.

3. The method according to claim 2, characterized in that Based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, determining whether the gallbladder contour in the ultrasound image is blocked by an occluder in each target block includes: Acquire second boundary coordinate points of the left and right boundaries of the gallbladder contour in the plane coordinate system through a segmentation network; Determine a second angle between the left and right edges of the gallbladder contour and the y-axis according to the second boundary coordinate point; Taking the center line of the ultrasonic image as the origin, obtaining the angle range of each target block; The second angle is compared with the angle range of each target block. If the second angle is within the angle range of any target block, it is determined that the gallbladder contour is blocked by the target block.

4. The method according to claim 1, characterized in that: According to the positional relationship between the shielded area and the gallbladder contour, the swing direction of the robotic arm is controlled, including: If the shielding area is located on the right side of the gallbladder outline, the robotic arm is controlled to swing in a first preset direction; if the shielding area is located on the left side of the gallbladder outline, the robotic arm is controlled to swing in a second preset direction.

5. The method according to claim 4, characterized in that According to the positional relationship between the shielded area and the gallbladder contour, the swing direction of the mechanical arm is controlled, and further comprising: Controlling the mechanical arm to swing and scan to perform a first total number of obstruction determinations on the gallbladder contour to determine whether there is an obstruction on the gallbladder contour; If it is determined within the first total number of times that there is an obstruction on the gallbladder contour, the current scanning position of the robot arm is obtained; If the current scanning position is above the ribs, the robotic arm is controlled to swing around the y-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder; If the current scanning position is below the ribs, the robotic arm is controlled to swing around the x-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder.

6. The method according to claim 5, characterized in that If the current scanning position is above the ribs, the robotic arm is controlled to swing around the y-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder, including: If the current scanning position is above the ribs, controlling the robotic arm to rotate around the y-axis of the tool coordinate system at the end of the robotic arm for a second total number of times to automatically scan the gallbladder; Before each rotation, the robot arm calculates the posture angle vector required for the current rotation as (0, Δr yi , 0); Δr yi The calculation formula is as follows: Δr yi =you*r ystep Among them, r ystep Indicates the angular step of the current rotation of the manipulator around the y-axis of the tool coordinate system of the manipulator end. If the occluded area is on the right side of the gallbladder contour, dir represents the first preset direction. If the occluded area is on the left side of the gallbladder contour, dir represents the second preset direction Δr yi Indicates the angle that the robot arm needs to swing around the y-axis of the end tool coordinate system for the current rotation.

7. The method according to claim 6, characterized in that If the current scanning position is above the ribs, controlling the robotic arm to rotate around the y-axis of the tool coordinate system at the end of the robotic arm for a second total number of times to automatically scan the gallbladder, further comprising: Each time the robot rotates, the current robot arm is controlled to translate toward the x-axis direction of the tool coordinate system at the end of the robot arm, where the translation vector is (Δp fxtcpi , 0, Δp ztcpi ); Δp fxtcpi The calculation formula is as follows: Among them, Δp fxtcpi Indicates the distance moved along the x-axis of the tool coordinate system at the end of the robot due to lateral force, Δp ztcpi Indicates the displacement of the robot arm along the x-axis of the tool coordinate system at the end of the robot arm. x0 represents the preset force threshold, f xi It represents the force of the robot arm along the x-axis of the tool coordinate system at the end of the robot arm, and δ represents the preset proportional coefficient.

8. The method according to claim 5, characterized in that If the current scanning position is below the ribs, the robotic arm is controlled to swing around the x-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder, including: If the current scanning position is below the ribs, controlling the robotic arm to rotate a third total number of times around the x-axis of the tool coordinate system at the end of the robotic arm to automatically scan the gallbladder; Among them, before each rotation, the posture angle vector that the robot arm needs to move in the current rotation is calculated as (Δr xi , 0, 0); Δr xi The calculation formula is as follows: Δr xi =you*r xstep Among them, r xstep Indicates the angular step of the current rotation of the manipulator around the x-axis of the tool coordinate system of the manipulator end. If the occluded area is on the right side of the gallbladder contour, dir represents the first preset direction. If the occluded area is on the left side of the gallbladder contour, dir represents the second preset direction. Δr xi Indicates the angle that the robot arm needs to swing around the x-axis of the end tool coordinate system for the current rotation.

9. The method according to any one of claims 1 to 8, characterized in that: The human abdomen is scanned according to a preset path, an ultrasound image is acquired by the ultrasound probe during the scanning process, and the gallbladder contour in the ultrasound image is segmented by a segmentation network, and the position of the gallbladder segment formed by the continuous gallbladder contour in the world coordinate system is calculated, including: The gallbladder contour in the liver area of ​​the abdomen is segmented by a segmentation network, and the aspect ratio of the gallbladder contour corresponding to each section is calculated; If the aspect ratio corresponding to the current section meets the preset range, the gallbladder section contour information corresponding to the current section is stored in the corresponding container; wherein the gallbladder section contour information includes the gallbladder area, aspect ratio, position vector of the ultrasound probe and posture matrix of the ultrasound probe corresponding to the current section; All continuous sections are regarded as a gallbladder segment, and all gallbladder segments in the container are traversed to determine the gallbladder segment information corresponding to each gallbladder segment; wherein the gallbladder segment information includes the length of the current gallbladder segment, the target area, the position vector of the ultrasound probe corresponding to the target area, the posture matrix of the ultrasound probe corresponding to the target area, and the number of sections corresponding to the current gallbladder segment, and the target area is the maximum value of the gallbladder areas corresponding to the sections in the current gallbladder segment; acquiring a maximum value of the length of the gallbladder segment among all the gallbladder segments as a first reference value, and a maximum value of the target area of ​​the gallbladder segment as a second reference value; Calculating the ratio of the length to the first reference value, and the ratio of the target area to the second reference value in each gallbladder segment; According to the length proportion and area proportion corresponding to each gallbladder segment, the weighted score of each gallbladder segment is calculated by weighted score, and the gallbladder segment corresponding to the maximum weighted score is taken as the target gallbladder segment; The position vector and posture matrix corresponding to the target gallbladder segment are used as the position and posture corresponding to the target gallbladder segment.

10. An ultrasonic robot gallbladder automatic scanning device, characterized in that: include: The ultrasonic robot and terminal device include a mechanical arm and an ultrasonic probe. The ultrasonic probe is arranged at the end of the mechanical arm. The terminal device is connected to the mechanical arm and the ultrasonic probe respectively. The terminal device is specifically used for: Scanning the abdomen of a human body according to a preset path, acquiring an ultrasound image through the ultrasound probe during the scanning process, segmenting the gallbladder contour in the ultrasound image through a segmentation network, and calculating the position and posture of the gallbladder segment formed by the continuous gallbladder contour in the world coordinate system; After the human abdomen is scanned, the target gallbladder segment is calculated according to the weight, the ultrasound probe is controlled to reach the position corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is found; After the gallbladder is found, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block where an occluder exists; Based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, determining whether the gallbladder contour in the ultrasound image is blocked; If the gallbladder outline is blocked, the swing direction of the mechanical arm is controlled according to the current position of the ultrasound probe and the position of the ribs, and the gallbladder is automatically scanned.

Citation Information

Patent Citations

  • Biliary tract angiography image position marking method based on ultrasonic guidance

    CN116452591A

  • Method, device and equipment for adjusting observation pose of ultrasonic probe and storage medium

    CN117752362A

  • Autonomous ultrasonic robot kidney scanning method, device and equipment and storage medium

    CN117838189A

  • Automatic ultrasonic robot gall bladder scanning method, device and equipment and storage medium

    CN117982173A

  • Rib shielding judgment method and device for ultrasonic image, equipment and storage medium

    CN118014955A