Ultrasound robotic gallbladder automatic scanning method and device

Abstract

The application relates to the technical field of ultrasonic robots, and discloses an ultrasonic robot gallbladder automatic scanning method and device, which comprises the following steps: recording the position of a gallbladder contour during abdominal scanning, finally calculating an optimal gallbladder position, then controlling a probe to reach the optimal gallbladder position, and then performing an elliptical search; after the gallbladder is searched, it is judged whether the gallbladder contour is blocked by an occlusion in an ultrasonic image; if the gallbladder contour is blocked, an occlusion area of the gallbladder contour blocked by the occlusion in the ultrasonic image is obtained, and the swinging direction of a mechanical arm is controlled according to the positional relationship between the occlusion area and the gallbladder contour, and the gallbladder is automatically scanned. The swinging direction of the mechanical arm is controlled based on the positional relationship between the occlusion area and the gallbladder contour, so that the occlusion is avoided, and the ultrasonic probe can scan the complete gallbladder.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ultrasonic robots, and particularly relates to an ultrasonic robot gallbladder automatic scanning method and device. BACKGROUND

[0002] The gallbladder is a pear-shaped sac structure located behind the liver under the right rib. In the gallbladder automatic scanning process, the ultrasonic robot usually first scans the abdominal liver, and then calculates and records the gallbladder contour information in the scanning process. When the abdominal liver scanning is completed, the optimal position of the gallbladder is returned, and the gallbladder is scanned at multiple angles. However, the gallbladder may be blocked by the rib during the gallbladder scanning, resulting in poor gallbladder ultrasonic imaging. SUMMARY

[0003] In order to overcome the problem that the gallbladder may be blocked by the rib during the gallbladder scanning, resulting in poor gallbladder ultrasonic imaging quality, the present application provides an ultrasonic robot gallbladder automatic scanning method and device.

[0004] In a first aspect, in order to solve the above technical problems, the present application provides an ultrasonic robot gallbladder automatic scanning method, comprising:

[0005] Scanning the abdominal part of the human body along a preset path, acquiring an ultrasonic image through the ultrasonic probe during the scanning process, and segmenting the gallbladder contour in the ultrasonic image through a segmentation network to calculate the pose of the gallbladder segment constituted by the continuous gallbladder contour in the world coordinate system;

[0006] After the abdominal part of the human body is scanned, a target gallbladder segment is calculated according to the weight, the ultrasonic probe is controlled to reach the pose corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is searched;

[0007] When the gallbladder is searched, the ultrasonic image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block in which there is an occlusion;

[0008] Based on the positional relationship between the gallbladder contour and each target block in the ultrasonic image, it is determined whether the gallbladder contour is occluded by the occlusion in each target block;

[0009] If the gallbladder contour is occluded, the occluded area of the gallbladder contour in the ultrasonic image is acquired, and the swinging direction of the mechanical arm is controlled according to the positional relationship between the occluded area and the gallbladder contour, and the gallbladder is automatically scanned.

[0010] In a second aspect, the present application provides an ultrasound robot gallbladder automatic scanning device, comprising: an ultrasound robot and a terminal device, the ultrasound robot comprising a mechanical arm and an ultrasound probe, the ultrasound probe being arranged at the end of the mechanical arm, and the terminal device being connected with the mechanical arm and the ultrasound probe respectively, and the terminal device being specifically used for:

[0011] scanning the abdominal cavity of a human body along a preset path, acquiring an ultrasound image through the ultrasound probe during the scanning process, segmenting a gallbladder contour in the ultrasound image through a segmentation network, and calculating a pose of a gallbladder segment constituted by continuous gallbladder contours in a world coordinate system;

[0012] after the scanning of the abdominal cavity of the human body is completed, calculating a target gallbladder segment according to the weight, controlling the ultrasound probe to reach a pose corresponding to the target gallbladder segment, and performing an elliptical trajectory search until the gallbladder is searched;

[0013] after the gallbladder is searched, dividing the ultrasound image into blocks, and determining a target block based on pixel values of each block; wherein the target block is a block in which an occlusion exists;

[0014] determining whether the gallbladder contour is occluded by the occlusion in each target block in the ultrasound image based on a positional relationship between the gallbladder contour and each target block in the ultrasound image;

[0015] if the gallbladder contour is occluded, acquiring an occlusion area of the gallbladder contour occluded by the occlusion in the ultrasound image, and controlling a swinging direction of the mechanical arm and automatically scanning the gallbladder according to a positional relationship between the occlusion area and the gallbladder contour.

[0016] The present application has the beneficial effect that when the gallbladder is scanned, the ultrasound image is divided into blocks, and a target block is determined based on pixel values of each block, so that whether the gallbladder contour is occluded by the occlusion in each target block can be determined through the positional relationship between the gallbladder contour and each target block in the ultrasound image, and if there is an occlusion, the swinging direction of the mechanical arm is controlled according to the positional relationship between the occlusion area and the gallbladder contour, and the gallbladder is automatically scanned. The present application controls the swinging direction of the mechanical arm based on the positional relationship between the occlusion area and the gallbladder contour, thereby avoiding the occlusion, so that the ultrasound probe can scan the complete gallbladder. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the present application will be further described below with reference to the drawings and embodiments.

[0018] Figure 1 FIG. 1 is a flowchart of an ultrasound robot gallbladder automatic scanning method according to an embodiment of the present application;

[0019] Figure 2A schematic diagram of an ultrasound image being blocked by an object;

[0020] Figure 3 A schematic diagram of a plane coordinate system o-xy established;

[0021] Figure 4 A schematic diagram of a first boundary coordinate point in a plane coordinate system;

[0022] Figure 5 A schematic diagram of a first angle in a plane coordinate system;

[0023] Figure 6 A schematic diagram of an included angle corresponding to a left-right boundary in an ultrasound image in a plane coordinate system;

[0024] Figure 7 A schematic diagram of a sub-block in an ultrasound image;

[0025] Figure 8 A schematic diagram of a second angle in a plane coordinate system;

[0026] Figure 9 A schematic diagram of a swinging direction of an ultrasound probe;

[0027] Figure 10 A schematic diagram of a current scanning position being above a rib;

[0028] Figure 11 A schematic diagram of a current scanning position being below a rib;

[0029] Figure 12 A structural schematic diagram of an ultrasound robot gallbladder automatic scanning device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following embodiments are further explanations and supplements of the present application and do not constitute any limitation on the present application.

[0031] An ultrasound robot gallbladder automatic scanning method and device according to an embodiment of the present application are described below with reference to the accompanying drawings.

[0032] As shown in Figure 1 , the present application provides an ultrasound robot gallbladder automatic scanning method, comprising:

[0033] S1, scanning a human abdomen according to a preset path, acquiring an ultrasound image through the ultrasound probe in the scanning process, and segmenting a gallbladder contour in the ultrasound image through a segmentation network to calculate a pose of a gallbladder segment constituted by continuous gallbladder contours in a world coordinate system.

[0034] S2, after the abdominal scan of the human body is completed, a target gallbladder segment is calculated according to the weight, the ultrasonic probe is controlled to reach a pose corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is searched.

[0035] S3, when the gallbladder is searched, the ultrasonic image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block in which an occluder exists.

[0036] S4, based on the position relationship between the gallbladder contour and each target block in the ultrasonic image, it is determined whether the gallbladder contour is occluded by the occluder in each target block.

[0037] S5, if the gallbladder contour is occluded, an occluded area of the gallbladder contour occluded by the occluder in the ultrasonic image is obtained, and the swing direction of the mechanical arm is controlled according to the position relationship between the occluded area and the gallbladder contour, and the gallbladder is automatically scanned.

[0038] In the embodiment, when the gallbladder is scanned, the ultrasonic image is divided into blocks, and a target block is determined based on the pixel value of each block. The position relationship between the gallbladder contour and each target block in the ultrasonic image can be used to determine whether the gallbladder contour is occluded by the occluder in each target block. If there is an occlusion, the swing direction of the mechanical arm is controlled according to the position relationship between the occluded area and the gallbladder contour, and the gallbladder is automatically scanned. The application controls the swing direction of the mechanical arm based on the position relationship between the occluded area and the gallbladder contour, thereby avoiding the occluder and enabling the ultrasonic probe to scan the complete gallbladder.

[0039] In the embodiment, the segmentation model is a common segmentation model, such as a U-Net segmentation network.

[0040] In the embodiment, the positions of the gallbladder in the ultrasonic image are all transformed into the world coordinate system, and the position of the gallbladder in the world coordinate system is calculated as follows:

[0041] When the gallbladder contour is obtained, the gallbladder contour is fitted with an ellipse, and the center point (c x , c y ) of the gallbladder contour is obtained. The center point is converted to the physical space coordinates (P y , P z ) by the following formula:

[0042] P y =c x ×(W / validW)

[0043] P z =c y ×(W / validW);

[0044] Where W is the physical distance corresponding to the width of the ultrasound image, and validW is the pixel distance corresponding to the width of the ultrasound image.

[0045] Obtain the coordinates P of the gallbladder in the end-effector coordinate system of the robotic arm. TCP = (0, P y P z ).

[0046] P TCP Transform to the world coordinate system to obtain the position P of the gallbladder in the world coordinate system. B The formula is as follows:

[0047] P B =R c P TCP +P c

[0048] Among them, R c P represents the pose of the robotic arm's end effector based on the world coordinate system. c The position of the robotic arm's end effector based on the world coordinate system.

[0049] In this embodiment, to quickly locate the gallbladder, the search trajectory of the robotic arm's end effector is designed according to the equation of an ellipse, i.e., an elliptical trajectory search. The gallbladder's position is affected by respiration, primarily changing vertically (along the y-axis of the world coordinate system). Therefore, the focus of the elliptical trajectory is designed on the y-axis of the world coordinate system. The elliptical trajectory can be:

[0050]

[0051] Where a and b are two parameters of the ellipse equation, calculated as follows:

[0052] With the current position P of the ultrasound probe cur Establish a coordinate system o with the center point. In order to ensure that the search range is large enough and to take into account the search efficiency, set the two foci of the ellipse (0, m) and (0, -m) with a distance of half the probe, and set the intersection of the ellipse on the x-axis (m, 0) and (-m, 0) with a distance of half the probe, where m represents the width of half the probe and the reference value is 0.02m.

[0053] Based on the above description, the parameter values ​​of the elliptical trajectory equations a and b can be calculated, which in turn allows the robotic arm's end effector to search for an elliptical trajectory in the xy-plane of the world coordinate system, as detailed below:

[0054] The position of the probe at the end effector of the robotic arm at the next moment is (Δx, Δy), where Δy is the position of the ultrasonic probe from point P. curThe distance of starting to move along the y-axis of the world coordinate system (a fixed step can be used for movement, and the recommended value of the movement step is 0.001 m), and Δx represents the displacement amount of the probe at the end of the mechanical arm in the x-axis direction of the world coordinate system relative to P cur according to the above-mentioned elliptical equation and Δy.

[0055] In addition, it should be noted that when the mechanical arm searches according to the elliptical trajectory, if the gallbladder cannot be found after one search or the number of existing gallbladders does not reach a certain threshold, the gallbladder scanning is not performed, otherwise the gallbladder scanning stage is entered.

[0056] Optionally, after the gallbladder is searched, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block, including:

[0057] A plane coordinate system is established along the intersection point of the boundary of the ultrasound image, and first boundary coordinate points on the left and right sides of the ultrasound image are obtained based on the plane coordinate system;

[0058] Taking the y-axis of the plane coordinate system as the center line, the first angles of the left and right boundaries of the ultrasound image to the y-axis are determined according to the arctangent trigonometric function and the boundary coordinate points;

[0059] The ultrasound image is divided into blocks based on the first angles corresponding to the left and right boundaries of the ultrasound image;

[0060] The image pixels of each block are added to determine an addition result;

[0061] The block corresponding to the addition result less than the pixel addition threshold is taken as the target block.

[0062] As shown in Figure 2 , the hatched area is an occlusion, which is generally caused by the non-adhesion of the ultrasound probe or the rib occlusion. Therefore, it is necessary to judge the occlusion and the area where the occlusion is located, and the specific judgment is as follows:

[0063] As shown in Figure 3 , a plane coordinate system o-xy is established along the intersection point o of the boundary of the ultrasound image, where the coordinate origin is (x0, y0).

[0064] As shown in Figure 4 , the first boundary coordinate points on the left and right sides of the ultrasound image obtained from the coordinate system o-xy are (x l , y l ) and (x r , y r ) respectively.

[0065] As shown in Figure 5 , taking the y-axis of the plane coordinate system as the center line, the arctangent trigonometric function is established according to the first boundary coordinate points and the y-axis, so that the first angles of the left and right boundaries of the ultrasound image to the y-axis are calculated respectively as θl θ r .

[0066] like Figure 6 As shown, according to θ l θ r The angle between the left and right boundaries in the ultrasound image can then be calculated as θ = |θ r -θ l |θ l <0.

[0067] To determine the specific location of gallbladder obstruction, the ultrasound image is divided into sector-shaped blocks, with the angle value τ for each block being:

[0068]

[0069] Where n is the number of blocks, which can be set according to the actual situation. Preferably, in this embodiment, n can be 8, that is, the ultrasound image is divided into 8 blocks, such as... Figure 7 As shown.

[0070] Calculate the sum of the pixel values ​​within each block of the image, denoted as S. j (j = 1, 2, ..., n, representing the block numbers from left to right of the image). Set the pixel summation threshold (recommended value is 50000), if S j If the sum of pixels is greater than the pixel sum threshold, it means that block j has no occlusion; otherwise, if S... j If the sum of pixels is less than the pixel sum threshold, then block j is occluded, and this block is taken as the target block.

[0071] Optionally, based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, it is determined whether the gallbladder contour in the ultrasound image is occluded by occluders in each target block, including:

[0072] The second boundary coordinates of the left and right boundaries of the gallbladder contour in the planar coordinate system are obtained by segmentation network;

[0073] Based on the coordinates of the second boundary point, determine the second angle from the left and right edges of the gallbladder contour to the y-axis;

[0074] Using the centerline of the ultrasound image as the origin, the angular range of each target block is obtained;

[0075] The second angle is compared with the angle range of each target block. If the second angle is within the angle range of a certain target block, it is determined that the gallbladder contour is occluded by the target block.

[0076] In this embodiment, the second boundary coordinates of the left and right boundaries of the gallbladder contour in the planar coordinate system are obtained through a segmentation network, which are (x, y, z) respectively. cl y cl ), (xcr y cr ).

[0077] like Figure 8 As shown, the second angles from the left and right edges of the gallbladder contour to the y-axis (the origin of the plane coordinate system is (x0, y0)) are θ and θ', respectively. cl θ cr , where θ cl θ cr The calculation formula is as follows:

[0078]

[0079] With the center line of the ultrasound image as the origin, the angular range of the j-th target block is:

[0080]

[0081] Where θ jmin θ represents the minimum value of the angular range of the j-th target block; jmax This represents the maximum value of the angle range for the j-th target block. If θ cl ≥θ jmin And θ cl ≤θ jmax , or θ cr ≥θ jmin And θ cr ≤θ jmax If the value is 0, it indicates that the gallbladder outline in the ultrasound image is obscured at the j-th target segment.

[0082] Optionally, the swing direction of the robotic arm is controlled according to the positional relationship between the obstructed area and the gallbladder contour, including:

[0083] If the obscured area is located to the right of the gallbladder contour, the robotic arm is controlled to swing in a first preset direction; if the obscured area is located to the left of the gallbladder contour, the robotic arm is controlled to swing in a second preset direction, as follows:

[0084]

[0085] Where dir represents the direction of oscillation. If θ cr ≥θ jmin And θ cr ≤θ jmax If θ = 1, it means the obscured area is to the right of the gallbladder outline, and the swing direction is dir = 1, which is the first preset direction; if θ cl ≥θ jmin And θ cl ≤θ jmax If the value is 0, it means that the obscured area is on the left side of the gallbladder outline. At this time, the swing direction is dir = -1, which is the second preset direction.

[0086] Optionally, controlling the swing direction of the robotic arm based on the positional relationship between the obstructed area and the gallbladder contour further includes:

[0087] When the robotic arm swings and scans, it performs the first total number of obstruction checks on the gallbladder outline to determine whether there are any obstructions on the gallbladder outline.

[0088] If an obstruction is detected in the gallbladder outline within the first total number of scans, the current scanning position of the robotic arm is obtained.

[0089] If the current scanning position is above the ribs, the robotic arm is controlled to swing around the y-axis of the end-effector coordinate system and automatically scan the gallbladder.

[0090] If the current scanning location is below the ribs, the robotic arm is controlled to swing around the x-axis of the end-effector coordinate system and automatically scan the gallbladder.

[0091] like Figure 9 As shown, by controlling the robotic arm (ultrasound probe) to rotate around the x-axis or y-axis of the end-effector coordinate system, the completeness of gallbladder contour scanning can be improved to some extent.

[0092] In this embodiment, the first total number of times can be set according to the actual situation.

[0093] Optionally, if the current scanning position is above the ribs, the robotic arm is controlled to swing around the y-axis of the end-effector coordinate system to automatically scan the gallbladder, including:

[0094] If the current scanning position is above the ribs, control the robotic arm to rotate around the y-axis of the end-effector coordinate system a second total number of times to automatically scan the gallbladder;

[0095] Before each rotation, the required posture angle vector for the robotic arm to move in the current rotation is calculated as (0, Δr). yi ,0);

[0096] Δr yi The calculation formula is as follows:

[0097] Δr yi =dir*r ystep

[0098] Where, r ystep dir represents the angular step size of the current rotation of the robotic arm around the y-axis of the end-effector tool coordinate system. If the occluded area is to the right of the gallbladder contour, then dir represents the first preset direction; if the occluded area is to the left of the gallbladder contour, then dir represents the second preset direction. Δr yirepresents an angle of the current rotation of the mechanical arm around the y-axis of the end tool coordinate system.

[0099] As shown in Figure 10 , if the current scanning position is above the rib, the mechanical arm is controlled to swing around the y-axis of the end tool coordinate system of the mechanical arm to avoid the rib and scan the gallbladder, and the posture angle vector of the mechanical arm is (0, Δr yi , 0) each time.

[0100] wherein r ystep may be set according to actual conditions, and preferably r ystep = 0.008 rad in the embodiment.

[0101] In the embodiment, the rotation range of the mechanical arm around the y-axis of the end tool coordinate system of the mechanical arm, i.e. the maximum search range r ymax around the y-axis, is required to be controlled, and r ymax may be set according to actual conditions, and preferably r ymax = 0.8 rad in the embodiment. Then, the total number of rotations, i.e. the second total number, is determined according to the ratio between r ymax and r ystep , so as to ensure that the rotation range of the mechanical arm around the y-axis is controlled within r ymax .

[0102] Optionally, if the current scanning position is above the rib, the mechanical arm is controlled to rotate around the y-axis of the end tool coordinate system of the mechanical arm for the second total number of times to automatically scan the gallbladder, and further comprising:

[0103] controlling the current mechanical arm to translate along the x-axis of the end tool coordinate system of the mechanical arm each time, wherein the translation vector is (Δp fxtcpi , 0, Δp ztcpi );

[0104] The calculation formula of Δp fxtcpi is as follows:

[0105]

[0106] wherein Δp fxtcpi represents the distance of lateral force along the x-axis of the end tool coordinate system of the mechanical arm, Δp ztcpi represents the displacement of the current rotation of the mechanical arm along the x-axis of the end tool coordinate system of the mechanical arm, f x0 represents a preset force threshold, f xi represents the force of the mechanical arm along the x-axis of the end tool coordinate system of the mechanical arm each time, and δ represents a preset proportion coefficient.

[0107] In this embodiment, by adding a translation vector, the ultrasound probe is automatically aligned with the intercostal space at each rotation, so that the ultrasound probe at the end of the mechanical arm can be smoothly scanned on the gallbladder, and the integrity of the gallbladder scan is improved.

[0108] In this embodiment, f x0 , and δ are set according to actual conditions, preferably, in this embodiment, f x0 = 10N and δ = 0.002.

[0109] In this embodiment, at each rotation, the posture angle vector of the mechanical arm is (0, Δr yi , 0), and the translation vector is (Δp fxtcpi , 0, Δp ztcpi ), so as to achieve smooth control of the posture of the ultrasound probe at the end of the mechanical arm.

[0110] Optionally, if the current scanning position is below the ribs, the mechanical arm is controlled to swing around the x-axis of the tool coordinate system at the end of the mechanical arm to automatically scan the gallbladder, comprising:

[0111] If the current scanning position is below the ribs, the mechanical arm is controlled to rotate around the x-axis of the tool coordinate system at the end of the mechanical arm for a third total number of times to automatically scan the gallbladder.

[0112] Wherein, before each rotation, the posture angle vector that the mechanical arm needs to move at the current rotation is calculated as (Δr xi , 0, 0).

[0113] The calculation formula of Δr xi is as follows:

[0114] Δr xi = dir*r xstep

[0115] Wherein, r xstep represents the angle step of the current rotation of the mechanical arm swinging around the x-axis of the tool coordinate system at the end of the mechanical arm, dir represents the first preset direction if the obstacle is on the right side of the gallbladder contour, and dir represents the second preset direction if the obstacle is on the left side of the gallbladder contour, and Δr xi represents the angle of the current rotation of the mechanical arm swinging around the x-axis of the tool coordinate system at the end of the mechanical arm.

[0116] In this embodiment, as shown in Figure 11 , if the current scanning position is below the ribs, the mechanical arm is controlled to swing around the x-axis of the tool coordinate system at the end of the mechanical arm to ensure that the probe can avoid the ribs.

[0117] Wherein, r xstep may be set according to actual conditions, preferably, in this embodiment, r xstep = 0.008 rad.

[0118] In this embodiment, the rotation range of the mechanical arm around the x-axis of the mechanical arm end tool coordinate system, i.e., the maximum search range r xmax , r xmax is required to be controlled xmax According to the actual situation, preferably, r xmax = 1.0 rad in this embodiment, and the total number of rotations, i.e., the third total number, is determined according to the ratio between r xstep and r xmax , so as to control the rotation range of the mechanical arm around the x-axis within r xmax .

[0119] Optionally, the abdomen of the human body is scanned according to a preset path, an ultrasound image is acquired through the ultrasound probe during the scanning process, and a gallbladder outline in the ultrasound image is segmented through a segmentation network to calculate a pose of a gallbladder segment constituted by continuous gallbladder outlines in a world coordinate system, including:

[0120] The gallbladder outline of the liver region of the abdomen is segmented through the segmentation network, and an aspect ratio of the gallbladder outline corresponding to each section is calculated;

[0121] If the aspect ratio corresponding to the current section meets a preset range, the gallbladder section outline information corresponding to the current section is stored in a corresponding container; wherein the gallbladder section outline information includes a gallbladder area corresponding to the current section, an aspect ratio, a position vector of the ultrasound probe, and an attitude matrix of the ultrasound probe;

[0122] Continuous sections are taken as a gallbladder segment, and all gallbladder segments in the container are traversed to determine gallbladder segment information corresponding to each gallbladder segment; wherein the gallbladder segment information includes a length of the current gallbladder segment, a target area, a position vector of the ultrasound probe corresponding to the target area, an attitude matrix of the ultrasound probe corresponding to the target area, and a number of sections corresponding to the current gallbladder segment, and the target area is the maximum value among the gallbladder areas corresponding to the sections in the current gallbladder segment;

[0123] The maximum value of the lengths of the gallbladder segments in all gallbladder segments is taken as a first reference value, and the maximum value of the target areas of the gallbladder segments in all gallbladder segments is taken as a second reference value;

[0124] The length proportion of the length of each gallbladder segment to the first reference value and the area proportion of the target area to the second reference value are calculated;

[0125] According to the length proportion and the area proportion corresponding to each gallbladder segment, a weighted score of each gallbladder segment is calculated through a weighted score method, and a gallbladder segment corresponding to a maximum weighted score is taken as a target gallbladder segment;

[0126] The position vector and the attitude matrix of the target gallbladder segment are taken as the pose of the target gallbladder segment.

[0127] In this embodiment, the ultrasound robot first scans the liver, then records and analyzes the shape of the gallbladder during the scanning process, and returns to the optimal position of the gallbladder after scanning the liver to scan the gallbladder. When scanning the liver region, an ultrasound image section is obtained from the ultrasound probe one by one. In addition, the continuous section refers to the section from the first acquisition of the gallbladder aspect ratio meeting the preset range to the continuous multiple sections without scanning the gallbladder aspect ratio meeting the preset range. The above continuous scanned gallbladder aspect ratio meeting the preset range is taken as the continuous section, that is, a gallbladder segment.

[0128] The length l and the width w of the fitted gallbladder contour are obtained through the U-Net segmentation network, so as to calculate the aspect ratio lwRatio = l / w (w≠0) of the gallbladder contour corresponding to each section.

[0129] A container partList is established to store the information of each gallbladder segment. partList[i] represents the gallbladder information corresponding to the i-th gallbladder segment in the container. If the gallbladder aspect ratio meets the preset range, the gallbladder contour information is stored in partList[i]. The preset range is [r0, 1), that is, the value of lwRatio falls within the range [r0, 1), and the gallbladder area, aspect ratio, position vector of the ultrasound probe and attitude matrix of the ultrasound probe corresponding to the section are stored in partList[i], so as to obtain partList[i] = {…, (area, lwRatio, P c ,R c )}, area represents the gallbladder area, P c 、R c respectively represent the position vector of the ultrasound probe and the attitude matrix of the ultrasound probe. If the gallbladder data is not stored in partList[i] for continuous k times, it means that the current continuous gallbladder segment has been scanned, at this time i is increased by 1, and k is a fixed threshold value, and the recommended value is 5.

[0130] In this embodiment, r0 is a fixed threshold value, which is set according to the actual situation, and preferably, in this embodiment, r0 = 0.6.

[0131] When the liver region is scanned according to the above method, all partList[i] is traversed, and the gallbladder segment information corresponding to each gallbladder segment can be calculated and stored in the container partStats[i] corresponding to the gallbladder segment. partStats[i] represents the gallbladder segment information corresponding to the i-th gallbladder segment, and partStats[i] = {maxArea i ,P cmaxi ,Rcmaxi ,L i}, where maxArea i P represents the target area, which is the maximum value of the gallbladder area corresponding to each section in partList[i] for the i-th gallbladder segment. cmaxi R cmaxi L represents the position vector and attitude matrix of the ultrasound probe corresponding to the target area in partList[i] for the i-th gallbladder segment, respectively. i This represents the length of the i-th gallbladder segment.

[0132] At this point, since there are numerous gallbladder segments, it is necessary to find the most suitable gallbladder orientation from among these segments as the starting position for the ultrasound probe's search. Therefore, by traversing partStats[i], the maximum length of the gallbladder segment among all segments can be obtained as the first reference value L. max and each maxArea i The maximum target area is used as the second reference value, allMaxArea.

[0133] Calculate the length percentage r of each gallbladder segment. Li =L i / L max and area ratio r ai =maxArea i / allMaxArea,r Li r represents the length percentage of the i-th gallbladder segment. ai This represents the area percentage of the i-th gallbladder segment.

[0134] The weighted score of the i-th gallbladder segment is calculated as follows:

[0135] score=αr Li +βr ai

[0136] α+β=1

[0137] Wherein, score represents the weighted score, and α and β are preset weight coefficient values, which are set according to the actual situation. Preferably, in this embodiment, α = 0.6 and β = 0.4.

[0138] Based on the weighted score calculation method described above, the target gallbladder segment with the highest score, partStats[i], is obtained. best ], and from partStats[i best Get P from ] cmaxi R cmaxi The position vector and attitude matrix of the end-effector of the robotic arm represent the pose of the target gallbladder.

[0139] In this embodiment, the terminal device can be a computer, which is not described here.

[0140] As shown in the figure, an ultrasonic robot gallbladder automatic scanning device comprises an ultrasonic robot and a terminal device, the ultrasonic robot comprises a mechanical arm and an ultrasonic probe, the ultrasonic probe is arranged at the end of the mechanical arm, and the terminal device is connected with the mechanical arm and the ultrasonic probe respectively, and the terminal device is specifically used for: Figure 12 scanning the abdominal cavity according to a preset path, acquiring an ultrasonic image through the ultrasonic probe during the scanning process, segmenting a gallbladder contour in the ultrasonic image through a segmentation network, and calculating a pose of a gallbladder segment in a world coordinate system;

[0141] after the abdominal cavity scanning is completed, calculating a target gallbladder segment according to a weight, controlling the ultrasonic probe to reach a pose corresponding to the target gallbladder segment, and performing an elliptical trajectory search until the gallbladder is searched;

[0142] when the gallbladder is searched, dividing the ultrasonic image into blocks, and determining a target block based on pixel values of each block; wherein the target block is a block in which an occlusion exists;

[0143] based on a positional relationship between the gallbladder contour and each target block in the ultrasonic image, determining whether the gallbladder contour is occluded by the occlusion in each target block;

[0144] if the gallbladder contour is occluded, acquiring an occlusion area of the gallbladder contour occluded by the occlusion in the ultrasonic image, and controlling a swing direction of the mechanical arm and automatically scanning the gallbladder according to a positional relationship between the occlusion area and the gallbladder contour.

[0145] Optionally, the terminal module is specifically used for:

[0146] establishing a plane coordinate system along a boundary intersection point of the ultrasonic image, and acquiring first boundary coordinate points on the left and right sides of the ultrasonic image based on the plane coordinate system;

[0147] taking the y-axis of the plane coordinate system as a center line, determining first angles of the left and right boundaries to the y-axis in the ultrasonic image according to the first boundary coordinate points and an inverse tangent trigonometric function;

[0148] dividing the ultrasonic image into blocks based on the first angles of the left and right boundaries in the ultrasonic image;

[0149] adding image pixels of each block to determine an addition result;

[0150] taking a block corresponding to an addition result less than a pixel addition threshold as a target block.

[0151]

[0152] ​Optionally, the terminal module is specifically used for:

[0153] acquiring, by the segmentation network, second boundary coordinate points of the left and right boundaries of the gallbladder profile in the planar coordinate system;

[0154] determining, according to the second boundary coordinate points, second angles of the left and right edges of the gallbladder profile to the y-axis;

[0155] acquiring an angle range of each target sub-block with the center line of the ultrasound image as an origin;

[0156] comparing the second angle with the angle range of each target sub-block, and if the second angle is within the angle range of a target sub-block, judging that the gallbladder profile is blocked by the target sub-block.

[0157] Optionally, the terminal module is specifically used for, comprising:

[0158] if the blocking area is located on the right side of the gallbladder profile, controlling the mechanical arm to swing in a first preset direction, and if the blocking area is located on the left side of the gallbladder profile, controlling the mechanical arm to swing in a second preset direction.

[0159] Optionally, the terminal module is further used for:

[0160] controlling the mechanical arm to perform a first total number of times of blocking object judgment on the gallbladder profile during swing scanning, and judging whether the gallbladder profile has a blocking object;

[0161] if it is judged that the gallbladder profile has a blocking object within the first total number of times, acquiring a current scanning position of the mechanical arm;

[0162] if the current scanning position is located above the rib, controlling the mechanical arm to swing around the y-axis of the mechanical arm end tool coordinate system and automatically scan the gallbladder;

[0163] if the current scanning position is located below the rib, controlling the mechanical arm to swing around the x-axis of the mechanical arm end tool coordinate system and automatically scan the gallbladder.

[0164] Optionally, the terminal module is specifically used for:

[0165] if the current scanning position is located above the rib, controlling the mechanical arm to rotate around the y-axis of the mechanical arm end tool coordinate system for a second total number of times and automatically scan the gallbladder;

[0166] wherein, before each rotation, the posture angle vector that the mechanical arm needs to move for the current rotation is (0, Δr yi , 0);

[0167] The calculation formula of Δr yi is as follows:

[0168] Δr yi= dir * r ystep

[0169] wherein, r ystep represents the angle step of the current rotation of the mechanical arm around the y-axis of the end tool coordinate system, dir represents the first preset direction if the shielding area is on the right side of the gallbladder contour, and dir represents the second preset direction if the shielding area is on the left side of the gallbladder contour, and Δr yi represents the angle of the current rotation of the mechanical arm around the y-axis of the end tool coordinate system.

[0170] Optionally, the terminal module is further configured to:

[0171] control the current mechanical arm to move in the x-axis direction of the end tool coordinate system of the mechanical arm each time the mechanical arm rotates, wherein the moving vector is (Δp fxtcpi , 0, Δp ztcpi ) ;

[0172] The calculation formula of Δp fxtcpi is as follows:

[0173]

[0174] wherein, Δp fxtcpi represents the distance of the lateral force along the x-axis of the end tool coordinate system of the mechanical arm, Δp ztcpi represents the displacement of the current movement of the mechanical arm along the x-axis of the end tool coordinate system of the mechanical arm, f x0 represents a preset force threshold, f xi represents the force of the mechanical arm along the x-axis of the end tool coordinate system of the mechanical arm, and δ represents a preset proportion coefficient.

[0175] Optionally, the terminal module is specifically configured to:

[0176] if the current scanning position is below the rib, control the mechanical arm to rotate around the x-axis of the end tool coordinate system of the mechanical arm for a third total number of times to automatically scan the gallbladder;

[0177] wherein, the posture angle vector of the current rotation of the mechanical arm is calculated as (Δr xi , 0, 0) before each rotation;

[0178] The calculation formula of Δr xi is as follows:

[0179] Δr xi = dir * r xstep

[0180] wherein, r xsteprepresents an angle step of the current secondary rotation of the mechanical arm swinging around the x-axis of the mechanical arm end tool coordinate system, if the shielding area is on the right side of the gallbladder contour, dir represents a first preset direction, and if the shielding area is on the left side of the gallbladder contour, dir represents a second preset direction, Δr xi represents an angle of the current secondary rotation of the mechanical arm swinging around the x-axis of the end tool coordinate system.

[0181] Optionally, the terminal module is specifically used for:

[0182] The gallbladder contour of the liver region of the abdomen is segmented by the segmentation network, and the aspect ratio of the gallbladder contour corresponding to each slice is calculated.

[0183] If the aspect ratio corresponding to the current slice meets a preset range, the gallbladder slice contour information corresponding to the current slice is stored in the corresponding container; wherein the gallbladder slice contour information includes the gallbladder area corresponding to the current slice, the aspect ratio, the position vector of the ultrasonic probe, and the attitude matrix of the ultrasonic probe.

[0184] The continuous slices are taken as a gallbladder segment, and all gallbladder segments in the container are traversed to determine the gallbladder segment information corresponding to each gallbladder segment; wherein the gallbladder segment information includes the length of the current gallbladder segment, the target area, the position vector of the ultrasonic probe corresponding to the target area, the attitude matrix of the ultrasonic probe corresponding to the target area, and the number of slices corresponding to the current gallbladder segment, and the target area is the maximum value in the gallbladder areas corresponding to the slices in the current gallbladder segment.

[0185] The maximum value of the gallbladder segment length in all gallbladder segments is taken as a first reference value, and the maximum value of the target area of the gallbladder segment in all gallbladder segments is taken as a second reference value.

[0186] The length proportion of the length in each gallbladder segment to the first reference value and the area proportion of the target area to the second reference value are calculated.

[0187] According to the length proportion and the area proportion corresponding to each gallbladder segment, the weighted score of each gallbladder segment is calculated by a weighted score method, and the gallbladder segment corresponding to the maximum weighted score is taken as a target gallbladder segment.

[0188] The position vector and the attitude matrix corresponding to the target gallbladder segment are taken as the pose of the target gallbladder segment.

[0189] Those skilled in the art know that the present application can be embodied as a system, method or computer program product. Therefore, the present disclosure can be embodied in the form of a complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "circuitry", "module" or "system". In addition, in some embodiments, the present application can also be embodied in the form of a computer program product in one or more computer readable media, which contains computer readable program codes. The computer readable storage medium may, for example, be but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above.

[0190] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0191] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. An ultrasonic robotic gallbladder automatic scanning method, characterized by, The method comprises the following steps: scanning the abdomen of a human body along a preset path, acquiring an ultrasound image through an ultrasound probe during the scanning process, segmenting a gallbladder contour in a liver region of the abdomen through a segmentation network, and calculating an aspect ratio of the gallbladder contour corresponding to each slice; if the aspect ratio corresponding to the current slice meets a preset range, storing the gallbladder slice contour information corresponding to the current slice in a corresponding container; wherein the gallbladder slice contour information comprises a gallbladder area corresponding to the current slice, the aspect ratio, a position vector of the ultrasound probe, and an attitude matrix of the ultrasound probe; all consecutive slices are taken as a gallbladder segment, and all gallbladder segments in the container are traversed to determine the gallbladder segment information corresponding to each gallbladder segment; wherein the gallbladder segment information comprises the length of the current gallbladder segment, a target area, a position vector of the ultrasound probe corresponding to the target area, an attitude matrix of the ultrasound probe corresponding to the target area, and the number of slices corresponding to the current gallbladder segment, and the target area is the maximum value among the gallbladder areas corresponding to each slice in the current gallbladder segment; the maximum value of the length of the gallbladder segment among all gallbladder segments is taken as a first reference value, and the maximum value of the target area of the gallbladder segment is taken as a second reference value; the length ratio of the length of each gallbladder segment to the first reference value and the area ratio of the target area to the second reference value are calculated; according to the length ratio and the area ratio corresponding to each gallbladder segment, the weighted score of each gallbladder segment is calculated through the weighted score, and the gallbladder segment corresponding to the maximum weighted score is taken as a target gallbladder segment; the position vector and the attitude matrix corresponding to the target gallbladder segment are taken as the pose of the target gallbladder segment, the ultrasound probe is controlled to reach the pose of the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is searched; after the gallbladder is searched, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block with an occluder; based on the position relationship between the gallbladder contour and each target block in the ultrasound image, it is determined whether the gallbladder contour is occluded by the occluder in each target block; if the gallbladder contour is occluded, the occluded area of the gallbladder contour in the ultrasound image is obtained, and the swinging direction of the mechanical arm is controlled according to the position relationship between the occluded area and the gallbladder contour, and the gallbladder is automatically scanned.

2. The method of claim 1, wherein, When the gallbladder is searched, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block, comprising: establishing a plane coordinate system along the intersection point of the ultrasound image boundary, and obtaining the first boundary coordinate point on the left and right sides of the ultrasound image based on the plane coordinate system; taking the y-axis of the plane coordinate system as the center line, determining the first angle of the left and right boundaries to the y-axis in the ultrasound image according to the first boundary coordinate point and the inverse tangent trigonometric function; dividing the ultrasound image based on the first angle corresponding to the left and right boundaries in the ultrasound image; adding the image pixels of each block to determine the sum result; the block corresponding to the sum result less than the pixel sum threshold is taken as the target block.

3. The method of claim 2, wherein, Determine whether the gallbladder contour is blocked by the occluder in each target block based on the position relationship between the gallbladder contour and each target block in the ultrasound image, comprising: Obtain the second boundary coordinate point of the left and right boundaries of the gallbladder contour in the plane coordinate system through the segmentation network; Determine the second angle of the left and right edges of the gallbladder contour to the y-axis according to the second boundary coordinate point; Take the center line of the ultrasound image as the origin to obtain the angle range of each target block; Compare the second angle with the angle range of each target block, if the second angle is within the angle range of any target block, it is judged that the gallbladder contour is blocked by the target block.

4. The method of claim 1, wherein, Control the swing direction of the mechanical arm according to the position relationship between the occlusion area and the gallbladder contour, comprising: If the occlusion area is located on the right side of the gallbladder contour, control the mechanical arm to swing to the first preset direction, if the occlusion area is located on the left side of the gallbladder contour, control the mechanical arm to swing to the second preset direction.

5. The method of claim 4, wherein, Control the swing direction of the mechanical arm according to the position relationship between the occlusion area and the gallbladder contour, further comprising: Control the mechanical arm to swing and scan the gallbladder contour for a first total number of times to determine whether there is an occluder on the gallbladder contour; If it is determined that there is an occluder on the gallbladder contour within the first total number of times, obtain the current scanning position of the mechanical arm; If the current scanning position is located above the rib, control the mechanical arm to swing around the y-axis of the mechanical arm end tool coordinate system to automatically scan the gallbladder; If the current scanning position is located below the rib, control the mechanical arm to swing around the x-axis of the mechanical arm end tool coordinate system to automatically scan the gallbladder.

6. The method of claim 5, wherein, If the current scanning position is located above the rib, control the mechanical arm to swing around the y-axis of the mechanical arm end tool coordinate system to automatically scan the gallbladder, comprising: If the current scanning position is located above the rib, control the mechanical arm to rotate around the y-axis of the mechanical arm end tool coordinate system for a second total number of times to automatically scan the gallbladder; Wherein, the posture angle vector that the mechanical arm needs to move before each rotation is calculated as (θx, θy, θz) ) The calculation formula is as follows: wherein, represents the angle step of the current rotation of the robot arm around the y-axis of the robot arm end tool coordinate system, and if the occlusion region is on the right side of the gallbladder contour, then represents the first preset direction, and if the occlusion region is on the left side of the gallbladder contour, then represents the second preset direction, represents the angle of the current rotation of the robot arm around the y-axis of the end tool coordinate system.

7. The method of claim 6, wherein, If the current scanning position is located above the rib, control the mechanical arm to rotate around the y-axis of the mechanical arm end tool coordinate system for a second total number of times to automatically scan the gallbladder, further comprising: Each rotation controls the current robotic arm to translate along the x-axis of the end-effector coordinate system, where the translation vector is ( ); The calculation formula is as follows: wherein, represents a distance of movement along the x-axis of the robot tool coordinate system under lateral force, represents a displacement amount of the current movement of the robot along the x-axis of the robot tool coordinate system, represents a preset force threshold value, represents a force of the robot along the x-axis of the robot tool coordinate system, represents a preset proportionality coefficient.

8. The method of claim 5, wherein, If the current scanning position is located below the rib, control the mechanical arm to swing around the x-axis of the mechanical arm end tool coordinate system to automatically scan the gallbladder, comprising: If the current scanning position is located below the rib, control the mechanical arm to rotate around the x-axis of the mechanical arm end tool coordinate system for a third total number of times to automatically scan the gallbladder; Wherein, the posture angle vector that the mechanical arm needs to move before each rotation is calculated as ); The calculation formula is as follows: wherein, represents the angle step of the current rotation of the robot arm around the x-axis of the robot arm end tool coordinate system, if the occlusion region is on the right side of the gallbladder contour, then represents the first preset direction, if the occlusion region is on the left side of the gallbladder contour, then represents the second preset direction, represents the angle of the current rotation of the robot arm around the x-axis of the end tool coordinate system.

9. An ultrasonic robotic gallbladder auto-scan device, characterized by, Comprising: An ultrasound robot and a terminal device, the ultrasound robot comprising a mechanical arm and an ultrasound probe, the ultrasound probe being arranged at the end of the mechanical arm, the terminal device being connected with the mechanical arm and the ultrasound probe respectively, the terminal device being specifically used for: According to the preset path, the abdomen of the human body is scanned, and in the scanning process, an ultrasound image is acquired by the ultrasound probe, and a gallbladder contour of a liver region of the abdomen is segmented by a segmentation network, and a length-width ratio of the gallbladder contour corresponding to each section is calculated; if the length-width ratio corresponding to the current section meets a preset range, the gallbladder section contour information corresponding to the current section is stored in the corresponding container; wherein the gallbladder section contour information includes the gallbladder area corresponding to the current section, the length-width ratio, the position vector of the ultrasound probe and the attitude matrix of the ultrasound probe; All continuous sections are taken as a gallbladder segment, and all gallbladder segments in the container are traversed to determine the gallbladder segment information corresponding to each gallbladder segment; wherein the gallbladder segment information includes the length of the current gallbladder segment, the target area, the position vector of the ultrasound probe corresponding to the target area, the attitude matrix of the ultrasound probe corresponding to the target area and the number of sections corresponding to the current gallbladder segment, and the target area is the maximum value in the gallbladder area corresponding to each section in the current gallbladder segment; The maximum value of the length of the gallbladder segment in all gallbladder segments is taken as a first reference value, and the maximum value of the target area of the gallbladder segment is taken as a second reference value; the length proportion of the length in each gallbladder segment to the first reference value and the area proportion of the target area to the second reference value are calculated; according to the length proportion and the area proportion corresponding to each gallbladder segment, the weighted score of each gallbladder segment is calculated by the weighted score, and the gallbladder segment corresponding to the maximum weighted score is taken as the target gallbladder segment; The position vector and the attitude matrix corresponding to the target gallbladder segment are taken as the pose corresponding to the target gallbladder segment, the ultrasound probe is controlled to reach the pose corresponding to the target gallbladder segment, and an elliptical trajectory search is performed until the gallbladder is searched; When the gallbladder is searched, the ultrasound image is divided into blocks, and a target block is determined based on the pixel value of each block; wherein the target block is a block with an occlusion; Based on the positional relationship between the gallbladder contour and each target block in the ultrasound image, it is determined whether the gallbladder contour is occluded in the ultrasound image; If the gallbladder contour is occluded, the swing direction of the mechanical arm is controlled according to the current position of the ultrasound probe and the position of the rib, and the gallbladder is automatically scanned.

Images