Trajectory display method, surgical robot and computer-readable storage medium

Abstract

The present application relates to a trajectory display method, a surgical robot and a computer-readable storage medium. The method rotates an ultrasonic probe so that a medical device installed on a guide tool is in a sagittal plane scanned by the ultrasonic probe, and then displays the predicted motion trajectory of the medical device in an ultrasonic image obtained by scanning the sagittal plane by the ultrasonic probe, thereby solving the problem that the predicted motion trajectory of the medical device cannot be displayed in the ultrasonic image and improving the sampling success rate.

Description

Technical Field

[0001] The present application relates to the technical field of medical devices, and in particular to a trajectory display method, a surgical robot, and a computer-readable storage medium. Background Art

[0002] Biopsy (abbreviated as "biopsy") refers to the technology of performing pathological examination on the diseased tissue in the patient's body in response to the needs of diagnosis or treatment. Puncture biopsy is a type of biopsy technology, which uses a medical device, such as a biopsy needle (or puncture needle, etc.) to percutaneously puncture the patient's diseased tissue for sampling. At present, when doctors perform puncture biopsies (such as prostate puncture biopsies, etc.), they manually push the medical device into the patient's body for sampling under the guidance of a guide tool, such as a needle guide. During the sampling process, after the guide tool is positioned, the medical device is passed through the guide tool. In theory, the center of the needle sampling section can be placed on the sampling point (or target point) of the patient's lesion area. After the medical device is completely inserted into the patient's prostate, the image of the medical device head can be seen on the ultrasound image obtained by scanning the cross section of the ultrasound probe.

[0003] However, since the current ultrasound probe only scans the cross section to obtain the ultrasound image of the corresponding cross section, it cannot display the expected movement trajectory of the medical device, and therefore cannot provide guidance to the doctor for puncture. If the patient moves after the guide tool is positioned, it will cause a deviation between the expected initial point and the target point. In this case, continuing the puncture will result in a high sampling failure rate. Moreover, after the sampling fails, the puncture sampling needs to be performed again, causing secondary damage to the patient and low surgical accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a trajectory display method, a surgical robot and a computer-readable storage medium that can solve the problem of not being able to display the expected motion trajectory of the medical device in the ultrasound image in response to the above-mentioned technical problems.

[0005] A trajectory display method, the method comprising:

[0006] calculating the coordinates of the guiding tool in the image coordinate system corresponding to the ultrasound probe;

[0007] Controlling the rotation of the ultrasound probe according to the coordinates of the guiding tool in the image coordinate system and a first sagittal plane currently scanned by the ultrasound probe;

[0008] Acquiring an ultrasonic image scanned by the rotated ultrasonic probe;

[0009] Based on the coordinates of the guiding tool in the image coordinate system and the length of the medical device installed in the guiding tool, the expected motion trajectory of the medical device is displayed in the ultrasound image. In one embodiment, the method further includes:

[0010] Get the location of the target point;

[0011] Controlling the initial point of the guiding tool to align with the target point, and positioning the guiding tool based on the position of the target point to obtain the coordinates of the guiding tool in the robot coordinate system;

[0012] The calculation of the coordinates of the guiding tool in the image coordinate system corresponding to the ultrasound probe includes:

[0013] Obtain the coordinate transformation relationship between the robot coordinate system and the image coordinate system;

[0014] The coordinates of the guiding tool in the robot coordinate system are converted into coordinates in the image coordinate system based on the coordinate conversion relationship.

[0015] In one embodiment, the position of the target point includes the coordinates of the target point in the robot coordinate system; the controlling the initial point of the guiding tool to align with the target point, and positioning the guiding tool based on the position of the target point to obtain the coordinates of the guiding tool in the robot coordinate system includes:

[0016] Obtaining the coordinates of the initial point of the guiding tool in the robot coordinate system;

[0017] Based on the coordinates of the target point in the robot coordinate system and the coordinates of the initial point of the guiding tool in the robot coordinate system, obtaining a vertical angle and a horizontal angle of the initial point of the guiding tool relative to the target point;

[0018] According to the vertical angle and the horizontal angle, the initial point of the guiding tool is controlled to align with the target point, and the guiding tool is positioned based on the position of the target point to obtain the coordinates of the guiding tool in the robot coordinate system.

[0019] In one embodiment, controlling the rotation of the ultrasound probe according to the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe comprises:

[0020] Based on the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe, the rotation angle of the ultrasound probe is obtained, and the ultrasound probe is controlled to rotate by the rotation angle so that the medical device installed in the guiding tool is in the second sagittal plane scanned by the ultrasound probe.

[0021] In one embodiment, obtaining the rotation angle of the ultrasound probe based on the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe comprises:

[0022] Acquire the coordinates of the ultrasound probe in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe;

[0023] Based on the coordinates of the guiding tool in the image coordinate system, obtaining the coordinates of the medical device installed in the guiding tool in the image coordinate system;

[0024] Based on the coordinates of the ultrasound probe in the image coordinate system and the coordinates of the medical device in the image coordinate system, a second sagittal plane is obtained, where the second sagittal plane is a plane where the ultrasound probe and the medical device are located;

[0025] An angle between the first sagittal plane and the second sagittal plane is obtained, where the angle is a rotation angle of the ultrasound probe.

[0026] In one embodiment, the method further comprises:

[0027] Acquire a region of at least one target in the ultrasound image;

[0028] Prompt information is output based on the distance between the predicted motion trajectory and the area of ​​the target in the ultrasound image.

[0029] In one embodiment, the at least one target includes a first target, and acquiring a region of the at least one target in the ultrasound image includes:

[0030] Acquire the position of the first target in the ultrasound image;

[0031] A region of the first target in the ultrasound image is obtained in the ultrasound image based on a preset three-dimensional shape and a position of the first target in the ultrasound image.

[0032] In one embodiment, the at least one target further includes a second target, and acquiring the region of the at least one target in the ultrasound image includes:

[0033] Taking the set starting point as the center, a boundary search is performed outward according to the pixel value to obtain a plurality of first boundary pixel points, and a first curved surface is generated by the plurality of the first boundary pixel points, wherein the value of the first boundary pixel point is greater than or equal to a preset first threshold value;

[0034] Subdividing the first curved surface using a subdivision model to obtain a plurality of subdivided pixel points;

[0035] Taking the multiple subdivided pixel points as the starting point, a boundary search is performed toward the starting point according to the pixel value size to obtain multiple second boundary pixel points, and a second surface is generated by the multiple second boundary pixel points, the value of the second boundary pixel point is less than or equal to a preset second threshold, and the second surface is used to represent the area of ​​the second target in the ultrasound image.

[0036] In one embodiment, the step of performing boundary search outwards according to pixel values ​​with the set starting point as the center to obtain a plurality of first boundary pixel points includes:

[0037] Taking the starting point as the center, a boundary search is performed outward with a preset step size according to the spherical coordinate system. If the pixel value of the searched pixel point is greater than or equal to a preset first threshold, the searched pixel point is the first boundary pixel point.

[0038] In one embodiment, the prompt information includes first prompt information and second prompt information, and outputting the prompt information based on the distance between the predicted motion trajectory and the area of ​​the target in the ultrasound image includes:

[0039] Selecting a plurality of trajectory points in the predicted motion trajectory;

[0040] Obtaining the distance between each of the trajectory points and the target;

[0041] If the distance between any one of the trajectory points and the area of ​​the target in the ultrasound image is less than or equal to a preset distance threshold, outputting the first prompt information, wherein the first prompt information is used to indicate that the predicted motion trajectory poses a threat to the target;

[0042] If the distances between all the trajectory points and the area of ​​the target in the ultrasound image are greater than the preset distance threshold, the second prompt information is output, and the second prompt information is used to indicate that the expected motion trajectory poses no threat to the target.

[0043] A surgical robot, comprising:

[0044] An ultrasound probe, used for acquiring ultrasound images;

[0045] a guide tool having a medical device installed therein; and

[0046] A control device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in any one of the above embodiments when executing the computer program;

[0047] A display is used to display the expected motion trajectory of the medical device and / or the area of ​​at least one target in the ultrasound image in the ultrasound image.

[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of the above embodiments.

[0049] The above-mentioned trajectory display method, surgical robot and computer-readable storage medium rotate the ultrasonic probe so that the medical device installed on the guiding tool is in the sagittal plane scanned by the ultrasonic probe, and then displays the expected motion trajectory of the medical device in the ultrasonic image obtained by the ultrasonic probe scanning the sagittal plane, thereby solving the problem of not being able to display the expected motion trajectory of the medical device in the ultrasonic image and improving the sampling success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A diagram showing an application environment of a trajectory display method in an embodiment;

[0051] Figure 2 A schematic diagram of a flow chart of a trajectory display method in one embodiment;

[0052] Figure 3 A schematic diagram of coordinate conversion between a robot coordinate system and an image coordinate system in a trajectory display method in an embodiment;

[0053] Figure 4 A schematic diagram of coordinates for positioning a guiding tool in a trajectory display method in an embodiment;

[0054] Figure 5 A schematic diagram of controlling the ultrasound probe to rotate to the second sagittal plane in a trajectory display method in an embodiment;

[0055] Figure 6 A schematic diagram of a flow chart of a trajectory display method in another embodiment;

[0056] Figure 7 is a schematic diagram of a transverse ultrasound image and a sagittal ultrasound image in one embodiment;

[0057] Figure 8 A schematic diagram of displaying the predicted motion trajectory and the structure of an organ in a sagittal ultrasound image in one embodiment;

[0058] Fig. 9 A schematic diagram of a process of segmenting an ultrasound image in a trajectory display method in an embodiment;

[0059] Fig.10 A schematic diagram of a process of outputting prompt information in a trajectory display method in an embodiment;

[0060] Fig.11 A schematic diagram of a flow chart of outputting prompt information in a trajectory display method in another embodiment;

[0061] Fig.12 is a schematic flow chart of an ultrasound image segmentation method in one embodiment;

[0062] Fig.13 is a structural block diagram of a trajectory display device in an embodiment;

[0063] Fig.14 FIG. 4 is a structural block diagram of an ultrasonic image segmentation device in an embodiment.

[0064] Fig.15 is a structural block diagram of a surgical robot in one embodiment;

[0065] Fig.16 is a hardware structure diagram of a surgical robot in another embodiment;

[0066] Fig.17 A schematic diagram of a software serial port read and write control motor flow of a surgical robot in one embodiment;

[0067] Fig.18 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] As mentioned above, since the current ultrasound probe only scans the cross section to obtain the ultrasound image of the corresponding cross section, it cannot display the medical device, such as the expected movement trajectory of the puncture needle, such as the expected needle insertion trajectory. Therefore, it is impossible to provide doctors with operational guidance, such as guidance on puncture. If the patient moves after the guide tool is positioned, it will cause a deviation between the expected initial point and the target point. In this case, continuing the puncture will result in a high sampling failure rate. Moreover, after the sampling fails, the puncture sampling needs to be repeated, causing secondary damage to the patient and low surgical accuracy.

[0070] Therefore, the present application provides a trajectory display method, an ultrasonic image segmentation method, an apparatus, a surgical robot, a computer device, a computer-readable storage medium, and a computer program product, which can display the expected motion trajectory of the medical device in the ultrasonic image obtained by scanning the sagittal plane with the ultrasonic probe, and provide guidance to the doctor for puncture, thereby solving the problem that the expected motion trajectory of the medical device cannot be displayed in the ultrasonic image, and improving the sampling success rate. In other words, before the doctor performs the puncture operation, if the expected initial point and the target point deviate due to the movement of the patient, etc., which is manifested as a deviation in the expected motion trajectory of the medical device, or the expected motion trajectory of the medical device is not within the allowed needle insertion area, etc., the doctor can re-plan the target point to avoid secondary damage to the patient and improve the success rate of the doctor's operation.

[0071] Taking prostate biopsy as an example, Figure 1 FIG. 4 is a schematic diagram of a sagittal plane of a prostate puncture biopsy performed by a surgical robot provided in this embodiment. Figure 1 As shown, the surgical robot 900 may include an ultrasonic probe 910 for collecting ultrasonic images. In some other optional embodiments, the ultrasonic probe may be replaced by other devices for collecting ultrasonic images; a motor 920 for driving the ultrasonic probe to move or rotate, such as a stepping and rotating motor, for driving the ultrasonic probe to scan the patient's rectum to obtain a prostate ultrasonic image; a guide tool 930, in which a medical device 931 is installed; a mechanical arm 940, for moving and positioning the guide tool 930 to assist the doctor in performing a puncture operation or for the mechanical arm 940 to drive the guide tool 930 to perform an operation, such as a puncture operation, which is used to indicate that the medical device 931 in the guide tool 930 punctures from an initial point 902 on the surface of the prostate 901 to a target point 903 inside the prostate 901 to achieve biopsy sampling. It should be noted that the guide tool 930 may be a needle guide, which may be a needle guide sheath, and the medical device 931 may be a puncture needle.

[0072] Furthermore, before performing the above-mentioned puncture operation, the surgical robot can execute the trajectory display method provided in the embodiment of the present application to display the expected motion trajectory of the medical device in the ultrasound image obtained by the ultrasound probe 910 scanning the sagittal plane, and provide guidance to the doctor for puncture, thereby solving the problem of not being able to display the expected motion trajectory of the medical device in the ultrasound image and improving the sampling success rate.

[0073] In one embodiment, Figure 2 As shown, a trajectory display method is provided, which is applied to Figure 1 The prostate biopsy puncture in the example of FIG. 1 includes the following steps:

[0074] S101. Calculate the coordinates of the guiding tool in the image coordinate system corresponding to the ultrasound probe.

[0075] The calculation of the coordinates of the guiding tool in the image coordinate system corresponding to the ultrasound probe may include: obtaining the coordinate conversion relationship between the robot coordinate system and the image coordinate system; and converting the coordinates of the guiding tool in the robot coordinate system into coordinates in the image coordinate system based on the coordinate conversion relationship.

[0076] In this embodiment, the robot coordinate system is based on the surgical robot. Specifically, the robot coordinate system can be expressed as the robot arm coordinate system or the global coordinate system of the surgical robot, such as the robot arm coordinate system with the base of the robot arm as the origin coordinate, etc. The image coordinate system is based on the ultrasound probe, that is, the image coordinate system can be expressed as the ultrasound image coordinate system collected by the ultrasound probe.

[0077] In step S101, the coordinates of the guide tool in the robot coordinate system can be determined based on the distance or angle of a specific position of the guide tool relative to the coordinates of the origin of the robot arm, wherein the specific position can be determined as needed, for example, it can be the center point of the guide tool or the connection point between the guide tool and the medical device. The specific position is not specifically limited here, and those skilled in the art can determine the specific position as needed. Specifically, the coordinates of the guide tool in the robot coordinate system can be expressed as three-dimensional coordinates. Furthermore, since the medical device is mounted on the guide tool, the positions of the two are relatively fixed, so according to the coordinates of the guide tool in the robot coordinate system and the rotation angle of the guide tool, the coordinates of the medical device in the guide tool in the robot coordinate system and the needle insertion direction or length direction of the medical device can be determined.

[0078] like Figure 3 The figure shows the coordinate transformation relationship between the robot coordinate system T1 and the image coordinate system T2. Specifically, the coordinate transformation relationship between the robot coordinate system and the image coordinate system can be expressed by the following formula:

[0079]

[0080] Among them, (x, y, z) is the coordinate of point A in the robot coordinate system, (x0, y0, z0) is the coordinate of point A in the image coordinate system, and M is the rotation matrix or coordinate conversion matrix from the image coordinate system to the robot coordinate system. t0 ,y t0 , z t0 ) is the compensation coordinate from the image coordinate system to the robot coordinate system, (x t0 ,y t0 , z t0) can be determined based on the length of the ultrasonic probe fProbeLength and the robot calibration deviation. Usually, x t0 and t0 Small, can be set to 0, z t0 =fProbeLength. λ is the scaling parameter in the image coordinate system, which is adjusted according to user operations and the default value is set to 1.

[0081] More specifically, the rotation matrix M can be determined when the surgical robot performs coordinate calibration, and the coordinate calibration can be expressed as coordinate calibration according to the relative distance or rotation angle between the ultrasound probe and the robotic arm. For example, the rotation matrix around the x, y and z axes in a three-dimensional rectangular coordinate system can be expressed by the following formula:

[0082]

[0083] Among them, R x (α) is the rotation matrix around the x-axis, R y (θ) is the rotation matrix around the y-axis, R z (γ) is the rotation matrix around the z-axis, and α, θ, and γ are the rotation angles obtained by calibrating the actual coordinates of the surgical robot.

[0084] In one embodiment, it is considered that the guide tool needs to be positioned before the puncture operation so that the initial point of the guide tool is aligned with the target point, and the target point can be a lesion point, a target point or a target area marked by a doctor in the patient's body (such as the prostate, etc.). The initial point can be the needle insertion point during the puncture operation. Specifically, the method further includes:

[0085] S201, obtaining the position of the target point;

[0086] S202, controlling the initial point of the guiding tool to align with the target point, positioning the guiding tool based on the position of the target point, and obtaining the coordinates of the guiding tool in the robot coordinate system.

[0087] In step S201, the position of the target point may include the coordinates of the target point in the robot coordinate system, or the coordinates of the target point in the image coordinate system, etc. For example, the coordinates of the target point in the image coordinate system can be obtained based on the marking made by the doctor in the ultrasound image, or the coordinates of the target point in the robot coordinate system can be obtained by converting the coordinates of the target point in the image coordinate system according to the above-mentioned coordinate conversion relationship.

[0088] In step S202, the initial point of the guide tool can be used to indicate the insertion point and insertion angle or direction of the medical device installed in the guide tool into the patient's body (such as the prostate, etc.). The initial point of the guide tool aligned with the target point can be used to indicate that the insertion direction or angle of the medical device of the guide tool is aligned with the target point, so that the medical device can puncture in a straight line to the target point position in the patient's body, thereby completing the sampling.

[0089] In one embodiment, step S202 may include:

[0090] S203: Obtain the coordinates of the initial point of the guiding tool in the robot coordinate system.

[0091] S204, based on the coordinates of the target point in the robot coordinate system and the coordinates of the initial point of the guiding tool in the robot coordinate system, obtain the vertical angle and horizontal angle of the initial point of the guiding tool relative to the target point. The vertical angle and horizontal angle here refer to the vertical angle and horizontal angle in the robot coordinate system, wherein the vertical angle may correspond to the z-axis direction in the robot coordinate system, and the horizontal angle may correspond to the y-axis direction in the robot coordinate system. The specific calculation method is as follows.

[0092] S205. According to the vertical angle and the horizontal angle, control the initial point of the guiding tool to align with the target point, and position the guiding tool based on the position of the target point to obtain the coordinates of the guiding tool in the robot coordinate system.

[0093] like Figure 4 As shown, in step S204, the vertical angle H and the horizontal angle V are angles in the robot coordinate system, which can be calculated by the following formula:

[0094] H=tan -1 (-(x t -x p ) / (z t -z p ))

[0095] V=tan -1 (-(y t -y p ) / (z t -z p ))

[0096] Where H is the horizontal angle, V is the vertical angle, P(x p ,y p , z p ) is the coordinate of the initial point in the robot coordinate system, T(x t ,y t , zt ) is the coordinate of the target point in the robot coordinate system.

[0097] In this embodiment, in step S205, the robot arm is controlled to drive the guiding tool to move or rotate according to the above-mentioned horizontal angle and vertical angle, so that the initial point of the guiding tool is aligned with the target point.

[0098] It should be pointed out that in order to ensure the accuracy of the coordinate transformation of the guiding tool, the above steps S201 to S202 are first executed to position the guiding tool so that the initial point of the guiding tool is aligned with the target point, and then step S101 is executed to perform coordinate transformation on the guiding tool to obtain the coordinates of the guiding tool in the image coordinate system.

[0099] S102 : Control the rotation of the ultrasound probe according to the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe.

[0100] Among them, controlling the rotation of the ultrasound probe according to the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe may include: obtaining the rotation angle of the ultrasound probe based on the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe, and controlling the rotation angle of the ultrasound probe so that the medical device installed in the guiding tool is in the second sagittal plane scanned by the ultrasound probe.

[0101] That is, in step S102, the ultrasound probe is in a state of scanning a sagittal plane (such as the sagittal plane of the prostate, etc.), and the sagittal plane currently scanned by the ultrasound probe is the first sagittal plane. If the ultrasound probe is in a state of scanning a transverse plane (such as the transverse plane of the prostate), the ultrasound probe is adjusted to scan the sagittal plane of the prostate.

[0102] Since the medical device installed in the guiding tool may not be in the first sagittal plane currently scanned by the ultrasound probe, in step S102, the rotation angle of the ultrasound probe can be used to indicate the relative rotation angle of the ultrasound probe from the first sagittal plane currently scanned to the medical device, that is, after the ultrasound probe rotates by the rotation angle, the target point, the medical device and its needle insertion direction (or length direction, etc.) are all in the sagittal plane scanned after the ultrasound probe is rotated (that is, the second sagittal plane).

[0103] In one embodiment, step S102 may include:

[0104] S301, acquiring the coordinates of the ultrasound probe in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe;

[0105] S302, obtaining the coordinates of the medical device installed in the guiding tool in the image coordinate system based on the coordinates of the guiding tool in the image coordinate system;

[0106] S303, obtaining a second sagittal plane based on the coordinates of the ultrasound probe in the image coordinate system and the coordinates of the medical device in the image coordinate system, where the second sagittal plane is a plane where the ultrasound probe and the medical device are located;

[0107] S304: Obtain an angle between the first sagittal plane and the second sagittal plane, where the angle is a rotation angle of the ultrasound probe.

[0108] It can be understood that since the coordinates of the ultrasound probe in the image coordinate system remain fixed, by rotating the ultrasound probe, the sagittal plane currently scanned by the ultrasound probe (i.e., the first sagittal plane mentioned above) can be rotated with the ultrasound probe as the axis until the ultrasound probe is rotated to the point where the sagittal plane it scans (i.e., the second sagittal plane mentioned above) coincides with the medical device and its needle insertion direction or length direction.

[0109] In step S302, the coordinates of the medical device installed in the guiding tool in the image coordinate system can be calculated based on the coordinates of the guiding tool in the image coordinate system, the length of the medical device, and the position or angle of the medical device relative to the guiding tool, wherein the length of the medical device and the position or angle relative to the guiding tool can be obtained by measurement, that is, after the medical device is installed, its position or angle relative to the guiding tool is fixed, and therefore can be determined by measurement and stored, so that it remains unchanged during the entire operation process and can be directly acquired and used.

[0110] In step S303, the coordinates of the medical device in the image coordinate system can be expressed as line segment coordinates, and the ultrasound probe and the medical device can determine a plane, which is the second sagittal plane, that is, the ultrasound probe and the medical device and their needle insertion direction or length direction are in the second sagittal plane.

[0111] In step S304, since the ultrasound probe is the intersection of the first sagittal plane and the second sagittal plane, the angle between the first sagittal plane and the second sagittal plane can be expressed as the rotation angle of the ultrasound probe, and by controlling the ultrasound probe to rotate the rotation angle, the ultrasound probe can be rotated to a position for scanning the second sagittal plane, so as to scan the second sagittal plane to obtain an ultrasound image. Figure 5 As shown, the left figure is a schematic diagram of the positional relationship between the ultrasound probe 910 scanning the first sagittal plane S1 and the medical device 931, and the right figure is a schematic diagram of the second sagittal plane S2 scanning and the overlap with the medical device 931 after the ultrasound probe 910 rotates the above-mentioned rotation angle A.

[0112] S103: Acquire an ultrasonic image scanned by the rotated ultrasonic probe.

[0113] It should be pointed out that before puncture sampling, the medical device has not punctured into the patient's body (such as the prostate). The ultrasound image obtained by scanning the second sagittal plane with the ultrasound probe can display the sagittal plane of the prostate. Therefore, in order to provide guidance to the doctor for puncture, etc., it is necessary to display the expected movement trajectory of the medical device in the ultrasound image obtained by scanning the second sagittal plane with the ultrasound probe.

[0114] S104: Displaying an estimated motion trajectory of the medical device in the ultrasound image based on the coordinates of the guiding tool in the image coordinate system and the length of the medical device installed in the guiding tool.

[0115] That is to say, based on the coordinates of the guiding tool in the image coordinate system, the rotation angle of the guiding tool, and the position or angle of the medical device relative to the guiding tool, the coordinates of the medical device in the image coordinate system and the insertion direction of the medical device can be determined. Then, based on the coordinates of the medical device in the image coordinate system, the insertion direction and the length of the medical device, the expected motion trajectory of the medical device can be drawn in the ultrasound image.

[0116] Furthermore, if Figure 6 As shown, the process of the trajectory display method in the embodiment of the present application is as follows:

[0117] S1. Marking of lesions or targets, etc.;

[0118] S2, positioning the guiding tool so that the initial point of the guiding tool is aligned with the lesion point;

[0119] S3, obtaining the coordinates of the guiding tool in the image coordinate system;

[0120] S4, calculating the rotation angle of the ultrasound probe from the first sagittal plane to the second sagittal plane;

[0121] S5, rotating the ultrasound probe and scanning the second sagittal plane to obtain an ultrasound image;

[0122] S6. Display the predicted motion trajectory in the ultrasound image.

[0123] Therefore, the trajectory display method provided in the embodiment of the present application can display the expected motion trajectory of the medical device in the ultrasound image obtained by the ultrasound probe scanning the sagittal plane of the prostate (such as the second sagittal plane mentioned above), and provide guidance to the doctor for puncture, thereby solving the problem of not being able to display the expected motion trajectory of the medical device in the ultrasound image and improving the sampling success rate.

[0124] Furthermore, the above-mentioned target point can also be displayed in the ultrasound image, so that the doctor can observe in the ultrasound image whether there is a significant deviation between the expected motion trajectory of the medical device and the target point, and thus decide whether to perform a puncture operation. If there is a deviation between the expected motion trajectory and the target point (such as the target point) due to patient movement, the doctor can re-plan the target point to avoid secondary damage to the patient and improve the success rate of the doctor's operation.

[0125] In some other optional embodiments, the method may further include: determining whether the target point deviates from the expected motion trajectory based on the distance between the target point and the expected motion trajectory. For example, if the distance between the target point and the expected motion trajectory is less than or equal to the deviation threshold, it indicates that there is no deviation between the target point and the expected motion trajectory or the deviation is within an acceptable range, and the doctor can perform a puncture operation. If the distance between the target point and the expected motion trajectory is greater than the deviation threshold, it indicates that there is a significant deviation between the target point and the expected motion trajectory, and the doctor can re-plan the target point or re-plan the motion trajectory of the medical device to avoid secondary damage to the patient and improve the success rate of the doctor's operation.

[0126] In addition, if Figure 7 As shown, the left picture is an ultrasound image obtained by scanning the cross section with an ultrasound probe, the black area in the left picture is the cross section of the prostate, and the white dots in the black area are medical devices. The right picture is an ultrasound image obtained by scanning the sagittal plane with an ultrasound probe, the black area in the right picture is the sagittal plane of the prostate, and the white track in the black area is the actual motion track of the medical device. That is to say, during the doctor's puncture operation, the actual motion track of the medical device can be displayed in real time in the ultrasound image obtained by scanning the second sagittal plane with the ultrasound probe, so as to guide the doctor to perform the puncture operation. If the actual motion track deviates significantly, the doctor can adjust the actual motion track of the medical device or stop the puncture operation, etc., so as to avoid causing harm to the patient and improve the success rate of the operation.

[0127] In one embodiment, the method further comprises:

[0128] S401, acquiring a region of at least one target in the ultrasound image;

[0129] S402: Output prompt information based on the distance between the predicted motion trajectory and the area of ​​the target in the ultrasound image.

[0130] In this embodiment, the above-mentioned target may be represented as an organ of the patient. For example, in an ultrasound image obtained by scanning the sagittal plane of the prostate, the target may include organs such as the rectum, urethra, and bladder. Taking prostate puncture sampling as an example, in step S401, at least one target may include a first target and a second target. The first target may be an organ of approximately regular shape, such as the rectum or urethra, and the second target may be an organ of irregular shape, such as the bladder. The region of the above-mentioned target in the ultrasound image may include three-dimensional coordinates in the ultrasound image. Figure 8 It is a schematic diagram showing the predicted motion trajectory and the sagittal plane of each organ in the ultrasound image, wherein L represents the predicted motion trajectory, 901 represents the prostate, 904 represents the rectum, 905 represents the urethra, and 906 represents the bladder.

[0131] In order to prevent the medical device from threatening or causing harm to various organs during the puncture sampling process, in step S402, if the distance between the predicted motion trajectory of the medical device and any organ is too close, or the predicted motion trajectory of the medical device passes through any organ, a prompt message is output that the predicted motion trajectory of the medical device threatens or will cause harm to the organ, so as to prompt the doctor to re-plan the target or adjust the motion trajectory of the medical device to avoid damaging the patient's organs and causing unnecessary harm to the patient.

[0132] In one embodiment, since the first target such as the rectum, urethra, etc. can be expressed as an approximately regular three-dimensional shape, such as a cylinder, etc., in order to facilitate determining the area of ​​the first target (such as an organ of a regular shape, etc.) in the ultrasound image, step S401 may include:

[0133] S501, obtaining a position of a first target in the ultrasound image;

[0134] S502: Based on a preset three-dimensional shape and a position of the first target in the ultrasound image, obtain a region of the first target in the ultrasound image.

[0135] Optionally, in step S501, the position of the first target in the ultrasound image can be determined according to a preset distance. The preset distance can be determined according to the relative position of the first target in the ultrasound image or the relative distance between the first target and the ultrasound probe. Taking the rectum as the first target as an example, when the human body lies flat, the rectum is located below the prostate, and the ultrasound probe scans the prostate upward through the rectum to obtain an ultrasound image. Therefore, the preset distance can be selected as 5 mm. This is only an example. It can actually be determined according to the patient's condition. The rectum can be represented as a position 5 mm from the bottom of the ultrasound image.

[0136] Optionally, in step S501, the position of the first target in the ultrasound image can be marked by a user (such as a doctor) in the ultrasound image. Taking the first target as the urethra as an example, the doctor can mark the plane outline of the urethra on the urethra area displayed in the ultrasound image, and the plane outline indicates the position of the urethra in the ultrasound image.

[0137] In step S502, the preset three-dimensional shape can be determined according to the three-dimensional shape of the organ, such as the rectum and the urethra are both cylinders. Further, in order to reflect the reasonable size of the organ in the ultrasound image, the preset three-dimensional shape can be scaled according to a preset scaling ratio, and then the scaled preset three-dimensional shape is generated at the position of the first target in the ultrasound image, and the area of ​​the first target in the ultrasound image (i.e., the three-dimensional space area) can be obtained. The preset scaling ratio can be determined according to the ratio of the organ to the ultrasound image.

[0138] In one embodiment, for an irregularly shaped organ such as a bladder, it is not suitable to use the above steps S501 to S502 to determine its three-dimensional spatial region in the ultrasound image. In order to solve the problem of how to determine the three-dimensional spatial region where the irregularly shaped organ is located in the ultrasound image, in step S401, the ultrasound image is segmented using an ultrasound image segmentation method to obtain the region of the second target (such as the irregularly shaped organ) in the ultrasound image. Specifically, step S401 may also include:

[0139] S601, taking a set starting point as the center, performing boundary search outward according to the pixel value to obtain a plurality of first boundary pixel points, and generating a first curved surface from the plurality of first boundary pixel points, wherein the value of the first boundary pixel point is greater than or equal to a preset first threshold;

[0140] S602, using a subdivision model to perform subdivision processing on the first curved surface to obtain a plurality of subdivision-processed pixel points;

[0141] S603, taking the multiple subdivided pixel points as the starting point, performing a boundary search toward the starting point according to the pixel value size, obtaining multiple second boundary pixel points, and generating a second surface from the multiple second boundary pixel points, wherein the value of the second boundary pixel point is less than or equal to a preset second threshold, and the second surface is used to represent the area of ​​the second target in the ultrasound image.

[0142] In step S601, the position of the second target in the ultrasound image may be obtained, and then a starting point may be selected at the position, wherein the position of the second target in the ultrasound image may be obtained by a user such as a doctor marking an area where the second target such as a bladder is located displayed in the ultrasound image, and then the starting point may be a center point or a center of gravity point at the position, etc. In some optional embodiments, the starting point may also be obtained by the user directly marking an area where the second target is located displayed in the ultrasound image.

[0143] Considering that there is a certain difference between the pixel value of the organ and the pixel value of the peripheral pixel points of its boundary in the ultrasound image, in step S601, a boundary search is performed using the pixel value size to obtain a plurality of first boundary pixel points. Specifically, step S601 may include:

[0144] Taking the starting point as the center, a boundary search is performed outward with a preset step size according to the spherical coordinate system. If the pixel value of the searched pixel point is greater than or equal to a preset first threshold, the searched pixel point is the first boundary pixel point.

[0145] That is to say, with the starting point as the center, according to the spherical coordinate system, search outward with a preset step size until a pixel value of a certain pixel point is found to be greater than or equal to a preset first threshold, and the pixel point is the searched first boundary pixel point, and then the first surface is generated based on the point cloud composed of multiple first boundary pixel points, that is, the first surface contains multiple first boundary pixel points.

[0146] Considering the influence of environmental noise when the ultrasound probe acquires images, the boundary pixel points of the organs in the acquired ultrasound images may be lost or interfered, resulting in the first curved surface obtained in step S601 being biased. If the first curved surface is output as the image segmentation result, the image segmentation accuracy is obviously low. Therefore, in order to improve the accuracy of image segmentation, in step S602, the first curved surface is subdivided by using a subdivision model to obtain a plurality of subdivided pixel points, and then in step S603, the plurality of subdivided pixel points are used as starting points, and a boundary search is performed toward the starting point according to the pixel value to obtain a plurality of second boundary pixel points, and a second curved surface is generated by the plurality of second boundary pixel points. Therefore, the second curved surface more accurately represents the three-dimensional space area of ​​the organ in the ultrasound image, and the second curved surface is output as the image segmentation result, which obviously improves the accuracy of image segmentation.

[0147] In this embodiment, the subdivision model can be a filter based on the Loop subdivision algorithm, such as the vtkLoopSubdivisoon filter, etc., which can insert edge points and update the original pixel points according to each pixel point in the above-mentioned first surface, continuously subdivide to increase the number of faces of the triangular surface, and obtain the vertices of the newly added triangular surface, which are the pixel points after subdivision. In other words, the subdivision model is used to subdivide multiple first boundary pixel points in the first surface to obtain multiple pixel points after subdivision. For example, in the subdivision model, let the pixel point set after each subdivision be P1, the newly inserted pixel point set be Eps, and the pixel point set after the original pixel point set is updated is P2, then P1=P2UEps. In some other optional embodiments, the subdivision model can be a filter based on a linear subdivision algorithm, or a filter based on a butterfly subdivision algorithm, etc., which is not limited here.

[0148] That is to say, the positions and pixel values ​​of the multiple pixel points obtained after the subdivision process in step S602 have changed relative to the positions of the first boundary pixel points in the first surface. Therefore, in order to more accurately determine the target boundary, in step S603, the multiple pixel points after the subdivision process are used as the starting point, and the boundary search is performed toward the starting point (i.e., inward) with a preset step length according to the spherical coordinate system until the pixel value of the searched pixel point is less than the preset second threshold, thereby obtaining multiple second boundary pixel points, i.e., the pixel value of the second boundary pixel point is less than the preset second threshold, and then the point cloud composed of the multiple second boundary pixel points is generated to obtain the second surface. Fig. 9 As shown, the starting point is O, the first boundary pixel point is X1, the first curved surface is Y1, the pixel point after subdivision processing is X2, the second boundary pixel point is X3, and the second curved surface is Y2.

[0149] The above-mentioned preset first threshold and preset second threshold can both be determined according to the pixel value size of the organ boundary. Specifically, the preset first threshold and the preset second threshold can be equal.

[0150] In one embodiment, if Fig.10 As shown, the prompt information includes first prompt information and second prompt information, and step S402 may include:

[0151] S701, selecting a plurality of trajectory points in the predicted motion trajectory;

[0152] S702, obtaining the distance between each trajectory point and the target;

[0153] S703: if the distance between any one of the trajectory points and the area of ​​the target in the ultrasound image is less than or equal to a preset distance threshold, output the first prompt information;

[0154] S704: If the distances between all the trajectory points and the area of ​​the target in the ultrasound image are greater than the preset distance threshold, output the second prompt information.

[0155] Specifically, the first prompt information is used to indicate that the predicted motion trajectory poses a threat to the target, and the second prompt information is used to indicate that the predicted motion trajectory poses no threat to the target.

[0156] In step S701, the predicted motion trajectory may be equally divided into multiple segments along the length direction, and then a trajectory point is selected from each segment to obtain multiple trajectory points. Alternatively, multiple trajectory points are selected sequentially from the tail end to the head end of the predicted motion trajectory according to a preset interval.

[0157] In step S702, the distance between the trajectory point and the target area in the ultrasound image can be expressed as the shortest distance between the trajectory point and the target in the three-dimensional space area in the ultrasound image. If the shortest distance is less than or equal to the preset distance threshold, the first prompt information is output, indicating that the expected motion trajectory is threatening to the organ or will cause damage, etc., thereby prompting the doctor to re-plan the target or adjust the motion trajectory of the medical device to avoid damage to the patient's organ and cause unnecessary harm to the patient. If the shortest distance between all of the trajectory points and the target area in the ultrasound image is greater than the preset distance threshold, the second prompt information is output, indicating that the expected motion trajectory is not threatening to the organ, and the doctor can perform a puncture operation to complete the biopsy sampling.

[0158] like Fig.11 As shown, the embodiment of the present application provides a process for providing information prompts based on the estimated motion trajectory as follows:

[0159] S11. Classify organs into regular-shaped organs and irregular-shaped organs;

[0160] S12, determining a region of a regular-shaped organ in the ultrasound image based on a preset three-dimensional shape;

[0161] S13, segmenting the region of the irregular-shaped organ in the ultrasound image using an ultrasound image segmentation method;

[0162] S14, calculating the distance between the predicted motion trajectory and each organ;

[0163] S15. Output prompt information.

[0164] Furthermore, if Fig.12 As shown, the embodiment of the present application also provides an ultrasound image segmentation method, comprising:

[0165] S801, acquiring an ultrasound image captured by an ultrasound probe;

[0166] S802, taking the starting point set in the ultrasound image as the center, performing boundary search outward according to the pixel value to obtain a plurality of first boundary pixel points, and generating a first curved surface from the plurality of first boundary pixel points, wherein the value of the first boundary pixel point is greater than or equal to a preset first threshold;

[0167] S803, using a subdivision model to perform subdivision processing on the first curved surface to obtain a plurality of subdivision-processed pixel points;

[0168] S804. Taking the multiple subdivided pixel points as the starting point, perform boundary search toward the starting point according to the pixel value to obtain multiple second boundary pixel points, and generate a second surface from the multiple second boundary pixel points. The value of the second boundary pixel point is less than or equal to a preset second threshold value. The second surface is used to represent the area of ​​the second target in the ultrasound image.

[0169] In step S801, the ultrasound image may be an ultrasound image obtained by scanning a sagittal plane of a body part of a patient (such as a prostate, etc.) with an ultrasound probe.

[0170] The target may be represented as an organ, such as an irregularly shaped organ such as a bladder. Steps S802 to S804 may refer to the principles or functions of steps S601 to S603 in the above trajectory display method, and will not be described in detail here.

[0171] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0172] Based on the same inventive concept, the embodiment of the present application also provides a trajectory display device for implementing the above-mentioned trajectory display method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more trajectory display device embodiments provided below can refer to the limitations of the trajectory display method above, and will not be repeated here.

[0173] In one embodiment, Fig.13As shown, a trajectory display device 100 is provided, comprising:

[0174] A coordinate conversion module 110 is used to calculate the coordinates of the guiding tool in the image coordinate system corresponding to the ultrasound probe;

[0175] A rotation module 120, configured to control the rotation of the ultrasound probe according to the coordinates of the guiding tool in the image coordinate system and a first sagittal plane currently scanned by the ultrasound probe;

[0176] An image acquisition module 130, used to acquire an ultrasonic image scanned by the rotated ultrasonic probe;

[0177] The trajectory display module 140 is used to display the predicted motion trajectory of the medical instrument in the ultrasound image based on the coordinates of the guiding tool in the image coordinate system and the length of the medical instrument installed in the guiding tool.

[0178] In one embodiment, the trajectory display device 100 is further used for:

[0179] Get the location of the target point;

[0180] The initial point of the guiding tool is controlled to align with the target point, and the guiding tool is positioned based on the position of the target point to obtain the coordinates of the guiding tool in the robot coordinate system.

[0181] The coordinate conversion module 110 is used to obtain the coordinate conversion relationship between the robot coordinate system and the image coordinate system; based on the coordinate conversion relationship, the coordinates of the guiding tool in the robot coordinate system are converted into coordinates in the image coordinate system.

[0182] In one embodiment, the position of the target point includes the coordinates of the target point in the robot coordinate system; the trajectory display device 100 is further used for:

[0183] Obtaining the coordinates of the initial point of the guiding tool in the robot coordinate system;

[0184] Based on the coordinates of the target point in the robot coordinate system and the coordinates of the initial point of the guiding tool in the robot coordinate system, obtaining a vertical angle and a horizontal angle of the initial point of the guiding tool relative to the target point;

[0185] According to the vertical angle and the horizontal angle, the initial point of the guiding tool is controlled to align with the target point, and the guiding tool is positioned based on the position of the target point to obtain the coordinates of the guiding tool in the robot coordinate system.

[0186] In one embodiment, the rotation module 120 is also used to obtain the rotation angle of the ultrasound probe based on the coordinates of the guide tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe, and control the ultrasound probe to rotate by the rotation angle so that the medical device installed in the guide tool is in the second sagittal plane scanned by the ultrasound probe.

[0187] In one embodiment, the rotation module 120 is further used for:

[0188] Acquire the coordinates of the ultrasound probe in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe;

[0189] Based on the coordinates of the guiding tool in the image coordinate system, obtaining the coordinates of the medical device installed in the guiding tool in the image coordinate system;

[0190] Based on the coordinates of the ultrasound probe in the image coordinate system and the coordinates of the medical device in the image coordinate system, a second sagittal plane is obtained, where the second sagittal plane is a plane where the ultrasound probe and the medical device are located;

[0191] An angle between the first sagittal plane and the second sagittal plane is obtained, where the angle is a rotation angle of the ultrasound probe.

[0192] In one embodiment, the trajectory display device 100 is further used for:

[0193] Acquire a region of at least one target in the ultrasound image;

[0194] Prompt information is output based on the distance between the predicted motion trajectory and the area of ​​the target in the ultrasound image.

[0195] In one embodiment, the at least one target includes a first target, and the trajectory display device 100 is further used for:

[0196] Acquire the position of the first target in the ultrasound image;

[0197] A region of the first target in the ultrasound image is obtained in the ultrasound image based on a preset three-dimensional shape and a position of the first target in the ultrasound image.

[0198] In one embodiment, the at least one target further includes a second target, and the trajectory display device 100 is further used for:

[0199] Taking the set starting point as the center, a boundary search is performed outward according to the pixel value to obtain a plurality of first boundary pixel points, and a first curved surface is generated by the plurality of the first boundary pixel points, wherein the value of the first boundary pixel point is greater than or equal to a preset first threshold value;

[0200] Subdividing the first curved surface using a subdivision model to obtain a plurality of subdivided pixel points;

[0201] Taking the multiple subdivided pixel points as the starting point, a boundary search is performed toward the starting point according to the pixel value size to obtain multiple second boundary pixel points, and a second surface is generated by the multiple second boundary pixel points, the value of the second boundary pixel point is less than or equal to a preset second threshold, and the second surface is used to represent the area of ​​the second target in the ultrasound image.

[0202] In one of the embodiments, the trajectory display device 100 is also used to obtain the rotation angle of the ultrasound probe based on the coordinates of the guidance tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe, and control the ultrasound probe to rotate the rotation angle so that the medical device installed in the guidance tool is in the second sagittal plane scanned by the ultrasound probe.

[0203] In one embodiment, the prompt information includes first prompt information and second prompt information, and the trajectory display device 100 is further used for:

[0204] Selecting a plurality of trajectory points in the predicted motion trajectory;

[0205] If the distance between any one of the trajectory points and the area of ​​the target in the ultrasound image is less than or equal to a preset distance threshold, outputting the first prompt information, wherein the first prompt information is used to indicate that the predicted motion trajectory poses a threat to the target;

[0206] If the distances between all the trajectory points and the area of ​​the target in the ultrasound image are greater than the preset distance threshold, the second prompt information is output, and the second prompt information is used to indicate that the expected motion trajectory poses no threat to the target.

[0207] Each module in the above-mentioned trajectory display device 100 can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0208] Based on the same inventive concept, the embodiment of the present application also provides an ultrasonic image segmentation device for implementing the ultrasonic image segmentation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more ultrasonic image segmentation device embodiments provided below can refer to the limitations on the ultrasonic image segmentation method above, and will not be repeated here.

[0209] In one embodiment, Fig.14 As shown, an ultrasound image segmentation device 200 is provided, comprising:

[0210] An image acquisition module 210 is used to acquire an ultrasound image captured by an ultrasound probe;

[0211] A first search module 220 is used to perform a boundary search outward according to the pixel value with the starting point set in the ultrasound image as the center to obtain a plurality of first boundary pixel points, and generate a first curved surface from the plurality of first boundary pixel points, wherein the value of the first boundary pixel point is greater than or equal to a preset first threshold;

[0212] A subdivision module 230, configured to perform subdivision processing on the first curved surface using a subdivision model to obtain a plurality of subdivided pixel points;

[0213] The second search module 240 is used to take the multiple subdivided pixel points as the starting point, perform a boundary search toward the starting point according to the pixel value size, obtain multiple second boundary pixel points, and generate a second surface from the multiple second boundary pixel points. The value of the second boundary pixel point is less than or equal to a preset second threshold value, and the second surface is used to represent the area of ​​the second target in the ultrasound image.

[0214] In one embodiment, the first search module 220 is further configured to:

[0215] Taking the starting point as the center, a boundary search is performed outward with a preset step size according to the spherical coordinate system. If the pixel value of the searched pixel point is greater than or equal to a preset first threshold, the searched pixel point is the first boundary pixel point.

[0216] Each module in the ultrasonic image segmentation device 100 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0217] Furthermore, if Fig.15As shown, in another embodiment, the present application provides a surgical robot 300, and the surgical robot 300 may include:

[0218] An ultrasonic probe 310, used for acquiring ultrasonic images;

[0219] a guide tool 320 in which a medical device 321 is installed; and

[0220] The control device 330 includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned trajectory display method and / or ultrasound image segmentation method when executing the computer program;

[0221] The display 340 is used to display the predicted motion trajectory of the medical device and / or the area of ​​at least one target in the ultrasound image in the ultrasound image.

[0222] In some other embodiments, the surgical robot 300 may also include a motor, which is controlled by the control device 330 and is used to drive the ultrasonic probe to move or rotate, such as a stepping and rotating motor, etc., which is used to drive the ultrasonic probe to scan the patient's rectum to obtain a prostate ultrasound image. The surgical robot 300 may also include a mechanical arm, which is controlled by the control device 330 and is used to move and position the guide tool to assist the doctor in performing a puncture operation or the mechanical arm drives the guide tool to perform a puncture operation, etc. The puncture operation is used to indicate that the medical instrument in the guide tool punctures from the initial point on the surface of the prostate to the target point inside the prostate to achieve biopsy sampling. In some other optional embodiments, the surgical robot 300 may also include other mechanisms that can perform operations, such as being controlled by the control device to perform corresponding operations, etc.

[0223] like Fig.16 The figure shows a hardware architecture diagram of a surgical robot of an embodiment. The surgical robot may include a power supply, a circuit board, a medical computer, an image grabber, an ultrasonic social security, multiple drivers and multiple motors. The power supply provides the power required for the operation of each hardware in the surgical robot. The ultrasonic device includes an ultrasonic probe for collecting ultrasonic images. The image grabber is used to capture the ultrasonic images collected by the ultrasonic device and transmit them to the medical computer. The hospital computer is used to display the ultrasonic images. The circuit board is used to control each driver to drive the corresponding motor to perform movement or rotation, etc. Multiple motors such as stepper motors, rotary motors, V-axis motors and H-axis motors, etc., wherein the stepper motor is used to drive the ultrasonic probe to move step by step, the rotary motor is used to drive the ultrasonic probe to rotate, the V-axis motor is used to drive the mechanical arm to rotate along the V-axis (such as a vertical angle) to guide the tool, and the H-axis motor is used to drive the mechanical arm to rotate along the H-axis (such as a horizontal angle) to guide the tool.

[0224] like Fig.17 As shown, the software serial port read and write control motor process of the surgical robot 300 is as follows:

[0225] S21, selecting a motor to be controlled from multiple motors;

[0226] S22, obtaining a motor control instruction, where the motor control instruction is used to indicate controlling the motor;

[0227] S23, writing the motor control instruction into the serial port;

[0228] S24, create a new thread and start executing motor control instructions;

[0229] S25, read the motor return value, motor position;

[0230] S26, if the return value is true, the motor reads the instruction normally;

[0231] S27. If the return value is false, the motor reads the command abnormally, writes the log, and closes the serial port.

[0232] It should be pointed out that the implementation solution provided by the surgical robot 300 is similar to the implementation solution recorded in the above-mentioned trajectory display method and / or ultrasonic image segmentation method. The specific principles or functions can refer to the above-mentioned trajectory display method and / or ultrasonic image segmentation method, and will not be repeated here.

[0233] In another embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.18 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a trajectory display method and / or an ultrasonic image segmentation method is implemented.

[0234] Those skilled in the art will understand that Fig.18 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0235] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned trajectory display method and / or ultrasound image segmentation method when executing the computer program.

[0236] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned trajectory display method and / or ultrasound image segmentation method are implemented.

[0237] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned trajectory display method and / or ultrasound image segmentation method:

[0238] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0239] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0240] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A trajectory display method, characterized in that: The method comprises: calculating the coordinates of the guiding tool in the image coordinate system corresponding to the ultrasound probe; Controlling the rotation of the ultrasound probe according to the coordinates of the guiding tool in the image coordinate system and a first sagittal plane currently scanned by the ultrasound probe; Acquiring an ultrasonic image scanned by the rotated ultrasonic probe; Based on the coordinates of the guiding tool in the image coordinate system and the length of the medical device installed in the guiding tool, displaying the expected motion trajectory of the medical device in the ultrasound image; Acquire a region of at least one target in the ultrasound image; outputting prompt information based on the distance between the predicted motion trajectory and the area of ​​the target in the ultrasound image; The at least one target includes a first target, and acquiring a region of the at least one target in the ultrasound image includes: Acquire the position of the first target in the ultrasound image; The method comprises: scaling the preset three-dimensional shape according to a preset scaling ratio, generating a scaled preset three-dimensional shape at the position of the first target in the ultrasound image, and obtaining the area of ​​the first target in the ultrasound image.

2. The method according to claim 1, characterized in that The method further comprises: Get the location of the target point; Controlling the initial point of the guiding tool to align with the target point, and positioning the guiding tool based on the position of the target point to obtain the coordinates of the guiding tool in the robot coordinate system; The calculation of the coordinates of the guiding tool in the image coordinate system corresponding to the ultrasound probe includes: Obtain the coordinate transformation relationship between the robot coordinate system and the image coordinate system; The coordinates of the guiding tool in the robot coordinate system are converted into coordinates in the image coordinate system based on the coordinate conversion relationship.

3. The method according to claim 2, characterized in that The position of the target point includes the coordinates of the target point in the robot coordinate system; the initial point of the control guide tool is aligned with the target point, and the guide tool is positioned based on the position of the target point to obtain the coordinates of the guide tool in the robot coordinate system, including: Obtaining the coordinates of the initial point of the guiding tool in the robot coordinate system; Based on the coordinates of the target point in the robot coordinate system and the coordinates of the initial point of the guiding tool in the robot coordinate system, obtaining a vertical angle and a horizontal angle of the initial point of the guiding tool relative to the target point; According to the vertical angle and the horizontal angle, the initial point of the guiding tool is controlled to align with the target point, and the guiding tool is positioned based on the position of the target point to obtain the coordinates of the guiding tool in the robot coordinate system.

4. The method according to claim 1, characterized in that The controlling the rotation of the ultrasound probe according to the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe comprises: Based on the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe, the rotation angle of the ultrasound probe is obtained, and the ultrasound probe is controlled to rotate by the rotation angle so that the medical device installed in the guiding tool is in the second sagittal plane scanned by the ultrasound probe.

5. The method according to claim 4, characterized in that The obtaining the rotation angle of the ultrasound probe based on the coordinates of the guiding tool in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe comprises: Acquire the coordinates of the ultrasound probe in the image coordinate system and the first sagittal plane currently scanned by the ultrasound probe; Based on the coordinates of the guiding tool in the image coordinate system, obtaining the coordinates of the medical device installed in the guiding tool in the image coordinate system; Based on the coordinates of the ultrasound probe in the image coordinate system and the coordinates of the medical device in the image coordinate system, a second sagittal plane is obtained, where the second sagittal plane is a plane where the ultrasound probe and the medical device are located; An angle between the first sagittal plane and the second sagittal plane is obtained, where the angle is a rotation angle of the ultrasound probe.

6. The method according to claim 1, characterized in that The at least one target further includes a second target, and acquiring a region of the at least one target in the ultrasound image includes: Taking the set starting point as the center, a boundary search is performed outward according to the pixel value to obtain a plurality of first boundary pixel points, and a first curved surface is generated by the plurality of the first boundary pixel points, wherein the value of the first boundary pixel point is greater than or equal to a preset first threshold value; Subdividing the first curved surface using a subdivision model to obtain a plurality of subdivided pixel points; Taking the multiple subdivided pixel points as the starting point, a boundary search is performed toward the starting point according to the pixel value size to obtain multiple second boundary pixel points, and a second surface is generated by the multiple second boundary pixel points, the value of the second boundary pixel point is less than or equal to a preset second threshold, and the second surface is used to represent the area of ​​the second target in the ultrasound image.

7. The method according to claim 1, characterized in that The method of performing boundary search outwards based on pixel values ​​with the set starting point as the center to obtain a plurality of first boundary pixel points includes: Taking the starting point as the center, a boundary search is performed outward with a preset step size according to the spherical coordinate system. If the pixel value of the searched pixel point is greater than or equal to a preset first threshold, the searched pixel point is the first boundary pixel point.

8. The method according to claim 1, characterized in that The prompt information includes first prompt information and second prompt information, and outputting the prompt information based on the distance between the predicted motion trajectory and the area of ​​the target in the ultrasound image includes: Selecting a plurality of trajectory points in the predicted motion trajectory; Obtaining the distance between each of the trajectory points and the target; If the distance between any one of the trajectory points and the area of ​​the target in the ultrasound image is less than or equal to a preset distance threshold, outputting the first prompt information, wherein the first prompt information is used to indicate that the predicted motion trajectory poses a threat to the target; If the distances between all the trajectory points and the area of ​​the target in the ultrasound image are greater than the preset distance threshold, the second prompt information is output, and the second prompt information is used to indicate that the expected motion trajectory poses no threat to the target.

9. A surgical robot, characterized in that: include: An ultrasound probe, used for acquiring ultrasound images; a guide tool, wherein the medical device is installed in the guide tool; as well as A control device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of claims 1 to 8 when executing the computer program; A display is used to display the expected motion trajectory of the medical device and / or the area of ​​at least one target in the ultrasound image in the ultrasound image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images