Three-dimensional reconstruction method for safe target area based on ultrasonic image

Abstract

Disclosed is a three-dimensional reconstruction method for a safe target area based on an ultrasonic image. The method comprises the following steps: S1, moving a probe to a target area by controlling the movement of the ultrasonic probe; S2, controlling the probe to start to rotate at a fixed angle, staying for a certain time after each rotation, collecting B-mode ultrasound images within the time, and performing target boundary recognition on the acquired images to obtain the boundary of safe target areas in a single image; S3, comparing the boundaries of the plurality of images at the same angle, and determining to form a comprehensive safe area in the field of view of the current ultrasonic probe; and S4, after the probe is rotated by 180 degrees, combining the safe areas in the images at multiple angles to construct a three-dimensional area, wherein the area is a safe target area in the treatment process. By using the method, the accurate three-dimensional shape of a safe target area (treatment area) is determined before the non-invasive procedure is about to start, so that the surgical system and equipment can perform treatment in the safe area.

Description

Ultrasound image-based safe target area three-dimensional reconstruction method TECHNICAL FIELD

[0001] The present application relates to an ultrasound image-based safe target area three-dimensional reconstruction method, belonging to the technical field of ultrasonic imaging. BACKGROUND

[0002] At present, there are various non-invasive surgical medical devices in various medical departments. These devices accurately position energy on target points in the target area through ultrasound, laser, radioactive rays and other physical means to produce therapeutic efficacy.

[0003] Before using these devices for treatment, a basic pre-examination is usually performed. Then, target point / target area planning is performed in the selected area. However, in reality, a considerable part of surgical devices need to consider the real-time shift of the target area caused by human respiratory movement during the planning process. Therefore, in addition to the accuracy of the target point ensured by the structure and system, these devices also need to have good adaptability to human respiratory movement. Some device A, which is worn on the surface of the human body, has low energy, and its therapeutic efficacy is only exerted to the skin or relatively shallow muscle, and the influence of respiratory movement can be ignored, such as some beauty devices or rehabilitation therapy devices. Some device B relies on the understanding of the device by the operator, combined with medical knowledge and experience, to intentionally reserve some space to ensure the safety of the target point treatment, such as extracorporeal shock wave lithotripsy. Some device C increases an external device for real-time tracking of user's respiration, dynamically adjusts the target point of the target area, so as to ensure the safety of the target point of the target area, such as gamma knife.

[0004] The organ movement affected by human respiratory movement is three-dimensional and dynamic. The accuracy of the device B depends on the operator's experience, and the accuracy will vary greatly depending on the cognition of different personnel, so it cannot guarantee high uniformity and safety. The device C generally uses fitting and correlation algorithms to associate respiratory movement with the focusing position of the target point, but the respiratory determination is generally achieved through external device detection. First, the additional device needs to improve the system cost. In addition, the placement of the external device and the system access need a specific method, and some even need the object to be in a fixed posture, so it is easy to handle improperly, which will affect the implementation results. SUMMARY

[0005] The purpose of the present application is to provide an ultrasound image-based safe target area three-dimensional reconstruction method, to establish a safe and reliable target point space for respiratory movement, and to apply this method to ultrasonic treatment systems and devices.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is: an ultrasound image-based safe target area three-dimensional reconstruction method, comprising the following steps:

[0007] S1, moving the probe to the target region by controlling the movement of the ultrasound probe;

[0008] S2, controlling the probe to start rotating at a fixed angle, after each rotation, staying for a set time, during the staying time, collecting B-ultrasound images, and identifying the target boundary of the obtained images to obtain the boundary of a single image safe target area;

[0009] S3, comparing the boundaries of multiple image contents at the same angle to determine the current ultrasound probe field of view comprehensive safety area;

[0010] S4, after the probe completes 180-degree rotation, combining the safety areas in the images of multiple angles to construct a three-dimensional area, which is the safety target area during the treatment process.

[0011] Further, in step S1, the probe posture control system is used to control the ultrasound module probe to move to the target region (some controllers support manual direct dragging of the probe to the specified region, and can ensure stable probe posture). Through the ultrasound module probe, the target region ultrasound image can be seen.

[0012] Further, in step S2, the initial position of the ultrasound probe is set to 0°, and the end joint of the mechanical arm is controlled to rotate 180°. The rotation logic is: divide 180° equally, and obtain ultrasound images at a fixed angle each time.

[0013] Further, considering data processing, uniform image acquisition, and three-dimensional reconstruction precision, a granularity angle D is selected, which satisfies two conditions: greater than the mechanical arm rotation accuracy, and divisible by 180; divide 180° into C equal parts according to D, then:

[0014] Then the probe control module rotates sequentially to intercept ultrasound images according to the set granularity angle D. Considering the relationship between equal division and cross section, the overall control system can obtain a set of (C+1) plane images, but 0° and 180° are theoretically the same plane, the difference is that the two ultrasound images are flipped left and right, so in actual application, C planes can be taken.

[0015] Further, the probe rotates sequentially to intercept ultrasound images according to the set granularity angle D; when the probe travels to the target position, it starts to collect ultrasound images, stays for a time T, and the time T is set in the following two ways:

[0016] Automatic determination: according to the human respiratory movement cycle, set a fixed time interval, and stop collecting when 3-5 respiratory cycles can be obtained;

[0017] Manual determination: the overall control system always acquires images until the external manual stops collecting through the system;

[0018] At the same time, within T time, images are acquired at F frames per second, and the ultrasound image is segmented through the ultrasound image recognition module to obtain a single image segmentation result. The result is a set of polygon points arranged in sequence, and different labels are used to distinguish different image segmentation contents.

[0019] Furthermore, in step S3, after the position image of an angle is acquired, a set of polygon boundary points of different labels at the angle can be obtained. Then, in the two-dimensional plane, a clipping algorithm is used to fill the inside and outside of the image according to the odd and even filling recognition, and the intersection and union of polygons with the same label are solved (the safe target area label can take the intersection, and other labels take the union).

[0020] Furthermore, in the process of boundary recognition, theoretically, the denser the recognition points, the more precise and accurate the recognition area, and the finer the 3D reconstruction. Currently, there are two main methods for increasing the point density of B-ultrasound image recognition areas:

[0021] (1) Ultrasonic image segmentation is implemented in the ultrasonic image recognition module, which uses image segmentation machine learning methods to segment the image. In the early stage of the construction of the ultrasonic image recognition module, when calibrating the machine learning data source, high-density points are used to calibrate the edges of the relevant areas, so that the point density of the results recognized by the ultrasonic image recognition module will be increased.

[0022] (2) For the recognition result of the ultrasound image recognition module, a dynamic interpolation method is used to interpolate between two given points, including using a B-spline curve for interpolation.

[0023] Furthermore, the above steps are repeated until the robot arm is controlled to complete a 180° rotation.

[0024] Furthermore, in step S4, after the probe completes a 180-degree rotation, C plane polygonal point sets of safe target areas and unsafe target areas have been obtained, and each plane safe target area point set is associated with an angle; because these areas are coaxial, under this condition, a point mapping from each plane to a three-dimensional space can be established.

[0025] Furthermore, we can establish a mapping from points on each plane to points in three-dimensional space. The specific method is as follows:

[0026] (1) Assume that the Z axis is the probe rotation axis, the rotation direction is counterclockwise, and the first plane angle is set to 0°, so that the plane is coplanar with the XZ axis.

[0027] (2) The angle α of the Nth set of images (N∈[1,C]) is D*(N-1), and in image G nIn the plane, the position of an image point is p(m, n), and the image size is fixed as (W, H); because in general cases, the origin of the image pixel point is the top vertex of the left upper corner, and the vertical coordinate increases along the image vertical direction. In order to facilitate calculation, the origin is first moved to the image center, at this time, the image vertical direction (height direction) is coaxial with Z, and then the vertical coordinate is flipped. At this time, the image origin is the same as the origin in the three-dimensional space, and the vertical coordinate value growth direction is the same, as shown in Figure 2. Assuming that the converted point of p(m, n) is p'(m', n'), then:

[0028] (3) The converted point p'(m', n') is placed in the three-dimensional space, as shown in Figure 3, and the point P(x, y, z) corresponds to it, wherein the coordinates of the point P satisfy: x=m'cosα y=-m'sinα z=n'.

[0029] Further, after completing the three-dimensional point mapping of the C safe target areas, the safe area point cloud is obtained, and then the point cloud to three-dimensional contour algorithm is used to obtain the approximate three-dimensional reconstructed safe target area.

[0030] Finally, the non-invasive surgery area can be planned according to the reconstructed three-dimensional full target area, and the surgical treatment is performed through the ultrasonic treatment module.

[0031] Compared with the prior art, the beneficial effects of the present application are:

[0032] (1) The three-dimensional reconstruction scheme of the present application is for the determination of the planning stage safe target area, and establishes a safe and reliable target point space for the respiratory motion of the object. The establishment of this three-dimensional space uniform process is safe and reliable, can adapt to different body positions, and at the same time, reduces the introduction of external auxiliary equipment, and reduces the cost and system complexity.

[0033] (2) In actual use, the three-dimensional reconstruction scheme of the present application can quantitatively calculate the reconstruction mode process, so different parameters such as rotation granularity angle can be adjusted according to the scene, so as to obtain a three-dimensional reconstruction effect more suitable for the business scene.

[0034] (3) The three-dimensional reconstruction scheme of the present application can reconstruct different segmented objects, and is not limited to the reconstruction of the safe target area in the present scheme. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a method flowchart of the present application;

[0036] Figure 2 is an image coordinate translation and flipping diagram;

[0037] Figure 3 is a two-dimensional coordinate mapping into a three-dimensional scene coordinate diagram;

[0038] Figure 4 is a 10-second screenshot based on a system angle of 135 degrees fixed position;

[0039] Figure 5 is a single file image segmentation result diagram (B region is a safe target area, and A region is a kidney lower pole) ;

[0040] Figure 6 is a three-dimensional point cloud three-view diagram organized based on the segmentation manner similar to that of Figure 5, combined with the safe target area and the kidney segmentation result of each angle (light color is the safe target area, and dark color is the kidney lower pole) ;

[0041] Figure 7 is a slice point cloud three-view diagram based on the coordinate set of Figure 6, converted to Z direction parallel to the XY plane (light color is the safe target area, and dark color is the kidney lower pole) ;

[0042] Figure 8 is a three-dimensional reconstruction result three-view diagram after adding a peripheral contour based on the point cloud of Figure 7 (light color is the safe target area, and dark color is the kidney lower pole). DETAILED DESCRIPTION

[0043] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0044] The present application provides a safe target area three-dimensional reconstruction method based on an ultrasound image, which is applied to an ultrasound treatment system and equipment before formal treatment.

[0045] First, the ultrasound probe is moved to a target area by precisely controlling the movement of the ultrasound probe. Then, the probe starts to rotate at a fixed angle. After each rotation, it stays for a certain period of time, during which B-mode images are collected, and the target boundary of the obtained images is identified to obtain the boundary of the safe target area (or related tissue) of a single image. Then, the boundaries of the contents of multiple images at the same angle are compared to determine the comprehensive safe area of the current ultrasound probe field of view. Finally, after the probe completes 180-degree rotation, the safe areas in the images of multiple angles are combined to construct a three-dimensional area, which is the safe target area during treatment.

[0046] During the target point confirmation process or during formal treatment, this area can ensure the safety and reliability of treatment. At the same time, during the image acquisition link, the ultrasound equipment used in the early stage of treatment is reused, so that certain costs can be reduced.

[0047] In the implementation process of the present application, the hardware is selected as follows:

[0048] Ultrasound module H1: a B-mode machine is used, and two probes (P1, P2) are configured, wherein the probe (P1) participating in the safe target area three-dimensional reconstruction satisfies:

[0049] Frequency range: 2.5 MHz to 6 MHz;

[0050] Depth range: 4.5 cm to 31 cm;

[0051] Image FPS range: 20 to 30.

[0052] In addition, the ultrasound module supports conventional adjustments of gain, power, focus, etc. Another probe (P2) can be used as a priori device to preliminarily determine the target region.

[0053] Probe control module H2: using a mechanical arm, a 6-axis mechanical arm, with the following basic parameters:

[0054] Working range: 1327 mm;

[0055] Repositioning accuracy: ±0.03 mm;

[0056] Maximum end load: 12 kg.

[0057] Ultrasound treatment module H3: using an electro-acoustic conversion system. Containing a control board and a transducer T.

[0058] The software part function development is implemented as follows:

[0059] Ultrasound control system S1: can control the ultrasound to make conventional parameter adjustments, and can obtain ultrasound images through the system.

[0060] Probe posture control system S2: can use multiple links of the probe control module H2 to move in three-dimensional space;

[0061] Ultrasound image recognition module S3: using image segmentation algorithm, can mark the region of the image at a specific position, and can distinguish between organs and safe target areas.

[0062] Transducer control system S4: can control the transducer to emit energy.

[0063] Overall control system S5: can schedule all software and hardware resources, provide a unified operation interface for users, and realize a three-dimensional reconstruction scheme.

[0064] Software S1-S4 is published in binary library, directly integrated into S5, thereby forming a unified software system S. The system S can complete all software operations of all three-dimensional safe target area reconstruction.

[0065] Hardware H1-H3 control board and system S running equipment are placed in a unified area. The operation area structure of H1 ultrasound probe P1 and H3 transducer T is guaranteed to be unified.

[0066] Specifically, the method steps of the present application are as follows:

[0067] S1, control the mechanical arm H2 to move to the target region through the system S, and finally adjust to the appropriate posture.

[0068] S2, adjust the ultrasound module H1 through the system S to a suitable frequency, depth and focus parameters, to obtain a relatively clear B-ultrasound image.

[0069] S3, start to intercept B-ultrasound images of the target region using the B-ultrasound probe through the system S, considering data integrity and actual situation of target personnel comfort and tolerance, the interception logic and related parameters are as follows:

[0070] Set the initial position to 0°, intercept every 15°, rotate counterclockwise by 180°, and set the rotation axis as the Z direction in the three-dimensional space;

[0071] Single interception surface stays for 10 seconds, i.e. 2-3 breathing periods under normal circumstances;

[0072] During the stay time, take a B-ultrasound image every 300 milliseconds; the intercepted image is shown in FIG. 4.

[0073] S4, after completing the B-ultrasound image interception of each fixed angle, perform image segmentation on the result image. The segmentation result is divided into kidney R and safe target area T according to the label, as shown in FIG. 5.

[0074] S5, combine the same angle and same label regions. Since there are multiple images of the same angle, for the safety target area T label, use the intersection of segmented regions to converge the range as much as possible, and for the kidney R, use the union to expand the range.

[0075] S6, use the above region point combination result to perform B-spline interpolation to obtain a dense ordered point set.

[0076] S7, repeat the above steps S3-S6 until the probe completes 180° rotation.

[0077] S8, use the above image segmentation point combination result to convert the image coordinates of all segmentation lines to scene coordinates, set the segmentation point in the two-dimensional image as p(m, n), and the converted point as p'(m', n'), and use the formula:

[0078] In the example, the image width W is 640 and the height H is 480, then: m' = m-320 n' = 240-n;

[0079] S9, use the above point to map to the three-dimensional coordinate system. Then the point p'(m', n') on the image corresponds to the point P(x, y, z) in the three-dimensional coordinate system, and the P point coordinates satisfy: x = m'cosα y = -m'sinα z = n'

[0080] Where m', n' are known, which are the point coordinates converted from two-dimensional image to space coordinates, and the angle a satisfies: a = t*15°

[0081] Where t is the rotation number, the initial position is 0, and 1 is added every 15° rotation.

[0082] The three-dimensional clicking is used to complete the safe target point cloud reconstruction as shown in Fig. 6.

[0083] S10, further, for the point set A in the above three-dimensional space, fusion is made into a three-dimensional model. The point cloud outer contour algorithm or Z direction slice reconstruction mode can be used. In the present example, considering the point sparse feature, the Z direction parallel to the XY plane slice mode is used to obtain the three-dimensional model by using the space Z axis rotation coordinate set. The method is as follows:

[0084] (1) in the Z direction, the XY parallel plane F Z (z is the Z axis coordinate value) is used to intersect with each calibration region in the B ultrasound section of each angle, and the point set of the outer contour of the different calibration region outer contour on Fz can be obtained;

[0085] (2) according to 1 pixel unit, the step (1) is repeated from top to bottom;

[0086] (3) the Z direction plane set F s -F e is obtained, wherein s >=-240, e <=240, the point position in each plane is combined, and the three-dimensional target dense point cloud as shown in Fig. 7 can be obtained. According to the dense point cloud surface reconstruction algorithm, the three-dimensional image as shown in Fig. 8 can be obtained.

[0087] S11, in view of the obtained three-dimensional safe region, the planned focus is intersected, and the energy in the safe region is focused, so that the treatment effect is achieved.

[0088] The basic principles, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the above examples do not limit the protection scope of the present application in any form, and any technical solutions obtained by equivalent replacement or the like fall within the protection scope of the present application. The parts not involved in the present application are the same as or can be realized by using the prior art.

Claims

1. A method for three-dimensional reconstruction of a safe target area based on ultrasound images, characterized in that The steps include: S1, by controlling the movement of the ultrasound probe, move the probe to the target area; S2. Control the probe to start rotating at a fixed angle. After each rotation, it stays for a set time. During the stay time, it acquires B-ultrasound images and performs target boundary recognition on the acquired images to obtain the boundaries of the safe target area in a single image. S3. Compare the boundaries of multiple images taken at the same angle to determine a comprehensive safety area within the current ultrasound probe field of view; S4. After the probe completes a 180-degree rotation, the safe areas in the images from multiple angles are combined to construct a three-dimensional area, which is the safe target area during treatment.

2. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 1, characterized in that: In step S2, the initial position of the ultrasound probe is set to 0°, and the end joint of the robotic arm is controlled to rotate 180°. The rotation logic is: 180° is divided into equal parts, and an ultrasound image is acquired each time the rotation is a fixed angle.

3. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 2, characterized in that: When acquiring ultrasound images, select a granularity angle D that satisfies two conditions: greater than the rotation accuracy of the robotic arm and divisible by 180°. Divide 180° into C equal parts according to D, and we have: Then the probe control module rotates and intercepts the ultrasound image according to the set granularity angle D.

4. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 3, characterized in that: The probe rotates sequentially according to the set granularity angle D to capture ultrasound images. When the probe reaches the target position, ultrasound image acquisition begins. The dwell time T can be set in the following two ways: Automatic determination: According to the human respiratory cycle, a fixed time interval is set, and the acquisition stops when 3 to 5 respiratory cycles are obtained; Manual judgment: The overall control system continues to acquire images until an external person stops the acquisition through the system; At the same time, within T time, images are acquired at F frames per second, and the ultrasound image is segmented to obtain a single image segmentation result. The result is a set of polygon points arranged in sequence, and different labels are used to distinguish different image segmentation contents.

5. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 4, characterized in that: In step S3, after the position image of an angle is acquired, a set of polygon boundary points of different labels at the angle is obtained. Then, in the two-dimensional plane, a clipping algorithm is used to fill the inside and outside of the identification image according to odd and even numbers to solve the intersection and union of polygons with the same label.

6. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 5, characterized in that: During the boundary recognition process, the denser the recognition points, the more precise and accurate the recognition area, and the finer the 3D reconstruction. There are two ways to increase the point density in the B-ultrasound image recognition area: (1) Ultrasonic image segmentation is implemented in the ultrasonic image recognition module, which uses image segmentation machine learning methods to segment the image. In the early stage of the construction of the ultrasonic image recognition module, when calibrating the machine learning data source, high-density points are used to calibrate the edges of the relevant areas, so that the point density of the results recognized by the ultrasonic image recognition module will be increased. (2) For the recognition result of the ultrasound image recognition module, a dynamic interpolation method is used to interpolate between two given points, including using a B-spline curve for interpolation.

7. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 6, characterized in that: Repeat the steps in claims 3 to 6 until the robot arm is controlled to complete a 180° rotation.

8. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 1, characterized in that: In step S4, after the probe completes a 180-degree rotation, C planar polygonal point sets of safe target areas and unsafe target areas are obtained, and each planar safe target area point set is associated with an angle; Because these areas are coaxial, under this condition, a mapping from points on each plane to points in three-dimensional space is established.

9. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 8, characterized in that: Establish a point mapping from each plane to a three-dimensional space. The specific steps are as follows: (1) Assume that the Z axis is the probe rotation axis, the rotation direction is counterclockwise, and the first plane angle is set to 0°, so that the plane is coplanar with the XZ axis. (2) The angle α of the Nth set of images (N∈[1,C]) is D*(N-1), and in image G n In the plane, let the position of an image point be p(m,n) and the image size be fixed to (W,H); First, move the origin to the center of the image. At this time, the image's vertical axis is coaxial with the Z axis. Then, flip the vertical coordinate. The image origin is the same as the origin in the three-dimensional space, and the vertical coordinate values ​​increase in the same direction. Let the point p(m,n) after the transformation be p'(m',n'), then: (3) Place the transformed point p'(m',n') in three-dimensional space, and there is a point P(x,y,z) corresponding to it, where the coordinates of point P satisfy: x=m′cosα y=-m′sinα z=n′.

10. The method for three-dimensional reconstruction of a safe target area based on ultrasound images according to claim 8 or 9, characterized in that: After completing the 3D point mapping of C safety target areas, the safety area point cloud is obtained, and then the point cloud to 3D contour algorithm is used to obtain the 3D reconstructed safety target area.

Images