Human organ ultrasonic three-dimensional reconstruction method and device based on respiratory displacement compensation

Abstract

The invention discloses a human organ ultrasonic three-dimensional reconstruction method and device based on respiratory displacement compensation, and the method comprises the steps: controlling a probe at the tail end of a mechanical arm of an ultrasonic robot to collect a plurality of test ultrasonic images in a preset test time period at a target organ of a human body; based on the plurality of test ultrasonic images, carrying out conversion from two-dimensional pixel points to three-dimensional point clouds, and establishing a breathing displacement model of the target organ; and controlling the probe to continuously collect ultrasonic images, and applying the breathing displacement model to all the ultrasonic images to obtain a target organ model after ultrasonic three-dimensional reconstruction. According to the method, the accuracy and the stability of three-dimensional reconstruction are improved, the problem of target organ displacement caused by respiratory movement is effectively solved, the applicability and the reliability of ultrasonic imaging in a dynamic environment are improved, and a more complete and accurate three-dimensional target organ model can be obtained.

Description

Technical Field

[0001] The invention relates to a method and device for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation. Background Art

[0002] With the development of medical technology, ultrasonic scanning imaging has been widely used in clinical diagnosis due to its non-invasive, real-time and cost-effective characteristics, especially in the examination of human organs. In order to realize the autonomous scanning of human organs by ultrasonic robots, it is often necessary to reconstruct the three-dimensional morphology of the organs in real time during the scanning process, so as to guide the robot to complete the search for the optimal section of the organ. However, human breathing will inevitably cause unknown organ displacement, which poses a great challenge to its real-time three-dimensional reconstruction during the scanning process. Taking the most typical gallbladder organ as an example, human breathing will cause it to produce a displacement fluctuation of 2-5cm, and traditional ultrasonic imaging methods are difficult to deal with the problem of organ displacement caused by breathing. Therefore, how to realize the three-dimensional reconstruction of human organs through real-time compensation of respiratory displacement is the key step to realize the autonomous scanning of human organs by ultrasonic robots. Summary of the invention

[0003] In order to achieve more accurate three-dimensional reconstruction of human organs, an embodiment of the present invention provides a method and device for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation.

[0004] In a first aspect, an embodiment of the present invention provides a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation, which may include:

[0005] At the target organ of the human body, controlling the probe at the end of the ultrasonic robot mechanical arm to collect multiple test ultrasonic images within a preset test period;

[0006] Based on the multiple test ultrasound images, convert two-dimensional pixel points into three-dimensional point clouds to establish a respiratory displacement model of the target organ;

[0007] The probe is controlled to continue acquiring ultrasonic images, and the respiratory displacement model is applied to all the ultrasonic images to obtain a target organ model after ultrasonic three-dimensional reconstruction.

[0008] In one or some optional implementations of the embodiments of the present application, the conversion of two-dimensional pixel points to three-dimensional point clouds based on the multiple test ultrasound images to establish the respiratory displacement model of the target organ includes:

[0009] Calculate the point cloud centroid of each test ultrasound image;

[0010] Based on the point cloud centroids of all the test ultrasound images, decomposition is performed in three directions of XYZ to obtain a scatter plot of the point cloud centroids in the three directions of XYZ;

[0011] Curve fitting is performed on the centroid scatter plot of the point cloud in each direction to obtain the respiratory displacement model of the target organ.

[0012] In one or some optional implementations of the embodiment of the present application, the step of calculating the point cloud centroid of each test ultrasound image includes:

[0013] Convert pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in a base coordinate system to obtain a point cloud set of the test ultrasound image;

[0014] Convert all points in the point cloud set of the test ultrasound image from a base coordinate system to a probe coordinate system to obtain a point set of the test ultrasound image;

[0015] The point cloud centroid of the test ultrasound image is calculated based on the point set of the test ultrasound image.

[0016] In one or some optional implementations of the embodiment of the present application, the step of converting the pixel coordinates of all pixels in the test ultrasound image into three-dimensional coordinates in a base coordinate system to obtain a point cloud set of the test ultrasound image includes:

[0017] Converting pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in a probe coordinate system;

[0018] Based on the acquired hand-eye matrix, all points of the test ultrasound image in the probe coordinate system are converted to the robot end tool coordinate system;

[0019] Based on the acquired robot arm posture, the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system is obtained;

[0020] Based on the transformation matrix from the robot end tool coordinate system to the base coordinate system, all points of the test ultrasound image in the robot end tool coordinate system are transformed to the base coordinate system to obtain a point cloud set of the test ultrasound image.

[0021] In one or some optional implementations of the embodiment of the present application, converting all points in the point cloud set of the test ultrasound image from a base coordinate system to a probe coordinate system to obtain a point set set of the test ultrasound image includes:

[0022] Based on the acquired robot arm posture, a transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm is obtained;

[0023] Based on the conversion matrix from the base coordinate system to the robot end tool coordinate system, the point cloud set of the test ultrasound image is converted from the base coordinate system to the robot end tool coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the robot end tool coordinate system;

[0024] Based on the acquired inverse matrix of the hand-eye matrix, all points of the test ultrasound image in the robot end tool coordinate system are converted to the probe coordinate system to obtain a point set of the test ultrasound image.

[0025] In one or some optional implementations of the embodiments of the present application, controlling the probe to continue acquiring ultrasound images, and applying the respiratory displacement model to all the ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction, includes:

[0026] For each ultrasound image, converting pixel coordinates of all pixel points in the ultrasound image into three-dimensional coordinates in a base coordinate system;

[0027] Convert all points of the ultrasound image in the base coordinate system to the probe coordinate system;

[0028] Based on the three-dimensional coordinates of all points of the ultrasound image in the probe coordinate system, the respiratory displacement model is applied to obtain a respiratory compensation point set corresponding to the ultrasound image;

[0029] Based on the respiratory compensation point sets corresponding to all the ultrasound images, a target organ model after ultrasound three-dimensional reconstruction is obtained.

[0030] In a second aspect, an embodiment of the present invention provides a human organ ultrasonic three-dimensional reconstruction device based on respiratory displacement compensation, which may include:

[0031] The first acquisition module is used to control the probe at the end of the ultrasonic robot mechanical arm to acquire multiple test ultrasonic images within a preset test period at the target organ of the human body;

[0032] A model building module, used for converting two-dimensional pixel points into three-dimensional point clouds based on the multiple test ultrasound images, and building a respiratory displacement model of the target organ;

[0033] The three-dimensional reconstruction module is used to control the probe to continue to collect ultrasonic images, and apply the respiratory displacement model to all the ultrasonic images to obtain a target organ model after ultrasonic three-dimensional reconstruction.

[0034] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the above-mentioned method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation.

[0036] In a fifth aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the computer program, the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described above is implemented.

[0037] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0038] The embodiment of the present invention provides a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation. The method controls the probe at the end of the ultrasonic robot mechanical arm to collect multiple test ultrasonic images within a preset test period at the human target organ, and converts two-dimensional pixel points to three-dimensional point clouds based on these images to establish a respiratory displacement model of the target organ. Subsequently, the probe is controlled to continue to collect ultrasonic images, and the respiratory displacement model is applied to all images to obtain the target organ model after ultrasonic three-dimensional reconstruction. This method achieves high-precision three-dimensional reconstruction of human organs by establishing an accurate respiratory displacement model and performing respiratory displacement compensation for each frame of ultrasonic image collected, improves the accuracy and stability of three-dimensional reconstruction, effectively solves the problem of organ displacement caused by respiratory movement, and improves the applicability and reliability of ultrasonic imaging in dynamic environments. In addition, by simplifying the computational complexity and improving the compensation efficiency, the method can achieve efficient and real-time three-dimensional reconstruction in actual clinical applications, provide more reliable imaging support for doctors, and promote the development of ultrasonic robot autonomous scanning technology.

[0039] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 A schematic flow chart of a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a mechanical arm of an ultrasonic robot provided in an embodiment of the present invention;

[0044] Figure 3 A scatter plot of the centroid of the point cloud in the X-axis direction provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the curve fitting result in the Y-axis direction provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of a curve fitting result in the Z-axis direction provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of the structure of a human organ ultrasonic three-dimensional reconstruction device based on respiratory displacement compensation provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0049] The inventors found that the traditional ultrasonic scanning imaging method cannot accurately capture and correct the displacement caused by breathing due to the lack of an effective real-time breathing compensation mechanism, which seriously affects the accuracy and stability of three-dimensional reconstruction, thereby limiting the application of ultrasonic robots in complex surgeries or treatment planning. Based on this, the inventors made the present invention after further research and development, providing a method and device for ultrasonic three-dimensional reconstruction of human organs based on breathing displacement compensation.

[0050] Embodiment 1

[0051] In the first embodiment of the present invention, a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation is provided. Figure 1 As shown, the method may include the following steps S101-S103:

[0052] S101: At a target organ of a human body, controlling a probe at the end of a mechanical arm of an ultrasonic robot to collect a plurality of test ultrasonic images within a preset test period.

[0053] S102: Based on the multiple test ultrasound images, convert the two-dimensional pixel points into three-dimensional point clouds to establish a respiratory displacement model of the target organ.

[0054] S103: Control the probe to continue acquiring ultrasound images, and apply the respiratory displacement model to all ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction.

[0055] The embodiment of the present invention provides a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation. The method controls the probe at the end of the ultrasonic robot mechanical arm to collect multiple test ultrasonic images within a preset test period at the human target organ, and converts two-dimensional pixel points to three-dimensional point clouds based on these images to establish a respiratory displacement model of the target organ. Subsequently, the probe is controlled to continue to collect ultrasonic images, and the respiratory displacement model is applied to all images to obtain the target organ model after ultrasonic three-dimensional reconstruction. This method achieves high-precision three-dimensional reconstruction of human organs by establishing an accurate respiratory displacement model and performing respiratory displacement compensation for each frame of ultrasonic image collected, improves the accuracy and stability of three-dimensional reconstruction, effectively solves the problem of organ displacement caused by respiratory movement, and improves the applicability and reliability of ultrasonic imaging in dynamic environments. In addition, by simplifying the computational complexity and improving the compensation efficiency, the method can achieve efficient and real-time three-dimensional reconstruction in actual clinical applications, provide more reliable imaging support for doctors, and promote the development of ultrasonic robot autonomous scanning technology.

[0056] In the embodiment of the present application, the schematic diagram of the ultrasonic robot mechanical arm is as follows Figure 2 As shown, a probe is connected to the end of the robot arm, and the XYZ at the probe is a schematic diagram of the direction of the tool coordinate system at the end of the robot arm, and the XYZ axis of the robot arm base is a schematic diagram of the direction of the base coordinate system, which is used to indicate the three-dimensional spatial position of the probe at the end of the robot arm.

[0057] In the above step S101, at the target organ of the human body, the probe at the end of the mechanical arm of the ultrasonic robot is controlled to collect a plurality of test ultrasonic images within a preset test period.

[0058] Specifically, during the scanning process of the ultrasonic robot, when the target organ is identified, the probe can be rotated according to the displacement direction of the target organ so that the displacement direction of the target organ is parallel to the collected ultrasonic image. Then, within a preset test period, the probe at the end of the ultrasonic robot's mechanical arm is controlled to remain stationary, and multiple test ultrasonic images are collected, and the multiple test ultrasonic images are sorted according to the collection time.

[0059] In a specific embodiment, the target organ is the gallbladder. Since the approximate displacement direction of the gallbladder is known, that is, it moves up and down in the human body, during the scanning process, when the probe identifies the gallbladder, the probe can be rotated based on the prior information of the main direction of displacement to ensure that the displacement direction of the gallbladder is parallel to the ultrasound image, and then the probe is controlled to be stationary to collect 125 test ultrasound images within 5 seconds.

[0060] In the embodiment of the present application, the above step S101 reduces the error caused by dynamic changes by statically recording the displacement law of the target tissue, and provides a reliable data basis for the establishment of the subsequent respiratory displacement model. At the same time, the direction of the probe is adjusted first according to the displacement direction of the target organ, so that the ultrasonic image can more clearly record the respiratory displacement of the target organ.

[0061] In the above step S102, based on multiple test ultrasound images, two-dimensional pixel points are converted into three-dimensional point clouds to establish a respiratory displacement model of the target organ. Specifically, the following steps S1021-S1023 are included:

[0062] S1021: Calculate the point cloud centroid of each test ultrasound image. Specifically, it includes the following steps S10211-S10213:

[0063] S10211: Convert the pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in the base coordinate system to obtain a point cloud set of the test ultrasound image. Specifically, the following steps S102111-S102114 are included:

[0064] S102111: Convert the pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in the probe coordinate system.

[0065] Specifically, it can be that, since the probe is equivalent to an extension along the Z axis of the tool coordinate system at the end of the robot arm, the test ultrasound image is basically parallel to the YZ plane of the tool coordinate system at the end of the robot arm. In order to reduce the difficulty of calculation, this method pre-sets the XYZ direction of the probe coordinate system to be roughly the same as the XYZ direction of the tool coordinate system at the end of the robot arm, and the X-axis (hereinafter referred to as pt.x) and Y-axis (hereinafter referred to as pt.y) directions of the pixel coordinate system in the test ultrasound image correspond to the Y and Z directions of the probe coordinate system, and because the pixel coordinate system in the test ultrasound image generally takes the upper left corner as the origin. It can be seen that in the pixel coordinate system in the test ultrasound image, the vertical pt.y direction is consistent with the Z direction of the probe coordinate system, while the horizontal pt.x direction is opposite to the Y axis of the probe coordinate system. Therefore, the conversion from the pixel coordinate system to the probe coordinate system can be realized based on the following formula 1, and the pixel coordinates of all pixels in the test ultrasound image are converted into three-dimensional coordinates in the probe coordinate system:

[0066] p ji =[0,-pt.x ji ·μ,pt.y ji ·μ] Formula 1

[0067] In the formula, p ji Indicates the three-dimensional coordinates of the probe coordinate system corresponding to the i-th pixel point in the j-th test ultrasound image, pt.x ji and pt.yji They represent the pixel coordinates of the i-th pixel point in the j-th test ultrasound image on the pt.x-axis and pt.y-axis respectively, and μ is the actual physical distance per unit pixel.

[0068] S102112: Based on the acquired hand-eye matrix, all points of the test ultrasound image in the probe coordinate system are converted to the robot arm end tool coordinate system.

[0069] Specifically, the three-dimensional coordinates of all points of the test ultrasound image in the probe coordinate system may be multiplied by the hand-eye matrix to obtain the three-dimensional coordinates of all points of the test ultrasound image in the tool coordinate system at the end of the robotic arm.

[0070] Among them, the hand-eye matrix is ​​the transformation matrix from the probe coordinate system to the tool coordinate system at the end of the robot arm. It is a fixed matrix that can be directly obtained according to the hardware configuration of the ultrasonic robot. The hand-eye matrix is ​​an important concept in robotics and is usually used to describe the spatial relationship between the robot hand (robot end effector) and the eye (sensor, such as a camera or ultrasonic probe).

[0071] S102113: Based on the acquired robot arm posture, obtain the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system.

[0072] In the embodiment of the present application, the method of obtaining the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system according to the posture of the robot arm as described in the above step S102113 is a prior art and will not be described in detail here.

[0073] S102114: Based on the transformation matrix from the robot end tool coordinate system to the base coordinate system, all points of the test ultrasound image in the robot end tool coordinate system are transformed to the base coordinate system to obtain a point cloud set of the test ultrasound image.

[0074] Specifically, the three-dimensional coordinates of all points of the test ultrasound image in the robot end tool coordinate system are multiplied by the transformation matrix from the robot end tool coordinate system to the base coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the base coordinate system, that is, the point cloud set.

[0075] In order to facilitate those skilled in the art to understand the present solution, the method of obtaining the point cloud set of the test ultrasound image described in the above step S10211 is explained more clearly in the form of a formula: Based on the following formula 2, the pixel coordinates of all pixels in the test ultrasound image are converted to the base coordinate system to obtain the point cloud set of the test ultrasound image:

[0076]

[0077] In the formula, B Pj represents the point cloud set of the jth test ultrasound image, represents the transformation matrix from the tool coordinate system at the end of the manipulator to the base coordinate system corresponding to the jth test ultrasound image, represents the hand-eye matrix, B represents the base coordinate system, S represents the probe coordinate system, TCP represents the tool coordinate system at the end of the robot arm, and p ji Indicates the three-dimensional coordinates of the probe coordinate system corresponding to the i-th pixel point in the j-th test ultrasound image, pt.x ji and pt.y ji They represent the pixel coordinates of the i-th pixel in the j-th test ultrasound image on the pt.x-axis and pt.y-axis respectively, μ is the actual physical distance per unit pixel, T represents transposition, and n is the total number of pixels in the j-th test ultrasound image.

[0078] In the embodiment of the present application, the above step S10211 converts the test ultrasound image into the base coordinate system. This ensures that all point cloud sets are calculated under a unified base coordinate system framework, which is beneficial to subsequent data fusion and coordination and improves data consistency and accuracy.

[0079] S10212: Convert all points in the point cloud set of the test ultrasound image from the base coordinate system to the probe coordinate system to obtain a point set of the test ultrasound image. Specifically, the following steps S102121-S102123 are included:

[0080] S102121: Based on the acquired robot arm posture, obtain the transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm.

[0081] Specifically, the conversion matrix from the base coordinate system to the tool coordinate system at the end of the robot arm can be obtained based on the acquired robot arm posture. Alternatively, based on the conversion matrix from the tool coordinate system at the end of the robot arm to the base coordinate system obtained according to the robot arm posture in the above step S102113, the inverse matrix is ​​taken to obtain the conversion matrix from the base coordinate system to the tool coordinate system at the end of the robot arm.

[0082] In the embodiment of the present application, the method of obtaining the transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm according to the robot arm posture described in the above step S102121 is a prior art and will not be described in detail here.

[0083] S102122: Based on the transformation matrix from the base coordinate system to the robot end tool coordinate system, the point cloud set of the test ultrasound image is transformed from the base coordinate system to the robot end tool coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the robot end tool coordinate system.

[0084] Specifically, the point cloud set of the test ultrasound image may be multiplied by the transformation matrix from the base coordinate system to the robot end tool coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the robot end tool coordinate system.

[0085] S102123: Based on the acquired inverse matrix of the hand-eye matrix, all points of the test ultrasound image in the tool coordinate system at the end of the robotic arm are converted to the probe coordinate system to obtain a point set of the test ultrasound image.

[0086] Specifically, the three-dimensional coordinates of all points of the test ultrasound image in the tool coordinate system at the end of the robot arm are multiplied by the inverse matrix of the hand-eye matrix to obtain the three-dimensional coordinates of all points of the test ultrasound image in the probe coordinate system, that is, the point set.

[0087] In order to facilitate those skilled in the art to understand the present solution, the method of converting the point cloud set of the test ultrasound image into the probe coordinate system to obtain the point set set described in step S10212 is explained more clearly in the form of a formula: Based on the following formula 3, all pixel coordinates in the point cloud set of the test ultrasound image are converted to the probe coordinate system to obtain the point set set of the test ultrasound image:

[0088]

[0089] In the formula, S P j represents the point set of the jth test ultrasound image, B P j represents the point cloud set of the jth test ultrasound image, represents the transformation matrix from the base coordinate system corresponding to the jth test ultrasound image to the tool coordinate system at the end of the robot arm, represents the inverse matrix of the hand-eye matrix, B represents the base coordinate system, S represents the probe coordinate system, and TCP represents the tool coordinate system at the end of the robot arm.

[0090] Thus, the point set of all test ultrasound images can be obtained S P = { S P j}, where j = 1, 2, 3, ..., m, and m is the total number of test ultrasound images.

[0091] S10213: Calculate the point cloud centroid of the test ultrasound image based on the point set of the test ultrasound image.

[0092] Specifically, it can be that for a test ultrasound image, the point set is S P j , which can be expressed as S P j ={p jl}, l = 1, 2, 3, ..., K, K is the total number of all points in the point set of the test ultrasound image, where p jl Represents a set of points S P j The point in the jl =[x jl ,y jl ,z jl ,],x jl ,y jl 、z jl Respectively represent point p jl The components in the XYZ directions in the probe coordinate system.

[0093] Therefore, the point set of the test ultrasound image is S P j It can be expressed as the following formula 4:

[0094]

[0095] According to the above formula 4, the point set of the test ultrasound image is S P j , based on the following formula 5, calculate the point cloud centroid of the test ultrasound image:

[0096]

[0097] In the formula, S C j represents the point cloud centroid of the jth test ultrasound image, K is the total number of all points in the point set of the jth test ultrasound image, and p jl Represents a set of points S P j The lth point in K represents a K×1 vector, in which all elements have the value of 1, and T represents transpose.

[0098] Therefore, for all the test ultrasound images, the point set S P = { S P j}, the point cloud centroid of all test ultrasound images can be calculated S C={ S C j}, where j = 1, 2, 3, ..., m, and m is the total number of test ultrasound images.

[0099] S1022: Based on the point cloud centroids of all test ultrasound images, decompose them in three directions of XYZ to obtain a scatter plot of the point cloud centroids in the three directions of XYZ.

[0100] S1023: Perform curve fitting on the point cloud centroid scatter plot in each direction to obtain a respiratory displacement model of the target organ.

[0101] In order to facilitate those skilled in the art to understand the present solution, the specific implementation process of the method provided in the above steps S1022-S1023 is illustrated below by way of example: When the target organ is the gallbladder, the result of curve fitting is as follows based on the centroid scatter diagram of the point cloud in the three directions of XYZ in the probe coordinate system: Figure 3-5 shown.

[0102] in, Figure 3 It is a scatter plot of the point cloud centroid in the X-axis direction. The X-axis direction is the direction perpendicular to the detection plane. The value remains basically unchanged and no curve fitting is required.

[0103] Figure 4 is the curve fitting result in the Y-axis direction, the blue points represent the point cloud centroid scatter plot, and the red curve is the curve fitting result. The blue points in the figure basically meet the characteristics of the sine curve, so the respiratory displacement model in the Y-axis direction can be fitted, as shown in the following formula 6:

[0104] y=A y sin(w y t y +b y )+C y Formula 6

[0105] Where, t y Indicates the time difference from the start of scanning to the current time, A y 、w y 、b y , C y represents a parameter in a sine function. The time when the scanning starts is the time when the probe starts to collect the test ultrasound image in the above step S101.

[0106] In a specific embodiment, the respiratory displacement model in the Y-axis direction is shown in Formula 7 below:

[0107] y=0.0094sin(0.0438t y +0.9901)+0.0113 Formula 7

[0108] Figure 5 is the curve fitting result in the Z direction, the blue points represent the point cloud centroid scatter plot, and the red curve is the curve fitting result. The blue points in the figure basically meet the characteristics of the sine curve, so the respiratory displacement model in the Z direction can be fitted, as shown in the following formula 8:

[0109] z=A z sin(w z t z +bz )+C z Formula 8

[0110] Where, t z Indicates the time difference from the start of scanning to the current time, A z 、w z 、b z , C z represents a parameter in a sine function. The time when the scanning starts is the time when the probe starts to collect the test ultrasound image in the above step S101.

[0111] In a specific embodiment, the respiratory displacement model in the Z direction is shown in the following formula 8:

[0112] z=-0.0024sin(0.0438t z +1.0151)+0.0024 Formula 8

[0113] In an embodiment of the present application, the process of establishing a respiratory displacement model described in the above steps S101-S102 accurately records the displacement of the target organ in the ultrasound image, thereby establishing a more accurate respiratory displacement model. This not only ensures the consistency and comparability of the data and improves the accuracy of subsequent processing, but also reduces errors caused by dynamic changes, and provides a reliable basis for subsequent real-time compensation. Specifically, this method further enhances the accuracy and robustness of the model by converting two-dimensional pixel points into three-dimensional point clouds, and based on this, establishes a respiratory displacement model of the target organ in the three directions of XYZ, which can better reflect the true motion characteristics of the target organ, thereby improving the accuracy and stability of three-dimensional reconstruction in practical applications, and providing doctors with more reliable imaging support.

[0114] In the above step S103, the probe is controlled to continue to collect ultrasound images, and the respiratory displacement model is applied to all ultrasound images to obtain a target organ model after ultrasound 3D reconstruction. Specifically, the following steps S1031-S1034 are included:

[0115] S1031: For each ultrasound image, convert the pixel coordinates of all pixel points in the ultrasound image into three-dimensional coordinates in a base coordinate system.

[0116] Specifically, the pixel coordinates of all points in the ultrasound image may be converted into three-dimensional coordinates in the probe coordinate system based on the following formula 9:

[0117] p i =[0, -μ×pt.x i , μ×pt.y i ] Formula 9

[0118] Where p represents the three-dimensional coordinate of the probe coordinate system corresponding to the i-th pixel point in the ultrasound image, pt.x and pt.y represent the pixel coordinates of the i-th pixel point in the ultrasound image on the pt.x axis and pt.y axis, respectively, and μ is the actual physical distance per unit pixel.

[0119] Based on the acquired hand-eye matrix, the three-dimensional coordinates of all points in the ultrasound image in the probe coordinate system are converted to the tool coordinate system at the end of the robotic arm.

[0120] Based on the acquired robot arm posture, the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system is obtained.

[0121] Based on the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system, all points of the ultrasound image in the tool coordinate system at the end of the robot arm are transformed to the base coordinate system.

[0122] In order to facilitate those skilled in the art to understand the present solution, the process described in step S1031 is explained more clearly in the form of a formula below: Based on the following formula 10, all pixel coordinates in the ultrasound image are converted into three-dimensional coordinates in the base coordinate system:

[0123]

[0124] Where P represents the set of three-dimensional coordinates of all points in the ultrasound image in the base coordinate system. represents the transformation matrix from the tool coordinate system at the end of the robot to the base coordinate system corresponding to the i-th pixel point in the ultrasound image, represents the hand-eye matrix, B represents the base coordinate system, S represents the probe coordinate system, TCP represents the tool coordinate system at the end of the robot arm, and p i represents the three-dimensional coordinates of the probe coordinate system corresponding to the i-th pixel point in the ultrasound image, T represents transposition, and n is the total number of pixels in the ultrasound image.

[0125] The specific implementation of the above step S1031 can refer to the above step S10211, which will not be elaborated here.

[0126] S1032: Convert all points of the ultrasound image in the base coordinate system into the probe coordinate system.

[0127] Specifically, based on the acquired robot arm posture, a transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm is obtained.

[0128] Based on the transformation matrix from the base coordinate system to the robot end tool coordinate system, the three-dimensional coordinates of all points in the ultrasound image in the base coordinate system are transformed to the robot end tool coordinate system to obtain the three-dimensional coordinates of all points in the ultrasound image in the robot end tool coordinate system.

[0129] Based on the inverse matrix of the acquired hand-eye matrix, all points of the ultrasound image in the tool coordinate system at the end of the robotic arm are transformed into the probe coordinate system to obtain a point set of the test ultrasound image.

[0130] The specific implementation of the above step S1032 can refer to the above step S10212, which will not be described in detail here.

[0131] S1033: Based on the three-dimensional coordinates of all points of the ultrasound image in the probe coordinate system, a respiratory displacement model is applied to obtain a respiratory compensation point set corresponding to the ultrasound image.

[0132] Specifically, the three-dimensional coordinates of all points of the ultrasound image in the probe coordinate system include components in three directions: X, Y, and Z. For each point of the ultrasound image in the probe coordinate system, the respiratory displacement model in the X-axis direction is applied to the component in the X-axis direction, the respiratory displacement model in the Y-axis direction is applied to the component in the Y-axis direction, and the respiratory displacement model in the Z-axis direction is applied to the component in the Z-axis direction, so as to obtain a set of respiratory compensation points corresponding to the ultrasound image.

[0133] Meanwhile, if there is no respiratory displacement model in certain axial directions, or the displacement caused by breathing is less than a preset threshold, the displacement change caused by breathing can be ignored, and the respiratory displacement model is not applied to the component in the axial direction.

[0134] In order to facilitate those skilled in the art to understand the present solution, the specific implementation process of step S1033 is explained more clearly below based on the examples given in the above steps S1022-S1023: Figure 3-5 From the curve fitting results shown, it can be seen that in the X-axis and Z-axis directions, the displacement caused by breathing is less than the preset threshold, and the X-direction cannot be fitted to obtain the respiratory displacement model. This is because the main displacement direction of the gallbladder is up and down movement in the human abdominal cavity, which is approximately equivalent to the Y direction of the probe. Therefore, only the Y-axis direction can be compensated for breathing. Therefore, based on the following formula 11, the component of the ultrasound image in the Y-axis direction in the probe coordinate system is applied to the respiratory displacement model in the Y-axis direction to obtain the component in the Y-axis direction after respiratory compensation:

[0135] Y i =Y i -{[A y sin(b y )+C y ]-[A y sin(w y t+b y )+C y ]} Formula 11

[0136] Where Y i represents the component of the i-th point in the Y-axis direction after breathing compensation, Y irepresents the component of the i-th point of the ultrasound image in the probe coordinate system in the Y-axis direction, t y Indicates the time difference between the start of scanning and the acquisition of ultrasound images, A y 、w y 、b y , C y Represents the parameter in the respiratory displacement model in the Y-axis direction. The time when the scan starts is the time when the probe starts to collect the test ultrasound image in the above step S101.

[0137] The components in the Y-axis direction after respiration compensation are used to replace the components in the Y-axis direction of the points in the ultrasound image, and the components in the X-axis and Z-axis directions remain unchanged, thereby obtaining a respiration compensation point set corresponding to the ultrasound image.

[0138] S1034: Based on the respiratory compensation point set corresponding to all ultrasound images, a target organ model after ultrasound three-dimensional reconstruction is obtained.

[0139] Specifically, the above steps S1031-S1033 may be performed on all ultrasound images to obtain the respiratory compensation point sets corresponding to all ultrasound images, and all respiratory compensation point sets may be integrated to obtain a complete and accurate three-dimensional target organ model to complete the ultrasonic three-dimensional reconstruction.

[0140] In the embodiment of the present application, in the scanning process, the above step S103 converts all pixel points in each frame of ultrasound image from two-dimensional coordinates to three-dimensional coordinates through hand-eye registration (Hand-Eye Registration or Hand-Eye Calibration) to align to the robot base coordinate system, thereby realizing three-dimensional reconstruction of the gallbladder without breathing compensation. Then, the reconstruction result without breathing compensation is converted to the ultrasound probe coordinate system, so as to apply the breathing displacement model in the ultrasound probe coordinate system to complete the breathing displacement compensation. Compared with directly performing complex displacement calculations in the original ultrasound image, this method effectively simplifies the entire compensation process, simplifies the calculation complexity, improves the compensation efficiency and accuracy, and finally realizes high-precision three-dimensional reconstruction of the gallbladder.

[0141] Embodiment 2

[0142] Based on the same inventive concept, the embodiment of the present invention also provides a human organ ultrasonic three-dimensional reconstruction device based on respiratory displacement compensation, referring to Figure 6 As shown, the device comprises:

[0143] The first acquisition module 101 is used to control the probe at the end of the ultrasonic robot mechanical arm to acquire multiple test ultrasonic images within a preset test period at the target organ of the human body;

[0144] A model building module 102, for converting two-dimensional pixel points into three-dimensional point clouds based on the plurality of test ultrasound images, and building a respiratory displacement model of the target organ;

[0145] The three-dimensional reconstruction module 103 is used to control the probe to continue to collect ultrasound images, and apply the respiratory displacement model to all the ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction.

[0146] Embodiment 3

[0147] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described in the above-mentioned embodiment 1 is implemented.

[0148] Embodiment 4

[0149] Based on the same inventive concept, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described in the above-mentioned embodiment 1.

[0150] Embodiment 5

[0151] Based on the same inventive concept, an embodiment of the present invention also provides a computer device, including a memory, a processor and a computer program stored in the memory. When the processor executes the computer program, it implements the ultrasonic three-dimensional reconstruction method of human organs based on respiratory displacement compensation as described in the above embodiment 1.

[0152] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0153] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0156] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation, characterized in that: include: At the target organ of the human body, controlling the probe at the end of the ultrasonic robot mechanical arm to collect multiple test ultrasonic images within a preset test period; Based on the multiple test ultrasound images, convert two-dimensional pixel points into three-dimensional point clouds to establish a respiratory displacement model of the target organ; The probe is controlled to continue acquiring ultrasonic images, and the respiratory displacement model is applied to all the ultrasonic images to obtain a target organ model after ultrasonic three-dimensional reconstruction.

2. The method according to claim 1, characterized in that The step of converting two-dimensional pixel points into three-dimensional point clouds based on the plurality of test ultrasound images to establish a respiratory displacement model of the target organ includes: Calculate the point cloud centroid of each test ultrasound image; Based on the point cloud centroids of all the test ultrasound images, decomposition is performed in three directions of XYZ to obtain a scatter plot of the point cloud centroids in the three directions of XYZ; Curve fitting is performed on the centroid scatter plot of the point cloud in each direction to obtain the respiratory displacement model of the target organ.

3. The method according to claim 2, characterized in that The calculation to obtain the point cloud centroid of each test ultrasound image includes: Convert pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in a base coordinate system to obtain a point cloud set of the test ultrasound image; Convert all points in the point cloud set of the test ultrasound image from a base coordinate system to a probe coordinate system to obtain a point set of the test ultrasound image; The point cloud centroid of the test ultrasound image is calculated based on the point set of the test ultrasound image.

4. The method according to claim 3, characterized in that The step of converting the pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in a base coordinate system to obtain a point cloud set of the test ultrasound image includes: Converting pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in a probe coordinate system; Based on the acquired hand-eye matrix, all points of the test ultrasound image in the probe coordinate system are converted to the robot end tool coordinate system; Based on the acquired robot arm posture, the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system is obtained; Based on the transformation matrix from the robot end tool coordinate system to the base coordinate system, all points of the test ultrasound image in the robot end tool coordinate system are transformed to the base coordinate system to obtain a point cloud set of the test ultrasound image.

5. The method according to claim 3, characterized in that: The step of converting all points in the point cloud set of the test ultrasound image from a base coordinate system to a probe coordinate system to obtain a point set of the test ultrasound image includes: Based on the acquired robot arm posture, a transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm is obtained; Based on the conversion matrix from the base coordinate system to the robot end tool coordinate system, the point cloud set of the test ultrasound image is converted from the base coordinate system to the robot end tool coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the robot end tool coordinate system; Based on the acquired inverse matrix of the hand-eye matrix, all points of the test ultrasound image in the robot end tool coordinate system are converted to the probe coordinate system to obtain a point set of the test ultrasound image.

6. The method according to claim 1, characterized in that The step of controlling the probe to continue acquiring ultrasound images and applying the respiratory displacement model to all the ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction includes: For each ultrasound image, converting pixel coordinates of all pixel points in the ultrasound image into three-dimensional coordinates in a base coordinate system; Convert all points of the ultrasound image in the base coordinate system to the probe coordinate system; Based on the three-dimensional coordinates of all points of the ultrasound image in the probe coordinate system, the respiratory displacement model is applied to obtain a respiratory compensation point set corresponding to the ultrasound image; Based on the respiratory compensation point sets corresponding to all the ultrasound images, a target organ model after ultrasound three-dimensional reconstruction is obtained.

7. A human organ ultrasonic three-dimensional reconstruction device based on respiratory displacement compensation, characterized in that: include: The first acquisition module is used to control the probe at the end of the ultrasonic robot mechanical arm to acquire multiple test ultrasonic images within a preset test period at the target organ of the human body; A model building module, used for converting two-dimensional pixel points into three-dimensional point clouds based on the multiple test ultrasound images, and building a respiratory displacement model of the target organ; The three-dimensional reconstruction module is used to control the probe to continue to collect ultrasonic images, and apply the respiratory displacement model to all the ultrasonic images to obtain a target organ model after ultrasonic three-dimensional reconstruction.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described in any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described in any one of claims 1 to 6 is implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described in any one of claims 1-6.

Images