Ultrasonic three-dimensional reconstruction method and device for human organs based on respiratory displacement compensation

Abstract

The present invention discloses a method and device for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation. The method comprises: controlling a probe at the end of an ultrasonic robot's mechanical arm to capture multiple test ultrasound images at a target organ within a preset test period; converting two-dimensional pixel points into three-dimensional point clouds based on the multiple test ultrasound images to establish a respiratory displacement model of the target organ; controlling the probe to continue capturing ultrasound images, and applying the respiratory displacement model to all of the ultrasound images to obtain an ultrasonically reconstructed three-dimensional target organ model. This method improves the accuracy and stability of three-dimensional reconstruction, effectively solves the problem of target organ displacement caused by respiratory motion, enhances the applicability and reliability of ultrasound imaging in dynamic environments, and can obtain a more complete and accurate three-dimensional target organ model.

Description

Technical Field

[0001] The present invention relates to a method and device for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation. Background Art

[0002] With the development of medical technology, ultrasound scanning and imaging has been widely used in clinical diagnosis due to its non-invasive, real-time, and cost-effective characteristics, especially in the examination of human organs. In order to achieve autonomous ultrasonic robotic scanning of human organs, it is often necessary to reconstruct the three-dimensional morphology of the organ in real time during the scanning process, thereby guiding the robot to complete the search for the optimal section of the organ. However, human breathing inevitably causes unknown organ displacement, which poses a great challenge to its real-time three-dimensional reconstruction during the scanning process. Taking the most typical gallbladder organ as an example, human breathing can cause it to produce displacement fluctuations of 2-5cm. Traditional ultrasound imaging methods are difficult to cope with the problem of organ displacement caused by breathing. Therefore, how to achieve three-dimensional reconstruction of human organs through real-time compensation for respiratory displacement is a key step in realizing autonomous ultrasonic robotic scanning of human organs. Summary of the Invention

[0003] In order to achieve more accurate three-dimensional reconstruction of human organs, an embodiment of the present invention provides a method and apparatus for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation.

[0004] In a first aspect, an embodiment of the present invention provides a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation, which may include:

[0005] At the target organ of the human body, the probe at the end of the ultrasonic robot's mechanical arm is controlled to collect multiple test ultrasound images within a preset test period;

[0006] Based on the multiple test ultrasound images, converting two-dimensional pixel points into three-dimensional point clouds to establish a respiratory displacement model of the target organ;

[0007] The probe is controlled to continue acquiring ultrasound images, and the respiratory displacement model is applied to all the ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction.

[0008] In one or some optional implementations of the embodiment of the present application, performing conversion of two-dimensional pixels into three-dimensional point clouds based on the multiple test ultrasound images to establish the respiratory displacement model of the target organ includes:

[0009] Calculate the point cloud centroid of each test ultrasound image;

[0010] Based on the point cloud centroids of all the test ultrasound images, decomposition is performed in the XYZ directions to obtain a scatter plot of the point cloud centroids in the XYZ directions;

[0011] Curve fitting is performed on the point cloud centroid scatter plot in each direction to obtain a respiratory displacement model of the target organ.

[0012] In one or some optional implementations of the embodiment of the present application, the calculating and obtaining the point cloud centroid of each test ultrasound image includes:

[0013] Converting pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in a base coordinate system to obtain a point cloud set of the test ultrasound image;

[0014] Convert all points in the point cloud set of the test ultrasound image from a base coordinate system to a probe coordinate system to obtain a point set set of the test ultrasound image;

[0015] The point cloud centroid of the test ultrasound image is obtained by calculation based on the point set of the test ultrasound image.

[0016] In one or some optional implementations of the embodiment of the present application, converting the pixel coordinates of all pixels in the test ultrasound image into three-dimensional coordinates in a base coordinate system to obtain a point cloud set of the test ultrasound image includes:

[0017] Converting the pixel coordinates of all pixels in the test ultrasound image into three-dimensional coordinates in a probe coordinate system;

[0018] Based on the acquired hand-eye matrix, all points of the test ultrasound image in the probe coordinate system are converted to the robot arm end tool coordinate system;

[0019] Based on the acquired robot arm posture, the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system is obtained;

[0020] Based on the transformation matrix from the manipulator end tool coordinate system to the base coordinate system, all points of the test ultrasound image in the manipulator end tool coordinate system are transformed to the base coordinate system to obtain a point cloud set of the test ultrasound image.

[0021] In one or some optional implementations of the embodiment of the present application, converting all points in the point cloud set of the test ultrasound image from a base coordinate system to a probe coordinate system to obtain a point set set of the test ultrasound image includes:

[0022] Based on the acquired robot arm pose, the transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm is obtained;

[0023] Based on the conversion matrix from the base coordinate system to the robot end tool coordinate system, convert the point cloud set of the test ultrasound image from the base coordinate system to the robot end tool coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the robot end tool coordinate system;

[0024] Based on the acquired inverse matrix of the hand-eye matrix, all points of the test ultrasound image in the robot arm end tool coordinate system are converted to the probe coordinate system to obtain a point set of the test ultrasound image.

[0025] In one or some optional implementations of the embodiment of the present application, controlling the probe to continue acquiring ultrasound images and applying the respiratory displacement model to all of the ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction includes:

[0026] For each ultrasound image, converting pixel coordinates of all pixel points in the ultrasound image into three-dimensional coordinates in a base coordinate system;

[0027] Converting all points of the ultrasound image in the base coordinate system into the probe coordinate system;

[0028] Based on the three-dimensional coordinates of all points of the ultrasound image in the probe coordinate system, applying the respiratory displacement model to obtain a respiratory compensation point set corresponding to the ultrasound image;

[0029] Based on the respiratory compensation point sets corresponding to all the ultrasound images, a target organ model after ultrasound three-dimensional reconstruction is obtained.

[0030] In a second aspect, an embodiment of the present invention provides a human organ ultrasonic three-dimensional reconstruction device based on respiratory displacement compensation, which may include:

[0031] The first acquisition module is used to control the probe at the end of the ultrasonic robot's mechanical arm to acquire multiple test ultrasound images within a preset test period at the target organ of the human body;

[0032] a model building module, configured to convert two-dimensional pixel points into three-dimensional point clouds based on the plurality of test ultrasound images, and to build a respiratory displacement model of the target organ;

[0033] The three-dimensional reconstruction module is used to control the probe to continue to collect ultrasonic images, and apply the respiratory displacement model to all the ultrasonic images to obtain a target organ model after ultrasonic three-dimensional reconstruction.

[0034] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the above-mentioned method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the above-mentioned method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation.

[0036] In a fifth aspect, an embodiment of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the computer program, the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described above is implemented.

[0037] The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least:

[0038] An embodiment of the present invention provides a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation. The method controls the probe at the end of the ultrasonic robot's mechanical arm at the target organ of the human body to collect multiple test ultrasound images within a preset test period, and converts two-dimensional pixel points to three-dimensional point clouds based on these images to establish a respiratory displacement model of the target organ. Subsequently, the probe is controlled to continue collecting ultrasound images, and the respiratory displacement model is applied to all images to obtain a target organ model after ultrasonic three-dimensional reconstruction. This method achieves high-precision three-dimensional reconstruction of human organs by establishing an accurate respiratory displacement model and performing respiratory displacement compensation on each frame of collected ultrasound image, thereby improving the accuracy and stability of three-dimensional reconstruction, effectively solving the problem of organ displacement caused by respiratory movement, and enhancing the applicability and reliability of ultrasound imaging in dynamic environments. In addition, by simplifying computational complexity and improving compensation efficiency, the method can achieve efficient and real-time three-dimensional reconstruction in actual clinical applications, providing doctors with more reliable imaging support and promoting the development of ultrasonic robot autonomous scanning technology.

[0039] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 A schematic flow chart of a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a mechanical arm of an ultrasonic robot provided in an embodiment of the present invention;

[0044] Figure 3 A scatter plot of the point cloud centroid in the X-axis direction provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the curve fitting results in the Y-axis direction provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the curve fitting results in the Z-axis direction provided by an embodiment of the present invention;

[0047] Figure 6 This is a schematic structural diagram of a device for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0049] The inventors discovered that traditional ultrasound scanning imaging methods, lacking an effective real-time respiratory compensation mechanism, are unable to accurately capture and correct for respiratory displacement, severely impacting the accuracy and stability of 3D reconstruction. This, in turn, limits the application of ultrasound robots in complex surgeries or treatment planning. Based on this, the inventors conducted further research and developed the present invention, which provides a method and apparatus for ultrasonic 3D reconstruction of human organs based on respiratory displacement compensation.

[0050] Example 1

[0051] In the first embodiment of the present invention, a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation is provided. Figure 1 As shown, the method may include the following steps S101-S103:

[0052] S101: At a target organ in a human body, controlling a probe at the end of an ultrasonic robot arm to collect a plurality of test ultrasonic images within a preset test period.

[0053] S102: Based on the multiple test ultrasound images, convert the two-dimensional pixel points into a three-dimensional point cloud to establish a respiratory displacement model of the target organ.

[0054] S103: Control the probe to continue acquiring ultrasound images, and apply the respiratory displacement model to all ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction.

[0055] An embodiment of the present invention provides a method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation. The method controls the probe at the end of the ultrasonic robot's mechanical arm at the target organ of the human body to collect multiple test ultrasound images within a preset test period, and converts two-dimensional pixel points to three-dimensional point clouds based on these images to establish a respiratory displacement model of the target organ. Subsequently, the probe is controlled to continue collecting ultrasound images, and the respiratory displacement model is applied to all images to obtain a target organ model after ultrasonic three-dimensional reconstruction. This method achieves high-precision three-dimensional reconstruction of human organs by establishing an accurate respiratory displacement model and performing respiratory displacement compensation on each frame of collected ultrasound image, thereby improving the accuracy and stability of three-dimensional reconstruction, effectively solving the problem of organ displacement caused by respiratory movement, and enhancing the applicability and reliability of ultrasound imaging in dynamic environments. In addition, by simplifying computational complexity and improving compensation efficiency, the method can achieve efficient and real-time three-dimensional reconstruction in actual clinical applications, providing doctors with more reliable imaging support and promoting the development of ultrasonic robot autonomous scanning technology.

[0056] In the embodiment of the present application, the schematic diagram of the ultrasonic robot mechanical arm is as follows Figure 2 As shown, a probe is connected to the end of the robotic arm. The XYZ at the probe is a schematic diagram of the direction of the tool coordinate system at the end of the robotic arm, and the XYZ axis of the robotic arm base is a schematic diagram of the direction of the base coordinate system, which is used to indicate the three-dimensional spatial position of the probe at the end of the robotic arm.

[0057] In the above step S101 , at the target organ of the human body, the probe at the end of the ultrasonic robot's mechanical arm is controlled to collect a plurality of test ultrasonic images within a preset test period.

[0058] Specifically, during the scanning process of the ultrasonic robot, when the target organ is identified, the probe can be rotated according to the displacement direction of the target organ so that the displacement direction of the target organ is parallel to the collected ultrasonic image. Then, within a preset test period, the probe at the end of the ultrasonic robot's mechanical arm is controlled to remain stationary, and multiple test ultrasonic images are collected, and the multiple test ultrasonic images are sorted according to the collection time.

[0059] In a specific embodiment, the target organ is the gallbladder. Since the approximate displacement direction of the gallbladder is known, that is, it moves up and down within the human body, during the scanning process, when the probe identifies the gallbladder, it can rotate the probe based on the prior information of the main displacement direction to ensure that the displacement direction of the gallbladder is parallel to the ultrasound image. The probe is then controlled to be stationary and 125 test ultrasound images are collected within 5 seconds.

[0060] In the embodiment of the present application, step S101 above, by statically recording the displacement patterns of the target tissue, reduces errors caused by dynamic changes and provides a reliable data foundation for the subsequent establishment of a respiratory displacement model. Furthermore, by preemptively adjusting the probe orientation based on the displacement direction of the target organ, the ultrasound image can more clearly record the respiratory displacement of the target organ.

[0061] In the above step S102, based on multiple test ultrasound images, two-dimensional pixel points are converted into three-dimensional point clouds to establish a respiratory displacement model of the target organ. Specifically, the following steps S1021-S1023 are included:

[0062] S1021: Calculate the point cloud centroid of each test ultrasound image. Specifically, it includes the following steps S10211-S10213:

[0063] S10211: Convert the pixel coordinates of all pixels in the test ultrasound image into three-dimensional coordinates in the base coordinate system to obtain a point cloud set of the test ultrasound image. Specifically, the steps S102111-S102114 are as follows:

[0064] S102111: Convert the pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in the probe coordinate system.

[0065] Specifically, it can be that, since the probe is equivalent to an extension along the Z axis of the tool coordinate system at the end of the manipulator, the test ultrasound image is basically parallel to the YZ plane of the tool coordinate system at the end of the manipulator. In order to reduce the difficulty of calculation, this method pre-sets the XYZ direction of the probe coordinate system to be roughly the same as the XYZ direction of the tool coordinate system at the end of the manipulator, and the X axis (hereinafter referred to as pt.x) and Y axis (hereinafter referred to as pt.y) directions of the pixel coordinate system in the test ultrasound image correspond to the Y and Z directions of the probe coordinate system, and because the pixel coordinate system in the test ultrasound image generally takes the upper left corner as the origin. It can be seen that in the pixel coordinate system in the test ultrasound image, the vertical pt.y direction is consistent with the Z direction of the probe coordinate system, while the horizontal pt.x direction is opposite to the Y axis of the probe coordinate system. Therefore, the conversion from the pixel coordinate system to the probe coordinate system can be realized based on the following formula 1, and the pixel coordinates of all pixel points in the test ultrasound image are converted into three-dimensional coordinates under the probe coordinate system:

[0066] p ji =[0,-pt.x ji ·μ,pt.y ji ·μ] Formula 1

[0067] Where p ji Indicates the three-dimensional coordinates of the probe coordinate system corresponding to the i-th pixel in the j-th test ultrasound image, pt.x ji and pt.yji They represent the pixel coordinates of the i-th pixel point in the j-th test ultrasound image on the pt.x-axis and pt.y-axis, respectively, and μ is the actual physical distance per unit pixel.

[0068] S102112: Based on the acquired hand-eye matrix, all points of the test ultrasound image in the probe coordinate system are converted to the end-of-arm tool coordinate system.

[0069] Specifically, the three-dimensional coordinates of all points of the test ultrasound image in the probe coordinate system may be multiplied by the hand-eye matrix to obtain the three-dimensional coordinates of all points of the test ultrasound image in the tool coordinate system at the end of the robotic arm.

[0070] The hand-eye matrix (HEM) is the transformation matrix from the probe coordinate system to the tool coordinate system at the end of the robotic arm. It is a fixed matrix that can be directly obtained based on the ultrasonic robot's hardware configuration. The HEM is a key concept in robotics, typically used to describe the spatial relationship between a robot's hand (the end effector of the robotic arm) and its eye (a sensor, such as a camera or ultrasound probe).

[0071] S102113: Based on the acquired robot arm posture, obtain the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system.

[0072] In the embodiment of the present application, the method of obtaining the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system according to the posture of the robot arm as described in the above step S102113 is an existing technology and will not be described in detail here.

[0073] S102114: Based on the transformation matrix from the robot end tool coordinate system to the base coordinate system, all points of the test ultrasound image in the robot end tool coordinate system are transformed into the base coordinate system to obtain a point cloud set of the test ultrasound image.

[0074] Specifically, the three-dimensional coordinates of all points in the test ultrasound image in the robot end tool coordinate system can be multiplied by the transformation matrix from the robot end tool coordinate system to the base coordinate system to obtain the three-dimensional coordinates of all points in the test ultrasound image in the base coordinate system, that is, the point cloud set.

[0075] To facilitate understanding of this solution by those skilled in the art, the method for obtaining the point cloud set of the test ultrasound image described in step S10211 is explained more clearly in the form of a formula below: Based on the following formula 2, the pixel coordinates of all pixels in the test ultrasound image are converted to the base coordinate system to obtain the point cloud set of the test ultrasound image:

[0076]

[0077] Where, B Pj represents the point cloud set of the jth test ultrasound image, represents the transformation matrix from the tool coordinate system at the end of the manipulator to the base coordinate system corresponding to the jth test ultrasound image, represents the hand-eye matrix, B represents the base coordinate system, S represents the probe coordinate system, TCP represents the tool coordinate system at the end of the robot arm, and p ji Indicates the three-dimensional coordinates of the probe coordinate system corresponding to the i-th pixel in the j-th test ultrasound image, pt.x ji and pt.y ji They represent the pixel coordinates of the i-th pixel in the j-th test ultrasound image on the pt.x-axis and pt.y-axis, respectively. μ is the actual physical distance per unit pixel. T represents transposition. n is the total number of pixels in the j-th test ultrasound image.

[0078] In an embodiment of the present application, the above-mentioned step S10211 converts the test ultrasound image into a base coordinate system. This ensures that all point cloud sets are calculated under a unified base coordinate system framework, which is beneficial to subsequent data fusion and coordination, and improves the consistency and accuracy of the data.

[0079] S10212: Convert all points in the point cloud set of the test ultrasound image from the base coordinate system to the probe coordinate system to obtain a point set of the test ultrasound image. Specifically, the process includes the following steps S102121-S102123:

[0080] S102121: Based on the acquired robot arm posture, obtain the transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm.

[0081] Specifically, the conversion matrix from the base coordinate system to the end-of-arm tool coordinate system may be obtained based on the acquired robot arm posture. Alternatively, the conversion matrix from the base coordinate system to the end-of-arm tool coordinate system may be obtained based on the obtained conversion matrix from the robot arm posture in step S102113, and the inverse matrix may be taken to obtain the conversion matrix from the base coordinate system to the end-of-arm tool coordinate system.

[0082] In the embodiment of the present application, the method of obtaining the transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm according to the posture of the robot arm as described in the above step S102121 is an existing technology and will not be described in detail here.

[0083] S102122: Based on the transformation matrix from the base coordinate system to the end-of-arm tool coordinate system, the point cloud set of the test ultrasound image is transformed from the base coordinate system to the end-of-arm tool coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the end-of-arm tool coordinate system.

[0084] Specifically, the point cloud set of the test ultrasound image may be multiplied by the transformation matrix from the base coordinate system to the robot end tool coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the robot end tool coordinate system.

[0085] S102123: Based on the acquired inverse matrix of the hand-eye matrix, all points of the test ultrasound image in the tool coordinate system at the end of the robotic arm are converted to the probe coordinate system to obtain a point set of the test ultrasound image.

[0086] Specifically, the three-dimensional coordinates of all points in the test ultrasound image in the tool coordinate system at the end of the robotic arm are multiplied by the inverse matrix of the hand-eye matrix to obtain the three-dimensional coordinates of all points in the test ultrasound image in the probe coordinate system, that is, the point set.

[0087] To facilitate those skilled in the art to understand this solution, the method of converting the point cloud set of the test ultrasound image into the probe coordinate system to obtain the point set set described in step S10212 is explained more clearly in the form of a formula: Based on the following formula 3, all pixel coordinates in the point cloud set of the test ultrasound image are converted into the probe coordinate system to obtain the point set set of the test ultrasound image:

[0088]

[0089] Where, S P j represents the point set of the j-th test ultrasound image, B P j represents the point cloud set of the jth test ultrasound image, represents the transformation matrix from the base coordinate system corresponding to the jth test ultrasound image to the tool coordinate system at the end of the manipulator, represents the inverse matrix of the hand-eye matrix, B represents the base coordinate system, S represents the probe coordinate system, and TCP represents the tool coordinate system at the end of the robot arm.

[0090] Thus, the point set of all test ultrasound images can be obtained S P={ S P j}, where j = 1, 2, 3, ..., m, and m is the total number of test ultrasound images.

[0091] S10213: Calculate the point cloud centroid of the test ultrasound image based on the point set of the test ultrasound image.

[0092] Specifically, it can be, for a test ultrasound image point set set S P j , which can be expressed as S P j ={p jl}, l = 1, 2, 3, ..., K, K is the total number of all points in the point set of the test ultrasound image, where p jl Represents a set of points S P j The point in the jl =[x jl ,y jl ,z jl ,],x jl 、y jl 、z jl Represents point p jl The components in the XYZ directions in the probe coordinate system.

[0093] Therefore, the point set of the test ultrasound image is S P j It can be expressed as the following formula 4:

[0094]

[0095] According to the above formula 4, the point set of the test ultrasound image is S P j , based on the following formula 5, calculate the point cloud centroid of the test ultrasound image:

[0096]

[0097] Where, S C j represents the point cloud centroid of the j-th test ultrasound image, K is the total number of all points in the point set of the j-th test ultrasound image, and p jl Represents a set of points S P j The lth point in , 1 K represents a K×1 vector in which all elements have the value 1, and T represents transpose.

[0098] Therefore, for all the test ultrasound image point sets S P={ S P j}, the point cloud centroid of all test ultrasound images can be calculated S C={ S C j}, where j = 1, 2, 3, ..., m, and m is the total number of test ultrasound images.

[0099] S1022: Decompose the point cloud centroids of all test ultrasound images in the X, Y, and Z directions to obtain a scatter plot of the point cloud centroids in the X, Y, and Z directions.

[0100] S1023: Curve fitting is performed on the point cloud centroid scatter plot in each direction to obtain a respiratory displacement model of the target organ.

[0101] In order to facilitate those skilled in the art to understand this solution, the specific implementation process of the method provided in steps S1022-S1023 is illustrated below: When the target organ is the gallbladder, the curve fitting result is as follows based on the point cloud centroid scatter plot in the XYZ directions in the probe coordinate system: Figure 3-5 shown.

[0102] in, Figure 3 It is the point cloud centroid scatter plot in the X-axis direction. The X-axis direction is the direction perpendicular to the detection plane. The value remains basically unchanged and no curve fitting is required.

[0103] Figure 4 The blue points represent the centroid scatter plot of the point cloud, and the red curve is the curve fitting result. The blue points in the figure basically meet the characteristics of the sine curve, so the respiratory displacement model in the Y-axis direction can be fitted, as shown in the following formula 6:

[0104] y=A y sin(w y t y +b y )+C y Formula 6

[0105] Where, t y Indicates the time difference from the start of scanning to the current moment, A y 、w y 、b y 、C y The scanning start time is the time when the probe starts to acquire the test ultrasound image in the above step S101.

[0106] In a specific embodiment, the respiratory displacement model in the Y-axis direction is shown in the following formula 7:

[0107] y=0.0094sin(0.0438t y +0.9901)+0.0113 Formula 7

[0108] Figure 5 The blue points represent the centroid scatter plot of the point cloud, and the red curve is the curve fitting result. The blue points in the figure basically meet the characteristics of the sine curve, so the respiratory displacement model in the Z direction can be fitted, as shown in the following formula 8:

[0109] z=A z sin(w z t z +bz )+C z Formula 8

[0110] Where, t z Indicates the time difference from the start of scanning to the current moment, A z 、w z 、b z 、C z The scanning start time is the time when the probe starts to acquire the test ultrasound image in the above step S101.

[0111] In a specific embodiment, the respiratory displacement model in the Z direction is shown in the following formula 8:

[0112] z=-0.0024sin(0.0438t z +1.0151)+0.0024 Formula 8

[0113] In an embodiment of the present application, the process of establishing a respiratory displacement model described in the above steps S101-S102 is to establish a more accurate respiratory displacement model by accurately recording the displacement of the target organ in the ultrasound image. This not only ensures the consistency and comparability of the data and improves the accuracy of subsequent processing, but also reduces the errors caused by dynamic changes, and provides a reliable basis for subsequent real-time compensation. Specifically, this method further enhances the accuracy and robustness of the model by converting two-dimensional pixel points into a three-dimensional point cloud and establishing a respiratory displacement model of the target organ in the XYZ three directions based on this, which can better reflect the real motion characteristics of the target organ, thereby improving the accuracy and stability of three-dimensional reconstruction in practical applications and providing doctors with more reliable imaging support.

[0114] In the above step S103, the probe is controlled to continue to acquire ultrasound images, and the respiratory displacement model is applied to all ultrasound images to obtain a target organ model after ultrasound 3D reconstruction. Specifically, the following steps S1031-S1034 are included:

[0115] S1031: For each ultrasound image, convert the pixel coordinates of all pixel points in the ultrasound image into three-dimensional coordinates in a base coordinate system.

[0116] Specifically, the pixel coordinates of all points in the ultrasound image may be converted into three-dimensional coordinates in the probe coordinate system based on the following formula 9:

[0117] p i =[0, -μ×pt.x i , μ×pt.y i ] Formula 9

[0118] Where p represents the three-dimensional coordinate of the probe coordinate system corresponding to the i-th pixel point in the ultrasound image, pt.x and pt.y represent the pixel coordinates of the i-th pixel point in the ultrasound image on the pt.x-axis and pt.y-axis, respectively, and μ is the actual physical distance per unit pixel.

[0119] Based on the acquired hand-eye matrix, the three-dimensional coordinates of all points in the ultrasound image in the probe coordinate system are converted to the tool coordinate system at the end of the robotic arm.

[0120] Based on the acquired robot arm pose, the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system is obtained.

[0121] Based on the transformation matrix from the tool coordinate system at the end of the manipulator to the base coordinate system, all points of the ultrasound image in the tool coordinate system at the end of the manipulator are transformed to the base coordinate system.

[0122] To facilitate those skilled in the art to understand this solution, the process described in step S1031 is explained more clearly in the form of a formula below: Based on the following formula 10, all pixel coordinates in the ultrasound image are converted into three-dimensional coordinates in the base coordinate system:

[0123]

[0124] Where P represents the set of three-dimensional coordinates of all points in the ultrasound image in the base coordinate system, represents the transformation matrix from the tool coordinate system at the end of the manipulator to the base coordinate system corresponding to the i-th pixel point in the ultrasound image, represents the hand-eye matrix, B represents the base coordinate system, S represents the probe coordinate system, TCP represents the tool coordinate system at the end of the robot arm, and p i represents the three-dimensional coordinates of the probe coordinate system corresponding to the i-th pixel in the ultrasound image, T represents transposition, and n is the total number of pixels in the ultrasound image.

[0125] The specific implementation of the above step S1031 can refer to the above step S10211, and will not be repeated here.

[0126] S1032: Convert all points of the ultrasound image in the base coordinate system into the probe coordinate system.

[0127] Specifically, based on the acquired robot arm posture, a transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm is obtained.

[0128] Based on the transformation matrix from the base coordinate system to the end-of-arm tool coordinate system, the three-dimensional coordinates of all points in the ultrasound image in the base coordinate system are transformed to the end-of-arm tool coordinate system to obtain the three-dimensional coordinates of all points in the ultrasound image in the end-of-arm tool coordinate system.

[0129] Based on the inverse matrix of the acquired hand-eye matrix, all points of the ultrasound image in the tool coordinate system at the end of the robotic arm are transformed into the probe coordinate system to obtain a point set of the test ultrasound image.

[0130] The specific implementation of the above step S1032 can refer to the above step S10212, which will not be described in detail here.

[0131] S1033: Based on the three-dimensional coordinates of all points of the ultrasound image in the probe coordinate system, a respiratory displacement model is applied to obtain a respiratory compensation point set corresponding to the ultrasound image.

[0132] Specifically, the three-dimensional coordinates of all points of the ultrasound image in the probe coordinate system include components in the X, Y, and Z directions. For each point of the ultrasound image in the probe coordinate system, the respiratory displacement model in the X-axis direction is applied to the component in the X-axis direction, the respiratory displacement model in the Y-axis direction is applied to the component in the Y-axis direction, and the respiratory displacement model in the Z-axis direction is applied to the component in the Z-axis direction to obtain the respiratory compensation point set corresponding to the ultrasound image.

[0133] At the same time, if there is no respiratory displacement model in certain axial directions, or the displacement caused by breathing is less than a preset threshold, the displacement change caused by breathing can be ignored, and the respiratory displacement model is not applied to the component in this axial direction.

[0134] In order to facilitate those skilled in the art to understand this solution, the following is a more clear explanation of the specific implementation process of step S1033 based on the examples given in the above steps S1022-S1023: Figure 3-5 The curve fitting results shown in the figure show that the displacement caused by breathing in the X-axis and Z-axis directions is less than the preset threshold, and the respiratory displacement model cannot be fitted in the X-axis. This is because the main displacement direction of the gallbladder is up and down movement within the human abdominal cavity, which is approximately equivalent to the Y-direction of the probe. Therefore, respiratory compensation can be performed only in the Y-axis direction. Therefore, based on the following formula 11, the Y-axis component of the ultrasound image point in the probe coordinate system is applied to the Y-axis component, and the Y-axis component after respiratory compensation is obtained:

[0135] Y i =Y i -{[A y sin(b y )+C y ]-[A y sin(w y t+b y )+C y ]} Formula 11

[0136] Where Y i It represents the component of the i-th point in the Y-axis direction after breathing compensation, Y irepresents the component of the i-th point of the ultrasound image in the probe coordinate system in the Y-axis direction, t y A represents the time difference between the start of scanning and the acquisition of ultrasound images. y 、w y 、b y 、C y The parameter in the respiratory displacement model in the Y-axis direction is represented by . The scanning start time is the time when the probe starts to acquire the test ultrasound image in the above step S101.

[0137] The Y-axis component after respiration compensation is used to replace the Y-axis component of the ultrasound image points, while the X-axis and Z-axis components remain unchanged, thereby obtaining a respiration compensation point set corresponding to the ultrasound image.

[0138] S1034: Obtaining a target organ model after ultrasound three-dimensional reconstruction based on the respiratory compensation point set corresponding to all ultrasound images.

[0139] Specifically, the above steps S1031-S1033 may be performed on all ultrasound images to obtain the respiratory compensation point sets corresponding to all ultrasound images, and all respiratory compensation point sets may be integrated to obtain a complete and accurate three-dimensional target organ model, thereby completing the ultrasound three-dimensional reconstruction.

[0140] In this embodiment of the present application, during the scanning process, step S103 converts all pixels in each ultrasound image frame from two-dimensional coordinates to three-dimensional coordinates through hand-eye registration (or hand-eye calibration) to align them with the robot base coordinate system, thereby achieving a 3D reconstruction of the gallbladder without respiration compensation. The reconstruction results without respiration compensation are then converted to the ultrasound probe coordinate system, facilitating the application of a respiratory displacement model within the ultrasound probe coordinate system to complete respiratory displacement compensation. Compared to performing complex displacement calculations directly on the original ultrasound image, this method effectively simplifies the entire compensation process, reduces computational complexity, improves compensation efficiency and accuracy, and ultimately achieves high-precision 3D reconstruction of the gallbladder.

[0141] Example 2

[0142] Based on the same inventive concept, the embodiment of the present invention also provides a human organ ultrasonic three-dimensional reconstruction device based on respiratory displacement compensation, referring to Figure 6 As shown, the device includes:

[0143] The first acquisition module 101 is used to control the probe at the end of the ultrasonic robot's mechanical arm to acquire multiple test ultrasound images within a preset test period at the target organ of the human body;

[0144] A model building module 102 is configured to convert two-dimensional pixels into three-dimensional point clouds based on the plurality of test ultrasound images, and to build a respiratory displacement model of the target organ;

[0145] The three-dimensional reconstruction module 103 is used to control the probe to continue acquiring ultrasound images, and apply the respiratory displacement model to all the ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction.

[0146] Example 3

[0147] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described in the above-mentioned embodiment 1 is implemented.

[0148] Example 4

[0149] Based on the same inventive concept, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation as described in the above embodiment 1.

[0150] Example 5

[0151] Based on the same inventive concept, an embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements the ultrasonic three-dimensional reconstruction method of human organs based on respiratory displacement compensation as described in the above embodiment 1.

[0152] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0153] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0156] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation, characterized in that: include: At the target organ of the human body, the probe at the end of the ultrasonic robot's mechanical arm is controlled to collect multiple test ultrasound images within a preset test period; Calculate the point cloud centroid of each test ultrasound image; Based on the point cloud centroids of all the test ultrasound images, decomposition is performed in the XYZ directions to obtain a scatter plot of the point cloud centroids in the XYZ directions; Performing curve fitting on the point cloud centroid scatter plot in each direction to obtain a respiratory displacement model of the target organ; The probe is controlled to continue acquiring ultrasound images, and the respiratory displacement model is applied to all the ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction.

2. The method according to claim 1, characterized in that The calculation to obtain the point cloud centroid of each test ultrasound image includes: Converting pixel coordinates of all pixel points in the test ultrasound image into three-dimensional coordinates in a base coordinate system to obtain a point cloud set of the test ultrasound image; Convert all points in the point cloud set of the test ultrasound image from a base coordinate system to a probe coordinate system to obtain a point set set of the test ultrasound image; The point cloud centroid of the test ultrasound image is obtained by calculation based on the point set of the test ultrasound image.

3. The method according to claim 2, characterized in that The step of converting the pixel coordinates of all pixels in the test ultrasound image into three-dimensional coordinates in a base coordinate system to obtain a point cloud set of the test ultrasound image includes: Converting the pixel coordinates of all pixels in the test ultrasound image into three-dimensional coordinates in a probe coordinate system; Based on the acquired hand-eye matrix, all points of the test ultrasound image in the probe coordinate system are converted to the robot arm end tool coordinate system; Based on the acquired robot arm posture, the transformation matrix from the tool coordinate system at the end of the robot arm to the base coordinate system is obtained; Based on the transformation matrix from the manipulator end tool coordinate system to the base coordinate system, all points of the test ultrasound image in the manipulator end tool coordinate system are transformed to the base coordinate system to obtain a point cloud set of the test ultrasound image.

4. The method according to claim 2, characterized in that The step of converting all points in the point cloud set of the test ultrasound image from a base coordinate system to a probe coordinate system to obtain a point set of the test ultrasound image includes: Based on the acquired robot arm pose, the transformation matrix from the base coordinate system to the tool coordinate system at the end of the robot arm is obtained; Based on the conversion matrix from the base coordinate system to the robot end tool coordinate system, convert the point cloud set of the test ultrasound image from the base coordinate system to the robot end tool coordinate system to obtain the three-dimensional coordinates of all points of the test ultrasound image in the robot end tool coordinate system; Based on the acquired inverse matrix of the hand-eye matrix, all points of the test ultrasound image in the robot arm end tool coordinate system are converted to the probe coordinate system to obtain a point set of the test ultrasound image.

5. The method according to claim 1, wherein The step of controlling the probe to continue acquiring ultrasound images and applying the respiratory displacement model to all the ultrasound images to obtain a target organ model after ultrasound three-dimensional reconstruction includes: For each ultrasound image, converting pixel coordinates of all pixel points in the ultrasound image into three-dimensional coordinates in a base coordinate system; Converting all points of the ultrasound image in the base coordinate system into the probe coordinate system; Based on the three-dimensional coordinates of all points of the ultrasound image in the probe coordinate system, applying the respiratory displacement model to obtain a respiratory compensation point set corresponding to the ultrasound image; Based on the respiratory compensation point sets corresponding to all the ultrasound images, a target organ model after ultrasound three-dimensional reconstruction is obtained.

6. A human organ ultrasonic three-dimensional reconstruction device based on respiratory displacement compensation, characterized in that: include: The first acquisition module is used to control the probe at the end of the ultrasonic robot's mechanical arm to acquire multiple test ultrasound images within a preset test period at the target organ of the human body; A model building module is used to calculate the point cloud centroid of each test ultrasound image; Based on the point cloud centroids of all the test ultrasound images, decomposition is performed in the XYZ directions to obtain a scatter plot of the point cloud centroids in the XYZ directions; Performing curve fitting on the point cloud centroid scatter plot in each direction to obtain a respiratory displacement model of the target organ; The three-dimensional reconstruction module is used to control the probe to continue to collect ultrasonic images, and apply the respiratory displacement model to all the ultrasonic images to obtain a target organ model after ultrasonic three-dimensional reconstruction.

7. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation according to any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation according to any one of claims 1 to 5 is implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the method for ultrasonic three-dimensional reconstruction of human organs based on respiratory displacement compensation according to any one of claims 1 to 5.

Images