Intraoperative ultrasound puncture guidance method and device
By transforming the tumor target from a two-dimensional image coordinate system to a three-dimensional head-mounted display coordinate system, and combining electromagnetic positioning and augmented reality technologies, the problem of angular deviation in two-dimensional ultrasound puncture guidance is solved, improving the efficiency and safety of puncture.
Patent Information
- Application Number
- CN202610607845.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-06
- Publication Date
- 2026-07-10
Smart Images
Figure CN122350830A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical ultrasound-guided surgery technology, and in particular to an intraoperative ultrasound-guided puncture method and device. Background Technology
[0002] Currently, most intraoperative ultrasound-guided punctures used in clinical practice rely on two-dimensional ultrasound images and the surgeon's experience to complete three-dimensional spatial positioning. The surgeon needs to determine the spatial location and depth of the tumor in the two-dimensional ultrasound image and convert it into a suitable puncture direction in the actual three-dimensional anatomical space. When the tumor is located deep or near important structures (such as blood vessels, bile ducts, etc.), the surgeon needs to control the insertion angle and path deviation of the puncture needle more precisely.
[0003] However, two-dimensional cross-sectional information cannot directly provide three-dimensional directional clues relative to organs and surgical fields, which can easily lead to angular deviations, deviations from the planned path, or the need for multiple attempts to reach the target point, affecting puncture efficiency and success rate. Furthermore, the puncture needle may increase friction between the needle body and tissue during multiple movements or rotations, causing additional tissue damage, bleeding, or complication risks. Therefore, existing intraoperative ultrasound-guided puncture methods reduce puncture efficiency, success rate, and safety. Summary of the Invention
[0004] This invention provides an intraoperative ultrasound-guided puncture method and device to address the shortcomings of existing technologies where two-dimensional cross-sectional information cannot directly provide three-dimensional directional guidance relative to the organ and surgical field. This leads to angular deviations, deviations from the planned path, or the need for multiple attempts to reach the target point, affecting puncture efficiency and success rate. Furthermore, repeated movement or rotation of the puncture needle may increase friction between the needle and tissue, causing additional tissue damage, bleeding, or complications. Therefore, existing intraoperative ultrasound-guided puncture methods suffer from reduced efficiency, success rate, and safety. This invention transforms the tumor target point from a two-dimensional image coordinate system to a three-dimensional head-mounted display coordinate system through the target probe's pose, providing the surgeon with accurate three-dimensional spatial location information. Furthermore, based on the needle tip pose, the device guides the operator to perform intraoperative ultrasound puncture, thereby improving the efficiency, success rate, and safety of intraoperative puncture.
[0005] This invention provides an intraoperative ultrasound-guided puncture method, comprising the following steps: Acquire the position of the tumor target in the image coordinate system, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system, and the needle tip pose of the puncture needle tip in the head display coordinate system; Based on the position in the image coordinate system and the pose of the target probe, the position of the tumor target point in the head-mounted display coordinate system is determined; Based on the position in the head-mounted display coordinate system and the needle tip pose, a display interface is generated, which is used to guide the operator to perform intraoperative ultrasonic puncture.
[0006] According to an intraoperative ultrasound-guided puncture method provided by the present invention, determining the position of the tumor target point in the head-mounted display coordinate system based on the position in the image coordinate system and the pose of the target probe includes: determining a first mapping relationship between the image coordinate system and the probe coordinate system based on the probe acoustic beam model, pixel size, and probe geometric calibration; determining the position of the tumor target point in the intraoperative ultrasound probe coordinate system based on the first mapping relationship and the position in the image coordinate system; and determining the position of the tumor target point in the head-mounted display coordinate system based on the position in the intraoperative ultrasound probe coordinate system and the pose of the target probe.
[0007] According to an intraoperative ultrasound-guided puncture method provided by the present invention, determining the position of the tumor target point in the head-mounted display coordinate system based on the position in the intraoperative ultrasound probe coordinate system and the pose of the target probe includes: determining the position of the tumor target point in the electromagnetic coordinate system based on the position in the probe coordinate system and the pose of the target probe; determining the alignment transformation between the electromagnetic coordinate system and the head-mounted display coordinate system; and determining the position of the tumor target point in the head-mounted display coordinate system based on the position in the electromagnetic coordinate system and the alignment transformation.
[0008] According to the present invention, an intraoperative ultrasound-guided puncture method is provided to obtain the position of the tumor target in an image coordinate system, comprising: obtaining a target ultrasound image corresponding to the target object; inputting the target ultrasound image into a pre-trained segmentation network to obtain a tumor region output by the pre-trained segmentation network; and determining the position of the tumor target in the image coordinate system based on the tumor region.
[0009] According to a method for intraoperative ultrasound-guided puncture provided by the present invention, the step of acquiring a target ultrasound image corresponding to a target object includes: acquiring an initial ultrasound image corresponding to the target object through an ultrasound imaging system; and performing data quality processing on the initial ultrasound image to obtain the target ultrasound image.
[0010] According to an intraoperative ultrasound-guided puncture method provided by the present invention, determining the position of the tumor target point in the image coordinate system based on the tumor region includes: determining the tumor target point in the tumor region; and performing morphological post-processing and temporal consistency constraints on the tumor target point to obtain the position in the image coordinate system.
[0011] According to the present invention, an intraoperative ultrasound-guided puncture method is provided to obtain the needle tip pose, comprising: detecting the mark of the puncture needle based on the camera sensor and spatial positioning capability of the head-mounted display system to obtain the mark pose; and determining the needle tip pose based on the mark pose and a second mapping relationship, wherein the second mapping relationship is used to characterize the correspondence between the mark pose and the needle tip pose.
[0012] According to the present invention, an intraoperative ultrasound puncture guidance method is provided to obtain the target probe pose of the intraoperative ultrasound probe in an electromagnetic coordinate system, comprising: generating an electrical signal based on a magnetic field signal generated by a magnetic field generator using an electromagnetic positioning sensor; and determining the target probe pose of the intraoperative ultrasound probe in an electromagnetic coordinate system based on the electrical signal and the magnetic field model.
[0013] According to an intraoperative ultrasound-guided puncture method provided by the present invention, determining the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system based on the electrical signal and the magnetic field model includes: determining a first probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system based on the electrical signal and the magnetic field model; filtering the first probe pose to obtain a second probe pose; and performing time alignment on the second probe pose to obtain the target probe pose.
[0014] According to the present invention, an intraoperative ultrasound-guided puncture method is provided, wherein generating a display interface based on the position in the head-mounted display coordinate system and the needle tip pose includes determining the puncture information required for the puncture needle tip based on the position in the head-mounted display coordinate system and the needle tip pose; and generating the display interface based on the puncture information.
[0015] According to the present invention, an intraoperative ultrasound-guided puncture method is provided, wherein the puncture information includes a tumor target point, a planned path of the puncture needle tip, a needle tip pose, and a deviation between the puncture needle tip and the tumor target point. The step of generating the display interface based on the puncture information includes: rendering the tumor target point, the planned path of the puncture needle tip, the needle tip pose, and the deviation between the puncture needle tip and the tumor target point on a head-mounted display system to obtain the display interface.
[0016] The present invention also provides an intraoperative ultrasound-guided puncture device, comprising the following modules: The acquisition module is used to acquire the position of the tumor target in the image coordinate system, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system, and the needle tip pose of the puncture needle tip in the head display coordinate system. The determination module is used to determine the position of the tumor target point in the head-mounted display coordinate system based on the position in the image coordinate system and the pose of the target probe; The guidance module is used to generate a display interface based on the position in the head-mounted display coordinate system and the needle tip pose. The display interface is used to guide the operator to perform intraoperative ultrasound puncture.
[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the intraoperative ultrasound-guided puncture method as described above.
[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intraoperative ultrasound-guided puncture method as described above.
[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the intraoperative ultrasound-guided puncture method as described above.
[0020] This invention provides an intraoperative ultrasound-guided puncture method and device. It acquires the position of the tumor target in the image coordinate system, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system, and the needle tip pose of the puncture needle in the head-mounted display coordinate system. Based on the position in the image coordinate system and the target probe pose, the position of the tumor target in the head-mounted display coordinate system is determined. Based on the position in the head-mounted display coordinate system and the needle tip pose, the operator is guided to perform intraoperative ultrasound puncture. Thus, by transforming the tumor target from the two-dimensional image coordinate system to the three-dimensional head-mounted display coordinate system through the target probe pose, the operator is provided with a true and accurate three-dimensional spatial location basis. Furthermore, based on the needle tip pose, the operator is guided to perform intraoperative ultrasound puncture, thereby improving the efficiency, success rate, and safety of intraoperative puncture. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0022] Figure 1 This is a flowchart illustrating the intraoperative ultrasound-guided puncture method provided by the present invention.
[0023] Figure 2 This is a schematic diagram of the intraoperative ultrasound puncture guidance system provided by the present invention.
[0024] Figure 3 This is a schematic diagram of the intraoperative ultrasonic puncture guidance device provided by the present invention.
[0025] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0027] Intraoperative ultrasound-guided puncture is a common technique in surgical procedures such as tumor biopsy, ablation needle placement, drainage, and catheter placement. Clinically, the ultrasound probe is usually placed directly on the surface of the target organ to acquire real-time ultrasound images. Target tissues such as tumors are identified on these images, and the needle insertion point, angle, and depth are determined accordingly to complete the puncture localization. Due to the advantages of ultrasound imaging, such as high real-time performance, no radiation, and sensitivity to soft tissues, ultrasound-guided puncture is considered one of the important methods for precise intraoperative puncture and has wide applications in various organs and lesion types.
[0028] Currently, most intraoperative ultrasound-guided punctures used in clinical practice rely on two-dimensional ultrasound sections and the surgeon's experience to complete three-dimensional spatial positioning. On the one hand, the surgeon needs to determine the spatial location and depth of the tumor in the two-dimensional ultrasound image and convert it into a suitable puncture direction in the actual three-dimensional anatomical space. On the other hand, the puncture needle appears as a slender, high-echo structure in the ultrasound image, which is significantly affected by the incident angle, scattering, noise, and artifacts. The needle tip is unstable, which often requires the surgeon to repeatedly adjust the probe section and puncture posture to maintain the visibility of the needle body / needle tip.
[0029] Existing technologies typically have two prominent problems in practical surgical applications.
[0030] First, there are difficulties in spatial guidance and angle control. For procedures that rely solely on two-dimensional ultrasound images, when the tumor is located deep or near important structures (such as blood vessels or bile ducts), the surgeon needs to control the insertion angle and path deviation of the puncture needle more precisely. However, two-dimensional sectional information cannot directly provide three-dimensional directional guidance relative to the organ and surgical field, which can easily lead to angle deviation, deviation from the planned path, or the need for multiple attempts to reach the target point, affecting puncture efficiency and success rate.
[0031] Secondly, there are safety and tissue damage risks during the procedure. To correct the aforementioned angular deviations or positioning uncertainties caused by the invisibility of the needle tip, the operator often needs to repeatedly move the puncture needle back and forth, change the needle's posture, or frequently move / rotate the ultrasound probe to recapture the same cross-section of the needle and the target point. This process not only increases the operation time but also increases the friction between the needle and tissue and the number of repeated punctures, potentially causing additional tissue damage, bleeding, or complications. Simultaneously, the operator also needs to frequently switch their line of sight between the surgical field and the external monitor, further increasing the operational burden and reducing operational stability.
[0032] Based on the aforementioned problems, this invention provides an intraoperative ultrasound-guided puncture method. By transforming the tumor target point from a two-dimensional image coordinate system to a three-dimensional head display coordinate system through the target probe posture, it provides the surgeon with a real and accurate three-dimensional spatial position basis. Furthermore, based on the needle tip posture, it guides the operator to perform intraoperative ultrasound puncture, thereby improving the efficiency, success rate, and safety of the operator's intraoperative puncture.
[0033] The following is combined Figures 1 to 2 The present invention describes an intraoperative ultrasound-guided puncture method that can be applied to puncture surgery at any tumor target. The subject executing this method can be an electronic device or an intraoperative ultrasound-guided puncture device installed in the electronic device. The intraoperative ultrasound-guided puncture device can be implemented by software, hardware, or a combination of both.
[0034] Figure 1 This is a flowchart illustrating the intraoperative ultrasound-guided puncture method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Obtain the position of the tumor target in the image coordinate system, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system, and the needle tip pose of the puncture needle tip in the head display coordinate system.
[0035] Here, the image coordinate system is a two-dimensional coordinate system; the electromagnetic positioning coordinate system is a three-dimensional spatial reference coordinate system established by the magnetic field generator in the electromagnetic positioning system, used to describe the spatial position and orientation of the electromagnetic sensor attached to the ultrasonic probe. The head-mounted display coordinate system is a three-dimensional Cartesian coordinate system in augmented reality (AR) devices, with the head-mounted display itself as the origin, describing the position and orientation of the head; its origin is dynamic.
[0036] Here, the tumor target can be any tumor, including but not limited to thyroid tumors, breast tumors, lung cancer tumors, etc.
[0037] Intraoperative ultrasound probes are used to emit ultrasound waves into the surgical area to acquire real-time two-dimensional ultrasound images of internal tissues, blood vessels, and lesions. Puncture needles are the instruments used in interventional procedures to reach tumor targets for sampling, drug administration, etc.
[0038] Here, the target probe pose can include the position and attitude of the probe, which can be understood as the probe's 3 translations (x, y, z) + 3 rotations (pitch / yaw / roll) in the spatial coordinate system, i.e., 6 degrees of freedom (DOF).
[0039] Here, needle tip pose includes the position and attitude of the needle tip. The position of the needle tip can be represented by three-dimensional coordinates (x, y, z), and the attitude can be represented by pitch, yaw, and roll.
[0040] Here, the position, target probe pose, and needle tip pose in the image coordinate system can be obtained directly using the device; or they can be obtained by processing the acquired initial data.
[0041] Step 102: Based on the position in the image coordinate system and the pose of the target probe, determine the position of the tumor target point in the head-mounted display coordinate system.
[0042] It should be noted that the position of the tumor target in the head-mounted display coordinate system is actually the position of the tumor target in the image coordinate system mapped to the position in the head-mounted display coordinate system.
[0043] It should be noted that in practical applications, the image coordinate system cannot be directly transformed to the head-mounted display coordinate system. Therefore, the target probe pose in the electromagnetic coordinate system is introduced as an intermediary to transform the tumor target point from the image coordinate system to the head-mounted display coordinate system.
[0044] Step 103: Generate a display interface based on the position in the head-mounted display coordinate system and the needle tip pose.
[0045] The display interface is used to guide the operator in performing intraoperative ultrasonic puncture.
[0046] Here, the position and needle tip pose in the head-mounted display coordinate system can be directly displayed, allowing the operator to perform surgical procedures based on these coordinates. Alternatively, more detailed puncture information can be determined and displayed based on the position and needle tip location in the head-mounted display coordinate system, enabling the operator to perform surgical procedures accordingly.
[0047] In this embodiment of the invention, the position of the tumor target in the image coordinate system, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system, and the needle tip pose of the puncture needle in the head-mounted display coordinate system are obtained. Based on the position in the image coordinate system and the target probe pose, the position of the tumor target in the head-mounted display coordinate system is determined. Based on the position in the head-mounted display coordinate system and the needle tip pose, the operator is guided to perform intraoperative ultrasound puncture. Thus, by transforming the tumor target from the two-dimensional image coordinate system to the three-dimensional head-mounted display coordinate system through the target probe pose, the operator is provided with a true and accurate three-dimensional spatial location basis. Furthermore, based on the needle tip pose, the operator is guided to perform intraoperative ultrasound puncture, thereby improving the efficiency, success rate, and safety of intraoperative puncture.
[0048] Furthermore, determining the position of the tumor target in the head-mounted display coordinate system based on its position in the image coordinate system and the pose of the target probe includes: determining a first mapping relationship between the image coordinate system and the probe coordinate system based on the probe acoustic beam model, pixel size, and probe geometric calibration; determining the position of the tumor target in the intraoperative ultrasound probe coordinate system based on the first mapping relationship and its position in the image coordinate system; and determining the position of the tumor target in the head-mounted display coordinate system based on its position in the intraoperative ultrasound probe coordinate system and the pose of the target probe.
[0049] Here, the probe beam model is used to characterize the spatial geometry and coordinate mapping of the ultrasonic beam, while probe geometric calibration is the process of solving for unknown parameters in the probe beam model through experiments and algorithms. Probe geometric calibration is a prerequisite for establishing an accurate probe beam model, and the probe beam model is the final output of probe geometric calibration. The probe beam model relies on pixel dimensions to achieve a quantitative mapping between image coordinates and physical space coordinates. That is, probe geometric calibration solves for the pixel dimensions, fills the pixel dimensions into the probe beam model, and the probe beam model achieves the first mapping relationship between the image coordinate system and the probe coordinate system.
[0050] Here, the position in the probe coordinate system can be obtained through the first transformation matrix, which is used to characterize the mapping relationship between the image coordinate system and the probe coordinate system.
[0051] Here, the position in the head-mounted display coordinate system can be calculated directly from the position in the probe coordinate system and the pose of the target probe, or it can be determined by transforming the position in the probe coordinate system and then using the transformed position and the pose of the target probe.
[0052] In this embodiment of the invention, based on the spatial correspondence between multiple coordinate systems, the tumor target points detected in the image can be transformed into the head-mounted display world coordinate system, providing accurate positional basis for augmented reality overlay display.
[0053] Furthermore, determining the position of the tumor target in the head-mounted display coordinate system based on its position in the intraoperative ultrasound probe coordinate system and the target probe pose includes: determining the position of the tumor target in the electromagnetic coordinate system based on its position in the probe coordinate system and the target probe pose; determining the alignment transformation between the electromagnetic coordinate system and the head-mounted display coordinate system; and determining the position of the tumor target in the head-mounted display coordinate system based on its position in the electromagnetic coordinate system and the alignment transformation.
[0054] Here, the position in the electromagnetic coordinate system can be calculated by substituting the position in the probe coordinate system and the target probe pose into the formula, or it can be obtained based on the mapping relationship between the probe coordinate system, the target probe pose, and the electromagnetic coordinate system.
[0055] Here, the alignment transformation between the electromagnetic coordinate system and the head-mounted display coordinate system can be obtained through calibration bodies, common reference points, or short-time registration procedures.
[0056] Here, alignment transformation is used to characterize the mapping relationship between the electromagnetic coordinate system and the head-mounted display coordinate system.
[0057] For example, the transformation of the coordinate system can be represented by the following formula (1): (1) in, This represents the transformation from the image coordinate system to the head-mounted display coordinate system. Indicates time, This represents the transformation from the electromagnetic coordinate system to the head-mounted display coordinate system. This represents the transformation from the image coordinate system to the electromagnetic coordinate system.
[0058] It should be noted that to convert from the image coordinate system to the electromagnetic coordinate system, you need to first convert the image coordinate system to the probe coordinate system, and then convert the probe coordinate system to the electromagnetic coordinate system.
[0059] In this embodiment of the invention, based on the relationship between the probe coordinate system and the electromagnetic coordinate system, and the alignment transformation between the electromagnetic coordinate system and the head-mounted display coordinate system, the detected tumor target point can be transformed into the head-mounted display world coordinate system, providing accurate positional basis for augmented reality overlay display.
[0060] Further, obtaining the position of the tumor target in the image coordinate system includes: obtaining the target ultrasound image corresponding to the target object; inputting the target ultrasound image into a pre-trained segmentation network to obtain the tumor region output by the pre-trained segmentation network; and determining the position of the tumor target in the image coordinate system based on the tumor region.
[0061] Here, the target object can include any tissue or organ.
[0062] Here, the target ultrasound image can be obtained using an ultrasound imaging system; or it can be obtained by processing an image acquired using an ultrasound imaging system.
[0063] For example, the target ultrasound image can be a multi-frame ultrasound image, and a pre-trained segmentation network automatically extracts the tumor / lesion region on each frame or keyframe of the target ultrasound image.
[0064] Here, pre-trained segmentation networks include, but are not limited to, fully convolutional networks, U-Net, pyramid pooling networks (PSPNet), etc.
[0065] Here, the tumor region can take the form of a mask or outline, etc.
[0066] Here, the tumor target can be the center point of the tumor region, a key boundary point, or a point selected interactively by the doctor.
[0067] For example, after determining the tumor region, the center point of the tumor region can be used as the tumor target point, and then the coordinates corresponding to the tumor target point can be determined to obtain its position in the image coordinate system.
[0068] In this embodiment of the invention, a pre-trained segmentation network is used to determine the tumor region, and the location of the tumor target is determined based on the tumor region. This provides the location of the target in the image for conversion and AR display, improving the efficiency, success rate and safety of the operator in performing intraoperative puncture.
[0069] Furthermore, acquiring the target ultrasound image corresponding to the target object includes: acquiring an initial ultrasound image corresponding to the target object through an ultrasound imaging system; and performing data quality processing on the initial ultrasound image to obtain the target ultrasound image.
[0070] Here, data quality processing can include noise reduction, gain normalization, etc.
[0071] For example, an ultrasound imaging system can acquire consecutive frames of initial ultrasound images, and perform noise reduction or gain normalization on each frame of initial ultrasound images to obtain multiple frames of target ultrasound images.
[0072] In this embodiment of the invention, by performing data quality processing on the acquired initial ultrasound images, the effects of ultrasound noise and artifacts are reduced, thereby improving the quality of the data and thus improving the accuracy of tumor target location.
[0073] Furthermore, determining the position of the tumor target in the image coordinate system based on the tumor region includes: determining the tumor target in the tumor region; and performing morphological post-processing and temporal consistency constraints on the tumor target to obtain its position in the image coordinate system.
[0074] Here, morphological post-processing is based on mathematical morphology image processing operations to correct, optimize, and plan tumor targets.
[0075] Temporal consistency constraints are used to force the positions of tumor targets that exhibit abrupt changes, jumps, or unreasonable occurrences in consecutive frames to remain stable and continuous.
[0076] For example, after determining the tumor target in each frame of the target ultrasound image, morphological post-processing and temporal consistency constraints are performed on the tumor target in each frame to obtain its position in the image coordinate system.
[0077] In this embodiment of the invention, morphological post-processing and temporal consistency constraints are applied to the segmented tumor targets, which enhances the stability and reliability of the tumor targets and further improves the efficiency, success rate and safety of intraoperative puncture by the operator.
[0078] Furthermore, obtaining the needle tip pose includes: detecting the mark on the puncture needle based on the camera sensor and spatial positioning capability of the head-mounted display system to obtain the mark pose; determining the needle tip pose based on the mark pose and a second mapping relationship, wherein the second mapping relationship is used to characterize the correspondence between the mark pose and the needle tip pose.
[0079] Here, the marker is placed on the puncture needle. During the movement of the puncture needle, the camera sensor and spatial positioning capability of the head-mounted display system are used to check and track the marker to obtain the marker pose.
[0080] Here, the correspondence between the marker pose and the needle tip pose can be pre-calibrated.
[0081] In this embodiment of the invention, by combining the camera sensor and spatial positioning capability of the head-mounted display system, the pose of the marker is tracked, and then the pinhead pose in the world coordinates of the head-mounted display is determined based on the pose of the marker. In this way, stable spatial output is achieved by combining the world coordinate anchor point of the head-mounted display.
[0082] Furthermore, obtaining the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system includes: generating an electrical signal based on the magnetic field signal generated by the magnetic field generator using an electromagnetic positioning sensor; and determining the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system based on the electrical signal and the magnetic field model.
[0083] Here, the magnetic field model is used to characterize the distribution, strength, direction, and interaction of magnetic fields.
[0084] Here, electromagnetic positioning generates a low-frequency electromagnetic field through a magnetic field generator. The sensor generates an electrical signal based on the magnetic field signal. Based on the electrical signal and the magnetic field model, the sensor measures the field strength / phase and other information and calculates the 6-degree-of-freedom pose, thereby obtaining the transformation relationship between the probe coordinate system and the electromagnetic coordinate system. Based on this transformation relationship, the target probe pose in the electromagnetic coordinate system is determined, improving the accuracy of the target probe pose and further determining the actual location of the tumor target in the patient's body.
[0085] Furthermore, determining the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system based on the electrical signal and magnetic field model includes: determining a first probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system based on the electrical signal and the magnetic field model; filtering the first probe pose to obtain a second probe pose; and performing time alignment on the second probe pose to obtain the target probe pose.
[0086] Here, filtering processes include, but are not limited to, first-order low-pass filtering and Kalman filtering.
[0087] It should be noted that the probe pose is usually a high-frequency continuous output, which requires time alignment.
[0088] In this embodiment of the invention, the probe pose of the high-frequency continuous output is filtered and time-aligned to reduce jitter and synchronize with the ultrasound frame, providing a continuous and traceable basis for the spatial mapping of probe-image-headset.
[0089] Furthermore, generating the display interface based on the position in the head-mounted display coordinate system and the needle tip pose includes determining the puncture information required for the puncture needle tip based on the position in the head-mounted display coordinate system and the needle tip pose; and generating the display interface based on the puncture information.
[0090] Here, puncture information is used to characterize the planned trajectory and deviation feedback of the puncture needle, such as target vector and distance, angle deviation, lateral offset, and safety constraints. Among them, the target vector and distance is the vector from the needle tip to the target point and the Euclidean distance, which is used to indicate the remaining depth / insertion volume; the angle deviation is the angle between the needle axis direction and the planned direction, which represents the yaw error; the lateral offset is the shortest distance from the needle tip to the planned path (straight line or curve from the entry point to the target point), which represents the path deviation; safety constraints: avoid prohibited areas of blood vessels / bile ducts, angle thresholds, minimum safe distances, etc., which trigger prompts or restrict planning.
[0091] Here, after the puncture information is determined, it can be displayed on the head-mounted display system or announced to the operator via voice.
[0092] For example, the puncture information can be calculated by substituting the position in the head-mounted display coordinate system and the needle tip pose into the formula, or it can be obtained through the mapping relationship between the position in the head-mounted display coordinate system, the needle tip position, and the puncture information.
[0093] In this embodiment of the invention, based on the position of the tumor target in the head display coordinate system and the puncture needle pose information, the recommended puncture path, needle insertion angle, remaining depth to the target, and deviation are calculated to form puncture navigation information that can be used for closed-loop guidance.
[0094] Furthermore, the puncture information includes the tumor target, the planned path of the puncture needle tip, the needle tip pose, and the deviation between the puncture needle tip and the tumor target. The step of generating the display interface based on the puncture information includes: rendering the tumor target, the planned path of the puncture needle tip, the needle tip pose, and the deviation between the puncture needle tip and the tumor target in the head-mounted display system to generate the display interface.
[0095] It should be noted that when puncture information is displayed on the head-mounted display system's interface, it can be displayed in the form of geometric objects. For example, tumor targets can be displayed as target spheres, planned paths can be displayed as path lines / cones, deviations can be displayed as correction arrows, numerical labels, and alarm statuses.
[0096] In this embodiment of the invention, the tumor target, planned trajectory, needle position / direction, and correction prompts are overlaid and displayed in an augmented reality manner in the surgeon's field of vision. The surgeon can obtain intuitive guidance within the field of vision without frequently looking down to check the external screen, thereby improving the stability and efficiency of the puncture operation. Simultaneously, the head-mounted display system supports interactive operations (such as locking the target, starting / pausing guidance, adjusting thresholds, etc.), thus achieving real-time, intuitive, and closed-loop navigation assistance for intraoperative puncture operations. Figure 2 This is a schematic diagram of the intraoperative ultrasound puncture guidance system provided by the present invention, as shown below. Figure 2As shown, the system includes an ultrasound imaging system 210, an electromagnetic positioning sensor 220, a high-performance processor 230, and a Hololens head-mounted display system 240. The high-performance processor 230 includes an intraoperative ultrasound image acquisition parameter information acquisition module 231, an intraoperative ultrasound image tumor automatic segmentation module 232, an intraoperative ultrasound probe pose acquisition module 233, a coordinate system registration module 234, a puncture information calculation module 235, and a data management module 236. The Hololens head-mounted display system 240 includes a visualization module 241 and a tracking module 242. Specifically, the ultrasound imaging system is used to acquire initial ultrasound images; the intraoperative ultrasound image acquisition parameter information acquisition module is used to preprocess the initial ultrasound images to obtain target ultrasound images; the intraoperative ultrasound image tumor automatic segmentation module is used to segment the target ultrasound images to obtain tumor target points; the electromagnetic positioning sensor and intraoperative ultrasound probe pose acquisition module is used to acquire the spatial pose information of the intraoperative ultrasound probe in real time; the coordinate system registration module is used to transform the tumor target points from the image coordinate system to the head-mounted display coordinate system to obtain the position of the tumor target points in the head-mounted display coordinate system; the tracking module is used to acquire the needle tip pose; the puncture information technology module is used to calculate puncture information based on the position in the head-mounted display coordinate system and the needle tip pose, and output the geometric objects and prompts required for visualization corresponding to the puncture information; the visualization module is used to spatially anchor and render the geometric objects and prompts required for visualization in the head-mounted display system; and the data management module is responsible for parameter configuration, system calibration result management, log recording, and playback verification.
[0097] The visualization module works by converting both the tumor target and needle tip pose to the HoloLens world coordinate system. AR content can then be directly placed in real space using world anchor points, achieving virtual-real alignment. Rendered content can include: tumor location markers, recommended puncture direction / trajectory, real-time needle direction, angle / depth / deviation values, and color / shape alarm prompts when thresholds are exceeded. Displayed on the head-mounted display, the operator can obtain intuitive navigation information without frequently turning their head to check the external monitor, thereby improving the stability and efficiency of the puncture procedure.
[0098] The data management module works by storing key data (ultrasound frames, segmentation results, probe pose, needle pose, registration matrix, and guidance output) in a unified manner according to timestamps, supporting postoperative review, algorithm iteration, error analysis, and quality control. Furthermore, calibration parameters can be version-managed to ensure rapid switching and consistency across different probes, markers, or surgical environments.
[0099] The intraoperative ultrasonic puncture guidance device provided by the present invention is described below. The intraoperative ultrasonic puncture guidance device described below and the intraoperative ultrasonic puncture guidance method described above can be referred to in correspondence.
[0100] Figure 3This is a schematic diagram of the intraoperative ultrasonic puncture guidance device provided by the present invention, as shown below. Figure 3 As shown, the intraoperative ultrasonic puncture guidance device 300 includes the following: The acquisition module 310 is used to acquire the position of the tumor target in the image coordinate system, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system, and the needle tip pose of the puncture needle tip in the head display coordinate system. The determination module 320 is used to determine the position of the tumor target in the head-mounted display coordinate system based on the position in the image coordinate system and the pose of the target probe; The guidance module 330 is used to generate a display interface based on the position in the head display coordinate system and the needle tip pose. The display interface is used to guide the operator to perform intraoperative ultrasound puncture.
[0101] In this embodiment of the invention, the determining module 320 is specifically used for: determining a first mapping relationship between the image coordinate system and the probe coordinate system based on the probe acoustic beam model, pixel size and probe geometric calibration; determining the position of the tumor target point in the intraoperative ultrasound probe coordinate system based on the first mapping relationship and the position in the image coordinate system; and determining the position of the tumor target point in the head display coordinate system based on the position in the intraoperative ultrasound probe coordinate system and the target probe pose.
[0102] In this embodiment of the invention, the determining module 320 is further specifically used for: determining the position of the tumor target in the electromagnetic coordinate system based on the position in the probe coordinate system and the pose of the target probe; determining the alignment transformation between the electromagnetic coordinate system and the head-mounted display coordinate system; and determining the position of the tumor target in the head-mounted display coordinate system based on the position in the electromagnetic coordinate system and the alignment transformation.
[0103] In this embodiment of the invention, the acquisition module 310 is specifically used for: acquiring a target ultrasound image corresponding to the target object; inputting the target ultrasound image into a pre-trained segmentation network to obtain a tumor region output by the pre-trained segmentation network; and determining the position of the tumor target point in the image coordinate system based on the tumor region.
[0104] In this embodiment of the invention, the acquisition module 310 is further specifically used to: acquire an initial ultrasound image corresponding to the target object through an ultrasound imaging system; and perform data quality processing on the initial ultrasound image to obtain the target ultrasound image.
[0105] In this embodiment of the invention, the acquisition module 310 is further specifically used to: determine the tumor target in the tumor region; and perform morphological post-processing and temporal consistency constraints on the tumor target to obtain its position in the image coordinate system.
[0106] In this embodiment of the invention, the acquisition module 310 is further specifically used to: detect the mark on the puncture needle based on the camera sensor and spatial positioning capability in the head-mounted display system to obtain the mark pose; and determine the needle tip pose based on the mark pose and the second mapping relationship, wherein the second mapping relationship is used to characterize the correspondence between the mark pose and the needle tip pose.
[0107] In this embodiment of the invention, the acquisition module 310 is further specifically used to: generate an electrical signal based on the magnetic field signal generated by the magnetic field generator through the electromagnetic positioning sensor; and determine the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system based on the electrical signal and the magnetic field model.
[0108] In this embodiment of the invention, the acquisition module 310 is further specifically used to: determine the first probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system based on the electrical signal and the magnetic field model; filter the first probe pose to obtain the second probe pose; and perform time alignment on the second probe pose to obtain the target probe pose.
[0109] In this embodiment of the invention, the guidance module 330 is further specifically used to: determine the puncture information required for the puncture needle tip based on the position in the head-mounted display coordinate system and the needle tip pose; and generate the display interface based on the puncture information.
[0110] In this embodiment of the invention, the puncture information includes the tumor target point, the planned path of the puncture needle tip, the needle tip pose, and the deviation between the puncture needle tip and the tumor target point. The guidance module 330 is further specifically used to: render the tumor target point, the planned path of the puncture needle tip, the needle tip pose, and the deviation between the puncture needle tip and the tumor target point on the head-mounted display system to obtain the display interface; and guide the operator to perform intraoperative ultrasound puncture based on the display interface.
[0111] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an intraoperative ultrasound-guided puncture method. This method includes: acquiring the position of the tumor target in an image coordinate system, the target probe pose of the intraoperative ultrasound probe in an electromagnetic coordinate system, and the needle tip pose of the puncture needle in a head-mounted display coordinate system; determining the position of the tumor target in the head-mounted display coordinate system based on the position in the image coordinate system and the target probe pose; and generating a display interface based on the position in the head-mounted display coordinate system and the needle tip pose, the display interface being used to guide the operator in performing intraoperative ultrasound puncture.
[0112] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0113] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the intraoperative ultrasound puncture guidance method provided by the above methods. The method includes: acquiring the position of the tumor target in an image coordinate system, the target probe pose of the intraoperative ultrasound probe in an electromagnetic coordinate system, and the needle tip pose of the puncture needle in a head-mounted display coordinate system; determining the position of the tumor target in the head-mounted display coordinate system based on the position in the image coordinate system and the target probe pose; and generating a display interface based on the position in the head-mounted display coordinate system and the needle tip pose, the display interface being used to guide the operator to perform intraoperative ultrasound puncture.
[0114] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the intraoperative ultrasound-guided puncture method provided by the methods described above. The method includes: acquiring the position of the tumor target in an image coordinate system, the target probe pose of the intraoperative ultrasound probe in an electromagnetic coordinate system, and the needle tip pose of the puncture needle in a head-mounted display coordinate system; determining the position of the tumor target in the head-mounted display coordinate system based on the position in the image coordinate system and the target probe pose; and generating a display interface based on the position in the head-mounted display coordinate system and the needle tip pose, the display interface being used to guide the operator in performing intraoperative ultrasound puncture.
[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for intraoperative ultrasound-guided puncture, characterized in that, include: Acquire the position of the tumor target in the image coordinate system, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system, and the needle tip pose of the puncture needle tip in the head display coordinate system; Based on the position in the image coordinate system and the pose of the target probe, the position of the tumor target point in the head-mounted display coordinate system is determined; Based on the position in the head-mounted display coordinate system and the needle tip pose, a display interface is generated, which is used to guide the operator to perform intraoperative ultrasonic puncture.
2. The intraoperative ultrasound-guided puncture method according to claim 1, characterized in that, Determining the position of the tumor target point in the head-mounted display coordinate system based on its position in the image coordinate system and the pose of the target probe includes: Based on the probe acoustic beam model, pixel size, and probe geometric calibration, a first mapping relationship between the image coordinate system and the probe coordinate system is determined. Based on the first mapping relationship and the position in the image coordinate system, the position of the tumor target point in the intraoperative ultrasound probe coordinate system is determined; Based on the position of the intraoperative ultrasound probe in the coordinate system and the pose of the target probe, the position of the tumor target point in the head display coordinate system is determined.
3. The intraoperative ultrasound-guided puncture method according to claim 2, characterized in that, Determining the position of the tumor target point in the head-mounted display coordinate system based on its position in the intraoperative ultrasound probe coordinate system and the target probe pose includes: Based on the position of the intraoperative ultrasound probe in the coordinate system and the pose of the target probe, the position of the tumor target in the electromagnetic coordinate system is determined; Determine the alignment transformation between the electromagnetic coordinate system and the head-mounted display coordinate system; The position of the tumor target in the head-mounted display coordinate system is determined based on its position in the electromagnetic coordinate system and the alignment transformation.
4. The intraoperative ultrasound-guided puncture method according to any one of claims 1 to 3, characterized in that, Obtaining the position of the tumor target in the image coordinate system includes: Obtain the target ultrasound image corresponding to the target object; The target ultrasound image is input into a pre-trained segmentation network to obtain the tumor region output by the pre-trained segmentation network. Based on the tumor region, the position of the tumor target point in the image coordinate system is determined.
5. The intraoperative ultrasound-guided puncture method according to any one of claims 1 to 3, characterized in that, Obtaining the needle tip pose includes: Based on the camera sensor and spatial positioning capability of the head-mounted display system, the markings on the puncture needle are detected to obtain the marking pose; The needle tip pose is determined based on the marker pose and the second mapping relationship, wherein the second mapping relationship is used to characterize the correspondence between the marker pose and the needle tip pose.
6. The intraoperative ultrasound-guided puncture method according to any one of claims 1 to 3, characterized in that, Obtaining the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system includes: An electromagnetic positioning sensor generates an electrical signal based on the magnetic field signal generated by a magnetic field generator. Based on the electrical signal and magnetic field model, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system is determined.
7. The intraoperative ultrasound-guided puncture method according to claim 6, characterized in that, The determination of the target probe pose in the electromagnetic coordinate system based on the electrical signal and magnetic field model includes: Based on the electrical signal and the magnetic field model, the first probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system is determined; The pose of the first probe is filtered to obtain the pose of the second probe; The pose of the second probe is time-aligned to obtain the pose of the target probe.
8. The intraoperative ultrasound-guided puncture method according to any one of claims 1 to 3, characterized in that, The process of generating a display interface based on the position in the head-mounted display coordinate system and the needle tip pose includes... Based on the position in the head-mounted display coordinate system and the needle tip pose, determine the puncture information required for the puncture needle tip; The display interface is generated based on the puncture information.
9. The intraoperative ultrasound-guided puncture method according to claim 8, characterized in that, The puncture information includes the tumor target point, the planned path of the puncture needle tip, the needle tip pose, and the deviation between the puncture needle tip and the tumor target point. Generating the display interface based on the puncture information includes: The tumor target, the planned path of the puncture needle tip, the needle tip pose, and the deviation between the puncture needle tip and the tumor target are rendered in the head-mounted display system to obtain the display interface.
10. An intraoperative ultrasonic puncture guidance device, characterized in that, include: The acquisition module is used to acquire the position of the tumor target in the image coordinate system, the target probe pose of the intraoperative ultrasound probe in the electromagnetic coordinate system, and the needle tip pose of the puncture needle tip in the head display coordinate system. The determination module is used to determine the position of the tumor target point in the head-mounted display coordinate system based on the position in the image coordinate system and the pose of the target probe; The guidance module is used to generate a display interface based on the position in the head-mounted display coordinate system and the needle tip pose. The display interface is used to guide the operator to perform intraoperative ultrasound puncture.