Kidney puncture guide method based on multimodal fusion, kidney puncture guide system, and computer device

Abstract

To provide a kidney puncture guide method based on multimodal fusion, a kidney puncture system, and a computer device.SOLUTION: In a first region of a puncture guidance interface, a relative positional relationship between a puncture needle and a kidney in a two-dimensional ultrasonic image and a relative positional relationship between a puncture needle and a kidney in a medical image slice are displayed. In addition, by using a relative positional relationship among an ultrasonic probe, a puncture needle, and a three-dimensional reconstruction model, in a second region of the puncture guidance interface, an ultrasonic probe model, a puncture needle model, and a three-dimensional reconstruction model are displayed in real time. As a result, a doctor can check not only a puncture process from a two-dimensional viewpoint, but also a relative positional relationship among the ultrasonic probe, the puncture needle, and a human body in the puncture process from a three-dimensional viewpoint. As a result, guidance information in the puncture process is enriched, and efficiency and accuracy of puncture are improved.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to the technical fields of medical devices and kidney puncture, and in particular to a kidney puncture guide method, a kidney puncture guide system and a computer device based on multimodal fusion. [Background technology]

[0002] Kidney biopsy is a commonly used procedure to determine the type of kidney disease.

[0003] Conventional puncture guidance systems always rely on optical sensors or six-axis sensors to position various instruments such as probes and puncture needles under ultrasound images. Although this solves the imaging problem, the resolution of ultrasound images is relatively low and the imaging effect on fine structures and lesions is not ideal. In actual surgery, the imaging effect on the target area is relatively poor. In addition, the single image source makes it difficult for doctors to accurately and completely grasp lesion information. As a result, the puncture efficiency is reduced and the puncture risk is increased. Summary of the Invention [Problem to be solved by the invention]

[0004] Based on this, in the face of the above technical challenges, there is a need to provide a renal puncture guidance method, a renal puncture guidance device, a computer device, a computer-readable storage medium, a computer program product, and a renal puncture guidance system based on multimodal fusion, which can improve the efficiency and accuracy of puncture. [Means for solving the problem]

[0005] In a first aspect, the present application provides a method for guiding a renal puncture based on multimodal fusion, the method comprising: acquiring a medical image sequence of the subject under examination, and performing three-dimensional reconstruction based on the medical image sequence to obtain a three-dimensional reconstructed model of the subject under examination, the renal structure of the subject under examination; acquiring a corresponding slice within the three-dimensional reconstruction model of the current ultrasound probe in response to the coordinate registration operation; displaying a fusion image of the slice and a real-time ultrasound image collected by the ultrasound probe in a first area of ​​the puncture guide interface; acquiring an advance angle of the puncture needle, and displaying a puncture guide line on the fusion image in real time based on the advance angle; determining a relative positional relationship between the ultrasound probe model, the puncture needle model, and the 3D reconstruction model based on real-time coordinate information of the ultrasound probe and the real-time coordinate information of the puncture needle; and displaying the ultrasound probe model, the puncture needle model, and the 3D reconstruction model in real time in a second area of ​​the puncture guide interface based on the relative positional relationship.

[0006] In a second aspect, the present application provides a renal puncture guide device based on multimodal fusion, the device comprising: an image acquisition module for acquiring a medical image sequence of the subject under examination and performing three-dimensional reconstruction based on the medical image sequence to obtain a three-dimensional reconstructed model of the subject under examination including a renal structure of the subject under examination; a slice acquisition module for causing a current ultrasound probe to acquire a corresponding slice within the three-dimensional reconstruction model in response to a coordinate registration operation of the coordinate system; a fusion module for displaying a fusion image of the real-time ultrasound image acquired by the ultrasound probe and the slice in a first area of ​​the puncture guidance interface; a guide module for acquiring an advance angle of the puncture needle and displaying a puncture guide line in the fusion image in real time based on the advance angle; and a 3D processing module for determining a relative positional relationship between the ultrasound probe model, the puncture needle model, and the 3D reconstruction model based on real-time coordinate information of the ultrasound probe and the real-time coordinate information of the puncture needle, and displaying the ultrasound probe model, the puncture needle model, and the 3D reconstruction model in real time in the second area of ​​the puncture guidance interface based on this relative positional relationship.

[0007] In a third aspect, the present application provides a computer device including a memory storing a computer program and a processor, the processor executing the computer program to implement the steps of the renal puncture guidance method based on multimodal fusion described in each of the above embodiments.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored therein, the computer program, when executed by a processor, implementing the steps of the renal puncture guidance method based on multimodal fusion described in each of the above embodiments.

[0009] In a fifth aspect, the present application provides a computer program product, the computer program product including a computer program that, when executed by a processor, implements the steps of the method for renal puncture guidance based on multimodal fusion described in each of the above embodiments.

[0010] In a sixth aspect, the present application provides a renal puncture guidance system based on multimodal fusion, including the above-mentioned computer device, a display terminal, an ultrasound probe, a puncture needle, and a magnetic field generator. Both the ultrasound probe and the puncture needle are equipped with electromagnetic sensors. The display terminal, the ultrasound probe, the puncture needle, and the magnetic field generator are each communicably connected to the computer device. [Effects of the Invention]

[0011] The above-mentioned renal puncture guidance method, device, computer device, storage medium, computer program product, and renal puncture guidance system based on multimodal fusion display the relative positional relationship between the puncture needle and the kidney in a two-dimensional ultrasound image and the relative positional relationship between the puncture needle and the kidney in a medical image slice in a first area of ​​a puncture guide interface, and then use the relative positional relationships between the ultrasound probe, the puncture needle, and the 3D reconstructed model to display the ultrasound probe model, the puncture needle model, and the 3D reconstructed model in real time in a second area of ​​the puncture guide interface, thereby allowing the doctor to not only view the puncture process from a two-dimensional angle, but also view the relative positional relationships between the ultrasound probe, the puncture needle, and the human body during the puncture process from a three-dimensional angle, thereby enriching the puncture guidance information during the puncture process and improving the puncture efficiency and puncture accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of a kidney puncture guide system according to one embodiment. FIG. [Figure 2] 1 is a flowchart of a method for guiding renal puncture based on multimodal fusion according to one embodiment. [Figure 3] FIG. 1 is a schematic diagram of the layout of a needle guide interface in one embodiment. [Figure 4] FIG. 10 is a schematic diagram of the layout of a needle guide interface in another embodiment. [Figure 5] 1 is a schematic diagram of a puncture guide interface according to one embodiment. FIG. [Figure 6] FIG. 10 is a schematic diagram of the layout of the needle guide interface. [Figure 7] FIG. 10 is a schematic diagram of the layout of the needle guide interface. [Figure 8] FIG. 1 is a schematic diagram of a puncture guide interface. [Figure 9] FIG. 1 is a schematic diagram of a puncture guide interface. [Figure 10]FIG. 1 is a block diagram of a renal puncture guide device based on multimodal fusion in one embodiment. [Figure 11] FIG. 2 is a diagram illustrating the internal configuration of a computer device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] In order to more clearly understand the objectives, technical solutions and advantages of the present application, the present application will be described in more detail below in conjunction with the drawings and examples. It should be understood that the specific examples described herein are only used to illustrate the present application, and are not used to limit the present application.

[0014] As shown in FIG. 1 , a renal puncture guidance system based on multimodal fusion according to an embodiment of the present application includes a computer device 10, a display terminal 20, an ultrasound probe 30, a puncture needle 40, and a magnetic field generator 50. The magnetic field generator 50 is installed at a fixed position in the application environment. Typically, the magnetic field generator 50 can be used independently if a power supply is available, or the magnetic field generator 50 can be controlled by controlling the power supply and switching signals via a control module built into the computer device using the power supplied to the computer device 10. Both the ultrasound probe 30 and the puncture needle 40 are equipped with electromagnetic sensors 60 for inducing magnetic fields and connecting to the computer device 10. The computer device 10 acquires coordinate information between the ultrasound probe and the puncture needle under the coordinates of the magnetic field generator 50. The display terminal 20, the ultrasound probe 30, the puncture needle 40, and the magnetic field generator 50 are all communicably connected to the computer device 10.

[0015] The computer device acquires a medical image sequence of an object under examination, and performs 3D reconstruction based on the medical image sequence to obtain a 3D reconstruction model of the object under examination, including the renal structures of the object under examination. In response to a coordinate registration operation between the coordinate system of the ultrasound probe and the coordinate system of the 3D reconstruction model, a corresponding slice of the current ultrasound probe in the 3D reconstruction model is acquired based on real-time coordinate information of the ultrasound probe. A fusion image of the real-time ultrasound image collected by the ultrasound probe and the slice is displayed in a first area of ​​the puncture guide interface. The advance angle of the puncture needle is acquired, and a puncture guide line is displayed in the fusion image in real time based on the advance angle. The relative positional relationship between the ultrasound probe model, the puncture needle model, and the 3D reconstruction model is determined based on the real-time coordinate information of the ultrasound probe and the real-time coordinate information of the puncture needle, and the ultrasound probe model, the puncture needle model, and the 3D reconstruction model are displayed in real time in a second area of ​​the puncture guide interface based on the relative positional relationship.

[0016] Therefore, in addition to displaying the relative positional relationship between the puncture needle and the kidney in the two-dimensional ultrasound image and the relative positional relationship between the puncture needle and the kidney in the medical image slice in the first area of ​​the puncture guide interface, the relative positional relationship between the ultrasound probe, the puncture needle, and the 3D reconstructed model is used to display the ultrasound probe model, the puncture needle model, and the 3D reconstructed model in real time in the second area of ​​the puncture guide interface, allowing the doctor to view the puncture process from a two-dimensional angle and also view the relative positional relationship between the ultrasound probe, the puncture needle, and the human body during the puncture process from a three-dimensional angle, thereby enriching the puncture guidance information during the puncture process and improving the puncture efficiency and accuracy.

[0017] The renal puncture guidance method based on multimodal fusion according to the embodiment of the present application can be applied to a computer device such as that shown in Fig. 1. The computer device 10 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc.

[0018] In one embodiment, the present disclosure further provides a renal puncture guidance method based on multimodal fusion, as shown in Figure 2. The method will be described by taking the computer device of Figure 1 as an example. The method includes the following steps:

[0019] In step 202, a medical image sequence of the subject is acquired, and a three-dimensional reconstruction is performed based on the medical image sequence to obtain a three-dimensional reconstructed model of the subject, including the renal lesion of the subject.

[0020] Here, a medical image is an image containing information about the internal structure of the human body obtained using medical imaging technology. In one embodiment, the medical image includes, but is not limited to, a CT image and an MR image. In contrast to ultrasound images, CT and MR can provide high-resolution images and display tissue structure and function. CT and MR can also provide multi-directional, native 3D cross-sectional images, such as stereoscopic images of the brain and spinal cord. Such omnidirectional imaging methods can help doctors more comprehensively understand the location and morphology of lesions and make more accurate diagnoses.

[0021] The medical images include structural information of the kidney of the subject. In one embodiment, a user inputs a medical image sequence of the subject through a computer device, and the computer device performs 3D reconstruction based on the medical image sequence to obtain a 3D reconstructed model of the subject. The 3D reconstructed model includes the kidney lesion of the subject.

[0022] Specifically, the step of acquiring a medical image sequence of the object under examination and performing 3D reconstruction based on the medical image sequence to obtain a 3D reconstructed model of the object under examination further includes multiple substeps. To make the above operations easier for the user, a corresponding operation interface can be provided for each step. The user can perform the operations of each step according to the instructions in the step-by-step operation interface. In one embodiment, the method includes the following steps:

[0023] 1. Import medical image sequences: Provide an import operation interface for users to import medical image sequences.

[0024] 2. Kidney scaling: A kidney scaling interface is provided for the user to determine the kidney position.

[0025] 3. Determine the initial frame of the kidney organ.

[0026] 4. Determine the last frame of the kidney organ. By providing a first frame and last frame confirmation interface, the user can screen the first frame and last frame of the kidney against the imported medical image sequence.

[0027] 5. Confirm kidney segmentation. The computer device displays a kidney segmentation interface in response to the confirmation command of the first and last frames. The user confirms the position of the kidney through the kidney segmentation interface.

[0028] 6. Identify the lesion target. The computer device switches to the target identification interface based on the kidney segmentation identification command. In the target identification interface, the user can selectively add a target. The target is the location of a suspected lesion, which helps the doctor to puncture the lesion target more quickly and effectively reduces the puncture time and the incidence of complications.

[0029] 7. Performing 3D reconstruction: In response to the operation of identifying the lesion target, the computer device performs 3D reconstruction based on the identified image sequence and the position information of the subject when taking the medical images, thereby obtaining a 3D reconstructed model.

[0030] In step 204, in response to the coordinate registration operation, a corresponding slice is obtained within the three-dimensional reconstruction model of the current ultrasound probe.

[0031] Here, a coordinate registration operation is used to align the coordinate system of the ultrasound probe with the 3D coordinates corresponding to the medical image data. An ultrasound image sequence is acquired through the coordinate registration operation. A computer device receives the ultrasound image sequence of the human body and extracts the 3D coordinates of the four vertices of each frame of the ultrasound image sequence to reconstruct the ultrasound data. At the same time, an ICP iterative algorithm is used to perform a 3D alignment operation on the reconstructed ultrasound data and the 3D reconstruction model to obtain a registration transformation matrix. This registration transformation matrix is ​​used to transform the 3D coordinates corresponding to the ultrasound data into the 3D coordinate system corresponding to the medical image data.

[0032] After completing the coordinate registration, the user can use the ultrasound probe to acquire real-time ultrasound images. At the same time, the computer device obtains real-time coordinate information of the ultrasound probe and obtains the cross-sectional position of the 3D reconstruction model of the ultrasound image based on the real-time coordinate information and the registration transformation matrix. Based on the cross-sectional position of the 3D reconstruction model of the ultrasound image, the 3D reconstruction model is cut to obtain a slice corresponding to the ultrasound image. Taking CT as an example, the slice is the corresponding CT slice in the 3D reconstruction model of the current ultrasound probe.

[0033] In step 206, a fusion image of the real-time ultrasound image collected by the ultrasound probe and the aforementioned slice is displayed in a first area of ​​the puncture guide interface.

[0034] In one embodiment, the computer device may include an interaction terminal and a display terminal. The interaction terminal is for displaying an operation interface. A user interacts with the computer device through the interaction terminal. The computer device obtains an operation command corresponding to the interaction. A puncture guidance interface is provided via the display terminal to guide the user to perform kidney puncture.

[0035] In one embodiment, the computer device may be provided with a display terminal having an interactive function such as a touch screen. The display interface of the display terminal is divided into two areas: an operation interface and a puncture guidance interface. The operation interface is for displaying operation guide information and operation buttons for each step of kidney puncture. The computer device obtains operation commands corresponding to interactions through the operation interface. In addition, the puncture guidance interface is displayed on the display terminal to provide the user with guidance for performing kidney puncture.

[0036] The puncture guidance interface may include a first area that displays a real-time ultrasound image acquired by the ultrasound probe and a fused image of a corresponding slice within a three-dimensional reconstruction model of the current ultrasound probe, thereby providing a user with kidney puncture guidelines based on the two images.

[0037] In step 208, the advance angle of the puncture needle is acquired, and a puncture guide line is displayed in real time on the fusion image based on the advance angle.

[0038] Specifically, the puncture needle is equipped with an electromagnetic sensor. The electromagnetic sensor induces a magnetic field and calculates coordinate information of the puncture needle in a magnetic field coordinate system. The computer converts the coordinate information of the puncture needle in the magnetic field coordinate system into the coordinate system of the 3D reconstructed model by coordinate transformation to obtain the advance angle of the puncture needle.

[0039] The magnetic field generator is used to generate an electromagnetic coordinate system. Electromagnetic sensors attached to the puncture rack and the puncture needle can obtain coordinate information of the puncture rack and the puncture needle in real time. Because the ultrasound probe is coupled to the puncture holder, the ultrasound probe receives ultrasound sequence images of the human body and transmits the coordinate information of the puncture holder and the puncture needle to a computer device. The computer device then obtains the needle advance angle based on the coordinate information of the puncture rack and the puncture needle, and displays a puncture guideline in the fusion image in real time based on the advance angle. The puncture guideline provides a guide for the physician when performing two-dimensional puncture. By observing the distance or angle deviation between the puncture guideline and the lesion in the fusion image, the physician can adjust the puncture angle in a timely manner to improve puncture efficiency and accuracy.

[0040] In step 210, the relative positional relationship between the ultrasound probe, the puncture needle, and the 3D reconstruction model is determined based on the real-time coordinate information of the ultrasound probe and the real-time coordinate information of the puncture needle, and the ultrasound probe model, the puncture needle model, and the 3D reconstruction model are displayed in real time in the second area of ​​the puncture guidance interface based on the relative positional relationship.

[0041] In practice, the ultrasound probe and the puncture needle are each equipped with an electromagnetic sensor, and the kidney puncture guide system is equipped with a magnetic field generator. The magnetic field generator generates an electromagnetic coordinate system. The magnetic field generator senses the electromagnetic sensors of the ultrasound probe and the puncture needle to obtain real-time coordinate information of the ultrasound probe and the puncture needle. To facilitate calculation, the transformation relationship between the electromagnetic coordinate system and the coordinate system of the 3D reconstruction model is pre-defined. Furthermore, this transformation relationship is used to obtain the relative positional relationship between the ultrasound probe, the puncture needle, and the 3D reconstruction model, and this relative positional relationship can be used to display the ultrasound probe model, the puncture needle model, and the 3D reconstruction model in real time in the second area of ​​the puncture guide interface.

[0042] That is, the second area of ​​the puncture guidance interface displays the relative positions of the 3D ultrasound probe model, the puncture needle model, and the kidney, whereas the first area of ​​the puncture guidance interface displays the relative positions of the puncture needle and the kidney lesion in the 2D ultrasound image and the relative positions of the puncture needle and the kidney lesion in the medical image slice.

[0043] In some embodiments, multimodal fusion can be performed to achieve better guidance. This typically involves simple matching fusion of ultrasound and a prior modality (e.g., CT or MR). The only information available to the user is a real-time ultrasound image and a corresponding prior modality image of the cross section. This makes it difficult to meet the demand for multi-source information in complex surgical environments. At the same time, a needle holder is typically installed on the ultrasound probe to establish a needle path. To guide the surgeon through the needle insertion, the display interface of the fusion device typically provides a fixed-angle needle guide line that corresponds to the angle adjustment of the needle holder. However, due to factors such as the surgeon's left-handed or right-handed nature and the patient's standing position, the actual holding orientation of the ultrasound probe may be opposite to the orientation indicated by the interface. Therefore, the needle guide line must be manually adjusted to match the actual orientation of the ultrasound probe. This reduces the fault tolerance rate during surgery and increases the risk of mis-puncture.

[0044] In this embodiment, in addition to displaying the relative positional relationship between the puncture needle and the kidney in the 2D ultrasound image and the relative positional relationship between the puncture needle and the kidney in the medical image slice in the first area of ​​the puncture guidance interface, the relative positional relationship between the ultrasound probe, the puncture needle, and the 3D reconstructed model is used to display the ultrasound probe model, the puncture needle model, and the 3D reconstructed model in real time in the second area of ​​the puncture guidance interface, allowing the doctor to observe the puncture process from a 2D angle and the relative positional relationship between the ultrasound probe, the puncture needle, and the human body during the puncture process from a 3D angle, thereby enriching the puncture guidance information during the puncture process and improving the puncture efficiency and accuracy.

[0045] In addition, for the 3D image of the second area, adjustment buttons are provided on the operation screen to make it easy for users to adjust 3D zoom, 360° rotation, and up / down / left / right movement, allowing them to intuitively preview different positions in 3D space.

[0046] In another embodiment, the renal puncture guidance method based on multimodal fusion further includes displaying a current ultrasound probe slice in the three-dimensional reconstruction model in real time in the scanning direction of the ultrasound probe model in the second region.

[0047] That is, the second area not only displays the relative positions of the ultrasound probe model, the puncture needle model, and the 3D reconstructed model in 3D mode, but also displays the slice of the current ultrasound probe in the 3D reconstructed model in real time in the scanning direction of the ultrasound probe model based on the coordinate information of the current ultrasound probe.

[0048] In this way, while moving the ultrasound probe, the user can not only observe the changing real-time ultrasound image and the corresponding medical image slice in real time in the first area of ​​the puncture guidance interface, but also observe the contact position of the ultrasound probe and the medical image slice corresponding to this position in three dimensions, combine the medical image slice with the ultrasound, and determine whether the medical image slice and the real-time ultrasound image correspond to the same position. The increased observation angle and amount of information can improve puncture efficiency and accuracy.

[0049] In one embodiment, the upper left side of the second area displays the rendered frame rate and needle length, the lower left side displays the electromagnetic signal strength, electromagnetic signal value, and guidance for x, y, and z alignment directions of the human body, and the lower right side displays the rendered volumes of all organs, in milliliters.

[0050] The strength of the electromagnetic signal is displayed, making it easy to understand the connection status of the magnetic field generator and ensuring the accuracy of the coordinate information of the needle tip sensor. Guidance on the x, y, and z alignment directions of the human body is displayed to align the body position, enabling quick alignment.

[0051] In another embodiment, the renal puncture guidance method based on multimodal fusion further includes displaying a current ultrasound probe slice in the three-dimensional reconstruction model in real time in a third area of ​​the puncture guidance interface.

[0052] In this embodiment, in the puncture guidance interface, the first area is used to display the fusion image and the 2D puncture guidance information, the second area is used to display the 3D puncture guidance information, and the third area is used to individually display the current ultrasound probe slice in the 3D reconstruction model. Therefore, by using the third area, the user can intuitively view the current ultrasound probe slice in the 3D reconstruction model without being disturbed by other factors, and can know the puncture guidance information in a multidimensional manner in accordance with the information displayed in the first and second areas.

[0053] In one embodiment, the layout of the puncture guidance interface is as shown in Figure 3 or Figure 4. However, the layout of the puncture guidance interface is not limited to the format of Figure 3 or Figure 4. Users can freely adjust the layout of the puncture guidance interface according to their usage habits.

[0054] In one embodiment, the renal puncture guidance method based on multimodal fusion further includes, in response to a needle tip enhancement command, displaying at least one of the needle tip position, safe puncture zone, and needle advance distance of the puncture needle in the fusion image based on real-time coordinates of the puncture needle.

[0055] As shown in Figure 5, when the interaction terminal receives an augmented command from the user to open the needle tip, it can display in real time in the first display area the needle advance status within the needle holder and the coordinate information collected in real time by the sensor attached to the needle. Based on this information, it calculates the corresponding needle tip position and displays it in the first display area as a yellow solid circle point. Based on the needle tip position, it then calculates a rectangular solid-line protection frame representing the safe needle insertion area, and places a light blue solid circle point on the extension of the straight line to help medical staff determine the needle advance distance and ensure quick and accurate needle advancement.

[0056] In this embodiment, the needle tip coordinates are converted into an ultrasound plane through multimodal medical image alignment, thereby guiding the ultrasound probe to puncture accurately, further improving the accuracy of puncture guidance, helping doctors puncture the lesion target more quickly, and effectively reducing the puncture time and complication rate.

[0057] In another embodiment, displaying a fusion image of the real-time ultrasound image acquired by an ultrasound probe and the slice in the first region of the puncture guidance interface includes displaying the real-time ultrasound image acquired by an ultrasound probe and the slice in the first region of the puncture guidance interface with different display parameters to obtain the fusion image, wherein the display parameters include at least one of contrast and transparency.

[0058] That is, the ultrasound image and the slices in the fused image are displayed with different transparency or contrast to allow the user to distinguish between them. In some embodiments, the operation interface further includes a display parameter adjustment interface. The user can adjust the display parameters of the ultrasound image and / or the slices through the operation interface. This allows the fused image to be displayed according to the user's needs.

[0059] In another embodiment, the renal puncture guidance method based on multimodal fusion further includes identifying kidney regions in the real-time ultrasound image and the slice in the fusion image, respectively; marking the contours of the kidney region in the real-time ultrasound image and the kidney region in the slice with different display parameters; and, when the real-time ultrasound image identifies a lesion, marking the contour of the lesion.

[0060] In this embodiment, the kidney regions of the ultrasound image and the slices in the fusion image are marked to distinguish the kidney region identification results between the two images. For example, as shown in FIG. 5, the outline of the kidney region of the ultrasound image is indicated by a yellow dotted line, and the outline of the kidney region of the CT image is indicated by a blue dotted line. At the same time, if the current cutting plane position of the ultrasound probe indicates the target, the system marks the target region with a red dotted line. The outline shape is determined by the drawn target shape and cutting plane angle. Therefore, the user can observe the puncture situation in real time based on the puncture angle, lesion location, etc., which is convenient for the doctor to determine the puncture angle.

[0061] In another embodiment, the renal puncture guidance method based on multimodal fusion further includes, in response to a dual display mode operation on a first display area, displaying a first sub-area and a second sub-area in the first display area, respectively, and displaying the real-time ultrasound image in the first sub-area and the fusion image in the second sub-area, or displaying the fusion image in the first sub-area and the real-time ultrasound image in the second sub-area.

[0062] In this embodiment, when a user triggers the dual display mode, the first region is divided into a first sub-region and a second sub-region. The first sub-region and the second sub-region are used to display images related to the ultrasound image in the dual display mode (also called dual B-mode). In one embodiment, the distribution of the first sub-region and the second sub-region is as shown in FIG. 6 or FIG. 7.

[0063] That is, when the dual display mode is not triggered, the first region displays the fusion image and the puncture guide line. After the dual display mode is triggered, the first region is divided into a first sub-region and a second sub-region. As shown in Figures 8 and 9, the dual display mode further includes two combinations. The interactive terminal can display only one of the combinations. Each combination further includes upper and lower sub-regions.

[0064] Combination 1: First sub-region: Ultrasound B-mode image Second subregion: ultrasound B-mode image and slice fusion image.

[0065] Combination 2: First subregion: Ultrasound B-mode image and slice fusion image Second subregion: ultrasound B-mode image.

[0066] In this embodiment, the dual display mode allows an ultrasound image and a fusion image to be simultaneously displayed in the first area, thereby enriching the puncture guidance information.

[0067] In one embodiment, the first display area is divided into areas such as top, left, center, right, and bottom, as shown in FIG.

[0068] (1) The upper area can display the company logo, patient information, department, and system time (not shown).

[0069] (2) Left area: Text display information related to ultrasound. Specifically, it includes the probe number (5C2AN in Figure 5 indicates the probe number), probe ID (8 in Figure 5 indicates the probe ID), fusion frame rate, and ultrasound parameters, including MI, TIS, depth, gain, frequency, etc.

[0070] (3) Central region: Includes planned needle tip path, puncture guide line, and fused image.

[0071] Furthermore, in the first region, the following is shown: 1) Planned needle tip path. The needle tip position is indicated by a yellow solid circle point. The safe puncture area is calculated from the needle tip position and is indicated by a blue solid rectangle. In addition, several light blue solid circle points are placed on the straight extension line to help medical personnel determine the needle advance distance. 2) Puncture guide line. In other words, the advance angle of the puncture holder can be indicated by the extension of the green dotted line. Here, the green dotted line in the center is the center line. 3) Fusion image: A fusion image of ultrasound and CT cross sections is displayed, where the organ outline in the ultrasound image is indicated by a yellow dotted line, the organ outline in the CT image is indicated by a blue dotted line, and the target area is indicated by a red dotted line.

[0072] (4) Right region: The right region is a scale of ultrasound depth and a black and white rectangular grayscale.

[0073] (5) Bottom area. The leftmost area displays the frame count information of the cached image (current frame count / total frame count) in real time, and the rightmost area displays the color representation of the US, CT, and target contours, making it easier to distinguish between different contours. Here, the blue snowflake shape represents the frozen state. In this state, the contents of the first and second display areas are all frozen.

[0074] The first display area allows the needle tip position to be clearly seen and is indicated by a yellow solid circle point, and also displays a rectangular solid line protection frame which is the safe puncture area of ​​the planned puncture path, and several light blue solid circle points for determining the needle advance distance.

[0075] In another embodiment, the step of acquiring a corresponding slice in the three-dimensional reconstructed model of the current ultrasound probe in response to a coordinate registration operation comprises: acquiring an ultrasound image sequence in response to a coordinate registration command triggered at an operating interface; performing coordinate calibration based on the ultrasound image sequence and the three-dimensional reconstruction model to obtain a registration transformation matrix; Transforming the coordinate system of the ultrasound probe under the coordinate system of the three-dimensional reconstruction model based on the registration transformation matrix; and Obtaining a real-time ultrasound image collected by the ultrasound probe, and obtaining a corresponding slice in the three-dimensional reconstruction model of the current ultrasound probe based on the registration transformation matrix.

[0076] Here, the coordinate registration operation is used to realize the alignment between the three-dimensional coordinates corresponding to the ultrasound data and the three-dimensional coordinates corresponding to the medical image data. Specifically, when the user terminal issues an alignment start command, the interactive terminal retrieves the ultrasound cross-sectional image data collected in real time by the host and the CT sequence image data imported by the user, and performs a three-dimensional alignment operation to obtain a registration transformation matrix.

[0077] Therefore, the coordinate registration operation involves a user collecting ultrasound image sequences according to a coordinate registration request, a computer receiving ultrasound image sequences of the human body, extracting the 3D coordinates of the four vertices of each frame of the ultrasound image sequence, and reconstructing the ultrasound data. At the same time, an ICP iterative algorithm is used to perform a 3D alignment operation on the reconstructed ultrasound data and the 3D reconstruction model to obtain a registration transformation matrix. The registration transformation matrix converts the 3D coordinates corresponding to the ultrasound data into a 3D coordinate system corresponding to the medical image data. A real-time ultrasound image collected by an ultrasound probe is acquired, and the corresponding slice in the 3D reconstruction model of the current ultrasound probe is obtained based on the registration transformation matrix.

[0078] In this embodiment, a registration operation is used to display a fusion image of the ultrasound cross section and the CT cross section in real time based on the position of the ultrasound probe, allowing for rapid and accurate search and location of the lesion.

[0079] Specifically, the present multimodal fusion-based renal puncture guidance method includes the following steps.

[0080] Step 1 involves introducing a medical image sequence.

[0081] Specifically, the DICOM import module imports CT sequence images, which are received human body nuclear magnetic resonance sequence images. The human body CT sequence images are acquired using a large-scale nuclear magnetic resonance device. The DICOM import module specifically performs the following steps:

[0082] First, the interactive terminal acquires the user's selection command and determines the CT sequence image of the kidney to be introduced.

[0083] Then, the interactive terminal reads and displays the sequence image information. In the default state, the interactive terminal automatically checks a set of sequence images, and the user can customize and select a sequence image according to the introduced rules. When the user inputs a selection command, the interactive terminal controls the display content of the corresponding window according to the corresponding command and the current display content, and modifies the information of the specified sequence image. The user can switch between modifying the display content of the window and the image information on the interactive terminal according to the actual situation.

[0084] Finally, based on the selected or chosen sequence image, the user determines whether the currently selected sequence image meets the subsequent fusion and reconstruction conditions, and based on the determination result, introduces the currently selected sequence image.

[0085] Step 2 is 3D reconstruction. Specifically, the 3D reconstruction involves receiving a CT sequence image, which is a nuclear magnetic resonance sequence image of the human body, and then 3D reconstructing and rendering it. The 3D reconstruction includes the following steps:

[0086] First, the interactive terminal acquires the reviewed and suitable CT sequence images for reconstruction, and the user needs to screen the first and last frames of the kidney against the sequence images.

[0087] Next, the interactive terminal switches to a kidney segmentation confirmation mode based on the confirmation command of the start frame, and the user specifies the start frame and end frame of the kidney in the DICOM data by segmenting the kidney.

[0088] Furthermore, the interactive terminal can switch to target confirmation mode based on the kidney segmentation confirmation command, allowing the user to selectively add a target, where the target refers to the location of a suspected lesion, to help doctors quickly puncture the lesion target, effectively reducing the puncture time and complication rate.

[0089] Finally, the interactive terminal performs 3D reconstruction and rendering based on the confirmed CT sequence images and the patient position information from DICOM.

[0090] Step 3 is to align the coordinates.

[0091] Specifically, the module receives ultrasound sequence images of the human body, extracts the 3D coordinates of four vertices from each frame of the ultrasound sequence image, and reconstructs the ultrasound data. A 3D alignment operation is performed on the reconstructed ultrasound data and the rendered CT data to obtain a registration transformation matrix. The registration transformation matrix is ​​used to convert the 3D coordinates corresponding to the ultrasound data into the 3D coordinate system corresponding to the CT data. Based on the alignment results, real-time image fusion of the ultrasound cross-section and the CT cross-section is realized. Here, the ultrasound sequence images of the human body are collected in real time by the host. The alignment and fusion module specifically performs the following steps:

[0092] First, when the user issues a command to start alignment, the interactive terminal calls up the ultrasound cross-sectional image data collected in real time by the host and the CT series image data entered by the user, and performs three-dimensional alignment processing to calculate the registration transformation matrix.

[0093] Next, based on the position information of the ultrasound probe, fusion images of the ultrasound cross section and the CT cross section are displayed in real time, enabling rapid and accurate detection and location of the lesion, which improves diagnostic efficiency in clinical settings and contributes to reducing the risk of medical errors.

[0094] In step 4, the puncture operation is guided.

[0095] Specifically, it receives ultrasound images of the human body acquired by the ultrasound probe and generates an electromagnetic coordinate system using a magnetic field generator. It generates a puncture navigation image based on the coordinate information acquired in real time by a sensor attached to the puncture holder and the current ultrasound image. It also calculates and displays the needle tip position based on the acquired coordinate information. The puncture guide module specifically performs the following steps:

[0096] Based on step 3, a 3D model of the human body (including the skeleton, organs, and tissues), an ultrasound probe model, a puncture needle model, and a puncture holder model can be displayed in real time. As the user moves the puncture holder and the puncture needle model, the interactive terminal receives the needle tip coordinate information acquired in real time by the puncture holder sensor and displays it on the fusion image. Here, the puncture needle model is equipped with an electromagnetic sensor, which is connected to a magnetic field generator system to acquire the real-time 3D coordinate of the needle tip. In this way, the electromagnetic sensor helps medical personnel determine the position of the needle tip and contributes to accurate positioning of the needle tip.

[0097] Step 5: View the system.

[0098] The puncture guidance interface includes a first display area, a second display area, and a third display area.

[0099] The first display area has two display modes. In the first display mode, based on step 3, a fusion image of the ultrasound cross section and the CT cross section corresponding to the position of the ultrasound probe is displayed in real time, and the position information of the puncture needle obtained in step S4 is displayed in real time on both fusion images. The second display mode triggers the dual display mode. Upon receiving a trigger signal, the first display area is divided into a first sub-area and a second sub-area. The display situation includes one of two combinations:

[0100] Combination 1: First sub-region: Ultrasound B-mode image Second sub-region: Ultrasound B-mode image and CT fusion image

[0101] Combination 2: First sub-region: Ultrasound B-mode image and CT fusion image Second sub-region: Ultrasound B-mode image

[0102] For the second display mode of the first display area, the interactive terminal displays two B-mode images in the first sub-area based on the ultrasound image acquired from the host, and displays a fusion image in the second sub-area by default. When the user issues a fusion display switching command, the interactive terminal displays the fusion image in the corresponding first or second sub-area based on the acquired command.

[0103] The second display area displays the 3D model of the human body (including the skeleton, organs and tissues) after 3D reconstruction, and displays the real-time positions of the ultrasound probe model, the puncture holder model, and the puncture needle model in 3D model coordinates, as well as the CT 3D reconstructed cross-sectional area in real time.

[0104] The third display area is used to display the current probe-cut slice of the CT 3D reconstruction in real time based on the ultrasound probe position.

[0105] The specific procedure for displaying the system is as follows.

[0106] First, the first display area displays a puncture guideline. The angle of the guideline corresponds to the needle angle of the hardware puncture holder. When adjusting the puncture angle of the puncture holder, the interactive terminal receives the display angle command input by the user and synchronously displays a puncture guideline at the same angle as the puncture holder in the first display area. This allows medical professionals to refer to the puncture guideline to determine the current puncture path and whether that path overlaps with the target.

[0107] Next, the system quickly determines the position of the convex probe on the surface of the target organ by rotating or translating the 3D ultrasound probe model in the second display area. This allows for real-time acquisition of ultrasound cross-sectional images of the target organ from various angles. Based on the acquired ultrasound images and the alignment results, the host calculates the corresponding probe cross-sectional CT image (which can be displayed in the third display area) in real time, and then fuses the ultrasound and CT images and displays them in the first display area. (The user can adjust the transparency of the CT image, and the interactive terminal reflects the transparency of the CT image.) The organ contour in the ultrasound image is indicated by a yellow dotted line, and the organ contour in the CT image is indicated by a blue dotted line. If the current cross-sectional position of the ultrasound probe can display the target, the system marks the target area with a red dotted line. The contour shape is determined by the target shape and cross-sectional angle.

[0108] Finally, the interactive terminal receives the needle tip opening enhancement command entered by the user, observes in real time in the second display area how the puncture needle advances within the puncture holder and the coordinate information collected in real time by the sensor attached to the puncture needle, calculates the corresponding needle tip position based on this coordinate information, and displays this needle tip position as a yellow solid circle point in the first display area. Furthermore, based on the needle tip position, it calculates a safe puncture area (a rectangular solid line protection frame) and displays multiple light blue solid circle points on the extension of the straight line. This provides medical professionals with auxiliary information to determine the needle advance distance, enabling quick and accurate needle advancement.

[0109] The kidney puncture guidance system and kidney puncture guidance method of the present application aligns multimodal medical images to convert the needle tip point coordinates into an ultrasound plane, and then guides ultrasound to perform precise puncture, thereby further improving the accuracy of puncture guidance and helping doctors puncture the lesion target more quickly, effectively reducing the puncture time and the incidence of complications.

[0110] It should be understood that, although the steps in the flowcharts relating to the above-described embodiments are displayed sequentially according to the direction of the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise clearly stated in the text, there is no strict order restriction on the execution of these steps, and they may be performed in other orders. Furthermore, at least some of the steps in the flowcharts relating to the above-described embodiments may include multiple sub-steps or multiple stages, and these sub-steps or stages may not necessarily be performed at the same time, but may be performed at different times. Furthermore, the execution order of these sub-steps or stages may not necessarily be sequential, and they may be performed alternately or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0111] Based on a similar inventive concept, embodiments of the present application also provide a multimodal fusion-based kidney puncture guide device for implementing the above-mentioned multimodal fusion-based kidney puncture guide method. The solution to the problem provided by this device is similar to the solution described in the above-mentioned method. Therefore, specific limitations of one or more embodiments of the multimodal fusion-based kidney puncture guide device provided below may refer to the limitations of the above-mentioned multimodal fusion-based kidney puncture guide method, and will not be redundantly described here.

[0112] In one embodiment, as shown in Figure 10, a renal puncture guide device based on multimodal fusion is provided. The device comprises: an image acquisition module 1002 for acquiring a medical image sequence of the subject under examination and performing 3D reconstruction based on the medical image sequence to obtain a 3D reconstructed model of the subject under examination including the renal structure of the subject under examination; a slice acquisition module 1004 for acquiring a corresponding slice in the three-dimensional reconstruction model with a current ultrasound probe in response to a coordinate registration operation of the coordinate system; a fusion module 1006 for displaying a fusion image of the real-time ultrasound image acquired by the ultrasound probe and the slice in a first area of ​​the puncture guidance interface; and a guide module 1008 for acquiring the advancement angle of the puncture needle and displaying a puncture guide line in the fusion image in real time based on the advancement angle.

[0113] and a three-dimensional processing module 1010 for determining the relative positional relationship between the ultrasound probe model, the puncture needle model, and the three-dimensional reconstruction model based on the real-time coordinate information of the ultrasound probe and the real-time coordinate information of the puncture needle, and displaying the ultrasound probe model, the puncture needle model, and the three-dimensional reconstruction model in real time in the second area of ​​the puncture guidance interface based on this relative positional relationship.

[0114] In one embodiment, the 3D processing module is also used to display in real time, in a second region, a slice of the current ultrasound probe in the 3D reconstruction model along the scanning direction of the ultrasound probe model.

[0115] In another embodiment, the guide module is also used to display in real time a slice of the current ultrasound probe within the 3D reconstruction model in a third area of ​​the puncture guidance interface.

[0116] In yet another embodiment, the guide module is also used to display at least one of the needle tip position, safe puncture zone, and needle advance distance of the puncture needle in the fused image based on the real-time coordinates of the puncture needle in response to a needle tip enhancement command.

[0117] In one embodiment, the fusion module displays the slices and the real-time ultrasound images acquired by the ultrasound probe in a first area of ​​the puncture guidance interface with different display parameters to generate a fusion image, where the display parameters include at least one of contrast and transparency.

[0118] In one embodiment, the system further includes a marking module, which identifies kidney regions in the real-time ultrasound image and the slices in the fusion image, respectively, and marks contours of the kidney regions in the real-time ultrasound image and the slices with different display parameters, and marks contours of a lesion if a lesion is identified in the real-time ultrasound image.

[0119] In one embodiment, the system further includes a switching module that, in response to a dual display mode operation for a first display region, displays a first sub-region and a second sub-region in the first display region, respectively, by displaying a real-time ultrasound image in the first sub-region and the fusion image in the second sub-region, or by displaying the fusion image in the first sub-region and the real-time ultrasound image in the second sub-region.

[0120] In one embodiment, the slice acquisition module acquires an ultrasound image sequence in response to a coordinate registration command triggered by the operation interface. Then, coordinate registration is performed based on the ultrasound image sequence and the 3D reconstruction model to obtain a registration transformation matrix. The registration transformation matrix is ​​used to transform the coordinate system of the ultrasound probe into the coordinate system of the 3D reconstruction model. Furthermore, the ultrasound probe acquires the acquired real-time ultrasound images, and the current ultrasound probe acquires corresponding slices in the 3D reconstruction model based on the registration transformation matrix.

[0121] Each module in the above-described renal puncture guide device based on multimodal fusion can be implemented in whole or in part by software, hardware, or a combination thereof. These modules can be integrated into a processor in a computer device as hardware, or can exist independently. They can also be stored in the memory of a computer device as software, allowing the processor to access them and perform the operations corresponding to each module.

[0122] In one embodiment, a computer device is provided. This computer device is a terminal, and its internal structure is shown in Figure 11. The computer device includes a processor, memory, a communication interface, a display, and an input device, all connected via a system bus. The processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and the computer program stored in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication may be achieved via Wi-Fi, a mobile cellular network, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program realizes the renal puncture guidance method based on multimodal fusion. The display of the computer device is a liquid crystal display or an electronic ink display. The input device of the computer device may be a touch layer coated on the display, a button, a trackball, or a touchpad installed on the housing of the computer device, or an external keyboard, touchpad, or mouse. Those skilled in the art will understand that the configuration shown in Figure 11 is merely a block diagram of a partial configuration related to the scheme of the present application, and is not intended to limit the computer device to which the scheme of the present application is applied. A specific computer device may have more or fewer components than those shown, or may combine some components or have a different component arrangement.

[0123] In one embodiment, a computer device is provided, including a memory storing a computer program and a processor. When the processor executes the computer program, the steps of the renal puncture guidance method based on multimodal fusion described in each of the above embodiments are realized. In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by the processor, the steps of the renal puncture guidance method based on multimodal fusion described in each of the above embodiments are realized.

[0124] In one embodiment, a computer program product is provided, which stores a computer program that, when executed by a processor, implements the steps of the renal puncture guidance method based on multimodal fusion described in the above embodiments.

[0125] Those skilled in the art should understand that all or part of the steps of the methods described in the above embodiments can be achieved by controlling associated hardware using a computer program. This computer program can be stored in a non-volatile computer-readable storage medium. When this computer program is executed, the steps described in each of the above method embodiments are realized. Any reference to a memory, database, or other medium used in each embodiment provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. The RAM may be in various forms, such as, but not limited to, static random access memory (SRAM) or dynamic random access memory (DRAM). A database according to various embodiments provided herein may include at least one of a relational database and a non-relational database. A non-relational database may include, but is not limited to, a distributed database based on blockchain. A processor according to various embodiments provided herein may be, but is not limited to, a general-purpose processor, a central processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, but as long as there is no contradiction in the combination of these technical features, it should be considered to fall within the scope described in this description.

[0127] The above-described embodiments only represent some of the embodiments of the present application, and the descriptions are more specific and detailed. Therefore, they should not be understood as limitations on the scope of the claims of the present application. Those skilled in the art may make some modifications and improvements without departing from the spirit of the present application. It should be noted that all such modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application should be governed by the appended claims.

Claims

1. 1. A multimodal fusion-based renal puncture guidance method, comprising: acquiring a medical image sequence of the subject under examination, and performing three-dimensional reconstruction based on the medical image sequence to obtain a three-dimensional reconstructed model of the subject under examination, the renal structure of the subject under examination; acquiring a corresponding slice within the three-dimensional reconstruction model of the current ultrasound probe in response to the coordinate registration operation; displaying a fusion image of the slice and a real-time ultrasound image collected by the ultrasound probe in a first area of ​​the puncture guide interface; acquiring an advance angle of the puncture needle, and displaying a puncture guide line on the fusion image in real time based on the advance angle; determining a relative positional relationship between an ultrasound probe model, a puncture needle model, and the 3D reconstruction model based on real-time coordinate information of the ultrasound probe and the puncture needle; and displaying the ultrasound probe model, the puncture needle model, and the 3D reconstruction model in real time in a second area of ​​the puncture guide interface based on the relative positional relationship.

2. 2. The renal puncture guidance method based on multimodal fusion according to claim 1, further comprising: displaying a corresponding slice in the three-dimensional reconstruction model of the current ultrasound probe in real time in the scanning direction of the ultrasound probe model in the second region.

3. The renal puncture guidance method based on multimodal fusion according to claim 1 or 2, further comprising: displaying a corresponding slice in the 3D reconstruction model of the current ultrasound probe in real time in a third area of ​​the puncture guidance interface.

4. 2. The kidney puncture guidance method based on multimodal fusion according to claim 1, further comprising: displaying at least one of the needle tip position, the safe puncture zone, and the needle advance distance of the puncture needle in the fusion image based on the real-time coordinates of the puncture needle in response to a needle tip enhancement command.

5. The step of displaying a fusion image of the slice and the real-time ultrasound image collected by the ultrasound probe in a first area of ​​the puncture guide interface includes: and displaying the real-time ultrasound image acquired by the ultrasound probe and the slices in a first area of ​​the puncture guide interface with different display parameters to obtain a fusion image; The method for guiding kidney needles based on multimodal fusion according to claim 1, 2 or 4, wherein the display parameters further include at least one of contrast and transparency.

6. identifying kidney regions in the real-time ultrasound image and the slices in the fused image, respectively; marking a kidney region of the real-time ultrasound image and a contour of the kidney region of the slice with different display parameters; and 10. The method of claim 1, further comprising: marking the contour of a lesion when the real-time ultrasound image identifies the lesion.

7. displaying a first sub-region and a second sub-region in the first display region in response to a dual display mode operation on the first display region; and 2. The renal puncture guidance method based on multimodal fusion of claim 1, further comprising: displaying the real-time ultrasound image in the first sub-region and the fusion image in the second sub-region; or displaying the fusion image in the first sub-region and the real-time ultrasound image in the second sub-region.

8. Responsive to the coordinate registration operation, obtaining a corresponding slice within the three-dimensional reconstruction model of the current ultrasound probe includes: acquiring an ultrasound image sequence in response to a coordinate registration command triggered at an operating interface; performing coordinate registration based on the ultrasound image sequence and the three-dimensional reconstruction model to obtain a registration transformation matrix; Transforming the coordinate system of the ultrasound probe under the coordinate system of the three-dimensional reconstruction model based on the registration transformation matrix; and 2. The renal puncture guidance method based on multimodal fusion according to claim 1, further comprising: acquiring real-time ultrasound images collected by the ultrasound probe; and acquiring corresponding slices in the three-dimensional reconstruction model of the current ultrasound probe based on the registration transformation matrix.

9. A computer device including a memory in which a computer program is stored and a processor, 10. A computer device, characterized in that the processor, when executing the computer program, implements the steps of the method for renal puncture guidance based on multimodal fusion according to claim 1, 2, 4, 7 or 8.

10. 1. A multimodal fusion-based renal puncture guidance system, comprising: a computer device, a display terminal, an ultrasound probe, a puncture needle, and a magnetic field generator according to claim 9; The ultrasonic probe and the puncture needle are both provided with electromagnetic sensors, A renal puncture guidance system based on multimodal fusion, characterized in that the display terminal, the ultrasound probe, the electromagnetic sensor, and the magnetic field generator are each communicatively connected to the computer device.

Images