Prostate puncture guide method based on multimodal fusion, prostate puncture guide apparatus, computer device, and prostate puncture guide system

Abstract

To improve surgical experience of a patient by shortening a puncture process time, improving puncture accuracy, and suppressing repeated puncture.SOLUTION: A prostate model and an ultrasonic probe model having at least one probing surface are displayed in the same display region, and contents dynamically changing in the display region are displayed in real time on the basis of coordinates acquired by the ultrasonic probe in real time and ultrasonic image information, so that a positional relationship between the at least one probing surface and a target identifier in the prostate model can be grasped, and an intersection between the probing surface corresponding to a puncture plate mounted on the ultrasonic probe and the target identifier can be quickly and accurately known. Further, according to the association relationship between the specific puncture hole on the puncture plate and the target identifier displayed in another display region, the sampling device is inserted into the specific puncture hole to complete the sampling quickly, thereby shortening the time for sampling, reducing the number of sampling needles, avoiding repeated puncturing, and reducing the operation difficulty and the learning time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present application relates to the technical fields of medical devices and prostate puncture, and in particular to a prostate puncture guide method, a prostate puncture guide device, a computer device, and a prostate puncture guide system based on multimodal fusion. [Background technology]

[0002] Prostate needle biopsy involves collecting prostate tissue via prostate puncture and conducting a pathological examination, and is extremely important for the clinical diagnosis of prostate cancer, particularly for the definitive diagnosis of prostate cancer.

[0003] Current needle guidance systems typically rely on position sensors to obtain biopsy needle position information, and then use related algorithms to predict the needle tip's position on ultrasound or fusion images. They also use highlighting or other special marks to enhance the needle tip's visibility, allowing accurate needle tip location even in poor needle tip imaging conditions. However, to ensure complete sampling, prostate puncture typically requires multiple needle insertions. Therefore, under existing technology, the puncture process can be time-consuming, inaccurate, and require repeated insertions for inexperienced physicians, patients with poor pain tolerance, or those with short anesthesia periods. This can cause severe pain during the puncture and prevent the procedure from being completed successfully. These issues further increase the difficulty of operation for medical professionals, increase their learning time, and reduce the patient's surgical experience. Summary of the Invention [Problem to be solved by the invention]

[0004] In light of this, it is necessary to provide a prostate puncture guide method, prostate puncture guide device, computer device, and prostate puncture guide system based on multimodal fusion that can solve the problems existing in the prior art, such as a time-consuming puncture process, low puncture accuracy, repeated punctures, and a poor surgical experience for patients. [Means for solving the problem]

[0005] A method for guiding prostate puncture based on multimodal fusion according to a first aspect of the present application includes: acquiring a medical image sequence of an object under examination, and performing three-dimensional reconstruction based on the medical image sequence to obtain a three-dimensional reconstructed model of the object under examination; the three-dimensional reconstruction model includes a prostate structure of a subject, the prostate structure including a plurality of target markers, the target markers including at least one of a prostate puncture mark and a target area mark; Acquiring real-time coordinates of an ultrasound probe and an ultrasound image of an object to be examined, and obtaining corresponding cross-sectional images in the three-dimensional reconstruction model of at least one probe plane of the ultrasound probe based on the real-time coordinates; displaying the three-dimensional reconstruction model in a first display area, and displaying an ultrasound probe model of the ultrasound probe based on the real-time coordinates; and displaying a corresponding cross-sectional image in the three-dimensional reconstruction model of the at least one probe plane of the ultrasound probe in at least one probe area of ​​the ultrasound probe model; displaying a fusion image of the cross-sectional image and the ultrasound image corresponding to at least one of the exploration planes in a second display area; If the search plane intersects with at least one of the target markers in the three-dimensional reconstruction model, projecting the target marker onto the fused image to obtain a two-dimensional image of the target marker; and associating the two-dimensional image of the target marker with a corresponding puncture hole of a puncture plate attached to the ultrasonic probe, and displaying the association between the two-dimensional image of the target marker and the puncture hole in the second display area.

[0006] In one embodiment, the step of projecting the target marker onto the fused image to obtain a two-dimensional image of the target marker comprises: Obtaining first coordinate information of the inspection plane and second coordinate information of the target marker intersecting the inspection plane; and and two-dimensionally projecting the landmark onto the fused image of the search plane based on the first coordinate information and the second coordinate information.

[0007] In one embodiment, at least one said prostate puncture mark is generated within a two-dimensional projection marked by said target area mark in said fused image.

[0008] In one embodiment, the method includes the steps of: in response to a first operation, acquiring at least one landmark selected by the first operation; highlighting the target marker in the first display area; and / or and hiding the target marker in the second display area.

[0009] In one embodiment, each sub-display area of ​​the third display area displays a cross-sectional image of a cross-section corresponding to at least one of the exploration planes.

[0010] In one embodiment, in response to a needle tip enhancement command, at least one of the needle tip position of the sampling device and a safe puncture zone is displayed in the fused image based on real-time coordinates of the sampling device.

[0011] In one embodiment, the ultrasound image and the cross-sectional image are displayed with different display parameters, and / or the contours of the prostate region in the ultrasound image and the cross-sectional image in the fusion image are marked with different marking parameters; The display parameters include at least one of contrast and transparency, and the marking parameters include a color identifier for an outline.

[0012] A prostate puncture guide device based on multimodal fusion according to a second aspect of the present application includes a 3D reconstruction module, an ultrasound data acquisition module, a model display module, a fusion image display module, and a puncture-related module, The 3D reconstruction module acquires a medical image sequence of the object under examination, and performs 3D reconstruction based on the medical image sequence to obtain a 3D reconstructed model of the object under examination; the three-dimensional reconstruction model includes a prostate structure of a subject, the prostate structure including a plurality of target markers, the target markers including at least one of a prostate puncture mark and a target area mark; The ultrasound data acquisition module acquires real-time coordinates of an ultrasound probe and an ultrasound image of an object to be examined, and acquires corresponding cross-sectional images of at least one probe plane of the ultrasound probe in the three-dimensional reconstruction model based on the real-time coordinates; the model display module displays the three-dimensional reconstruction model in a first display area, and displays an ultrasound probe model of the ultrasound probe according to the real-time coordinates, and displays a cross-sectional image corresponding to at least one exploration plane of the ultrasound probe model; the fusion image display module displays a fusion image of the cross-sectional image and the ultrasound image corresponding to at least one of the exploration planes in a second display area; and when the exploration plane intersects with at least one of the target markers in the three-dimensional reconstruction model, projects the target marker onto the fusion image to obtain a two-dimensional image of the target marker; The puncture-related module associates the two-dimensional image of the target marker with the corresponding puncture hole of the puncture plate attached to the ultrasound probe, and displays the association between the two-dimensional image of the target marker and the puncture hole in the second display area.

[0013] A computer device according to a third aspect of the present application includes a memory and a processor, The memory stores a computer program, When the processor executes the computer program, it implements the method according to any one of the above embodiments.

[0014] A prostate puncture guide system based on multimodal fusion according to a fourth aspect of the present application includes a magnetic field generator, an electromagnetic sensor, an ultrasound probe, a puncture plate, a sampling device, a display terminal, and the computer device according to claim 9; A puncture plate is attached to the ultrasonic probe, and electromagnetic sensors are installed on both the ultrasonic probe and the sampling device; The magnetic field generator, the display terminal, the ultrasound probe and the electromagnetic sensor are all electrically connected to a computing device. [Effects of the Invention]

[0015] The present application displays a prostate model of a subject and an ultrasound probe model having at least one probe plane in the same display area. Based on coordinate information and ultrasound image information collected in real time by the ultrasound probe, dynamic changes are displayed in real time within the display area, allowing the user to grasp the positional relationship between the at least one probe plane and a target marker within the prostate model. This allows the user to quickly and accurately determine the intersection of the probe plane and the target marker corresponding to the puncture plate attached to the ultrasound probe. At the same time, the relationship between specific puncture holes and the target marker on the puncture plate displayed in a separate display area is used to insert a sampling device into a specific puncture hole to quickly complete sampling. This shortens the time required for sample collection, reduces the number of sampling needles, and avoids repeated punctures, thereby reducing the difficulty and learning time for medical professionals and improving the surgical experience for patients. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a schematic diagram illustrating the configuration of a prostate puncture guide system according to one embodiment. [Figure 2] 1 is a flowchart of a method for guiding a prostate puncture in one embodiment. [Figure 3] FIG. 2 is a structural schematic diagram of a puncture plate in one embodiment. [Figure 4] FIG. 1 is a schematic diagram showing a related guide system between a prostate puncture mark and a puncture plate in one embodiment. [Figure 5] FIG. 10 is a schematic diagram showing a related guide system between the target area mark and the lancing plate in one embodiment. [Figure 6] FIG. 10 is a schematic diagram showing a related guide system between the target area mark and the lancing plate in another embodiment. [Figure 7] FIG. 10 is a schematic diagram of selecting a prostate puncture mark in one embodiment. [Figure 8] FIG. 1 is a schematic diagram of the layout of a needle guidance interface in one embodiment. [Figure 9]FIG. 10 illustrates the link between needle tip augmentation and a three-dimensional model in one embodiment. [Figure 10] FIG. 1 is a schematic diagram of a specific application scenario in one embodiment. [Figure 11] FIG. 1 is a configuration block diagram of a prostate puncture guide device in one embodiment. [Figure 12] FIG. 1 is a block diagram of a computer device according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application will be described in more detail with reference to the drawings and examples. It should be understood that the specific examples described herein are for the purpose of illustrating the present application, and are not intended to limit the present application.

[0018] As used herein, terms such as "comprises," "including," "having," and any variations thereof do not imply an exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the explicitly listed steps or units and may include other steps or units that are not explicitly listed or that are inherent to the process, method, product, or apparatus.

[0019] As used herein, the term "plurality" means two or more (including two), unless specifically limited otherwise.

[0020] Terms such as "first," "second," and the like, as used herein, are intended to distinguish similar objects in naming terms, but are not intended to limit the objects themselves, nor to imply relative importance, number, a particular order, or primary relationship of the technical features depicted. It should be understood that these terms are interchangeable under appropriate circumstances without departing from the scope of the present application. For example, a "first sub-region" could be referred to as a "second sub-region." Similarly, a "second sub-region" could be referred to as a "first sub-region."

[0021] The following industry terms are used in this specification:

[0022] Multimodal fusion / ultrasound fusion refers to the technology of combining US images with CT / MRI / PET cross-sectional images and displaying them synchronously on the same screen in real time. This technology incorporates CT / MRI / PET images into the ultrasound system, uses multiple positioning methods to match the ultrasound and CT / MRI / PET cross-sectional images, and uses magnetic field positioning to link the ultrasound and CT / MRI / PET images in real time. This combines the high resolution of CT / MRI / PET images with the real-time performance, ease of operation, and custom cross-sectional display features of US, allowing multiple imaging information to complement each other and accurately locate lesions.

[0023] CT (Computed Tomography), MRI (Magnetic Resonance Imaging) and PET (Positron Emission Tomography) will be explained in detail below. CT stands for Computed Tomography, a powerful medical condition detection device used to scan cross sections of the body. CT is based on the fact that different tissues in the human body absorb and transmit X-rays at different rates. It measures the human body with highly sensitive equipment, inputs the data obtained from the measurement into an electronic computer, and after processing the data, obtains a cross-sectional or three-dimensional image of the examined part of the human body, which can detect minute lesions anywhere in the body. CT is often used to examine the chest and abdomen of tumor patients. Chest CT scans display structures more clearly, and their sensitivity in detecting chest lesions and accuracy in identifying lesions are superior to standard chest X-rays. Chest CT scans are crucial, especially in diagnosing early-stage lung cancer. However, because CT scans use X-rays, the human body is exposed to a certain amount of radiation, and CT scans are not very clear when imaging soft tissues.

[0024] MRI stands for nuclear magnetic resonance imaging. MRI tests distinguish between different image features based on the content of hydrogen elements in the human body, and then use these different image features to determine the function of the corresponding organ and the nature of the lesion. MRI scans are primarily used to diagnose diseases of the cranial nervous system and to determine injuries to soft tissues and ligaments, such as common meningeal tumors, colloid tumors, astrocytic tumors, and cartilage ligament injuries.

[0025] Regarding slices, CT, MRI, and PET scans, they are used to acquire cross-sectional images of the human body at fixed angles. Based on the human body, slices are divided into sagittal, coronal, and transverse planes. The sagittal plane divides the human body into two parts, left and right. A cross section of the left and right sides is called the sagittal plane, and a cross section that is equal to the left and right sides is called the midsagittal plane. The coronal plane cuts the human body vertically along the left-right direction, dividing it into two parts, front and back. The transverse plane is a cross section that is parallel to the horizontal plane and perpendicular to the vertical axis when the human body is standing, cutting the human body horizontally along the up-down direction, dividing it into two parts, front and back.

[0026] The term "and / or" used in this application is merely an expression for describing a relation between related objects, and indicates that a three-way relationship may exist. For example, A and / or B can represent three situations: when only A exists, when A and B exist simultaneously, and when only B exists.

[0027] To provide a clearer understanding of the objectives and advantages of the present application, a detailed description will be given of the circumstances under which the inventors discovered the problems existing in the existing technology.

[0028] Prostate needle biopsy involves the extraction of prostate tissue for pathological examination. It plays a crucial role in clinical diagnosis of prostate disease, especially in the definitive diagnosis of prostate cancer. The most common needle techniques used in the industry include transrectal and transperineal needle biopsies. Each of these methods has its own advantages and disadvantages. Regardless of the method, ensuring complete sampling requires that the entire prostate be covered by the needle. A prostate needle biopsy typically requires the use of 10 to 12 needles. These needles must be inserted into different areas of the prostate. Therefore, anesthesia requirements are high, requiring not only accurate location of the anesthesia site but also the selection of an appropriate needle length for accurate anesthesia. Furthermore, the sampling time must be kept as short as possible. Insufficient anesthesia, excessive needle insertion time, repeated insertion attempts due to inaccurate insertion, or repeated insertion due to inability to identify previously inserted holes can cause severe pain for the patient during the biopsy and prevent the procedure from being completed successfully.

[0029] To solve the above problems, this application provides a prostate puncture guide method, a prostate puncture guide device, a computer device, and a prostate puncture guide system based on multimodal fusion. A prostate model of a subject and an ultrasound probe model with at least one probe plane are displayed in the same display area. Based on coordinate information and ultrasound image information collected in real time by the ultrasound probe, dynamically changing content within the display area is displayed in real time, thereby grasping the positional relationship between at least one probe plane and a target marker (target ID) within the prostate model. This allows for quick and accurate understanding of the intersection of the probe plane and target marker corresponding to the puncture plate attached to the ultrasound probe. Furthermore, based on the correlation between specific puncture holes and target markers on the puncture plate displayed in a separate display area, a sample collection device can be inserted into a specific puncture hole to quickly complete sample collection. This shortens the time required for sample collection, reduces the number of sampling needles, and avoids repeated punctures, thereby reducing the difficulty and learning time for medical professionals and improving the surgical experience for patients.

[0030] As shown in FIG. 1 , the prostate puncture guidance system based on multimodal fusion provided by the embodiment of the present application includes a computing device 10, a display terminal 20, an ultrasound probe 30, a sampling device 40, a magnetic field generator 50, an electromagnetic sensor 60, and a puncture plate 70. The magnetic field generator 50 is installed at a fixed position within the application environment. Typically, the magnetic field generator 50 can operate independently under power supply, and is operated by using the power supplied to the computing device 10 to control the power supply and switch signals through a control module built into the computing device. The ultrasound probe 30 and the sampling device 40 are each equipped with an electromagnetic sensor 60. A puncture plate 70 is fixedly attached to the ultrasound probe 30. Each electromagnetic sensor is used to detect magnetic fields. The three-dimensional coordinate information and direction information of each electromagnetic sensor 60 within the magnetic field region generated by the magnetic field generator 50 are acquired, and the information is then connected to the computing device 10 and transmitted back to the computing device 10. The computing device 10 then acquires the coordinate information of the ultrasound probe 30 and the sampling device 40 in the coordinate system of the magnetic field generator 50. The display terminal 20 , the ultrasound probe 30 , and the sampling device 40 are each electrically connected to the computer device 10 .

[0031] The computer device performs the following operations: first, acquire a medical image sequence of an object to be examined, and perform 3D reconstruction based on the medical image sequence to obtain a 3D reconstruction model of the object to be examined. The 3D reconstruction model includes a prostate structure of the object to be examined. The prostate structure includes a plurality of target markers. The target markers include at least one of a prostate puncture mark and a target area mark. Next, acquire real-time coordinates of an ultrasound probe and an ultrasound image of the object to be examined, and obtain a cross-sectional image in the 3D reconstruction model of at least one exploration plane of the ultrasound probe based on the real-time coordinates. Next, display the 3D reconstruction model in a first display area, and display an ultrasound probe model of the ultrasound probe based on the real-time coordinates, and display a cross-sectional image corresponding to the exploration plane in at least one exploration plane of the ultrasound probe model. Then, display a fusion image of the cross-sectional image and the ultrasound image corresponding to at least one of the exploration planes in a second display area, and if the exploration plane intersects with at least one target marker in the 3D reconstruction model, project the target marker onto the fusion image to obtain a 2D image of the target marker. Finally, the two-dimensional image of the target marker is associated with the corresponding puncture hole of the puncture plate attached to the ultrasound probe, and the association between the two-dimensional image of the target marker and the puncture hole is displayed in the second display area.

[0032] This allows a prostate model of the subject and an ultrasound probe model with at least one probe plane to be displayed in the same display area. Dynamic changes are displayed in real time based on coordinate information and ultrasound image information collected by the ultrasound probe. This allows the user to grasp the positional relationship between at least one probe plane and a target ID within the prostate model. This allows the user to quickly and accurately grasp the intersection of the probe plane and target ID corresponding to the puncture plate attached to the ultrasound probe. Furthermore, a separate display area displays the relationship between specific puncture holes on the puncture plate and the target ID. By inserting a sampling device into the corresponding puncture hole, sampling can be completed quickly. This method shortens sampling time, reduces the number of sampling needles, and avoids repeated punctures, reducing operational difficulty and learning time for medical professionals while improving the patient's surgical experience.

[0033] Although the steps shown in the flowchart of FIG. 2 are displayed sequentially according to the direction of the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise clearly stated in the text, there is no strict order restriction on the execution of these steps, and they may be performed in other orders. Furthermore, at least some of the steps in FIG. 2 may include multiple substeps or stages. These substeps or stages do not necessarily have to be completed at the same time, and may be performed at different times. Furthermore, the execution order of these substeps or stages does not necessarily have to be consecutive, and they may be performed in order or alternately with other steps or at least some of the substeps or stages of other steps.

[0034] FIG. 2 is a flowchart of a prostate puncture guidance method based on multimodal fusion provided by an embodiment of the present disclosure. This method is performed by a prostate puncture guidance device based on multimodal fusion. This device can be implemented by software and / or hardware and is generally integrated into an electronic device. One method of integration is to install a client on a corresponding electronic device. This client may be an application for displaying a puncture guidance interface, a web client, or a sub-application running within the execution environment of a parent application. This electronic device may be a mobile device such as a personal computer, laptop, smartphone, tablet, smartwatch, or personal digital assistant (PDA), or may be another device such as a desktop computer. It may also be various medical devices such as an ultrasound fusion device.

[0035] In the following examples, optional features and examples are provided for each example. The features described in the examples can be combined with each other to form multiple optional schemes. Each numbered example should not be considered as just one technical solution.

[0036] In one embodiment, a kind of multimodal fusion-based prostate puncture guidance method is provided, as shown in Figure 2. The method includes the following steps:

[0037] In step 201, a medical image sequence of an object to be examined is acquired, and three-dimensional reconstruction is performed based on the medical image sequence to obtain a three-dimensional reconstruction model of the object to be examined. The three-dimensional reconstruction model includes a prostate structure of the object to be examined. The prostate structure includes a plurality of target markers. The target markers include at least one of a prostate puncture mark and a target region mark.

[0038] Here, a medical image refers to an image containing structural information about the inside of the human body obtained by medical imaging technology. In one embodiment, the medical image includes, but is not limited to, a CT image, an MRI image, and a PET image. Compared with ultrasound images, CT and MRI can provide multidirectional, native, 3D cross-sectional images. Such omnidirectional imaging methods can help physicians more comprehensively understand the location and morphology of lesions and make more accurate diagnoses.

[0039] The medical images also include structural information of the patient's prostate region. In one embodiment, a user inputs a sequence of medical images of the patient into a computer device, and the computer device performs 3D reconstruction based on the sequence of medical images to obtain a 3D reconstructed model of the subject. The 3D reconstructed model includes the prostate structure of the subject. The 3D reconstructed model with the prostate structure generated by the 3D reconstruction can accurately reflect the true state of the prostate organ of the subject, which is convenient for medical personnel to perform accurate biopsy and sampling.

[0040] Specifically, the step of acquiring a medical image sequence of a patient and performing 3D reconstruction based on the medical image sequence to obtain a 3D reconstructed model of the patient may include multiple substeps. Here, to facilitate the user's operation, an operation interface corresponding to each step is provided. The user can perform each step according to the guidance of the operation interface. For example, one embodiment may include the following substeps:

[0041] 1. Importing medical image sequences. By providing an import operation interface, users can import medical image sequences. 2. Prostate Positioning: Prostate positioning interface is provided for the user to determine the position of the prostate. 3. Determine the first frame of the prostate organ. 4. Determine the last frame of the prostate organ: By providing a confirmation interface for the first and last frames, users can select the first and last frames of the prostate from the imported medical image sequence. 5. Prostate division confirmation: The computer device displays a prostate division interface according to the confirmation command of the first and last frames, and the user confirms the position of the prostate through the prostate division interface. 6. Lesion target confirmation. The computer device switches to a target confirmation interface in response to a prostate segmentation confirmation command. In the target confirmation interface, the user can selectively add a target. The target here refers to a suspected lesion site, which helps the physician more quickly puncture the lesion target and effectively reduces the puncture time and complication rate. 7. 3D Reconstruction: In response to the lesion target confirmation operation, the computer device performs 3D reconstruction based on the confirmed image sequence and the patient's position information when the medical image is taken, to obtain a 3D reconstruction model. 8. Needle placement by the system. The computer device switches to the system's needle placement interface in response to a confirmation command for the 3D reconstruction model. In the system's needle placement interface, the user can select the default layout of 12 needle puncture sites, or selectively add other puncture sites. After confirming the final number and layout of puncture sites, the computer device extracts the sequence with the largest cross-sectional area from the medical image sequence and uses it as the projection target. Using local deformation technology, the layout of the puncture sites is projected onto the sequence image, and then a 2D to 3D conversion is performed based on the position information, pixel information, and other information of each puncture site in the sequence image. A corresponding number of irregular ellipsoid-shaped prostate puncture marks with different volumes and positions are generated within the prostate structure of the 3D reconstruction model.

[0042] The target marker is a mark for marking a target area within the prostate structure. For the specific origin of the target marker, the prostate puncture mark, please refer to the detailed description in the above step 8 "Needle Placement by the System," and will not be repeated here. For the target area mark, please refer to the description in the above step 6 "Lesion Target Confirmation." The target area mark may be a suspicious lesion area drawn by the user or another region of interest drawn, and the principle of its generation in the prostate structure is similar to the principle of generating the puncture hole position in step 8. The user selects one or more target areas from some or all of the medical image sequences, and then converts them from two dimensions to three dimensions to generate a corresponding number of target area marks with irregular ellipsoid shapes and different volumes and positions within the prostate structure of the three-dimensional reconstruction model.

[0043] In one implementation scenario, the user first imports CT / MRI / PET or other image data of the subject and performs other preparatory steps, such as drawing prostate puncture marks, before entering the prostate puncture guidance step, which displays a 3D prostate model reconstructed based on the previously acquired CT / MRI / PET or other image data in a specific or full-screen display area of ​​a display device or monitor. This model contains multiple previously drawn prostate puncture marks, whose distribution area and positions are determined by the previously drawn target markers.

[0044] In one embodiment, the prostate needle mark is determined by the needle template of the system, and the number of needles is 12.

[0045] In step 202, real-time coordinates of an ultrasound probe and an ultrasound image of an object to be examined are obtained, and corresponding cross-sectional images in a 3D reconstruction model of at least one probe plane of the ultrasound probe are obtained based on the real-time coordinates.

[0046] The real-time coordinates of the ultrasound probe are located by an electromagnetic coordinate system emanating from a magnetic field generator based on electromagnetic sensors attached to the ultrasound probe.

[0047] A cross-sectional image refers to an image of a sequence of medical images of the object under examination acquired above.

[0048] The principle of acquiring the corresponding cross-sectional image in the 3D reconstruction model of the corresponding probe plane by acquiring the real-time coordinates of the ultrasonic probe is as follows.

[0049] Medical image sequences, such as CT, MRI, and PET images, are assigned corresponding coordinate information based on the coordinate system provided by the image acquisition device during acquisition. Because the electromagnetic coordinate system of ultrasound images does not match the coordinate system provided by each device, the two coordinate systems must be registered to unify them. Specifically, an ultrasound image sequence is acquired, and a computing device receives the ultrasound sequence images of the human body, extracts the 3D coordinates of the four vertices of each frame of the ultrasound sequence image, and reconstructs the ultrasound data. At the same time, an ICP iterative algorithm is used to perform a 3D registration operation between the reconstructed ultrasound data and the 3D reconstruction model to obtain a registration transformation matrix. This registration transformation matrix is ​​used to convert the 3D coordinates corresponding to the ultrasound data into the 3D coordinate system corresponding to the medical image data.

[0050] After completing the coordinate system registration, the user can use the ultrasound probe to capture real-time ultrasound images of each exploration plane. At the same time, the computing device uses the electromagnetic sensor to obtain real-time coordinate information of each exploration plane in the electromagnetic coordinate system. Based on the real-time coordinate information and the registration transformation matrix, the cross-sectional position of each exploration plane in the 3D model is obtained. Based on the cross-sectional position of each exploration plane in the 3D model, the 3D reconstruction model is cut to obtain MRI image cross sections corresponding to each exploration plane. Taking MRI as an example, the cross-sectional image is the MRI cross section of the 3D reconstruction model for the current exploration plane of the ultrasound probe.

[0051] In step 203, a 3D reconstruction model is displayed in a first display area, and an ultrasound probe model of the ultrasound probe is displayed based on real-time coordinates, and a cross-sectional image corresponding to at least one exploration plane of the ultrasound probe model is displayed on the exploration plane.

[0052] The first display area displays a 3D reconstruction model of the prostate structure, which may be located in the center of the display area or in another location according to the user's preferences.

[0053] The ultrasound probe model and the 3D reconstruction model are simultaneously displayed in the same space within the first display area based on the real-time coordinates of the ultrasound probe obtained by an electromagnetic sensor attached to the ultrasound probe. The display positions of both are determined by the real-time coordinates. This allows the true positional relationship between the ultrasound probe and the prostate organ to be quickly and accurately grasped. In addition, the ultrasound probe model simultaneously displays the exploration areas of multiple exploration planes to show the real exploration areas of the ultrasound probe. Furthermore, in the exploration planes, corresponding cross-sectional images are displayed synchronously. Taking MRI as an example, the multiple exploration planes each show an MRI cross-section in the 3D reconstruction model synchronously.

[0054] The first display area may be a full-screen display area of ​​an independent display device, a specific display area after partitioning, a full-screen display area or a partitioned display area of ​​a computer device with a display, or an edge of a user's web page or an integration of the entire display area or a partitioned display area of ​​a computer device.

[0055] In step 204, a fusion image of the cross-sectional image and the ultrasound image corresponding to at least one exploration plane is displayed in the second display area, and if the exploration plane intersects with at least one target marker in the 3D reconstruction model, the target marker is projected onto the fusion image to obtain a 2D image of the target marker.

[0056] A fusion image refers to a fusion image of an ultrasound image acquired in real time by an ultrasound probe and a medical image sequence (one of CT images, MRI images, or PET images) of cross-sectional images. Specifically, which image is fused is determined by the type of medical image sequence introduced in step 201. Based on the coordinate information acquired in real time by the ultrasound probe, a corresponding cross-sectional image is acquired from the 3D reconstruction model, and a pre-drawn organ contour is used as the registration target. For example, if the medical image sequence is an MRI image of the prostate organ, the corresponding prostate contours in the ultrasound image and the MRI image are drawn or automatically recognized, and the ultrasound image and the MRI image are registered and superimposed based on the prostate contour. In other words, by displaying two images superimposed in a single display area, a fusion display of different images is achieved.

[0057] Similarly, based on the coordinate information of the scanning plane and the coordinate information of the target marker, a cross-sectional projection of the target marker on the scanning plane, i.e., a two-dimensional image of the target marker on the scanning plane, is calculated. Because the ultrasound image is obtained on the scanning plane, a two-dimensional ultrasound image of the target marker is obtained at this time. Since the ultrasound image and the cross-sectional image have already been registered and fused according to the organ contour, the two-dimensional image of the target marker displayed on the ultrasound image is accurately synchronized with the corresponding fused image.

[0058] In step 205, the two-dimensional image of the target marker is associated with the corresponding puncture hole of the puncture plate attached to the ultrasound probe, and the associated relationship between the two-dimensional image of the target marker and the puncture hole is displayed in the second display area.

[0059] The puncture plate has a plurality of puncture holes arranged at predetermined intervals. The puncture plate is fixed to the ultrasound probe by a biopsy puncture shelf. Specifically, see FIG. 3.

[0060] The puncture plate 1 is provided with colored tape 4 and multiple puncture holes 5 spaced apart. All of the puncture holes 5 are aligned in the same direction. The colored tape 4 is located between adjacent puncture holes 5. By observing the colored tape 4, the operator can determine whether the puncture needle should pass through one of the puncture holes 5 on the colored tape 4 or the other puncture hole 5 on the colored tape 4, allowing them to more intuitively, conveniently, and accurately find the puncture hole where the puncture needle needs to be inserted. This saves time, improves surgical efficiency, and avoids the risk of inserting the puncture hole 5 incorrectly.

[0061] Specifically, the puncture holes 5 in the puncture plate 1 are aligned vertically, making it easy to select different puncture positions. Here, two puncture holes 5 are arranged, and one color tape 4 is placed between the two puncture holes 5. It is also possible to arrange more than two puncture holes 5. In this case, one color tape 4 can be placed between each two adjacent puncture holes 5, or one color tape 4 can be placed between any adjacent puncture holes 5. When multiple color tapes 4 are arranged, they are easy to distinguish because the colors of the color tapes 4 are different from each other.

[0062] Furthermore, the puncture plate 1 has scale lines 6 that correspond to the puncture holes 5. At least some of the scale lines 6 have numeral symbols 7. The scale lines 6 and numeral symbols 7 allow the operator to select the puncture position more accurately, improving surgical efficiency.

[0063] Preferably, when there are 12 puncture holes 5, 12 scale lines 6 are provided in one-to-one correspondence with the puncture holes 5, and numeral symbols 7 are provided on the scale lines 6 at even-numbered positions from bottom to top, and the numeral symbols 7 are set to increase sequentially from bottom to top. This structure is simple, easy to manufacture, and convenient for distinguishing the puncture holes.

[0064] The target marker includes at least one of a prostate puncture mark and a target area mark. These are all ellipsoids with different volumes, shapes, and positions. When the detection plane intersects with the target marker, a 2D projection of the target marker on the detection plane is obtained, and this 2D projection is displayed at the corresponding coordinates of the fusion image displayed in the first display area. This results in a 2D image of the target marker on the detection plane. Based on the coordinate information of this 2D image, the relationship between the two is displayed in the fusion image in the second display area, matching the actual positions of each puncture hole on the puncture plate. For example, as shown in Figure 4, the detection plane contacts prostate puncture marks 8 and 11 on the target marker. Based on the coordinate information of prostate puncture marks 8 and 11, they are calculated to correspond to puncture hole positions 6 and 4 on the puncture plate, respectively. The relationship between the two is indicated by displaying extension lines of each prostate puncture mark and the corresponding puncture hole positions. The extension lines guide the doctor to insert a needle from puncture hole positions 6 and 4 to achieve sampling of prostate puncture marks 8 and 11.

[0065] As shown in Figure 5, when the probe plane intersects with the target area mark in the target sign, two additional prostate puncture holes, numbered 13 and 14, with fixed shape and size, are generated in addition to the 12 prostate puncture holes already located based on information such as the area size and coordinates of the 2D image of the target area on the probe plane. The corresponding puncture hole positions on the puncture board are calculated, and the corresponding puncture hole positions are indicated by extension lines. The extension lines guide the medical staff to insert needles from puncture hole positions 6 and 4 to achieve sampling of the target area mark.

[0066] In the above-described solution, the first display area dynamically and synchronously displays the positional relationship between the ultrasound probe model and the 3D reconstruction model. The first display area allows the user to grasp the real-time positional relationship between the ultrasound probe and the prostate organ. Based on the exploration plane displayed on the ultrasound probe model, the exploration plane of the ultrasound probe can be precisely controlled to intersect with any target marker on the 3D reconstruction model. Furthermore, the second display area utilizes the fusion image of the ultrasound image and the cross-sectional image, and the relationship between the target marker in the fusion image and the puncture holes on the puncture plate attached to the ultrasound probe, to help the user quickly and accurately puncture and sample the target marker through the corresponding puncture hole on the puncture plate. This shortens the time required for sample collection, reduces the number of sampling needles, and avoids repeated punctures. It also reduces the operational difficulty and learning time for medical professionals, improves the patient's surgical experience, and reduces the risk of postoperative complications.

[0067] In one embodiment, to display a two-dimensional image of a target marker on a fusion image, first coordinate information of the detection surface and second coordinate information of the target marker intersecting with the detection surface are obtained, and the target marker is two-dimensionally projected onto the fusion image of the detection surface based on the first coordinate information and the second coordinate information.

[0068] Ultrasound probes used for prostate puncture are typically double-sided probes with two perpendicular scanning planes. A double-sided probe has two acoustic windows. Each acoustic window generates a scanning plane. The two scanning planes of a double-sided probe are orthogonal to each other. One acoustic window is located at the tip of the ultrasound probe, has a fan-shaped scanning area, and its scanning direction coincides with the transverse plane. The other probe is a long acoustic window installed along the axial direction of the ultrasound probe, has a rectangular scanning area, and its scanning direction coincides with the sagittal plane. Because the acoustic window is fixed to the ultrasound probe, the shape and size of its scanning plane are also fixed. From the electromagnetic coordinate information collected by an electromagnetic sensor attached to the rear end of the ultrasound probe, the coordinate information of the scanning planes of multiple probes can be obtained through simple calculations. Based on the coordinate information of each scanning plane and the 3D coordinate information of the target marker, the coordinate information of the interface between the two can be calculated. Based on this interface coordinate information, a 2D cross-sectional image of the scanning plane of the target marker can be calculated.

[0069] In one embodiment, at least one prostate puncture mark is generated within a two-dimensional projection marked with the target region of the fused image.

[0070] Specifically, if the target marker is a target area mark, the target area mark exists in the form of an irregular ellipsoid within the prostate structure of the 3D reconstruction model. If there are multiple target area marks, multiple irregular ellipsoids with different volumes will exist depending on the size of the target area outline, and the shape of the ellipsoid will correspond to the shape of the target area when the outline is drawn. Based on the coordinate information of the ellipsoid, the three-dimensional geometric center of the ellipsoid is calculated, and the coordinate information of this three-dimensional geometric center is obtained. As shown in FIG. 6, if the target area mark is projected onto the fusion image by two-dimensional projection, based on the coordinate information of this three-dimensional geometric center, this three-dimensional geometric center is displayed as a representation of the prostate puncture mark in the fusion image. That is, a regular circular prostate puncture mark 13 is displayed, and the corresponding hole position on the puncture plate is calculated. The relationship between the prostate puncture mark and the corresponding hole position is determined using extension lines, etc., to guide the medical professional through the corresponding hole position to complete the biopsy sampling of the target area mark. Since this prostate puncture mark is the three-dimensional geometric center of the target area mark, it is possible to improve the accuracy of sampling within the target area mark and to avoid repeatedly puncturing and sampling within the target area mark.

[0071] In one embodiment, in response to a first operation, at least one target marker selected by the first operation is acquired, and the target marker is displayed in a highlighted manner in the first display area or hidden in the second display area.

[0072] The first operation may be an operation in which a medical professional inputs a command via an interactive device such as a touch screen, keyboard, or mouse. Taking Fig. 7 as an example, the medical professional accesses an operation interface displaying each prostate puncture mark with a numeric code through the touch screen, and selects the corresponding prostate puncture mark by clicking one or more numeric numbers. Based on the acquired prostate puncture mark, the computer device sends a relevant command to the display terminal, controlling the display terminal to highlight and display the selected prostate puncture point in the first display area, simultaneously display the corresponding numeric number, and hide the selected prostate puncture point in the second display area.

[0073] The above technical proposal can be applied to multiple scenarios. For example, if a prostate puncture mark has already been punctured and sampled, or if a prostate puncture mark does not require puncture or sampling, a medical professional can use the interactive device to select the prostate puncture mark and highlight (e.g., highlight) the prostate puncture point in the 3D reconstruction model in the first display area. This notifies the user that the prostate puncture point has already been punctured and sampled, preventing repeated punctures. Furthermore, the prostate puncture point can be hidden in the second display area (fused image), and a mark (e.g., an extension line or other indicator) indicating the relationship between the prostate puncture point and the corresponding puncture site can be hidden. This more effectively prevents medical professionals from repeatedly puncturing the prostate.

[0074] Similarly, if the target marker is a target area mark, a corresponding target area mark is selected by a similar operation to avoid puncturing or sampling a target area mark that has already been punctured or that is not to be punctured.

[0075] In one embodiment, each of the sub-display areas of the third display area displays a cross-sectional image of a cross section corresponding to at least one of the probe planes.

[0076] In this embodiment, the first display area of ​​the puncture guide interface is used to display three-dimensional puncture guide information including an ultrasound probe model and a three-dimensional reconstruction model, the second display area is used to display two-dimensional puncture guide information including the fusion image and the target marker and the puncture plate, and the third display area is used to separately display the cross-sectional image of the current ultrasound probe in the three-dimensional reconstruction model, so that the user can intuitively check the cross-sectional image of the current ultrasound probe in the three-dimensional reconstruction model without being interfered with by other elements using the third display area, and can also understand the puncture guide information from various angles by referring to the display information in the first and second display areas.

[0077] In an embodiment of the present disclosure, the third display area further includes a first sub-display area and a second sub-display area, which respectively display a cross-sectional image corresponding to a first probe whose exploration direction coincides with the transverse plane and a cross-sectional image corresponding to a second probe whose exploration direction coincides with the sagittal plane, thereby providing medical professionals with more comprehensive and multifaceted reference information.

[0078] In another embodiment of the present disclosure, the layout of the puncture guidance interface is as shown in Fig. 8. The layout of the puncture guidance interface is not limited to Fig. 8. The user can freely adjust the layout of the puncture guidance interface according to their usage habits.

[0079] In one embodiment, in response to a needle tip enhancement command, at least one of the needle tip position of the sampling device and the safe puncture zone is displayed in the fused image based on the real-time coordinates of the sampling device.

[0080] The sampling device is typically a sample collection device commonly used in the art, such as a puncture needle or a biopsy needle, and generally involves cutting the target object by the asynchronous back-and-forth movement of the outer needle and the inner needle, retaining a sample of the target object after cutting in the needle reservoir of the inner needle, and then removing the sample retained in the needle reservoir to complete the sampling.

[0081] As shown in Figure 9, the interactive terminal receives an augmentation command to open the needle tip input by the user. In the second display area, the user can observe in real time the progress of the puncture needle in the prostate gland, and can also confirm in real time the position of the needle tip, indicated by the corresponding solid center point, calculated based on the coordinate information acquired in real time by the sensor attached to the puncture needle. From this needle tip position, a safe puncture area (a rectangular dotted line protection frame) is calculated, helping the medical professional determine the needle advance distance and achieving fast and accurate needle advancement.

[0082] In this embodiment, the needle tip point coordinates are transformed into an ultrasound plane through multimodal medical image registration, which can more accurately guide the ultrasound puncture and further improve the accuracy of the puncture guidance, thereby helping doctors puncture target marks (e.g., lesion sites or prostate puncture marks) more quickly and effectively reducing the puncture time and the probability of postoperative complications.

[0083] In one embodiment, the ultrasound image and the cross-sectional image may be displayed with different display parameters and / or the contours of the prostate region in the ultrasound image and the cross-sectional image in the fused image may be marked with different marking parameters, where the display parameters include at least one of contrast and transparency, and the marking parameters may include a color identifier for the contour line.

[0084] That is, to enable the user to distinguish between the ultrasound image and the cross-sectional image in the fusion image, the ultrasound image and the cross-sectional image may be displayed with different transparency or contrast, allowing the user to distinguish between them. In some embodiments, the operation interface further includes a display parameter adjustment interface. The user can adjust the display parameters of the ultrasound image and / or the cross-sectional image through the operation interface, thereby displaying the fusion image according to the user's needs. For example, only the ultrasound image may be displayed, only the cross-sectional image may be displayed, or both the ultrasound image and the cross-sectional image may be displayed simultaneously.

[0085] By marking the prostate region in the ultrasound image and the prostate region in the cross-sectional image in the fusion image, the identification results of the prostate region in the two images can be distinguished. For example, as shown in Figure 9, assuming the cross-sectional image is an MRI image, the contours of the prostate region in the ultrasound image and the prostate region in the MRI image are displayed with dotted lines of different colors. In addition, when the current exploration plane of the ultrasound probe intersects with the target marker, a corresponding 2D image can be displayed. The system marks the target region with a dotted line of one color and the prostate puncture point with a dotted line and number of another color. The contour shape is determined by the target marker and the cross-sectional angle. This allows medical professionals to observe the puncture status in real time based on the puncture angle, lesion location, etc., and easily identify the puncture angle.

[0086] As shown in Figure 10, in one specific application scenario, a patient undergoing guided prostatectomy lies on an operating table in a fixed position, with a multimodal fusion device placed next to him. This device integrates a magnetic field generator, an electromagnetic sensor, an ultrasound probe, a sampling device, a display terminal, and other devices. The magnetic field generator is placed at the upper left or upper right of the patient and generates a hemispherical magnetic field that covers the patient's entire body. Multiple electromagnetic sensors are detachably attached to the sampling device and ultrasound probe at one end and connected to the multimodal fusion device at the other end. As the sampling device and ultrasound probe move within the magnetic field, the electromagnetic sensors detect corresponding coordinate and direction information and return it to the multimodal fusion device. An MRI image of the patient's prostate gland previously introduced into the multimodal fusion device undergoes pre-registration and fusion steps to align the two different coordinate systems of the MRI image and the ultrasound image. Based on the coordinate and direction information returned by the electromagnetic sensor, the corresponding MRI image is extracted and displayed on the display of the multimodal fusion device, resulting in a fused image of the two. At this time, the display device simultaneously displays a 3D prostate model of the patient. The medical professional observes the relative position of the ultrasound probe's prostate puncture surface and the 3D prostate model and rotates the ultrasound probe so that the prostate puncture surface is in contact with the intended prostate puncture point or target area mark on the 3D prostate model. Next, the medical professional observes the guide lines on the fusion image that correspond to the prostate puncture point or target area mark and the specific hole position on the puncture plate. The specific puncture hole position is determined based on the guide lines, for example, by using the hole position number indicated by the extension of the guide lines or by matching the color of the guide lines to the hole color. The medical professional then operates the sampling device to puncture the specific puncture hole position and perform sampling. During the needle tip insertion process, an electromagnetic sensor attached to the sampling device returns the coordinate information of the sampling device in real time. After calculation, the multimodal fusion device displays the needle tip position on the fusion image and indicates the safe puncture area based on the stimulation distance after the needle tip is stimulated.Based on real-time tracking of the needle tip and real-time display of the safe puncture zone, the medical staff can grasp the movement trajectory of the needle tip of the sampling device in the patient's prostate organ in real time, and immediately correct the needle tip position if any deviation occurs. At the same time, based on the safe puncture zone, the medical staff can control the needle tip insertion depth to avoid penetrating the patient's organ.

[0087] In the above application scenarios, the technical solution disclosed herein can quickly complete sampling, shorten the time required for sample collection, reduce the number of sampling needles, avoid repeated punctures, reduce the operation difficulty and learning time for medical personnel, improve the surgical experience for patients, and reduce the probability of postoperative complications.

[0088] According to one or more embodiments of the present disclosure, a prostate puncture guidance device 200 based on multimodal fusion is provided, as shown in Figure 11. The device includes a 3D reconstruction module 201, an ultrasound data acquisition module 202, a model display module 203, a fused image display module 204, and a puncture-related module 205.

[0089] The 3D reconstruction module 201 acquires a medical image sequence of an object under examination, and performs 3D reconstruction based on the medical image sequence to obtain a 3D reconstructed model of the object under examination. The 3D reconstructed model includes a prostate structure of the object under examination. The prostate structure includes a plurality of target markers. The target markers include at least one of a prostate puncture mark and a target region mark.

[0090] The ultrasound data acquisition module 202 acquires real-time coordinates of the ultrasound probe and ultrasound images of the object under examination, and acquires corresponding cross-sectional images in a 3D reconstruction model of at least one exploration plane of the ultrasound probe based on the real-time coordinates.

[0091] The model display module 203 displays a 3D reconstruction model in a first display area, and displays an ultrasound probe model of the ultrasound probe based on real-time coordinates, and displays a cross-sectional image corresponding to at least one exploration plane of the ultrasound probe model.

[0092] The fusion image display module 204 displays a fusion image of the cross-sectional image and the ultrasound image corresponding to at least one exploration plane in the second display area, and when the exploration plane intersects with at least one target marker in the three-dimensional reconstruction model, projects the target marker onto the fusion image to obtain a two-dimensional image of the target marker.

[0093] The puncture-related module 205 associates the two-dimensional image of the target marker with the corresponding puncture hole of the puncture plate attached to the ultrasound probe, and displays the association between the two-dimensional image of the target marker and the puncture hole in the second display area.

[0094] The prostate puncture guide device based on the above multimodal fusion displays a prostate model of the subject and an ultrasound probe model with at least one probe plane in the same display area. Based on the coordinate information and ultrasound image information acquired in real time by the ultrasound probe, the device displays dynamically changing content in the display area in real time, allowing the user to grasp the positional relationship between the at least one probe plane and the target marker in the prostate model. This allows the user to quickly and accurately grasp the intersection of the probe plane and the target marker corresponding to the puncture plate attached to the ultrasound probe. Furthermore, based on the correlation between specific puncture holes and the target marker on the puncture plate displayed in a separate display area, the device can insert a sample collection device into the specific puncture hole to quickly complete sample collection. This shortens the time required for sample collection, reduces the number of sampling needles, and avoids repeated punctures, thereby reducing the difficulty and learning time for medical professionals and improving the surgical experience for patients.

[0095] In one embodiment, the fused image display module 204 generates at least one prostate puncture mark within the two-dimensional projection marked with the target region of the fused image.

[0096] In some embodiments, the fused image display module 204 displays the ultrasound image and the cross-sectional image with different display parameters and / or marks the contours of the prostate region of the ultrasound image and the cross-sectional image in the fused image with different marking parameters, where the display parameters include at least one of contrast and transparency, and the marking parameters include a color identifier (color ID) of the contour.

[0097] For a specific definition of the prostate puncture guide device based on multimodal fusion, please refer to the definition of the prostate puncture guide method based on multimodal fusion above, and therefore will not be repeated here. All or part of each module of the prostate puncture guide device based on multimodal fusion described above can be realized by software, hardware, or a combination thereof. To facilitate the processor calling the operations corresponding to each of the above modules, each of the above modules can be incorporated in the form of hardware relative to the processor in a computer device or can be independent, or can be stored in the form of software in memory in a computer device.

[0098] In one embodiment, a computer device is provided. This computer device may be a user terminal, and its internal structure is as shown in FIG. 12. The computer device includes a processor, a memory, a communication interface, a display, and an input device, all connected by a system bus. The processor is for providing calculation and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an execution environment for the operating system and the computer program stored in the non-volatile storage medium. The computer program is executed by the processor to realize a prostate puncture guide method based on multimodal fusion. The communication interface is for communicating with an external terminal via a network connection. The display device may be a liquid crystal display or an electronic ink display. The input device of the computer device may be a touch layer covered on the display device, or may be keys, a trackball, or a touchpad installed on the case of the computer device, or may be an external keyboard, touchpad, or mouse.

[0099] Those skilled in the art will appreciate that the structure shown in Figure 12 is merely a block diagram of a partial structure related to the solution of the present application, and does not limit the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than those shown in the figure, may combine some components, or may have a different component arrangement.

[0100] In one embodiment, the prostate puncture guide device based on multimodal fusion provided herein can function as a computer program. This computer program can be executed on a computer device such as that shown in FIG. 12. The memory of the computer device stores each program module constituting the prostate puncture guide device based on multimodal fusion described above, such as the 3D reconstruction module, ultrasound data acquisition module, model display module, fusion image display module, and puncture-related module shown in FIG. 11. The computer program composed of these program modules causes a processor to execute the steps of the prostate puncture guide method based on multimodal fusion in each embodiment of the present application described herein. For example, the computer device shown in FIG. 12 can perform step 201 using the 3D reconstruction module of the prostate puncture guide device based on multimodal fusion shown in FIG. 11, perform step 202 using the ultrasound data acquisition module, perform step 203 using the model display module, and perform step 204 using the fusion image display module.

[0101] Based on this, in one embodiment, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor, when executing the computer program, implementing the method for guiding prostate needles based on multimodal fusion provided in any embodiment of the present application.

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above-described embodiments can be executed by issuing instructions to associated hardware via a computer program. This program is stored in a non-volatile computer-readable storage medium. When this program is executed, the flow described in each of the above-described method embodiments is realized. Any reference to memory, storage, database, or other medium used in each embodiment provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of example, RAM can come in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM).

[0103] The technical features in the above embodiments can be arbitrarily combined. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, but as long as there is no contradiction in the combinations of these technical features, all should be considered within the scope described herein.

[0104] The above-described examples illustrate some embodiments of the present application, and the descriptions are more specific and detailed, but this does not limit the scope of the claims of the present application. A person skilled in the art can make some modifications and improvements without departing from the concept of the present invention. All such modifications and improvements shall fall within the scope of protection of the present application.

Claims

1. 1. A method for guiding prostate needles based on multimodal fusion, comprising: acquiring a medical image sequence of an object under examination, and performing three-dimensional reconstruction based on the medical image sequence to obtain a three-dimensional reconstructed model of the object under examination; the three-dimensional reconstruction model includes a prostate structure of the subject, the prostate structure including a plurality of target markers, the plurality of target markers including at least one of a prostate puncture mark and a target area mark; Acquiring real-time coordinates of an ultrasound probe and an ultrasound image of an object to be examined, and obtaining corresponding cross-sectional images in the three-dimensional reconstruction model of at least one probe plane of the ultrasound probe based on the real-time coordinates; displaying the three-dimensional reconstruction model in a first display area, and displaying an ultrasound probe model of the ultrasound probe based on the real-time coordinates; and displaying a cross-sectional image corresponding to the at least one exploration plane of the ultrasound probe in the three-dimensional reconstruction model in at least one exploration area of ​​the ultrasound probe model; displaying a fusion image of the cross-sectional image and the ultrasound image corresponding to at least one of the exploration planes in a second display area; If the search plane intersects with at least one of the target markers in the three-dimensional reconstruction model, projecting the target marker onto the fused image to obtain a two-dimensional image of the target marker; and associating the two-dimensional image of the target marker with a corresponding puncture hole of a puncture plate attached to the ultrasound probe, and displaying the association between the two-dimensional image of the target marker and the puncture hole in the second display area.

2. projecting the target marker onto the fused image to obtain a two-dimensional image of the target marker, Obtaining first coordinate information of the detection surface and second coordinate information of the target marker intersecting the detection surface; and The method for guiding prostate puncture based on multimodal fusion according to claim 1 , further comprising two-dimensionally projecting the target marker onto the fusion image of the exploration plane based on the first coordinate information and the second coordinate information.

3. The method for guiding prostate puncture based on multimodal fusion according to claim 1, further comprising generating at least one of the prostate puncture marks within a two-dimensional projection marked by the target area mark in the fusion image.

4. In response to a first operation, acquiring at least one target indicator selected by the first operation; highlighting the target marker in the first display area; and / or and hiding the target marker in the second display area. The method of claim 1, further comprising:

5. The prostate puncture guidance method based on multimodal fusion according to claim 1, characterized in that the cross-sectional image corresponding to the at least one exploration plane is displayed in each sub-display area of ​​the third display area.

6. The prostate puncture guidance method based on multimodal fusion according to claim 1, characterized in that, in response to a needle tip enhancement command, at least one of the needle tip position of the sampling device and the safe puncture zone is displayed in the fusion image based on the real-time coordinates of the sampling device.

7. Displaying the ultrasound image and the cross-sectional image with different display parameters, and / or marking the contours of the prostate region in the ultrasound image and the cross-sectional image in the fusion image with different marking parameters; The method for guiding prostate puncture based on multimodal fusion according to claim 1 , wherein the display parameters include at least one of contrast and transparency, and the marking parameters include a color identifier of an outline.

8. A multimodal fusion-based prostate puncture guide device, comprising: The apparatus includes a 3D reconstruction module, an ultrasound data acquisition module, a model display module, a fusion image display module, and a puncture-related module; The three-dimensional reconstruction module acquires a medical image sequence of the object under examination, and performs three-dimensional reconstruction based on the medical image sequence to obtain a three-dimensional reconstructed model of the object under examination; the three-dimensional reconstruction model includes a prostate structure of the subject, the prostate structure including a plurality of target markers, the target markers including at least one of a prostate puncture mark and a target area mark; The ultrasound data acquisition module acquires real-time coordinates of an ultrasound probe and an ultrasound image of an object to be examined, and acquires corresponding cross-sectional images of at least one probe plane of the ultrasound probe in the three-dimensional reconstruction model based on the real-time coordinates; The model display module displays the three-dimensional reconstruction model in a first display area, and displays an ultrasound probe model of the ultrasound probe based on the real-time coordinates, and displays a cross-sectional image corresponding to the at least one exploration plane of the ultrasound probe in the three-dimensional reconstruction model in at least one exploration area of ​​the ultrasound probe model; the fusion image display module displays a fusion image of the cross-sectional image and the ultrasound image corresponding to at least one of the exploration planes in a second display area; and when the exploration plane intersects with at least one of the target markers in the three-dimensional reconstruction model, projects the target marker onto the fusion image to obtain a two-dimensional image of the target marker; The puncture-related module associates the two-dimensional image of the target marker with the corresponding puncture hole of the puncture plate attached to the ultrasound probe, and displays the association between the two-dimensional image of the target marker and the puncture hole in the second display area.

9. 1. A computing device comprising: A memory and a processor, The memory stores a computer program, A computer device, characterized in that the processor, when executing the computer program, implements the method for guiding prostate puncture based on multimodal fusion described in any one of claims 1 to 7.

10. 1. A multimodal fusion-based prostate puncture guidance system, comprising: The prostate puncture guide system includes a magnetic field generator, an electromagnetic sensor, an ultrasound probe, a puncture plate, a sampling device, a display terminal, and the computer device of claim 9 ; A puncture plate is attached to the ultrasonic probe, and an electromagnetic sensor is installed on the ultrasonic probe and the sampling device, respectively; A prostate puncture guidance system based on multimodal fusion, characterized in that the magnetic field generator, the display terminal, the ultrasound probe, and the electromagnetic sensor are each electrically connected to the computing device.

Images