Magnetic resonance image real scene enhancement method and device and electronic equipment

Abstract

The invention discloses a real scene enhancement method and device for a real magnetic resonance image and electronic equipment, and belongs to the technical field of augmented reality. The method comprises the following steps: acquiring a real-time visual image and a real-time magnetic resonance image of a target object in a medical scene; acquiring a real three-dimensional model of the target object based on the real-time visual image, and acquiring a magnetic resonance three-dimensional model of the target object based on the real-time magnetic resonance image; and performing three-dimensional fusion operation on the reality three-dimensional model and the magnetic resonance three-dimensional model to obtain an augmented reality view of the target object. According to the magnetic resonance image real scene enhancement method disclosed by the invention, the problem that the observation effect of the real-time condition of an object in a medical scene is poor in the related technology is solved.

Description

Technical Field

[0001] This application belongs to the field of augmented reality technology, and in particular relates to a method, apparatus and electronic device for enhancing real-world scenes using magnetic resonance images. Background Technology

[0002] With the development of medical imaging technology, real-time magnetic resonance imaging (MRI) images generated by MRI equipment are acquired in medical surgical scenarios for real-time observation of intraoperative conditions.

[0003] However, under normal circumstances, only real-time magnetic resonance images or images of the intraoperative scene can be acquired separately, resulting in poor observation of the real-time status of objects and instruments during surgery. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, apparatus, and electronic device for enhancing real-world scenes in magnetic resonance imaging, to solve the problem of poor observation results of objects in real-time medical scenes in related technologies.

[0005] In a first aspect, this application provides a method for enhancing realistic scenes in magnetic resonance images, the method comprising: Acquire real-time visual and magnetic resonance images of target objects in medical settings; The real-world 3D model of the target object is obtained based on real-time visual images, and the magnetic resonance 3D model of the target object is obtained based on real-time magnetic resonance images. Perform a 3D fusion operation on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object.

[0006] According to the magnetic resonance image real-world scene enhancement method of this application, real-time visual images and real-time magnetic resonance images of a target object in a medical scene are acquired; a real-world 3D model of the target object is acquired based on the real-time visual image, and a magnetic resonance 3D model of the target object is acquired based on the real-time magnetic resonance image; a 3D fusion operation is performed on the real-world 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object; by fusing the real-time visual image and the real-time magnetic resonance image, an augmented reality view of the target object is obtained. Through the augmented reality view, both the surface condition and the internal condition of the target object in the medical scene can be observed, thus improving the observation effect in the medical scene.

[0007] According to one embodiment of this application, the real-time magnetic resonance image includes several two-dimensional magnetic resonance slice data; acquiring a real-world three-dimensional model of the target object based on a real-time visual image, and acquiring a magnetic resonance three-dimensional model of the target object based on the real-time magnetic resonance image, includes: Based on the medical imaging interaction toolkit and visualization toolkit, several two-dimensional magnetic resonance slice data are reconstructed into three dimensions to obtain a three-dimensional magnetic resonance model.

[0008] According to one embodiment of this application, acquiring a real-world 3D model of a target object based on real-time visual images and acquiring a magnetic resonance 3D model of the target object based on real-time magnetic resonance images includes: Based on the target data processing library, real-time visual images are transformed to obtain a 3D point cloud model; Obtain a real-world 3D model based on a 3D point cloud model.

[0009] According to one embodiment of this application, a three-dimensional fusion operation is performed on a real-world three-dimensional model and a magnetic resonance three-dimensional model to obtain an augmented reality view of the target object, including: Based on the target transformation matrix, a 3D fusion operation is performed on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object.

[0010] According to one embodiment of this application, after performing a 3D fusion operation on a real-world 3D model and a magnetic resonance 3D model to obtain an augmented reality view of the target object, the method includes: Upon detecting a target instrument marker, the position and attitude information of the target instrument are calculated based on the target instrument marker. Based on the target instrument's position and attitude information, the target instrument is rendered and displayed in an augmented reality view.

[0011] According to one embodiment of this application, after performing a 3D fusion operation on a real-world 3D model and a magnetic resonance 3D model to obtain an augmented reality view of the target object, the method includes: The augmented reality view is rendered and displayed based on the first window, the real-time visual image is rendered and displayed based on the second window, the real-time magnetic resonance image is rendered and displayed based on the third window, and the magnetic resonance 3D model is rendered and displayed based on the fourth window.

[0012] According to one embodiment of this application, the method further includes: Based on augmented reality views, obtain lesion markers and planned intervention paths; Render and display the planned intervention path and the predicted intervention path; the predicted intervention path is generated based on the lesion location, the position information and attitude information of the target instrument.

[0013] Secondly, this application provides a magnetic resonance image real-scene enhancement device, comprising: The first acquisition module is used to acquire real-time visual images and real-time magnetic resonance images of the target object in the medical scenario; The second acquisition module is used to acquire the real-world 3D model of the target object based on real-time visual images, and to acquire the magnetic resonance 3D model of the target object based on real-time magnetic resonance images. The third acquisition module is used to perform a three-dimensional fusion operation on the real three-dimensional model and the magnetic resonance three-dimensional model to obtain an augmented reality view of the target object.

[0014] According to the magnetic resonance image real-world scene enhancement device of this application, real-time visual images and real-time magnetic resonance images of a target object in a medical scene are acquired; a real-world 3D model of the target object is acquired based on the real-time visual images, and a magnetic resonance 3D model of the target object is acquired based on the real-time magnetic resonance images; a 3D fusion operation is performed on the real-world 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object; by fusing the real-time visual images and the real-time magnetic resonance images, an augmented reality view of the target object is obtained. Through the augmented reality view, both the surface condition and the internal condition of the target object in the medical scene can be observed, thus improving the observation effect in the medical scene.

[0015] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the magnetic resonance image real-scene enhancement method described in the first aspect.

[0016] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the magnetic resonance image real-scene enhancement method described in the first aspect.

[0017] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the magnetic resonance image real-scene enhancement method described in the first aspect.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating the method for enhancing real-world scenes in magnetic resonance images provided in this application embodiment; Figure 2 This is a second schematic flowchart of the method for enhancing real-world scenes in magnetic resonance images provided in the embodiments of this application; Figure 3 This is the third schematic flowchart of the magnetic resonance image real-scene enhancement method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the magnetic resonance image real-scene enhancement device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0021] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0022] The following description, in conjunction with the accompanying drawings, details the magnetic resonance image real-scene enhancement method, apparatus, and electronic device provided in this application through specific embodiments and application scenarios.

[0023] Among them, the method for enhancing the real-world scene of magnetic resonance images can be applied to a terminal, specifically executed by the hardware or software in the terminal.

[0024] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0025] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0026] The magnetic resonance image real-scene enhancement method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the magnetic resonance image real-scene enhancement method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The following uses an electronic device as the execution subject to illustrate the magnetic resonance image real-scene enhancement method provided in this application embodiment.

[0027] like Figure 1 As shown, the method for enhancing the real-world scene of a magnetic resonance image includes steps 110, 120, and 130.

[0028] Step 110: Obtain real-time visual images and real-time magnetic resonance images of the target object in the medical scene.

[0029] In practice, a medical scenario can be a surgical procedure. The target object can be the object undergoing the surgical procedure, such as human bones, muscles, brain tissue, or any theoretically feasible object.

[0030] In some embodiments, real-time magnetic resonance images can be acquired using a magnetic resonance imaging (MRI) device, and real-time visual images can be acquired using a depth camera (e.g., Intel RealSense D435i). The depth camera can be located inside the magnet aperture of the MRI device.

[0031] In some embodiments, a first communication connection can be established with a magnetic resonance imaging (MRI) device via a Socket network protocol, and real-time MRI images can be acquired based on the first communication connection. For example, the MRI device can encapsulate the real-time MRI image and original information such as slice thickness, spacing, and orientation matrix into a data packet, and send the data packet through the first communication connection.

[0032] In some embodiments, a multi-frame image sequence conforming to the DICOM standard can be obtained from local storage, and a deserialization operation can be performed on the multi-frame image sequence to restore a real-time magnetic resonance image.

[0033] Step 120: Obtain the real-world 3D model of the target object based on real-time visual images, and obtain the magnetic resonance 3D model of the target object based on real-time magnetic resonance images.

[0034] In some embodiments, after acquiring real-time magnetic resonance images, high-precision three-dimensional reconstruction can be performed based on the real-time magnetic resonance images to obtain a magnetic resonance three-dimensional model.

[0035] In some embodiments, after acquiring real-time visual images, high-precision 3D reconstruction can be performed based on the real-time visual images to obtain a real-world 3D model.

[0036] Step 130: Perform a 3D fusion operation on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object.

[0037] In some embodiments, after obtaining the augmented reality view, the augmented reality view can be rendered and displayed.

[0038] In some embodiments, after obtaining an augmented reality view, the target device can be rendered and displayed if it is present.

[0039] In some embodiments, after acquiring the augmented reality view, lesion markers and planned intervention paths can be acquired and displayed.

[0040] The magnetic resonance imaging real-world scene enhancement method according to the embodiments of this application acquires real-time visual images and real-time magnetic resonance images of a target object in a medical scene; acquires a real-world 3D model of the target object based on the real-time visual images, and acquires a magnetic resonance 3D model of the target object based on the real-time magnetic resonance images; performs a 3D fusion operation on the real-world 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object; and fuses the real-time visual images and real-time magnetic resonance images to obtain an augmented reality view of the target object. Through the augmented reality view, both the surface condition and the internal condition of the target object in the medical scene can be observed, thus improving the observation effect in the medical scene.

[0041] In some embodiments, the real-time magnetic resonance image includes several two-dimensional magnetic resonance slice data; based on the medical imaging interaction toolkit and visualization toolkit, the several two-dimensional magnetic resonance slice data are reconstructed in three dimensions to obtain a three-dimensional magnetic resonance model.

[0042] In some embodiments, interpolation and resampling operations can be performed on several two-dimensional magnetic resonance slice data based on the Medical Imaging Interaction Toolkit (MITK) and visualization toolkit (such as VTK) to obtain three-dimensional volume data, and a magnetic resonance three-dimensional model that can be arbitrarily sectioned, transparently rendered and interactively operated can be generated based on the three-dimensional volume data.

[0043] In some embodiments, a first physical coordinate system can be constructed for several two-dimensional magnetic resonance slice data based on the Medical Imaging Interaction Toolkit (MITK) and visualization toolkit (such as VTK), and the key tissue structures of the target object can be constructed based on the ray projection volume rendering algorithm and transfer function to construct a three-dimensional magnetic resonance model.

[0044] The magnetic resonance imaging augmentation method according to embodiments of this application acquires real-time visual images and real-time magnetic resonance images of a target object in a medical setting; acquires a realistic 3D model of the target object based on the real-time visual images; and reconstructs a 3D model of the magnetic resonance image by performing 3D reconstruction on several 2D magnetic resonance slice data based on a medical imaging interaction toolkit and a visualization toolkit; performs a 3D fusion operation on the realistic 3D model and the magnetic resonance 3D model to acquire an augmented reality view of the target object; and fuses the real-time visual images and real-time magnetic resonance images to acquire an augmented reality view of the target object. This augmented reality view allows observation of both the surface and internal structures of the target object in a medical setting, improving the observation effect in a medical setting.

[0045] In some embodiments, a 3D point cloud model is obtained by performing a transformation operation on a real-time visual image based on a target data processing library; and a real-world 3D model is obtained based on the 3D point cloud model.

[0046] In actual implementation, the target data processing library can be the Open3D library or any theoretically feasible data processing library; this application does not impose any specific restrictions on it.

[0047] In actual implementation, real-time visual images can be RGB-D data or any theoretically feasible visual image data; this application does not impose any specific restrictions on them.

[0048] In some embodiments, a 3D point cloud model of the target object can be obtained based on a target data processing library, and a second physical coordinate system can be constructed. Then, a format conversion operation is performed on the 3D point cloud model, converting its data format to vtkPolyData to generate a realistic 3D model.

[0049] The magnetic resonance imaging real-world scene enhancement method according to embodiments of this application acquires real-time visual images and real-time magnetic resonance images of a target object in a medical scene; performs a conversion operation on the real-time visual image based on a target data processing library to obtain a three-dimensional point cloud model; based on the three-dimensional point cloud model, obtains a real-world three-dimensional model; and performs three-dimensional reconstruction on several two-dimensional magnetic resonance slice data based on a medical image interaction toolkit and a visualization toolkit to obtain a magnetic resonance three-dimensional model; performs a three-dimensional fusion operation on the real-world three-dimensional model and the magnetic resonance three-dimensional model to obtain an augmented reality view of the target object; and fuses the real-time visual image and the real-time magnetic resonance image to obtain an augmented reality view of the target object. This augmented reality view allows observation of both the surface and internal conditions of the target object in a medical scene, improving the observation effect in a medical setting.

[0050] In some embodiments, a three-dimensional fusion operation is performed on the real three-dimensional model and the magnetic resonance three-dimensional model based on the target transformation matrix to obtain an augmented reality view of the target object.

[0051] In practice, the target transformation matrix can be obtained based on actual experiments. The target transformation matrix can be used to register the first physical coordinate system of the magnetic resonance 3D model with the second physical coordinate system of the real 3D model.

[0052] In some embodiments, based on the target transformation matrix, the real 3D model and the magnetic resonance 3D model are superimposed in the same physical coordinate system to generate an augmented reality view of the target object.

[0053] The magnetic resonance imaging augmentation method according to embodiments of this application acquires real-time visual images and real-time magnetic resonance images of a target object in a medical setting; performs a transformation operation on the real-time visual image based on a target data processing library to obtain a three-dimensional point cloud model; obtains a real-world three-dimensional model based on the three-dimensional point cloud model; and performs three-dimensional reconstruction on several two-dimensional magnetic resonance slice data based on a medical image interaction toolkit and a visualization toolkit to obtain a magnetic resonance three-dimensional model; performs a three-dimensional fusion operation on the real-world three-dimensional model and the magnetic resonance three-dimensional model based on a target transformation matrix to obtain an augmented reality view of the target object; and fuses the real-time visual image and the real-time magnetic resonance image to obtain an augmented reality view of the target object. This augmented reality view allows observation of both the surface and internal conditions of the target object in a medical setting, improving the observation effect in the medical setting.

[0054] In some embodiments, after performing a 3D fusion operation on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object, if a target instrument marker is detected, the position and attitude information of the target instrument are calculated based on the target instrument marker; and the target instrument is rendered and displayed in the augmented reality view based on the position and attitude information of the target instrument.

[0055] In some embodiments, the target device marker can be a marker deployed on the target device. The target device marker can be an ArUco marker or any theoretically feasible marker; this application does not impose specific limitations on this. The target device can be any theoretically feasible medical device such as a biopsy needle, scalpel, or hemostat.

[0056] In some embodiments, the position information of the target instrument can be calculated based on the target instrument marker, such as the position of the target instrument inside the target object, and the attitude information of the target instrument can be calculated, such as the tilt angle of the target instrument at this time.

[0057] In some embodiments, a device model can be constructed based on the position and attitude information of the target device, and the device model can be rendered and displayed in an augmented reality view.

[0058] The magnetic resonance imaging augmentation method according to embodiments of this application acquires real-time visual images and real-time magnetic resonance images of a target object in a medical setting; performs a transformation operation on the real-time visual image based on a target data processing library to obtain a three-dimensional point cloud model; obtains a real-world three-dimensional model based on the three-dimensional point cloud model; and performs three-dimensional reconstruction on several two-dimensional magnetic resonance slice data based on a medical image interaction toolkit and a visualization toolkit to obtain a magnetic resonance three-dimensional model; performs a three-dimensional fusion operation on the real-world three-dimensional model and the magnetic resonance three-dimensional model based on a target transformation matrix to obtain an augmented reality view of the target object; and fuses the real-time visual image and the real-time magnetic resonance image to obtain an augmented reality view of the target object. This augmented reality view allows observation of both the surface and internal conditions of the target object in a medical setting, improving the observation effect in the medical setting.

[0059] In some embodiments, after performing a 3D fusion operation on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object, the augmented reality view is rendered and displayed based on a first window, a real-time visual image is rendered and displayed based on a second window, a real-time magnetic resonance image is rendered and displayed based on a third window, and the magnetic resonance 3D model is rendered and displayed based on a fourth window.

[0060] In some embodiments, the first window, the second window, the third window, and the fourth window can be displayed overlaid or side by side, and this application does not impose specific limitations on this.

[0061] In some embodiments, lesion markers and planned interventional paths are acquired; the planned interventional path and the predicted interventional path are rendered and displayed; the predicted interventional path is generated based on the lesion location, the position information of the target instrument, and the attitude information.

[0062] In some embodiments, after acquiring the augmented reality view, lesion markers and planned intervention paths can be obtained based on an input device (e.g., a mouse, touch panel, touch display, etc.). Lesion markers represent the markings of lesions within the target object, used to indicate the relative position, volume, and other information of the lesion within the target object. The planned intervention path represents the expected path from the surface of the target object to the interior of the target object until contact with the lesion.

[0063] In some embodiments, after obtaining the position and attitude information of the target instrument, a predictive intervention path can be generated.

[0064] In some embodiments, the planned intervention path can be rendered and displayed based on a first form, and the predicted intervention path can be rendered and displayed based on a second form to create an intuitive comparison.

[0065] The magnetic resonance imaging augmentation method according to embodiments of this application acquires real-time visual images and real-time magnetic resonance images of a target object in a medical setting; performs a transformation operation on the real-time visual image based on a target data processing library to obtain a three-dimensional point cloud model; obtains a real-world three-dimensional model based on the three-dimensional point cloud model; and performs three-dimensional reconstruction on several two-dimensional magnetic resonance slice data based on a medical image interaction toolkit and a visualization toolkit to obtain a magnetic resonance three-dimensional model; performs a three-dimensional fusion operation on the real-world three-dimensional model and the magnetic resonance three-dimensional model based on a target transformation matrix to obtain an augmented reality view of the target object; and fuses the real-time visual image and the real-time magnetic resonance image to obtain an augmented reality view of the target object. This augmented reality view allows observation of both the surface and internal conditions of the target object in a medical setting, improving the observation effect in the medical setting.

[0066] To better understand the method for enhancing real-world scenes in magnetic resonance images provided in this application, further explanation is provided below. It should be understood that the following discussion is merely exemplary.

[0067] This application provides a method for enhancing real-world scenes in magnetic resonance images. The specific steps are as follows: Figure 2 As shown: Step 210: Obtain real-time visual images and real-time magnetic resonance images of the target object in the medical scene.

[0068] In practice, a medical scenario can be a surgical procedure. The target object can be the object undergoing the surgical procedure, such as human bones, muscles, brain tissue, or any theoretically feasible object.

[0069] In some embodiments, real-time magnetic resonance images can be acquired based on a magnetic resonance imaging device, and real-time visual images can be acquired based on a depth camera (e.g., Intel RealSense D435i). The depth camera can be located inside the magnet aperture of the magnetic resonance imaging device.

[0070] In some embodiments, a first communication connection can be established with a magnetic resonance imaging (MRI) device via a Socket network protocol, and real-time magnetic resonance images can be acquired based on the first communication connection. For example, the MRI device can encapsulate the real-time magnetic resonance image and original information such as slice thickness, spacing, and orientation matrix into a data packet, and send the data packet through the first communication connection.

[0071] In some embodiments, a multi-frame image sequence conforming to the DICOM standard can be obtained from local storage, and a deserialization operation can be performed on the multi-frame image sequence to restore a real-time magnetic resonance image.

[0072] In some embodiments, the real-time magnetic resonance image includes several two-dimensional magnetic resonance slice data.

[0073] Step 220: Based on the medical imaging interaction toolkit and visualization toolkit, perform interpolation and resampling operations on several two-dimensional magnetic resonance slice data to obtain three-dimensional volume data, and generate a magnetic resonance three-dimensional model that can be arbitrarily sectioned, transparently rendered and interactively operated based on the three-dimensional volume data.

[0074] In some embodiments, a first physical coordinate system can be constructed for several two-dimensional magnetic resonance slice data based on the Medical Imaging Interaction Toolkit (MITK) and visualization toolkit (such as VTK), and the key tissue structures of the target object can be constructed based on the ray projection volume rendering algorithm and transfer function to construct a three-dimensional magnetic resonance model.

[0075] Step 230: Perform conversion operations on the real-time visual image based on the target data processing library to obtain a 3D point cloud model; based on the 3D point cloud model, obtain a real 3D model.

[0076] In some embodiments, such as Figure 3 As shown, it can perform image processing on real-time visual images, obtain two-dimensional reality images, and render and display them.

[0077] In actual implementation, the target data processing library can be the Open3D library or any theoretically feasible data processing library; this application does not impose any specific restrictions on it.

[0078] In actual implementation, real-time visual images can be RGB-D data or any theoretically feasible visual image data; this application does not impose any specific restrictions on them.

[0079] In some embodiments, a 3D point cloud model of the target object can be obtained based on a target data processing library, and a second physical coordinate system can be constructed. Then, a format conversion operation is performed on the 3D point cloud model, converting its data format to vtkPolyData to generate a realistic 3D model.

[0080] Step 240: Based on the target transformation matrix, perform a 3D fusion operation on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object.

[0081] In practice, the target transformation matrix can be obtained based on actual experiments. The target transformation matrix can be used to register the first physical coordinate system of the magnetic resonance 3D model with the second physical coordinate system of the real 3D model.

[0082] In some embodiments, based on the target transformation matrix, the real 3D model and the magnetic resonance 3D model are superimposed in the same physical coordinate system to generate an augmented reality view of the target object.

[0083] Step 250: If a target instrument marker is detected, calculate the position and attitude information of the target instrument based on the target instrument marker; render and display the target instrument in the augmented reality view based on the position and attitude information of the target instrument.

[0084] In some embodiments, the target device marker can be a marker deployed on the target device. The target device marker can be an ArUco marker or any theoretically feasible marker; this application does not impose specific limitations on this. The target device can be any theoretically feasible medical device such as a biopsy needle, scalpel, or hemostat.

[0085] In some embodiments, the position information of the target instrument can be calculated based on the target instrument marker, such as the position of the target instrument inside the target object, and the attitude information of the target instrument can be calculated, such as the tilt angle of the target instrument at this time.

[0086] In some embodiments, a device model can be constructed based on the position and attitude information of the target device, and the device model can be rendered and displayed in an augmented reality view.

[0087] Step 260: Render and display the augmented reality view based on the first window, render and display the real-time visual image based on the second window, render and display the real-time magnetic resonance image based on the third window, and render and display the magnetic resonance 3D model based on the fourth window.

[0088] In some embodiments, the first window, the second window, the third window, and the fourth window can be displayed overlaid or side by side, and this application does not impose specific limitations on this.

[0089] In some embodiments, lesion markers and planned interventional paths are acquired; the planned interventional path and the predicted interventional path are rendered and displayed; the predicted interventional path is generated based on the lesion location, the position information of the target instrument, and the attitude information.

[0090] In some embodiments, after acquiring the augmented reality view, lesion markers and planned intervention paths can be obtained based on an input device (e.g., a mouse, touch panel, touch display, etc.). Lesion markers represent the markings of lesions within the target object, used to indicate the relative position, volume, and other information of the lesion within the target object. The planned intervention path represents the expected path from the surface of the target object to the interior of the target object until contact with the lesion.

[0091] In some embodiments, after obtaining the position and attitude information of the target instrument, a predictive intervention path can be generated.

[0092] In some embodiments, the planned intervention path can be rendered and displayed based on a first form, and the predicted intervention path can be rendered and displayed based on a second form to create an intuitive comparison.

[0093] This application also provides a magnetic resonance image real-scene enhancement device.

[0094] like Figure 4 As shown, the magnetic resonance image real-scene enhancement device 400 includes: a first acquisition module 410, a second acquisition module 420 and a third acquisition module 430.

[0095] The first acquisition module 410 is used to acquire real-time visual images and real-time magnetic resonance images of the target object in a medical scenario. The second acquisition module 420 is used to acquire the real three-dimensional model of the target object based on real-time visual images and to acquire the magnetic resonance three-dimensional model of the target object based on real-time magnetic resonance images. The third acquisition module 430 is used to perform a three-dimensional fusion operation on the real three-dimensional model and the magnetic resonance three-dimensional model to obtain an augmented reality view of the target object.

[0096] According to the magnetic resonance image augmentation device of this application, the device acquires real-time visual images and real-time magnetic resonance images of a target object in a medical scene; acquires a real three-dimensional model of the target object based on the real-time visual images, and acquires a magnetic resonance three-dimensional model of the target object based on the real-time magnetic resonance images; performs a three-dimensional fusion operation on the real three-dimensional model and the magnetic resonance three-dimensional model to obtain an augmented reality view of the target object; and fuses the real-time visual images and real-time magnetic resonance images to obtain an augmented reality view of the target object. Through the augmented reality view, both the surface condition and the internal condition of the target object in the medical scene can be observed, thus improving the observation effect in the medical scene.

[0097] In some embodiments, the real-time magnetic resonance image includes a plurality of two-dimensional magnetic resonance slice data; the second acquisition module 420 includes: The first acquisition unit is used to perform three-dimensional reconstruction of several two-dimensional magnetic resonance slice data based on the medical image interaction toolkit and visualization toolkit to obtain a three-dimensional magnetic resonance model.

[0098] In some embodiments, the second acquisition module 420 further includes: The second acquisition unit is used to perform conversion operations on real-time visual images based on the target data processing library to acquire a 3D point cloud model. The third acquisition unit is used to acquire a real-world 3D model based on a 3D point cloud model.

[0099] In some embodiments, the third acquisition module 430 includes: The fourth acquisition unit is used to perform a three-dimensional fusion operation on the real three-dimensional model and the magnetic resonance three-dimensional model based on the target transformation matrix to obtain an augmented reality view of the target object.

[0100] In some embodiments, the magnetic resonance image real-scene enhancement device 400 further includes: The calculation module is used to calculate the position and attitude information of the target instrument based on the target instrument marker when the target instrument marker is detected. The rendering module is used to render and display the target instrument in an augmented reality view based on its position and orientation information.

[0101] In some embodiments, the magnetic resonance image real-scene enhancement device 400 further includes: The first display module is used to render and display an augmented reality view based on a first window, render and display a real-time visual image based on a second window, render and display a real-time magnetic resonance image based on a third window, and render and display a magnetic resonance 3D model based on a fourth window.

[0102] In some embodiments, the magnetic resonance image real-scene enhancement device 400 further includes: The fourth acquisition module is used to acquire lesion markers and planned intervention paths; The second display module is used to render and display the planned intervention path and the predicted intervention path; the predicted intervention path is generated based on the lesion location, the position information and posture information of the target instrument.

[0103] The magnetic resonance image augmentation device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific devices.

[0104] The magnetic resonance image real-scene enhancement device in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit it.

[0105] The magnetic resonance image real-scene enhancement device 400 provided in this application embodiment can achieve... Figures 1 to 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0106] In some embodiments, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-described magnetic resonance image real scene enhancement method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0107] It should be noted that the computer equipment in this application embodiment includes the mobile electronic equipment and non-mobile electronic equipment described above.

[0108] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described magnetic resonance image real-scene enhancement method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0109] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0110] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described magnetic resonance image real-scene enhancement method.

[0111] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0112] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described magnetic resonance image real-scene enhancement method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0113] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0114] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0116] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0117] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0118] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for enhancing realistic scenes in magnetic resonance images, characterized in that, include: Acquire real-time visual and magnetic resonance images of target objects in medical settings; A real-world 3D model of the target object is obtained based on the real-time visual image, and a magnetic resonance 3D model of the target object is obtained based on the real-time magnetic resonance image. A 3D fusion operation is performed on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object.

2. The method for enhancing realistic scenes in magnetic resonance images according to claim 1, characterized in that, The real-time magnetic resonance image includes several two-dimensional magnetic resonance slice data; the step of obtaining a realistic three-dimensional model of the target object based on the real-time visual image, and obtaining a magnetic resonance three-dimensional model of the target object based on the real-time magnetic resonance image, includes: Based on the medical imaging interaction toolkit and visualization toolkit, several two-dimensional magnetic resonance slice data are reconstructed into three dimensions to obtain a three-dimensional magnetic resonance model.

3. The method for enhancing realistic scenes in magnetic resonance images according to claim 1, characterized in that, The step of obtaining a real-world 3D model of the target object based on the real-time visual image and obtaining a magnetic resonance 3D model of the target object based on the real-time magnetic resonance image includes: The real-time visual image is transformed based on the target data processing library to obtain a 3D point cloud model. Based on the aforementioned 3D point cloud model, a real-world 3D model is obtained.

4. The method for enhancing real-world scenes in magnetic resonance images according to claim 1, characterized in that, The step of performing a 3D fusion operation on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object includes: Based on the target transformation matrix, a three-dimensional fusion operation is performed on the real three-dimensional model and the magnetic resonance three-dimensional model to obtain an augmented reality view of the target object.

5. The method for enhancing realistic scenes in magnetic resonance images according to claim 1, characterized in that, After performing a 3D fusion operation on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object, the method includes: Upon detecting a target instrument marker, the position and attitude information of the target instrument are calculated based on the target instrument marker; Based on the position information and posture information of the target instrument, the target instrument is rendered and displayed in the augmented reality view.

6. The method for enhancing realistic scenes in magnetic resonance images according to any one of claims 1-5, characterized in that, After performing a 3D fusion operation on the real 3D model and the magnetic resonance 3D model to obtain an augmented reality view of the target object, the method includes: The augmented reality view is rendered and displayed based on the first window, the real-time visual image is rendered and displayed based on the second window, the real-time magnetic resonance image is rendered and displayed based on the third window, and the magnetic resonance 3D model is rendered and displayed based on the fourth window.

7. The method for enhancing realistic scenes in magnetic resonance images according to claim 6, characterized in that, The method further includes: Based on the augmented reality view, lesion markers and planned intervention paths are obtained; The planned intervention path and the predicted intervention path are rendered and displayed; the predicted intervention path is generated based on the lesion location, the target instrument's position information, and its orientation information.

8. A magnetic resonance imaging real-scene enhancement device, characterized in that, include: The first acquisition module is used to acquire real-time visual images and real-time magnetic resonance images of the target object in the medical scenario; The second acquisition module is used to acquire a real three-dimensional model of the target object based on the real-time visual image, and to acquire a magnetic resonance three-dimensional model of the target object based on the real-time magnetic resonance image. The third acquisition module is used to perform a three-dimensional fusion operation on the real three-dimensional model and the magnetic resonance three-dimensional model to obtain an augmented reality view of the target object.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the magnetic resonance image real-scene enhancement method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for enhancing the real-world scene of a magnetic resonance image as described in any one of claims 1-7.

Images