Multimodal imaging system and multimodal imaging method

Abstract

This application provides a multimodal imaging system and a multimodal imaging method, which can be applied to the field of image processing technology. The multimodal imaging system includes: an optical imaging unit, used to obtain an optical image of a target object by performing optical detection on the target object under the action of a signal isolation control unit; a magnetic resonance imaging unit, used to obtain a magnetic resonance image of the target object by performing magnetic resonance detection on the target object under the action of a signal isolation control unit; a signal isolation control unit, used to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signals of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit; and an image fusion unit, used to fuse the optical image and the magnetic resonance image to obtain a multimodal image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a multimodal imaging system and a multimodal image imaging method. Background Technology

[0002] Among related technologies, imaging techniques can be combined for disease diagnosis, treatment monitoring, and medical research. For example, images obtained based on magnetic resonance imaging (MRI) have advantages such as no ionizing radiation, high soft tissue contrast, and multi-parameter imaging capabilities, clearly presenting the internal anatomical structure and physiological function information of the target object. However, MRI technology also has limitations such as relatively limited spatial resolution and insufficient specific detection capabilities at the molecular level. Summary of the Invention

[0003] In view of this, embodiments of this application provide a multimodal imaging system and a multimodal imaging method.

[0004] One aspect of this application provides a multimodal imaging system, comprising: an optical imaging unit, configured to obtain an optical image of a target object by performing optical detection on the target object under the action of a signal isolation control unit; a magnetic resonance imaging unit, configured to obtain a magnetic resonance image of the target object by performing magnetic resonance detection on the target object under the action of a signal isolation control unit; a signal isolation control unit, configured to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signals of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit; and an image fusion unit, configured to fuse the optical image and the magnetic resonance image to obtain a multimodal image.

[0005] According to an embodiment of this application, a signal isolation control unit includes: an electromagnetic shielding subunit disposed outside the optical imaging unit, comprising: a metal shielding cover for shielding the electromagnetic field generated by the magnetic resonance imaging unit within the range of the optical imaging unit; an electromagnetic absorption layer for absorbing the electromagnetic field generated by the magnetic resonance imaging unit within the range of the optical imaging unit; a signal isolation subunit including: an optical coupling isolation device for isolating the electrical signals of the optical imaging unit; and a timing control subunit for controlling the detection timing of the optical imaging unit and the detection timing of the magnetic resonance imaging unit according to a preset timing control logic.

[0006] According to an embodiment of this application, an image fusion unit includes: an image registration subunit, configured to perform feature point matching between the optical image and the magnetic resonance image based on feature points in the optical image and feature points in the magnetic resonance image, obtain a spatial transformation matrix based on the feature point-matched optical image and magnetic resonance image, and perform a spatial transformation on the optical image based on the spatial transformation matrix so that the image space of the spatially transformed optical image matches the image space of the magnetic resonance image; and an image fusion subunit, configured to perform wavelet decomposition on the magnetic resonance image and the spatially matched optical image respectively to obtain low-frequency components and high-frequency components, fuse the low-frequency components and high-frequency components respectively, and perform inverse wavelet transform on the fused low-frequency components and the fused high-frequency components to obtain a multimodal image.

[0007] According to an embodiment of this application, the optical imaging unit includes a multi-mode optical detection subunit, used to perform optical detection on a target object according to a target optical detection mode among multiple optical detection modes, and obtain an optical image of the target object.

[0008] According to embodiments of this application, the multi-mode optical detection subunit includes at least one of a visible light detection module, a bioluminescence detection module, an excitation fluorescence detection module, or a 3D optical imaging module; the visible light detection module performs optical detection on the target object through a visible light optical detection mode, the bioluminescence detection module performs optical detection on the target object through a bioluminescence detection mode, the excitation fluorescence detection module performs optical detection on the target object through an excitation fluorescence optical detection mode, and the 3D optical imaging module performs optical detection on the target object through a 3D optical detection mode.

[0009] According to embodiments of this application, the visible light detection module includes a visible light source and a visible light detector. The visible light source is used to emit visible light, and the visible light detector is used to receive the visible light signal reflected by the target object. The bioluminescence detection module includes a first detector, which is used to receive a first fluorescence signal generated by a fluorescent marker in the target object. The excitation fluorescence detection module includes a multi-wavelength excitation source and a second detector. The multi-wavelength excitation source is used to emit excitation light of a preset wavelength, and the second detector is used to receive a second fluorescence signal generated under the action of the excitation light. The 3D optical imaging module includes a multi-view optical acquisition sub-module and a 3D reconstruction processing sub-module. The multi-view optical acquisition sub-module is used to continuously acquire multi-view optical sub-images of the target object through lenses with multiple different viewpoints, and the 3D reconstruction processing unit is used to perform stereo matching and three-dimensional reconstruction on the multi-view optical sub-images.

[0010] According to an embodiment of this application, the optical imaging unit further includes: an optical switching subunit, used to switch multiple optical detection modes so that the multi-mode optical detection subunit performs optical detection on the target object according to the target optical detection mode among the multiple optical detection modes.

[0011] According to an embodiment of this application, the multimodal imaging system further includes: an integrated support unit for connecting the magnetic resonance imaging unit and the optical imaging unit, wherein the material of the integrated support unit includes a non-magnetic material.

[0012] According to an embodiment of this application, a magnetic resonance imaging unit includes: a main magnet for generating a static magnetic field; a gradient coil for generating a gradient magnetic field; a radio frequency transmitting coil for transmitting radio frequency pulses to excite hydrogen nuclei of a target object to generate resonance; a radio frequency receiving coil for receiving resonance signals generated by the hydrogen nuclei of the target object under resonance conditions; and a magnetic resonance control subunit for processing the resonance signals and generating magnetic resonance images.

[0013] Another aspect of this application provides a multimodal image imaging method, comprising: using an optical imaging unit under the action of a signal isolation control unit to perform optical detection on a target object to obtain an optical image of the target object; using a magnetic resonance imaging unit under the action of a signal isolation control unit to perform magnetic resonance detection on the target object to obtain a magnetic resonance image of the target object; using a signal isolation control unit to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signal of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit; and using an image fusion unit to fuse the optical image and the magnetic resonance image to obtain a multimodal image.

[0014] Another aspect of this application provides an electronic device, including: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] Another aspect of this application provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] Another aspect of this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-described method.

[0017] According to embodiments of this application, the multimodal imaging system of this application uses a signal isolation control unit to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signals of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit. This overcomes the mutual interference problem between the optical imaging unit and the magnetic resonance imaging unit. Furthermore, the signal isolation control unit controls the optical imaging unit and the magnetic resonance imaging unit to work according to a certain timing sequence. Finally, the optical image and the magnetic resonance image are fused by the image fusion unit to obtain a multimodal image. Since the optical imaging unit and the magnetic resonance imaging unit work under the action of the signal isolation control unit, the magnetic resonance imaging unit does not generate electromagnetic field interference to the optical image, and the optical imaging unit does not generate electrical signal interference to the magnetic resonance image, thereby improving the imaging quality of the optical image and the magnetic resonance image. Based on this, the quality of the multimodal image obtained is also improved. Attached Figure Description

[0018] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 A block diagram of a multimodal imaging system according to an embodiment of this application is shown schematically;

[0020] Figure 2 A schematic diagram of a multimodal imaging system according to an embodiment of this application is shown.

[0021] Figure 3 A flowchart illustrating a multimodal image imaging method according to an embodiment of this application is shown schematically; and

[0022] Figure 4 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of this application, is illustrated schematically. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0028] In the embodiments of this application, the user's authorization or consent was obtained before obtaining or collecting the user's personal information.

[0029] Different modalities of medical imaging have their own unique advantages and limitations. For example, optical imaging techniques include visible light imaging, bioluminescence imaging, and excitation fluorescence imaging. Optical imaging techniques have advantages such as high sensitivity, high spatial resolution, and the ability to achieve molecular-specific labeling, and can be used to detect molecular activities and pathological changes in organisms. However, optical imaging has a shallow tissue penetration depth, making it difficult to obtain imaging information of deep tissues, and it cannot provide global anatomical structure localization.

[0030] Therefore, to compensate for the shortcomings of single-modality images, images from different modalities can be combined to achieve complementary advantages. Magnetic resonance-optical multimodal image fusion is currently a research hotspot. However, the integration between magnetic resonance imaging units and optical imaging units presents challenges. For example, strong magnetic fields, gradient fields, and radio frequency fields are generated during magnetic resonance imaging. These magnetic and electromagnetic fields can interfere with the operation of the optical imaging unit, leading to signal distortion and reduced signal-to-noise ratio in optical detection. Simultaneously, components such as the light source and detector of the optical imaging unit may also interfere with the magnetic field uniformity and radio frequency signal reception of the magnetic resonance imaging unit, affecting the quality of the magnetic resonance images.

[0031] Furthermore, the optical imaging units in existing integrated solutions are often single-function, mostly supporting only one optical imaging mode, which cannot meet the diverse imaging requirements of different target objects and different detection needs. For example, some solutions can only achieve visible light imaging and cannot detect labeled bioluminescent molecules; while some fluorescence imaging solutions are limited to excitation fluorescence at specific wavelengths, resulting in narrow applicability.

[0032] In view of this, embodiments of this application provide a multimodal imaging system. A signal isolation control unit shields the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolates the electrical signals of the optical imaging unit, and controls the working timing of the optical and magnetic resonance imaging units. This overcomes the mutual interference problem between the two units. Furthermore, the signal isolation control unit controls the optical and magnetic resonance imaging units to operate according to a specific timing sequence. Finally, an image fusion unit fuses the optical and magnetic resonance images to obtain a multimodal image. Because the optical and magnetic resonance imaging units operate under the control of the signal isolation control unit, the magnetic resonance imaging unit does not generate electromagnetic field interference to the optical image, and the optical imaging unit does not generate electrical signal interference to the magnetic resonance image, thereby improving the imaging quality of both the optical and magnetic resonance images. Based on this, the quality of the resulting multimodal image is also improved.

[0033] Figure 1 A block diagram of a multimodal imaging system according to an embodiment of this application is shown schematically.

[0034] like Figure 1 As shown, the multimodal imaging system 100 may include an optical imaging unit 110, a magnetic resonance imaging unit 120, a signal isolation control unit 130, and an image fusion unit 140.

[0035] The optical imaging unit 110 can be used to obtain an optical image of a target object by performing optical detection on the target object under the action of the signal isolation and control unit.

[0036] The magnetic resonance imaging unit 120 can be used to obtain a magnetic resonance image of a target object by performing magnetic resonance detection on the target object under the action of the signal isolation and control unit.

[0037] The signal isolation control unit 130 can be used to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signals of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit.

[0038] The image fusion unit 140 can be used to fuse optical images and magnetic resonance images to obtain multimodal images.

[0039] The optical imaging unit can perform optical detection on the target object under the action of the signal isolation and control unit. For example, it can generate a signal by interacting with the target object through light, thereby obtaining an optical image of the target object.

[0040] The magnetic resonance imaging unit can perform magnetic resonance detection on the target object under the action of the signal isolation and control unit. For example, it can use a strong magnetic field and radio frequency pulse to excite the hydrogen nuclei in the target object to generate resonance signals, thereby obtaining a magnetic resonance image of the target object.

[0041] The signal isolation control unit can shield the electromagnetic fields generated by the magnetic resonance imaging unit within the operating range of the optical imaging unit, and isolate the electrical signals of the optical imaging unit. This ensures that the electrical signals of the optical imaging unit do not affect the magnetic resonance imaging unit, and that the electromagnetic field signals of the magnetic resonance imaging unit do not affect the optical imaging unit. Furthermore, the signal isolation control unit can control the operating timing of the magnetic resonance imaging unit within the optical imaging unit, ensuring that both operate in an orderly manner.

[0042] The image fusion unit can fuse optical images and magnetic resonance images. For example, it can fuse optical images and magnetic resonance images of the same part of the target object to obtain a multimodal image of the target object.

[0043] According to embodiments of this application, the multimodal imaging system of this application uses a signal isolation control unit to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signals of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit. This overcomes the mutual interference problem between the optical imaging unit and the magnetic resonance imaging unit. Furthermore, the signal isolation control unit controls the optical imaging unit and the magnetic resonance imaging unit to work according to a certain timing sequence. Finally, the optical image and the magnetic resonance image are fused by the image fusion unit to obtain a multimodal image. Since the optical imaging unit and the magnetic resonance imaging unit work under the action of the signal isolation control unit, the magnetic resonance imaging unit does not generate electromagnetic field interference to the optical image, and the optical imaging unit does not generate electrical signal interference to the magnetic resonance image, thereby improving the imaging quality of the optical image and the magnetic resonance image. Based on this, the quality of the multimodal image obtained is also improved.

[0044] Figure 2 A schematic diagram of a multimodal imaging system according to an embodiment of this application is shown.

[0045] like Figure 2 As shown, the multimodal imaging system 100 may include an optical imaging unit 110, a magnetic resonance imaging unit 120, and an integrated scaffold unit 210. Figure 2 The absence of a signal isolation control unit and an image fusion unit does not indicate that... Figure 2 The multimodal imaging system in the text does not include a signal isolation control unit and an image fusion unit.

[0046] According to embodiments of this application, an integrated scaffold unit can be used to connect a magnetic resonance imaging unit and an optical imaging unit, wherein the material of the integrated scaffold unit includes a non-magnetic material.

[0047] The integrated stent unit can achieve mechanical integration of the magnetic resonance imaging unit and the optical imaging unit. The integrated stent unit can be made of non-magnetic materials to avoid interference with the magnetic resonance imaging unit.

[0048] The integrated support unit can be provided with a bearing area for the target object 220 for placing the target object, and a mounting base for the optical imaging unit for fixing the optical imaging unit.

[0049] The integrated support unit is made of non-magnetic material, which avoids affecting the magnetic field uniformity of the magnetic resonance imaging unit and ensures the quality of magnetic resonance images.

[0050] According to an embodiment of this application, the magnetic resonance imaging unit 120 may include a main magnet 121, a gradient coil 122, a radio frequency transmitting coil 123, a radio frequency receiving coil 124, and a magnetic resonance control subunit 125.

[0051] The main magnet can be used to generate a static magnetic field. The gradient coil can be used to generate a gradient magnetic field. The radio frequency (RF) transmitting coil can be used to transmit RF pulses, exciting the hydrogen nuclei of the target object to produce resonance. The RF receiving coil can be used to receive the resonance signal generated by the hydrogen nuclei of the target object under resonance conditions. The magnetic resonance control subunit can be used to process the resonance signal and generate magnetic resonance images. The magnetic resonance control unit can also control the operation of the main magnet, gradient coil, RF transmitting coil, and RF receiving coil.

[0052] Based on the embodiments of this application, continue to refer to Figure 2 The optical imaging unit may include a multi-mode optical detection subunit 111. The multi-mode optical detection subunit can be used to perform optical detection on a target object according to a target optical detection mode among multiple optical detection modes, and obtain an optical image of the target object.

[0053] According to embodiments of this application, the multi-mode optical detection subunit may include at least one of a visible light detection module, a bioluminescence detection module, an excitation fluorescence detection module, or a 3D optical imaging module.

[0054] The visible light detection module can perform optical detection of the target object through the visible light optical detection mode; the bioluminescence detection module can perform optical detection of the target object through the bioluminescence detection mode; the excitation fluorescence detection module can perform optical detection of the target object through the excitation fluorescence optical detection mode; and the 3D optical imaging module can perform optical detection of the target object through the 3D optical detection mode.

[0055] According to embodiments of this application, a visible light detection module may include a visible light source and a visible light detector. The visible light source can be used to emit visible light, and the visible light detector can be used to receive the visible light signal reflected by the target object.

[0056] The visible light source can be an LED visible light source, and the wavelength of visible light can be 400-760nm. The visible light detector can be a complementary metal-oxide-semiconductor (CMOS) image sensor.

[0057] The bioluminescence detection module may include a first detector. The first detector can be used to receive a first fluorescence signal generated by a fluorescent marker within the target object.

[0058] Fluorescent markers can be pre-added to the target object using chemical or biological methods. Then, a first detector detects the fluorescent markers within the target object and receives the resulting first fluorescence signal.

[0059] The excitation fluorescence detection module may include a multi-wavelength excitation source and a second detector. The multi-wavelength excitation source can be used to emit excitation light of a preset wavelength. The second detector can be used to receive the second fluorescence signal generated under the action of the excitation light.

[0060] The multi-wavelength excitation source can be a tunable laser, with a preset wavelength, for example, in the range of 300-800 nm. The second detector can be a hyperspectral imager, used to receive the second fluorescence signal generated under the action of the excitation light and acquire the fluorescence spectrum information of the second fluorescence signal.

[0061] The first detector and the second detector can be the same detector, that is, the bioluminescence detection module and the excitation fluorescence detection module can use the same detector, thereby reducing the number of devices in the system and reducing the complexity of the system.

[0062] The 3D optical imaging module can include a multi-view optical acquisition submodule and a 3D reconstruction processing submodule. The multi-view optical acquisition submodule can be used to continuously acquire multi-view optical sub-images of the target object through lenses with multiple different viewpoints, and the 3D reconstruction processing unit can be used to perform stereo matching and 3D reconstruction on the multi-view optical sub-images.

[0063] The multi-view optical acquisition submodule can employ an array of non-magnetic optical lenses (made of non-magnetic material). For example, it can continuously acquire multi-view optical images of the target object using 3-5 lenses with different perspectives. The 3D reconstruction processing submodule can perform stereo matching and 3D reconstruction based on the multi-view optical images to generate a 3D optical image of the target object. The multiple perspectives of the multi-view optical acquisition submodule can be achieved by keeping the target object stationary while rotating the optical camera.

[0064] Based on the embodiments of this application, continue to refer to Figure 2 The optical imaging unit may further include an optical switching subunit 112. The optical switching subunit can be used to switch between multiple optical detection modes, so that the multi-mode optical detection subunit can perform optical detection on the target object according to the target optical detection mode among the multiple optical detection modes.

[0065] The optical switching subunit can employ non-magnetic optical switching components to ensure stable operation in the strong magnetic field environment of the magnetic resonance imaging unit. The optical switching subunit can achieve rapid switching between the visible light detection module, the bioluminescence detection module, or the excitation fluorescence detection module.

[0066] The optical switching subunit can also switch between 2D and 3D imaging in various optical detection modes based on the 3D optical imaging module.

[0067] Continue to refer to Figure 2The optical imaging unit may also include an optical control subunit 113, which can be used to control the operation of the multi-mode optical detection subunit. The optical control subunit can be used for switching the light source on and off, adjusting the wavelength, acquiring the detector signal, and setting 3D reconstruction parameters in each detection mode.

[0068] To ensure that the detection lens of the optical imaging unit is aligned with the imaging area of ​​the target object, the multi-mode optical detection subunit can be placed on a rotating frame with the rotation center coinciding with the center of the bearing area of ​​the target object. The multi-mode optical detection subunit can rotate more than 270 degrees around the center of the bearing area, and can perform optical detection on the target object from different positions from 0 to 270 degrees.

[0069] According to embodiments of this application, the signal isolation control unit may include an electromagnetic shielding subunit, a signal isolation subunit, and a timing control subunit.

[0070] The electromagnetic shielding subunit can be installed outside the optical imaging unit. The electromagnetic shielding subunit is used to shield the optical imaging unit, preventing electromagnetic fields from the magnetic resonance imaging unit from interfering with the optical signals.

[0071] The electromagnetic shielding subunit may include a metal shielding cover and an electromagnetic absorption layer. The metal shielding cover can be used to shield the electromagnetic fields generated by the magnetic resonance imaging unit within the range of the optical imaging unit. The electromagnetic absorption layer can be used to absorb the electromagnetic fields generated by the magnetic resonance imaging unit within the range of the optical imaging unit.

[0072] The metal shielding cover can be made of copper, and it can block electromagnetic field interference from the magnetic resonance imaging unit. The electromagnetic absorption layer can be further used to absorb leaked electromagnetic waves.

[0073] The signal isolation subunit may include an optical isolation device. The optical isolation device can be used to isolate the electrical signals of the optical imaging unit.

[0074] The timing control subunit is used to control the detection timing of the optical imaging unit and the magnetic resonance imaging unit according to the preset timing control logic.

[0075] The timing control subunit can use a programmable gate array (PGA) as the core control chip. Through preset timing logic, it precisely controls the detection timing (e.g., trigger time) of the magnetic resonance imaging unit and the detection timing (e.g., detection time) of the optical imaging unit, ensuring temporal continuity between the magnetic resonance images and the optical images. Simultaneously, by setting positioning markers in the bearing area of ​​the target object, spatial continuity between the two modalities of the images is achieved.

[0076] According to embodiments of this application, the image fusion unit may include an image registration subunit and an image fusion subunit.

[0077] The image registration subunit can be used to match feature points in optical images and magnetic resonance images based on feature points in optical images and feature points in magnetic resonance images. Based on the feature point matching optical images and magnetic resonance images, a spatial transformation matrix is ​​obtained, and the optical images are spatially transformed based on the spatial transformation matrix so that the image space of the spatially transformed optical images matches the image space of the magnetic resonance images.

[0078] The image registration subunit can employ a registration algorithm based on feature point matching to spatially align the optical image with the magnetic resonance image. For example, the image registration subunit can use a registration algorithm based on feature point matching using scale-invariant feature transform (SIFT) to spatially align the optical image with the magnetic resonance image.

[0079] Specifically, the image registration subunit can extract feature points from both magnetic resonance images and optical images, and extract stable feature points from the images. For example, it can capture specific locations of the target object (such as the teeth or eyes of a mouse) as reference feature points, or it can use fixed points on the integrated scaffold unit as reference points.

[0080] The extracted feature points are described and generated, constructing feature point description vectors (e.g., 128-dimensional). The Fast Library for Approximate Nearest Neighbors (FLANN) algorithm is used to match the description vectors of the two sets of feature points, obtaining initial matching pairs. The Random Sample Consensus (RANSAC) algorithm is then used to eliminate incorrect matches from the initial matching pairs, obtaining correct feature point matching pairs. Based on the correct feature point matching pairs, the spatial transformation matrix between the magnetic resonance image and the optical image can be calculated using the least squares method. For example, the coordinate systems of the optical image can be aligned and paired through translation and rotation transformations, fixing the physical distance between the visible light imaging center and the magnetic resonance imaging center. The optical image is then spatially transformed according to the spatial transformation matrix to achieve spatial alignment with the magnetic resonance image.

[0081] The image fusion subunit can be used to perform wavelet decomposition on magnetic resonance images and optical images after image spatial matching to obtain low-frequency components and high-frequency components, fuse the low-frequency components and high-frequency components respectively, and perform inverse wavelet transform on the fused low-frequency components and fused high-frequency components to obtain multimodal images.

[0082] The image fusion subunit can employ a multi-scale fusion algorithm to fuse the two registered modalities, generating a multimodal image with both high anatomical resolution and high molecular specificity. For example, wavelet multi-scale fusion algorithms can be used to decompose the two registered images separately, obtaining low-frequency and high-frequency components. Then, a weighted average fusion rule is applied to the low-frequency components, and a gradient threshold-based fusion rule is applied to the high-frequency components. Finally, an inverse wavelet transform is performed on the fused low-frequency and high-frequency components to obtain the multimodal image.

[0083] The image fusion unit may also include a data preprocessing subunit, which performs preprocessing operations such as noise reduction and enhancement on magnetic resonance images and optical images to improve image quality. For example, the data preprocessing subunit may use a Gaussian filtering algorithm to reduce noise in magnetic resonance images and a histogram equalization algorithm to enhance optical image data, thereby improving image quality.

[0084] The image fusion unit, through preprocessing, registration, and fusion, can achieve precise registration and fusion of magnetic resonance imaging and optical images. The generated multimodal images combine the high anatomical resolution of magnetic resonance imaging with the high sensitivity and high molecular specificity of optical images, which helps to improve the accuracy and reliability of disease diagnosis.

[0085] Figure 3 A flowchart illustrating a multimodal image imaging method according to an embodiment of this application is shown schematically.

[0086] like Figure 3 As shown, the method includes operations S310 to S340.

[0087] When operating S310, the optical imaging unit, under the action of the signal isolation control unit, performs optical detection on the target object to obtain an optical image of the target object.

[0088] When operating S320, the magnetic resonance imaging unit, under the action of the signal isolation control unit, performs magnetic resonance detection on the target object to obtain the magnetic resonance image of the target object.

[0089] When operating S330, the signal isolation control unit is used to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signals of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit.

[0090] In operation of S340, the image fusion unit is used to fuse optical images and magnetic resonance images to obtain multimodal images.

[0091] The target object can be placed in the support area of ​​the integrated support unit, and the position of the integrated support unit can be adjusted to ensure that the target object is within the imaging field of view of both the optical imaging unit and the magnetic resonance imaging unit. After optical inspection is completed, the integrated support unit can be moved into the imaging field of view of the magnetic resonance imaging unit for magnetic resonance inspection.

[0092] The operating parameters and timing of the magnetic resonance imaging (MRI) unit and the optical imaging unit are controlled by a signal isolation and control unit. The optical detection mode of the optical imaging unit is selected according to the detection requirements. For example, when observing the two-dimensional anatomical structure of the target object, the visible light 2D imaging mode is selected. When detecting the three-dimensional distribution of fluorescent markers within the target object, the bioluminescence 3D imaging mode is selected. When fluorescence spectroscopy analysis is required, the excitation fluorescence imaging mode is selected.

[0093] After obtaining optical and magnetic resonance images, an image fusion unit can be used to fuse the optical and magnetic resonance images to obtain multimodal images.

[0094] During the acquisition of optical and magnetic resonance images, the signal isolation control unit works continuously to block the electromagnetic field of the magnetic resonance imaging unit from interfering with the optical imaging unit in real time, while isolating the electrical signal transmission between the two to ensure the quality of image acquisition.

[0095] This application achieves efficient integration of magnetic resonance imaging with multiple optical imaging methods. The optical imaging unit integrates multiple optical detection modes such as visible light detection, bioluminescence detection, and excitation fluorescence detection. It also supports flexible switching between 2D and 3D imaging under various optical detection modes, which broadens the application range of the system and can meet diverse biomedical detection needs, such as two-dimensional anatomical structure observation, three-dimensional distribution detection of molecular specific markers, fluorescence spectroscopy analysis, and three-dimensional structural characterization of biological samples.

[0096] Figure 4 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of this application, is illustrated schematically. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0097] like Figure 4As shown, an electronic device 400 according to an embodiment of this application includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0098] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0099] According to embodiments of this application, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0100] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0101] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0102] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0103] For example, according to embodiments of this application, a computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0104] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this application.

[0105] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0106] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0107] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0109] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A multimodal imaging system, characterized in that, include: An optical imaging unit is used to obtain an optical image of a target object by performing optical detection on the target object under the action of a signal isolation and control unit. A magnetic resonance imaging unit is used to obtain a magnetic resonance image of a target object by performing magnetic resonance detection on the target object under the action of a signal isolation and control unit. The signal isolation control unit is used to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signals of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit. An image fusion unit is used to fuse the optical image and the magnetic resonance image to obtain a multimodal image.

2. The multimodal imaging system according to claim 1, characterized in that, The signal isolation control unit includes: An electromagnetic shielding subunit, disposed outside the optical imaging unit, includes: A metal shielding cover is used to shield the electromagnetic field generated by the magnetic resonance imaging unit within the area of ​​the optical imaging unit; An electromagnetic absorption layer is used to absorb the electromagnetic field generated by the magnetic resonance imaging unit within the range of the optical imaging unit. The signal isolation subunit includes: Optical isolation devices are used to isolate the electrical signals of the optical imaging unit; The timing control subunit is used to control the detection timing of the optical imaging unit and the detection timing of the magnetic resonance imaging unit according to the preset timing control logic.

3. The multimodal imaging system according to claim 1, characterized in that, The image fusion unit includes: The image registration subunit is used to match feature points in the optical image and the magnetic resonance image based on feature points in the optical image and feature points in the magnetic resonance image, obtain a spatial transformation matrix based on the feature point matched optical image and magnetic resonance image, and perform spatial transformation on the optical image based on the spatial transformation matrix so that the image space of the spatially transformed optical image matches the image space of the magnetic resonance image. The image fusion subunit is used to perform wavelet decomposition on the magnetic resonance image and the optical image after image spatial matching to obtain low-frequency components and high-frequency components, fuse the low-frequency components and the high-frequency components respectively, and perform inverse wavelet transform on the fused low-frequency components and the fused high-frequency components to obtain the multimodal image.

4. The multimodal imaging system according to claim 1, characterized in that, The optical imaging unit includes: A multi-mode optical detection subunit is used to perform optical detection on the target object according to the target optical detection mode among multiple optical detection modes, so as to obtain the optical image of the target object.

5. The multimodal imaging system according to claim 4, characterized in that, The multi-mode optical detection subunit includes at least one of a visible light detection module, a bioluminescence detection module, an excitation fluorescence detection module, or a 3D optical imaging module. The visible light detection module performs optical detection on the target object using a visible light optical detection mode; the bioluminescence detection module performs optical detection on the target object using a bioluminescence detection mode; the excitation fluorescence detection module performs optical detection on the target object using an excitation fluorescence optical detection mode; and the 3D optical imaging module performs optical detection on the target object using a 3D optical detection mode.

6. The multimodal imaging system according to claim 5, characterized in that, The visible light detection module includes a visible light source and a visible light detector. The visible light source is used to emit visible light, and the visible light detector is used to receive the visible light signal reflected by the target object. The bioluminescence detection module includes a first detector, which is used to receive a first fluorescence signal generated by a fluorescent marker in the target object; The excitation fluorescence detection module includes a multi-wavelength excitation light source and a second detector. The multi-wavelength excitation light source is used to emit excitation light of a preset wavelength, and the second detector is used to receive the second fluorescence signal generated under the action of the excitation light. The 3D optical imaging module includes a multi-view optical acquisition sub-module and a 3D reconstruction processing sub-module. The multi-view optical acquisition sub-module is used to continuously acquire multi-view optical sub-images of the target object through lenses with multiple different viewpoints. The 3D reconstruction processing unit is used to perform stereo matching and three-dimensional reconstruction on the multi-view optical sub-images.

7. The multimodal imaging system according to claim 5, characterized in that, The optical imaging unit further includes: An optical switching subunit is used to switch between multiple optical detection modes so that the multi-mode optical detection subunit performs optical detection on the target object according to the target optical detection mode among the multiple optical detection modes.

8. The multimodal imaging system according to claim 1, characterized in that, The multimodal imaging system also includes: An integrated support unit is used to connect the magnetic resonance imaging unit and the optical imaging unit, wherein the material of the integrated support unit includes a non-magnetic material.

9. The multimodal imaging system according to claim 1, characterized in that, The magnetic resonance imaging unit includes: The main magnet is used to generate a static magnetic field. Gradient coils are used to generate gradient magnetic fields; Radio frequency transmitting coils are used to transmit radio frequency pulses to excite the hydrogen nuclei of the target object to generate resonance; Radio frequency receiving coil, used to receive the resonance signal generated by the hydrogen nuclei of the target object under resonance conditions; A magnetic resonance control subunit is used to process the resonance signal and generate the magnetic resonance image.

10. A multimodal image imaging method, characterized in that, include: Using an optical imaging unit under the action of a signal isolation and control unit, an optical image of the target object is obtained by optical detection of the target object; Using the magnetic resonance imaging unit under the action of the signal isolation and control unit, the magnetic resonance image of the target object is obtained by performing magnetic resonance detection on the target object; The signal isolation control unit is used to shield the electromagnetic field generated by the magnetic resonance imaging unit within the working range of the optical imaging unit, isolate the electrical signals of the optical imaging unit, and control the working timing of the optical imaging unit and the magnetic resonance imaging unit. The optical image and the magnetic resonance image are fused using an image fusion unit to obtain a multimodal image.

Images