A photoacoustic-magnetic multimodal imaging system and method for prostate cancer identification

Abstract

The application relates to the technical field of medical imaging, in particular to a photoacoustic magnetic multi-modal imaging system and method for prostate cancer identification, which comprises the following steps: the photoacoustic imaging unit is used for photoacoustic imaging of microvessels in a prostate part; the ultrasonic elastic imaging unit is used for ultrasonic elastic imaging of the prostate part; the magnetic resonance imaging unit is used for magnetic resonance imaging of the prostate part; the multi-modal image fusion unit is used for information fusion of photoacoustic imaging, ultrasonic elastic imaging and magnetic resonance imaging of the prostate part, so as to obtain a multi-modal fusion image of the prostate part. The application adopts high-quality imaging equipment for the prostate part, guarantees high-quality acquisition of the multi-modal images in the imaging stage, and after the high-quality image acquisition is completed, a bidirectional supervision registration network is used to realize registration and fusion of photoacoustic, ultrasonic elastic images and high-resolution magnetic resonance images, so that high-quality images are provided for image identification of prostate cancer.

Description

Technical Field

[0001] This invention relates to the field of medical imaging technology, specifically to a photoacoustic-magnetic multimodal imaging system and method for prostate cancer identification. Background Technology

[0002] Currently, lesion identification in medicine primarily relies on imaging techniques, such as magnetic resonance imaging (MRI), ultrasound, and photoacoustic imaging. Taking prostate cancer identification as an example, MRI or ultrasound images of the prostate area are acquired and used to identify prostate cancer. In practical applications, relying on a single modality of imagery is insufficient for accurate identification due to information limitations. Therefore, combining multimodal images is typically necessary to achieve precise lesion identification. Besides using multimodal imaging to improve lesion identification accuracy, the accuracy also depends on the image quality; higher image quality leads to greater accuracy in lesion identification.

[0003] Existing imaging technologies mostly use general-purpose equipment for prostate image acquisition, lacking high-quality multimodal imaging design for the prostate. As a result, it is difficult to ensure high-resolution acquisition of prostate information during the imaging stage, affecting the recognition results. After imaging, there is a lack of methods for high-quality registration of multimodal images, making it difficult to ensure a comprehensive display of prostate information during the multimodal information fusion stage, which ultimately affects the lesion recognition results again. Summary of the Invention

[0004] The purpose of this invention is to provide a photoacoustic-magnetic multimodal imaging system and method for prostate cancer identification, in order to solve the technical problems in the prior art that lack high-quality imaging design for multimodal images of the prostate site and lack a method design for high-quality registration of multimodal images, which adversely affects the lesion identification results.

[0005] To solve the above-mentioned technical problems, the present invention specifically provides the following technical solution:

[0006] A photoacoustic-magnetic multimodal imaging system for prostate cancer identification includes: a photoacoustic imaging unit, an ultrasound elastography unit, a magnetic resonance imaging unit, and a multimodal image fusion module.

[0007] The photoacoustic imaging unit includes an OPO multi-wavelength laser, a 360° diffused optical fiber, an ultrasonic probe array, and a control host, and is used for microvascular photoacoustic imaging of the prostate area.

[0008] The ultrasound elastography unit includes an acoustic radiation force excitation control module, an ultrasound probe array, a multi-channel high-speed acquisition module, and a control host, and is used for ultrasound elastography of the prostate area.

[0009] The magnetic resonance imaging unit includes a semi-flexible prostate magnetic resonance radio frequency coil and a magnetic resonance device for magnetic resonance imaging of the prostate area.

[0010] The multimodal image fusion unit is a computer device loaded with the medical image processing software 3D Slicer, used to fuse information from microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the prostate area to obtain a multimodal fused image of the prostate area.

[0011] As a preferred embodiment of the present invention, the OPO multi-wavelength laser in the photoacoustic imaging unit is the imaging light source for microvascular photoacoustic imaging, with a wavelength range of 680-2500mm.

[0012] The 360° diffused optical fiber is combined with a urinary catheter to be introduced into the prostate area through the urethra to uniformly irradiate the inner surface of the prostate area and excite photoacoustic signals.

[0013] The ultrasonic probe array acquires photoacoustic signals to obtain photoacoustic signal data.

[0014] The control host performs optical power compensation, filtering, and peak detection on the photoacoustic signal data, and then reconstructs the photoacoustic image based on the beamforming algorithm or the delay summation algorithm to obtain the microvascular photoacoustic imaging.

[0015] As a preferred embodiment of the present invention, the acoustic radiation force excitation control module in the ultrasonic elastography unit includes a transmitting circuit, a receiving circuit, and a data transmission circuit. The transmitting circuit consists of three parts: a field-effect transistor driver, a field-effect transistor, and an impedance matching circuit. The receiving circuit consists of four parts: a transmit / receive switch, a configurable signal amplification, a configurable analog filtering circuit, and a high-resolution analog-to-digital converter. The data transmission circuit uses a PCIE interface for high-speed communication with the control host.

[0016] The acoustic radiation force excitation control module is used to drive the ultrasonic probe array to emit one or more long pulse ultrasonic waves, forming a 100-microsecond focus at the prostate site, generating acoustic radiation force on the tissue at the focal point, driving tissue deformation and forming a shear wave source;

[0017] The multi-channel high-speed acquisition module uses fast ultra-high frame rate plane wave imaging technology to track the motion of shear waves in tissues and receive ultrasound radio frequency signal images.

[0018] The control host calculates the tissue deformation caused by shear waves from the radio frequency signal image using a tissue displacement estimation algorithm, estimates the propagation velocity of the shear waves at various points in the tissue, and then calculates the shear modulus of the tissue to obtain the ultrasound elastography.

[0019] As a preferred embodiment of the present invention, the semi-flexible prostate magnetic resonance radio frequency coil in the magnetic resonance imaging unit includes a coil body and a preamplifier. The coil body includes three overlapping loops, each loop being a signal acquisition channel. The acquired radio frequency signals are transmitted to the preamplifier through a coaxial cable.

[0020] A semi-flexible prostate magnetic resonance radiofrequency coil, embedded in the prostate area, is inserted into a magnetic resonance device for scanning to obtain the magnetic resonance image.

[0021] As a preferred embodiment of the present invention, the medical image processing software 3DSlicer in the multimodal image fusion unit uses a pre-established bidirectional supervised registration network to perform bidirectional registration and fusion of microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging in the prostate area between feature registration and grayscale registration methods to obtain the multimodal fused image.

[0022] As a preferred embodiment of the present invention, the bidirectional supervised registration network includes a first registration substructure and a second registration substructure, both of which are composed of a neural network based on feature registration and a neural network based on grayscale registration.

[0023] In the first registration substructure, the feature registration-based neural network is placed before the grayscale registration-based neural network. The feature registration-based neural network in the first registration substructure uses magnetic resonance imaging as a floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate a feature registration image of magnetic resonance imaging.

[0024] The expression for the feature-registration-based neural network in the first registration substructure is:

[0025] ;

[0026] ;

[0027] ;

[0028] In the formula, For magnetic resonance imaging, Photoacoustic imaging of microvessels. For ultrasound elastography, for and Deformation field of feature registration between them for and Deformation field of feature registration between them Image registration for features of magnetic resonance imaging. For neural networks that predict deformation fields, It is a space transformation function;

[0029] The loss function of the feature-registration-based neural network in the first registration substructure is:

[0030] ;

[0031] In the formula, The characteristic loss of the first registration substructure, For feature extraction networks, It is an L1 norm form;

[0032] In the first registration substructure, the neural network based on gray-scale registration uses the feature registration image of magnetic resonance imaging as the floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate the feature gray-scale registration image of magnetic resonance imaging.

[0033] The expression for the neural network based on grayscale registration in the first registration substructure is as follows:

[0034] ;

[0035] ;

[0036] ;

[0037] In the formula, for and Deformation field for grayscale registration between them for and Deformation field for grayscale registration between them For the characteristic grayscale registration image of magnetic resonance imaging;

[0038] The loss function of the neural network based on grayscale registration in the first registration substructure is:

[0039] ;

[0040] In the formula, The grayscale loss of the first registration substructure;

[0041] In the second registration sub-network, the neural network based on grayscale registration is placed before the neural network based on feature registration. The neural network based on grayscale registration in the second registration sub-structure uses magnetic resonance imaging as a floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate a grayscale registered image of magnetic resonance imaging.

[0042] The expression for the neural network based on grayscale registration in the second registration substructure is as follows:

[0043] ;

[0044] ;

[0045] ;

[0046] In the formula, for and Deformation field for grayscale registration between them for and Deformation field for grayscale registration between them Gray-scale registered images for magnetic resonance imaging;

[0047] The loss function of the neural network based on grayscale registration in the second registration substructure is:

[0048] ;

[0049] In the formula, This refers to the grayscale loss of the second registration substructure;

[0050] In the second registration substructure, the feature registration-based neural network uses magnetic resonance imaging as a floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate a grayscale feature registration image of magnetic resonance imaging.

[0051] The expression for the feature-registration-based neural network in the second registration substructure is as follows:

[0052] ;

[0053] ;

[0054] ;

[0055] In the formula, for and Deformation field of feature registration between them for and Deformation field of feature registration between them Image registration for grayscale features of magnetic resonance imaging;

[0056] The loss function of the feature-registration-based neural network in the second registration substructure is:

[0057] ;

[0058] In the formula, The characteristic loss of the second registration substructure;

[0059] The loss function for the mutual constraints between the first registration substructure and the second registration substructure is:

[0060] ;

[0061] In the formula, This represents the constraint loss between the first and second registration substructures.

[0062] As a preferred embodiment of the present invention, the present invention provides a photoacoustic-magnetic multimodal imaging method for prostate cancer identification, applied to a photoacoustic-magnetic multimodal imaging system for prostate cancer identification, comprising the following steps:

[0063] Photoacoustic imaging of microvessels in the prostate area using a photoacoustic imaging unit;

[0064] Ultrasound elastography of the prostate area was performed using an ultrasound elastography unit.

[0065] Magnetic resonance imaging of the prostate area was performed using a magnetic resonance imaging unit.

[0066] Information from microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging in the prostate area is fused to obtain a multimodal fused image of the prostate area.

[0067] In a preferred embodiment of the present invention, the OPO multi-wavelength laser in the photoacoustic imaging unit emits pulsed laser light that is uniformly irradiated on the inner surface of the prostate via a 360° diffused optical fiber to excite photoacoustic signals. The photoacoustic signals are then acquired by an ultrasonic probe array to obtain photoacoustic signal data. Finally, the photoacoustic signal data is subjected to optical power compensation, filtering, and peak detection by the control host. The photoacoustic image is then reconstructed based on the beamforming algorithm or the delay and sum algorithm, which serves as the microvascular photoacoustic imaging.

[0068] As a preferred embodiment of the present invention, the acoustic radiation force excitation control module in the ultrasonic elastography unit is used to drive the ultrasonic probe array to emit one or more long pulse ultrasonic waves, forming a focus of about 100 microseconds in the prostate area, generating acoustic radiation force on the tissue at the focal point, driving tissue deformation and forming a shear wave source;

[0069] The multi-channel high-speed acquisition module uses fast ultra-high frame rate plane wave imaging technology to track the motion of shear waves in tissues and receive ultrasound radio frequency signal images.

[0070] The control host is used to calculate the tissue deformation caused by shear waves from the radio frequency signal image through a tissue displacement estimation algorithm, estimate the propagation velocity of the shear waves at various points in the tissue, and then calculate the shear modulus of the tissue to obtain the ultrasound elastography.

[0071] As a preferred embodiment of the present invention, the multi-modal fusion image is obtained by using the medical image processing software 3D Slicer to register microvascular photoacoustic imaging and ultrasound elastography as reference images and magnetic resonance imaging as a floating image.

[0072] Compared with the prior art, the present invention has the following advantages:

[0073] This invention employs a high-quality imaging device for the prostate area, ensuring high-quality acquisition of images of each modality during the imaging stage. After acquiring high-quality images, a bidirectional supervised registration network is used to perform bidirectional registration using features and grayscale, achieving registration and fusion of photoacoustic and ultrasound elastography images (functional images) with high-resolution magnetic resonance images (anatomical images), thus achieving accurate functional image localization and providing high-quality images for the imaging identification of prostate cancer. Attached Figure Description

[0074] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0075] Figure 1 This is a block diagram of a photoacoustic-magnetic multimodal imaging system provided in an embodiment of the present invention;

[0076] Figure 2 A diagram of a photoacoustic imaging unit device provided in an embodiment of the present invention;

[0077] Figure 3 This is a block diagram of the acoustic radiation force excitation control module provided in an embodiment of the present invention;

[0078] Figure 4 A diagram of a semi-flexible prostate magnetic resonance radiofrequency coil device provided in an embodiment of the present invention;

[0079] Figure 5 A diagram illustrating the use of a semi-flexible prostate magnetic resonance radiofrequency coil provided in an embodiment of the present invention;

[0080] Figure 6 This is a diagram of the bidirectional supervised registration network structure provided in an embodiment of the present invention;

[0081] Figure 7This is a flowchart of the photoacoustic-magnetic multimodal imaging method provided in an embodiment of the present invention. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] like Figure 1 As shown, the present invention provides a photoacoustic-magnetic multimodal imaging system for prostate cancer identification, comprising: a photoacoustic imaging unit, an ultrasound elastography unit, a magnetic resonance imaging unit, and a multimodal image fusion module.

[0084] Photoacoustic imaging unit (e.g.) Figure 2 (As shown) It includes an OPO multi-wavelength laser, a 360° diffused fiber, an ultrasound probe array, and a control unit, for microvascular photoacoustic imaging of the prostate area;

[0085] In the photoacoustic imaging unit, the OPO multi-wavelength laser is the imaging light source for microvascular photoacoustic imaging, with a wavelength range of 680-2500mm.

[0086] A 360° diffused optical fiber is combined with a urinary catheter to be introduced into the prostate area through the urethra to uniformly irradiate the inner surface of the prostate area and excite photoacoustic signals.

[0087] An ultrasonic probe array acquires photoacoustic signals to obtain photoacoustic signal data;

[0088] The control host performs optical power compensation and filtering on the photoacoustic signal data, then uses Hilbert transformed packet peak detection, and reconstructs the photoacoustic image based on the beamforming algorithm or the delay and sum algorithm to obtain microvascular photoacoustic imaging.

[0089] The basic principle of photoacoustic imaging (PAI) can be simply summarized as imaging by detecting broadband ultrasound (photoacoustic) waves excited by pulsed light.

[0090] In this invention, an OPO multi-wavelength laser emits pulsed laser light that is uniformly irradiated on the inner surface of the prostate via a 360° diffused optical fiber to excite photoacoustic signals. The photoacoustic signals are then collected by an ultrasound probe array (or transrectal ultrasound probe array, which realizes transrectal ultrasound scanning) to obtain photoacoustic signal data. Finally, the host computer performs optical power compensation, filtering, and peak detection on the photoacoustic signal data, and then reconstructs the photoacoustic image based on the Beamforming algorithm or the Delay and Sum algorithm as microvascular photoacoustic imaging.

[0091] The ultrasound elastography unit includes an acoustic radiation force excitation control module, an ultrasound probe array, a multi-channel high-speed acquisition module, and a control host, and is used for ultrasound elastography of the prostate area.

[0092] Acoustic radiation force excitation control module in ultrasonic elastography unit (e.g.) Figure 3 (As shown) It includes a transmitting circuit, a receiving circuit, and a data transmission circuit. The transmitting circuit consists of three parts: a field-effect transistor driver, a field-effect transistor, and an impedance matching component. The receiving circuit consists of four parts: a transmit / receive switching switch, a configurable signal amplification, a configurable analog filter, and a high-resolution analog-to-digital conversion component. The data transmission circuit uses a PCIe interface to communicate with the control host at high speed.

[0093] The acoustic radiation force excitation control module is used to drive the ultrasound probe array to emit one or more long pulse ultrasound waves, which form a focus of about 100 microseconds in the prostate area, generate acoustic radiation force on the tissue at the focal point, promote tissue deformation and form a shear wave source;

[0094] The multi-channel high-speed acquisition module uses fast ultra-high frame rate plane wave imaging technology to track the motion of shear waves in tissues and receive ultrasound radio frequency signal images;

[0095] The control host calculates the tissue deformation caused by shear waves from the radio frequency signal image using a tissue displacement estimation algorithm, estimates the propagation velocity of the shear waves at various points in the tissue, and then calculates the shear modulus of the tissue to obtain ultrasound elastography.

[0096] The magnetic resonance imaging unit includes a semi-flexible prostate magnetic resonance radiofrequency coil (such as...) Figure 4 (as shown) and a magnetic resonance imaging device for magnetic resonance imaging of the prostate region;

[0097] A semi-flexible prostate MRI radiofrequency coil is embedded in the prostate area within a magnetic resonance imaging (MRI) unit and simultaneously inserted into the MRI scanner for scanning, resulting in MRI images. The radiofrequency receiving coil, as the primary component for signal acquisition, is crucial for the system to obtain high-resolution images. However, due to the rapid signal attenuation with distance, traditional surface coils struggle to achieve high-resolution imaging of the prostate, which is located deep within the body. Therefore, this invention provides a transrectal prostate coil that can function independently as a local coil for prostate imaging (e.g., ...). Figure 5 (Left image), it can also work with traditional surface coils (such as...) Figure 5 (Right image).

[0098] In summary, this invention employs a high-quality imaging device for the prostate area, ensuring high-quality acquisition of images of various modalities during the imaging stage.

[0099] The multimodal image fusion unit is a computer device loaded with the medical image processing software 3D Slicer. It is used to fuse information from microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the prostate area to obtain a multimodal fused image of the prostate area.

[0100] The semi-flexible prostate magnetic resonance radio frequency coil in the magnetic resonance imaging unit includes a coil body and a preamplifier. The coil body includes three overlapping loops, each loop being a signal acquisition channel. The acquired radio frequency signals are transmitted to the preamplifier through a coaxial cable.

[0101] A semi-flexible prostate magnetic resonance radiofrequency coil is placed in the prostate area, and the magnetic resonance image is obtained by scanning with a magnetic resonance device.

[0102] The medical image processing software 3D Slicer in the multimodal image fusion unit uses a pre-established bidirectional supervised registration network with microvascular photoacoustic imaging and ultrasound elastography as reference images and magnetic resonance imaging as floating images. It performs bidirectional registration and fusion of microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging of the prostate area between feature registration mode and grayscale registration mode to obtain a multimodal fused image.

[0103] like Figure 6 As shown, the bidirectional supervised registration network includes a first registration substructure and a second registration substructure. Both the first registration substructure and the second registration substructure are composed of a neural network based on feature registration and a neural network based on grayscale registration.

[0104] To achieve high-quality registration of microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging images, this invention employs two different registration methods. The first method uses image pixel grayscale similarity for registration, which has the advantages of being computationally simple, easy to implement, and insensitive to changes in illumination. The second method uses image feature similarity for registration. Commonly used features include feature points, line segments, contours, and statistical moments, which have the advantages of low computational cost, good robustness, and insensitivity to complex geometric deformations. Combining feature and grayscale information methods can fully utilize the advantages of both, improving the accuracy and robustness of registration and achieving high-quality registration.

[0105] In combining feature registration and grayscale registration, this invention employs a mutually constrained combination order. The first registration substructure performs feature registration followed by grayscale registration, while the second registration substructure performs grayscale registration followed by feature registration. The difference between the registration results produced by the first and second registration substructures is used as the loss. This ensures that during the training process of the two substructures, the registration results of the two substructures continuously approach each other, with lower-quality registration results moving closer to higher-quality ones, forming a progressive process of mutual improvement. Ultimately, this allows both substructures to achieve their highest performance and obtain high-quality registration results, resulting in a mutually supervised global optimization process.

[0106] In the first registration substructure, the feature registration-based neural network is placed before the grayscale registration-based neural network. The feature registration-based neural network in the first registration substructure uses magnetic resonance imaging as a floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate a feature registration image of magnetic resonance imaging.

[0107] The expression for the feature-registration-based neural network in the first registration substructure is:

[0108] ;

[0109] ;

[0110] ;

[0111] In the formula, For magnetic resonance imaging, Photoacoustic imaging of microvessels. For ultrasound elastography, for and Deformation field of feature registration between them for and Deformation field of feature registration between them Image registration for features of magnetic resonance imaging. For neural networks that predict deformation fields, It is a space transformation function;

[0112] The loss function of the feature-registration-based neural network in the first registration substructure is:

[0113] ;

[0114] In the formula, The characteristic loss of the first registration substructure, For feature extraction networks, It is an L1 norm form;

[0115] In the first registration substructure, the neural network based on gray-scale registration uses the feature registration image of magnetic resonance imaging as the floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate the feature gray-scale registration image of magnetic resonance imaging.

[0116] The expression for the neural network based on grayscale registration in the first registration substructure is:

[0117] ;

[0118] ;

[0119] ;

[0120] In the formula, for and Deformation field for grayscale registration between them for and Deformation field for grayscale registration between them For the characteristic grayscale registration image of magnetic resonance imaging;

[0121] The loss function of the neural network based on gray-level registration in the first registration substructure is:

[0122] ;

[0123] In the formula, This represents the grayscale loss of the first registration substructure.

[0124] In this invention, the first registration substructure executes a combination of feature registration followed by grayscale registration, implemented by two neural networks: a feature registration-based neural network and a grayscale registration-based neural network. The feature registration-based neural network performs feature registration on microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging (MRI) images, registering the MRI image with the microvascular photoacoustic imaging and ultrasound elastography images respectively. Then, it registers the two MRI images... and The images are fused to obtain the final registered magnetic resonance images. ,by For loss ( This is used to extract features from images for registration. Among the three modalities, blood vessels are most commonly used as a common feature for feature registration. Other feature points can also be selected as needed to ensure successful feature registration. It has the highest local feature similarity with the original microvascular photoacoustic imaging and ultrasound elastography in the first registration substructure, thus achieving the highest quality feature registration result in the first registration substructure.

[0125] Existing segmentation networks, such as U-Net and FastR-CNN, can be used, or a custom segmentation network can be used. The same neural network... Existing deformation field localization networks (GANs) or custom deformation field localization networks can be used. For example, a deformation field localization network might resemble the encoder-decoder structure of a segmentation network, containing an encoder, decoder, skip connections, and a bottleneck module. The encoder layer extracts features from the prostate image input to the network. The decoder layer establishes a spatial deformation field based on the correspondences between the extracted features. The encoder and decoder employ residual convolutional blocks in the network layers, performing downsampling through convolutions with a stride of 2 and upsampling through transposed convolutions with a stride of 2. Skip connections, integrating residual convolutional blocks and channel attention mechanisms, pass more information extracted from the encoder to the decoder layer, while avoiding excessive semantic differences between the encoder's low-level semantic features and the decoder's high-level semantic features. The bottleneck module integrates a dual attention mechanism to enhance the network's ability to map features between images. Finally, the network outputs a spatial deformation field for registration.

[0126] The grayscale registration-based neural network follows the feature registration-based neural network. Its function is to perform grayscale registration on microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging (MRI) images based on the already completed feature registration. Specifically, it registers the feature-registered MRI image with the microvascular photoacoustic imaging and ultrasound elastography images, and then combines the two registered MRI images... and The images are fused to obtain the final registered magnetic resonance images. ,by To minimize loss, ensure that grayscale registration based on feature registration yields the desired result. It has the highest local gray-level similarity with the original microvascular photoacoustic imaging and ultrasound elastography in the first registration substructure, thus achieving the highest quality local gray-level registration result in the first registration substructure.

[0127] The above describes the neural networks based on feature registration and grayscale registration in the first registration substructure. and The loss is used for training, aiming to achieve the highest local performance in feature registration and grayscale registration. The two evolve on their own, generating a self-supervised local optimization process. This ensures that the performance of the first registration substructure is continuously optimized, thereby continuously producing higher quality registration results than the second registration substructure, which then catches up. This process supervises and guides the performance improvement of the second registration substructure.

[0128] In the second registration sub-network, the gray-scale registration-based neural network is placed before the feature registration-based neural network. The gray-scale registration-based neural network in the second registration sub-structure uses magnetic resonance imaging as a floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate a gray-scale registered image of magnetic resonance imaging.

[0129] The expression for the neural network based on grayscale registration in the second registration substructure is:

[0130] ;

[0131] ;

[0132] ;

[0133] In the formula, for and Deformation field for grayscale registration between them for and Deformation field for grayscale registration between them Gray-scale registered images for magnetic resonance imaging;

[0134] The loss function of the neural network based on gray-level registration in the second registration substructure is:

[0135] ;

[0136] In the formula, This refers to the grayscale loss of the second registration substructure;

[0137] In the second registration substructure, the feature registration-based neural network uses magnetic resonance imaging as a floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate a grayscale feature registration image of magnetic resonance imaging.

[0138] The expression for the feature-registration-based neural network in the second registration substructure is:

[0139] ;

[0140] ;

[0141] ;

[0142] In the formula, for and Deformation field of feature registration between them for and Deformation field of feature registration between them Image registration for grayscale features of magnetic resonance imaging;

[0143] The loss function of the feature-registration-based neural network in the second registration substructure is:

[0144] ;

[0145] In the formula, The characteristic loss of the second registration substructure;

[0146] In this invention, the second registration substructure performs a combination of grayscale registration followed by feature registration, implemented by two neural networks: a grayscale registration-based neural network and a feature registration-based neural network. The grayscale registration-based neural network performs grayscale registration on microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging (MRI) images, registering the MRI image with the microvascular photoacoustic imaging and ultrasound elastography images respectively. Then, it registers the two MRI images... and The images are fused to obtain the final registered magnetic resonance images. ,by To minimize loss, ensure that grayscale registration is achieved. It has the highest local gray-level similarity with the original microvascular photoacoustic imaging and ultrasound elastography in the second registration substructure, thus achieving the highest quality feature registration result in the second registration substructure.

[0147] The feature-registration-based neural network follows the gray-level registration-based neural network. Its function is to perform feature registration on microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging (MRI) images based on the already completed gray-level registration. Specifically, it registers the gray-level registered MRI image with the microvascular photoacoustic imaging and ultrasound elastography images, and then combines the two registered MRI images... and The images are fused to obtain the final registered magnetic resonance images. ,by To minimize loss, ensure that feature registration based on grayscale registration is obtained. It has the highest local feature similarity with the original microvascular photoacoustic imaging and ultrasound elastography in the second registration substructure, thus achieving the highest quality feature registration result in the second registration substructure.

[0148] The above describes the neural networks based on grayscale registration and feature registration in the second registration substructure. and Training is performed using loss to achieve the highest local performance in grayscale registration and feature registration. The two evolve on their own, generating a self-supervised local optimization process. This ensures that the performance of the second registration substructure continuously optimizes itself, thereby continuously producing higher quality registration results than the first registration substructure, which the first registration substructure can then catch up with, thus achieving a supervised and guided performance improvement process for the first registration substructure.

[0149] The loss function for the mutual constraints between the first and second registration substructures is:

[0150] ;

[0151] In the formula, This represents the constraint loss between the first and second registration substructures.

[0152] This invention uses the difference between the registration results produced by the first registration substructure and the second registration substructure as the loss, so that during the training process of the two substructures, the registration results of the two substructures continuously approach each other, and the low-quality registration results move closer to the high-quality registration results, forming a progressive process of mutual pursuit, and finally enabling the performance of both substructures to reach the highest level and obtain high-quality registration results, thus generating a mutually supervised global optimization process.

[0153] This invention combines global optimization with local optimization to achieve performance optimization of the two substructures, thereby obtaining high-quality optoacoustic-magnetic multimodal registration results.

[0154] like Figure 7 As shown, this invention provides a photoacoustic-magnetic multimodal imaging method for prostate cancer identification, applied to a photoacoustic-magnetic multimodal imaging system for prostate cancer identification, comprising the following steps:

[0155] Photoacoustic imaging of microvessels in the prostate area using a photoacoustic imaging unit;

[0156] Ultrasound elastography of the prostate area was performed using an ultrasound elastography unit.

[0157] Magnetic resonance imaging of the prostate area was performed using a magnetic resonance imaging unit.

[0158] Information from microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging in the prostate area is fused to obtain a multimodal fused image of the prostate area.

[0159] In the photoacoustic imaging unit, the OPO multi-wavelength laser emits pulsed laser light that is uniformly irradiated on the inner surface of the prostate via a 360° diffused optical fiber to excite photoacoustic signals. The photoacoustic signals are then acquired by an ultrasound probe array to obtain photoacoustic signal data. Finally, the host computer performs optical power compensation, filtering, and peak detection on the photoacoustic signal data, and then reconstructs the photoacoustic image based on the Beamforming algorithm or the Delay and Sum algorithm, which serves as microvascular photoacoustic imaging.

[0160] In the ultrasound elastography unit, the acoustic radiation force excitation control module is used to drive the ultrasound probe array to emit one or more long pulse ultrasound waves, which form a focus of about 100 microseconds in the prostate area, generate acoustic radiation force on the tissue at the focal point, promote tissue deformation and form a shear wave source;

[0161] The multi-channel high-speed acquisition module uses fast ultra-high frame rate plane wave imaging technology to track the motion of shear waves in tissues and receive ultrasound radio frequency signal images.

[0162] The control unit is used to calculate the tissue deformation caused by shear waves from the radio frequency signal image through a tissue displacement estimation algorithm, estimate the propagation velocity of the shear waves at various points in the tissue, and then calculate the shear modulus of the tissue to obtain ultrasound elastography.

[0163] A semi-flexible prostate magnetic resonance radiofrequency coil in the magnetic resonance imaging unit is placed in the prostate area and simultaneously sent into the magnetic resonance device for scanning, thereby obtaining magnetic resonance imaging.

[0164] The multimodal fusion image was obtained by registering microvascular photoacoustic imaging and ultrasound elastography as reference images and magnetic resonance imaging as a floating image using the medical image processing software 3D Slicer.

[0165] This invention employs a high-quality imaging device for the prostate area, ensuring high-quality acquisition of images of each modality during the imaging stage. After acquiring high-quality images, a bidirectional supervised registration network is used to perform bidirectional registration using features and grayscale, achieving registration and fusion of photoacoustic and ultrasound elastography images (functional images) with high-resolution magnetic resonance images (anatomical images), thus achieving accurate functional image localization and providing high-quality images for the imaging identification of prostate cancer.

[0166] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A photoacoustic magnetic multimodality imaging system for prostate cancer identification, characterized in that, The application relates to a multi-mode image fusion device for a prostate site. The device comprises a photoacoustic imaging unit, an ultrasonic elastography imaging unit, a magnetic resonance imaging unit and a multi-mode image fusion unit. The photoacoustic imaging unit comprises an OPO multi-wavelength laser, a 360-degree diffused optical fiber, an ultrasonic probe array and a control host, and is used for photoacoustic imaging of microvessels in a prostate site. The ultrasonic elastography imaging unit comprises an acoustic radiation force excitation control module, an ultrasonic probe array, a multi-channel high-speed acquisition module and a control host, and is used for ultrasonic elastography imaging of the prostate site. The magnetic resonance imaging unit comprises a semi-flexible prostate magnetic resonance radio frequency coil and a magnetic resonance device, and is used for magnetic resonance imaging of the prostate site. The multi-mode image fusion unit is a computer device loaded with a medical image processing software 3D Slicer, and is used for information fusion of photoacoustic imaging, ultrasonic elastography imaging and magnetic resonance imaging of the prostate site, so as to obtain a multi-mode fusion image of the prostate site. The medical image processing software 3D Slicer in the multi-mode image fusion unit performs bidirectional registration fusion on the photoacoustic imaging, the ultrasonic elastography imaging and the magnetic resonance imaging of the prostate site between a feature registration mode and a grayscale registration mode by taking the photoacoustic imaging and the ultrasonic elastography imaging as reference images and taking the magnetic resonance imaging as a floating image through a pre-established bidirectional supervised registration network, so as to obtain the multi-mode fusion image. The bidirectional supervised registration network comprises a first registration substructure and a second registration substructure, and the first registration substructure and the second registration substructure are both composed of a feature registration-based neural network and a grayscale registration-based neural network. The feature registration-based neural network in the first registration substructure is arranged in front of the grayscale registration-based neural network, wherein the feature registration-based neural network in the first registration substructure takes the magnetic resonance imaging as a floating image and takes the photoacoustic imaging and the ultrasonic elastography imaging as reference images to generate a feature registration image of the magnetic resonance imaging. The grayscale registration-based neural network in the first registration substructure takes the feature registration image of the magnetic resonance imaging as a floating image and takes the photoacoustic imaging and the ultrasonic elastography imaging as reference images to generate a feature grayscale registration image of the magnetic resonance imaging. The grayscale registration-based neural network in the second registration substructure is arranged in front of the feature registration-based neural network, wherein the grayscale registration-based neural network in the second registration substructure takes the magnetic resonance imaging as a floating image and takes the photoacoustic imaging and the ultrasonic elastography imaging as reference images to generate a grayscale registration image of the magnetic resonance imaging. The feature registration-based neural network in the second registration substructure takes the grayscale registration image of the magnetic resonance imaging as a floating image and takes the photoacoustic imaging and the ultrasonic elastography imaging as reference images to generate a grayscale feature registration image of the magnetic resonance imaging.

2. The photoacoustic-magnetic multimodality imaging system for prostate cancer identification according to claim 1, characterized in that: The OPO multi-wavelength laser in the photoacoustic imaging unit is an imaging light source for the photoacoustic imaging of the microvessels, and the wavelength range is 680-2500nm. The 360-degree diffused optical fiber is combined with a urinary catheter, and is used for introduction into the prostate site through the urethra to uniformly irradiate the inner surface of the prostate site to excite photoacoustic signals. The ultrasonic probe array collects the photoacoustic signals to obtain photoacoustic signal data. The control host carries out optical power compensation, filtering and packet peak detection on the photoacoustic signal data, and then reconstructs a photoacoustic image based on a beam forming algorithm or a delay sum algorithm to obtain the microvessel photoacoustic imaging.

3. The photoacoustic-magnetic multimodality imaging system for prostate cancer identification according to claim 2, characterized in that: The acoustic radiation force excitation control module in the ultrasound elastography unit comprises a transmitting circuit, a receiving circuit and a data transmission circuit, wherein the transmitting circuit is composed of a field effect tube driver, a field effect tube and impedance matching, the receiving circuit is composed of a transceiver switching switch, a configurable signal amplification, a configurable analog filtering and a high-resolution analog-digital conversion, and the data transmission circuit adopts a PCIE interface to perform high-speed communication with the control host; The acoustic radiation force excitation control module is used to drive the ultrasonic probe array to emit one or more long-pulse ultrasonic waves, form focusing at the prostate site, generate acoustic radiation force on the tissue at the focal point, push the tissue to deform and form a shear wave source; The multi-channel high-speed acquisition module uses a fast ultrahigh frame rate plane wave imaging technology to track the movement of the shear wave in the tissue and receive the ultrasonic radio frequency signal image; The control host is used to calculate the tissue deformation caused by the shear wave from the radio frequency signal image through a tissue displacement estimation algorithm, estimate the propagation speed values of the shear wave at various positions in the tissue, calculate the shear modulus values of the tissue, and obtain the ultrasound elastography.

4. The photoacoustic-magnetic multimodality imaging system for prostate cancer identification of claim 3, wherein: The semi-flexible prostate magnetic resonance radio frequency coil in the magnetic resonance imaging unit comprises a coil main body and a preamplifier, wherein the coil main body comprises three overlapped loops, and each loop is a signal acquisition channel. The semi-flexible prostate magnetic resonance radio frequency coil is used to be placed at the prostate site and obtain the magnetic resonance imaging through scanning of a magnetic resonance device.

5. The photoacoustic magnetic multimodal imaging system for prostate cancer identification according to claim 1, wherein: The expression of the neural network based on feature registration in the first registration substructure is: ； ； ； wherein is a magnetic resonance image, is a microvascular photoacoustic image, is an ultrasound elastography image, is is a deformation field, is a deformation field, is a deformation field, is a deformation field, is a deformation field, is a feature registered image of magnetic resonance imaging, is a neural network for predicting a deformation field, is a spatial transformation function; The loss function of the neural network based on feature registration in the first registration substructure is: ； wherein is a loss of features of the first registration substructure, is a feature extraction network, is an L1 norm formula; The expression of the neural network based on grayscale registration in the first registration substructure is: ； ； ； wherein is between a deformation field for gray-scale registration between is between a deformation field for gray-scale registration between is a feature gray-scale registration image for magnetic resonance imaging; The loss function of the neural network based on grayscale registration in the first registration substructure is: ； wherein is the loss of gray level for the first registration substructure; The expression of the neural network based on grayscale registration in the second registration substructure is: ； ； ； wherein is and a deformation field for gray-scale registration between is and a deformation field for gray-scale registration between is a gray-scale registered image of magnetic resonance imaging; The loss function of the neural network based on grayscale registration in the second registration substructure is: ； wherein is the loss of gray level for the second registration substructure; The expression of the neural network based on feature registration in the second registration substructure is: ； ； ； wherein is between a deformation field that features aligns between is between a deformation field that features aligns between is a gray-scale feature aligned image of magnetic resonance imaging; The loss function of the neural network based on feature registration in the second registration substructure is: ； wherein is the loss of features of the second registration substructure; The loss function of mutual constraint between the first registration substructure and the second registration substructure is: ； wherein is the loss of constraints between the first and second registration substructures.

6. A photoacoustic magnetic multimodality imaging method for prostate cancer identification, characterized in that, The method applied to the photoacoustic magnetic multimodal imaging system for prostate cancer identification according to any one of claims 1-5 comprises the following steps: Performing microvessel photoacoustic imaging of the prostate site through the photoacoustic imaging unit; Performing ultrasound elastography of the prostate site through the ultrasound elastography unit; Performing magnetic resonance imaging of the prostate site through the magnetic resonance imaging unit; The microvascular photoacoustic imaging, the ultrasonic elastography and the magnetic resonance imaging of the prostate part are information fused to obtain a multi-mode fusion image of the prostate part.

7. The photoacoustic-magnetic multimodality imaging method for prostate cancer identification according to claim 6, characterized in that: The OPO multi-wavelength laser in the photoacoustic imaging unit emits pulsed laser to uniformly irradiate the inner surface of the prostate part via a 360° diffusing optical fiber to excite photoacoustic signals, the photoacoustic signals are collected by an ultrasonic probe array to obtain photoacoustic signal data, the photoacoustic signal data are subjected to light power compensation, filtering and peak envelope detection by a control host, and then a photoacoustic image is reconstructed based on a beam forming algorithm or a delay and sum algorithm to serve as the microvascular photoacoustic imaging. 8.The photoacoustic-magnetic multimodality imaging method for prostate cancer identification of claim 7, wherein: The acoustic radiation force excitation control module in the ultrasonic elastography unit is used to drive the ultrasonic probe array to emit one or more long-pulse ultrasonic waves to form a focus at the prostate part, to generate acoustic radiation force on the tissue at the focus to push the tissue to deform and form a shear wave source. The multi-channel high-speed acquisition module is used to track the movement of the shear wave in the tissue by using a fast ultrahigh frame rate plane wave imaging technology, and to receive an ultrasonic radio frequency signal image. The control host calculates the tissue deformation caused by the shear wave from the radio frequency signal image by using a tissue displacement estimation algorithm, estimates the propagation speed of the shear wave at each place in the tissue, calculates the shear modulus value of the tissue, and obtains the ultrasonic elastography. 9.The photoacoustic-magnetic multimodality imaging method for prostate cancer identification of claim 8, wherein: The multi-mode fusion image is obtained by using a medical image processing software 3D Slicer to register the microvascular photoacoustic imaging and the ultrasonic elastography as reference images and the magnetic resonance imaging as a floating image.

Images