A multi-modal image fusion method based on prostate cancer tissue morphology

Abstract

The present application relates to the technical field of medical imaging processing, in particular to a multi-modal image fusion method based on prostate cancer tissue morphology, comprising the following steps: obtaining microvessel photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging of the same prostate site; taking the microvessel photoacoustic imaging and the ultrasonic elastography as reference images and the magnetic resonance imaging as a floating image to construct a bidirectional supervision registration network; and performing lightweight processing on the bidirectional supervision registration network through transfer learning to establish a fast registration network. The present application first obtains a high-precision registration result of the magnetic resonance, photoacoustic and ultrasonic images through a complex model structure, and then forms a small model structure through transfer learning to learn the high-precision performance of the complex model structure, so that the simple structure and the high-precision performance of the model are compatible, the requirement of the deployment environment is reduced, and the universality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical imaging processing, in particular to a multi-modal image fusion method based on prostate cancer tissue morphology. BACKGROUND

[0002] Current medical lesion recognition is mostly image-based, that is, medical images such as magnetic resonance images, ultrasound images, and photoacoustic images are used to recognize lesions. Taking the recognition of prostate cancer as an example, the magnetic resonance images or ultrasound images of the prostate part are obtained, and the prostate cancer is recognized in the magnetic resonance images or ultrasound images. In actual application, due to the limitation of information quantity, it is difficult to obtain accurate recognition results relying on a single modality of image, and it is usually necessary to combine multiple modalities of image for recognition to obtain accurate lesion recognition results.

[0003] In the prior art, magnetic resonance imaging, photoacoustic imaging, and ultrasonic imaging are used to improve the recognition accuracy in prostate lesion recognition. Correspondingly, these images need to be registered and fused. The entire process needs to be manually operated and processed for a long time to achieve the process, and currently, machine learning artificial intelligence models are mostly used to improve efficiency. For example, the existing artificial intelligence model can complete the registration of multi-modal images through the gray level or feature level. In this process, in order to pursue high-precision registration results, a relatively complex model structure is generated, and the requirements for the deployment environment are also high, which greatly reduces the universality. Therefore, the prior art cannot balance the complexity of the structure and the high-precision performance of the model, and the universality of the model is reduced. SUMMARY

[0004] The present application relates to the technical field of medical imaging processing, in particular to a multi-modal image fusion method based on prostate cancer tissue morphology.

[0005] To solve the above technical problems, the present application specifically provides the following technical solutions:

[0006] A multi-modal image fusion method based on prostate cancer tissue morphology, comprising the following steps:

[0007] Obtaining microvascular photoacoustic imaging, ultrasonic elastography, and magnetic resonance imaging of the same prostate part;

[0008] Taking the microvascular photoacoustic imaging and ultrasonic elastography as reference images and the magnetic resonance imaging as a floating image, a bidirectional supervised registration network for feature and gray level registration and fusion of the microvascular photoacoustic imaging, ultrasonic elastography, and magnetic resonance imaging is constructed;

[0009] The bidirectional supervised registration network is lightened by migration learning, and a fast registration network for modality conversion between microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging and a multi-modal fusion image output by the bidirectional supervised registration network is established.

[0010] As a preferred scheme of the present application, the bidirectional supervised registration network comprises a first registration substructure and a second registration substructure, and the first registration substructure and the second registration substructure are both composed of a feature registration based neural network and a gray-scale registration based neural network;

[0011] The loss function of mutual constraint between the first registration substructure and the second registration substructure is:

[0012] ;

[0013] In the formula, is the constraint loss between the first registration substructure and the second registration substructure, is a feature gray-scale registration image of magnetic resonance imaging, is a gray-scale feature registration image of magnetic resonance imaging.

[0014] As a preferred scheme of the present application, the feature registration based neural network in the first registration substructure is arranged in front of the gray-scale registration based neural network, wherein the feature registration based neural network in the first registration substructure takes magnetic resonance imaging as a floating image and takes microvascular photoacoustic imaging and ultrasonic elastography as reference images to generate a feature registration image of magnetic resonance imaging.

[0015] The expression of the feature registration based neural network in the first registration substructure is:

[0016] ;

[0017] ;

[0018] ;

[0019] In the formula, is magnetic resonance imaging, is microvascular photoacoustic imaging, is ultrasonic elastography, is a feature registration deformation field between and a feature registration deformation field between and , is a feature registration image of magnetic resonance imaging, is a neural network for predicting a deformation field, is a spatial transformation function.

[0020] The loss function of the feature registration-based neural network in the first registration substructure is:

[0021] ;

[0022] In the formula, is the feature loss of the first registration substructure, is a feature extraction network, is an L1 norm formula;

[0023] The feature registration-based neural network in the first registration substructure takes the feature registration image of the magnetic resonance imaging as the floating image, and takes the microvascular photoacoustic imaging and the ultrasonic elastography as the reference image to generate the feature grayscale registration image of the magnetic resonance imaging.

[0024] The expression of the feature registration-based neural network in the first registration substructure is:

[0025] ;

[0026] ;

[0027] ;

[0028] In the formula, is the feature loss of the first registration substructure, is the deformation field of the grayscale registration between and is the deformation field of the grayscale registration between and is the feature grayscale registration image of the magnetic resonance imaging; The loss function of the feature registration-based neural network in the first registration substructure is:

[0029]

[0030] ;

[0031] In the formula, is the grayscale loss of the first registration substructure.

[0032] As a preferred scheme of the present application, the grayscale registration-based neural network in the second registration subnetwork is arranged in front of the feature registration-based neural network, wherein the grayscale registration-based neural network in the second registration substructure takes the magnetic resonance imaging as the floating image, and takes the microvascular photoacoustic imaging and the ultrasonic elastography as the reference image to generate the grayscale registration image of the magnetic resonance imaging.

[0033] The expression of the grayscale registration-based neural network in the second registration substructure is: ​​​

[0034] ;

[0035] ;

[0036] ;

[0037] In the formula, For magnetic resonance imaging, Photoacoustic imaging of microvessels. For ultrasound elastography, for and Deformation field for grayscale registration between them for and Deformation field for grayscale registration between them Gray-scale registered images for magnetic resonance imaging;

[0038] The loss function of the neural network based on grayscale registration in the second registration substructure is:

[0039] ;

[0040] In the formula, For the grayscale loss of the second registration substructure, It is an L1 norm form;

[0041] In the second registration substructure, the feature registration-based neural network uses magnetic resonance imaging as a floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate a grayscale feature registration image of magnetic resonance imaging.

[0042] The expression for the feature-registration-based neural network in the second registration substructure is as follows:

[0043] ;

[0044] ;

[0045] ;

[0046] In the formula, for and Deformation field of feature registration between them for and Deformation field of feature registration between them Image registration for grayscale features of magnetic resonance imaging;

[0047] The loss function of the feature-registration-based neural network in the second registration substructure is:

[0048] ;

[0049] In the formula, is a feature loss of the second registration substructure, is a feature extraction network.

[0050] As a preferred scheme of the present application, the gray feature registration image of magnetic resonance imaging or the feature gray registration image of magnetic resonance imaging is a multi-modal fusion image output by the bidirectional supervised registration network.

[0051] As a preferred scheme of the present application, the fast registration network comprises two neural networks for image modal conversion, namely a first modal conversion network and a second modal conversion network, wherein the first modal conversion network is used for converting microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging into a multi-modal fusion image;

[0052] The second modal conversion network is used for converting the multi-modal fusion image into microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging.

[0053] As a preferred scheme of the present application, the first modal conversion network takes microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging as input and outputs a multi-modal fusion image,

[0054] The structural expression of the first modal conversion network is:

[0055] ;

[0056] In the formula, is a multi-modal fusion image output by the first modal conversion network, are microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging respectively, is a neural network for image modal conversion;

[0057] The loss function of the first modal conversion network is:

[0058] ;

[0059] In the formula, is a conversion loss of the first modal conversion network, is a gray feature registration image of magnetic resonance imaging, is a feature gray registration image of magnetic resonance imaging, is an L1 norm formula.

[0060] In a preferred embodiment of the present invention, the second modality conversion network takes a multimodal fusion image as input and outputs microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging.

[0061] The structural expression of the second mode conversion network is:

[0062] ;

[0063] ;

[0064] In the formula, The second mode conversion network is based on The output includes microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging. The second mode conversion network is based on The output includes microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging. For neural networks used in image modality conversion, Image registration for grayscale features of magnetic resonance imaging. For the characteristic grayscale registration image of magnetic resonance imaging;

[0065] The loss function of the second mode conversion network is:

[0066] ;

[0067] In the formula, Let X be the conversion loss of the second mode conversion network. variable identifier, For the second mode conversion network according to The image corresponding to the output X, For the second mode conversion network according to The image corresponding to the output X, The image corresponding to X, It is an L1 norm.

[0068] As a preferred embodiment of the present invention, the total loss of training the fast registration network is... for:

[0069] ;

[0070] In the formula, The conversion loss of the first mode conversion network, This represents the conversion loss of the second mode conversion network.

[0071] As a preferred embodiment of the present invention, the first mode conversion network is used as the fast registration network after training is completed.

[0072] Compared with the prior art, the present application has the following beneficial effects:

[0073] The present application first obtains a high-precision registration result of magnetic resonance, photoacoustic and ultrasound images through a complex model structure, and then forms a small model structure through transfer learning to learn the high-precision performance of the complex model structure, so as to realize the combination of simple structure and high-precision performance of the model, reduce the requirement of the deployment environment, and improve the universality. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only exemplary, and other drawings can be obtained by the provided drawings without creative labor for those skilled in the art.

[0075] Figure 1 The multi-modal image fusion method flow chart provided for the embodiment of the present application;

[0076] Figure 2 The bidirectional supervision registration network block diagram provided for the embodiment of the present application;

[0077] Figure 3 The fast registration network block diagram provided for the embodiment of the present application. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0079] As shown in Figure 1 The present application provides a multi-modal image fusion method based on prostate cancer tissue morphology, comprising the following steps:

[0080] Obtaining microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging of the same prostate part;

[0081] Taking the microvascular photoacoustic imaging and the ultrasonic elastography as reference images and the magnetic resonance imaging as a floating image, a bidirectional supervision registration network for feature and grayscale registration fusion of the microvascular photoacoustic imaging, the ultrasonic elastography and the magnetic resonance imaging is constructed;

[0082] The two-way supervised registration network is lightened by migration learning, and a fast registration network for modality conversion between microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging and the multi-modal fusion image output by the two-way supervised registration network is established.

[0083] In order to realize high-quality registration of microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance images, two different ways are adopted for registration, the first is image pixel gray similarity registration, which has the advantages of simple calculation, easy implementation and insensitivity to light changes, the second is image feature similarity registration, the commonly used features mainly include feature points, line segments, contours and statistical moments, etc., which has the advantages of small calculation amount, good robustness and insensitivity to complex geometric deformation, and the method combining features and gray information can fully utilize the advantages of both, improve the accuracy and robustness of registration, and achieve high-quality registration.

[0084] In the process of combining feature registration and gray registration, the mutual constraint of the combination sequence is adopted, the first registration substructure executes the combination sequence of feature registration first and gray registration later, the second registration substructure executes the combination sequence of gray registration first and feature registration later, and the difference between the registration results generated by the first registration substructure and the second registration substructure is taken as the loss, so that in the training process of the two substructures, the registration results of the two substructures constantly approach each other, the low-quality registration result approaches the high-quality registration result, forming a progressive process of you chasing me, finally making the performance of the two substructures reach the highest to obtain high-quality registration results, and generating a mutual supervision global optimization process.

[0085] As shown in Figure 2 The two-way supervised registration network includes a first registration substructure and a second registration substructure, and the first registration substructure and the second registration substructure are both composed of a feature registration based neural network and a gray registration based neural network;

[0086] The feature registration based neural network in the first registration substructure is arranged in front of the gray registration based neural network, wherein the feature registration based neural network in the first registration substructure takes magnetic resonance imaging as a floating image and microvascular photoacoustic imaging and ultrasonic elastography as reference images to generate a feature registration image of the magnetic resonance imaging;

[0087] The expression of the feature registration based neural network in the first registration substructure is as follows:

[0088] ;

[0089] ;

[0090] ;

[0091] In the formula, For magnetic resonance imaging, Photoacoustic imaging of microvessels. For ultrasound elastography, for and Deformation field of feature registration between them for and Deformation field of feature registration between them For feature registration images of magnetic resonance imaging. For neural networks that predict deformation fields, It is a space transformation function;

[0092] The loss function of the feature-registration-based neural network in the first registration substructure is:

[0093] ;

[0094] In the formula, The characteristic loss of the first registration substructure, For feature extraction networks, It is an L1 norm form;

[0095] In the first registration substructure, the neural network based on gray-scale registration uses the feature registration image of magnetic resonance imaging as the floating image and microvascular photoacoustic imaging and ultrasound elastography as reference images to generate the feature gray-scale registration image of magnetic resonance imaging.

[0096] The expression for the neural network based on grayscale registration in the first registration substructure is:

[0097] ;

[0098] ;

[0099] ;

[0100] In the formula, for and Deformation field for grayscale registration between them for and Deformation field for grayscale registration between them For the characteristic grayscale registration image of magnetic resonance imaging;

[0101] The loss function of the neural network based on gray-level registration in the first registration substructure is:

[0102] ;

[0103] In the formula, This represents the grayscale loss of the first registration substructure.

[0104] In this invention, the first registration substructure executes a combination of feature registration followed by grayscale registration, implemented by two neural networks: a feature registration-based neural network and a grayscale registration-based neural network. The feature registration-based neural network performs feature registration on microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging (MRI) images, registering the MRI image with the microvascular photoacoustic imaging and ultrasound elastography images respectively. Then, it registers the two MRI images... and The images are fused to obtain the final registered magnetic resonance images. ,by For loss ( This is used to extract features from images for registration. Among the three modalities, blood vessels are most commonly used as a common feature for feature registration. Other feature points can also be selected as needed to ensure successful feature registration. It has the highest local feature similarity with the original microvascular photoacoustic imaging and ultrasound elastography in the first registration substructure, thus achieving the highest quality feature registration result in the first registration substructure.

[0105] The grayscale registration-based neural network follows the feature registration-based neural network. Its function is to perform grayscale registration on microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging (MRI) images based on the already completed feature registration. Specifically, it registers the feature-registered MRI image with the microvascular photoacoustic imaging and ultrasound elastography images, and then combines the two registered MRI images... and The images are fused to obtain the final registered magnetic resonance images. ,by To minimize loss, ensure that grayscale registration based on feature registration yields the desired result. It has the highest local gray-level similarity with the original microvascular photoacoustic imaging and ultrasound elastography in the first registration substructure, thus achieving the highest quality local gray-level registration result in the first registration substructure.

[0106] The above describes the neural networks based on feature registration and grayscale registration in the first registration substructure. and The loss is used for training, aiming to achieve the highest local performance in feature registration and grayscale registration. The two evolve on their own, generating a self-supervised local optimization process. This ensures that the performance of the first registration substructure is continuously optimized, thereby continuously producing higher quality registration results than the second registration substructure, which then catches up. This process supervises and guides the performance improvement of the second registration substructure.

[0107] The neural network based on gray-scale registration in the second registration sub-network is in front of the neural network based on feature registration, wherein the neural network based on gray-scale registration in the second registration sub-network takes magnetic resonance imaging as a floating image, takes microvascular photoacoustic imaging and ultrasonic elastography as reference images, and generates a gray-scale registration image of the magnetic resonance imaging.

[0108] The expression of the neural network based on gray-scale registration in the second registration sub-network is:

[0109] ;

[0110] ;

[0111] ;

[0112] In the formula, is magnetic resonance imaging, is microvascular photoacoustic imaging, is ultrasonic elastography, is and is a deformation field of gray-scale registration between and is and is a deformation field of gray-scale registration between and is a gray-scale registration image of the magnetic resonance imaging;

[0113] The loss function of the neural network based on gray-scale registration in the second registration sub-network is:

[0114] ;

[0115] In the formula, is a gray-scale loss of the second registration sub-network, is an L1 norm formula;

[0116] The neural network based on feature registration in the second registration sub-network takes magnetic resonance imaging as a floating image, takes microvascular photoacoustic imaging and ultrasonic elastography as reference images, and generates a gray-scale feature registration image of the magnetic resonance imaging.

[0117] The expression of the neural network based on feature registration in the second registration sub-network is:

[0118] ;

[0119] ;

[0120] ;

[0121] In the formula, For the deformation field between the features of , For the deformation field between the features of , For the gray feature registration image of magnetic resonance imaging

[0122] The loss function of the feature registration based neural network in the second registration substructure is:

[0123] ;

[0124] In the formula, is the feature loss of the second registration substructure, is the feature extraction network.

[0125] The gray feature registration image of magnetic resonance imaging or the feature gray registration image of magnetic resonance imaging is output as a multimodal fusion image by the bidirectional supervised registration network.

[0126] In the present application, the second registration substructure performs the combined sequence of gray registration and feature registration, which is realized by two neural networks, namely the gray registration based neural network and the feature registration based neural network. The gray registration based neural network is used to perform gray registration on the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance image, and respectively register the magnetic resonance image with the microvascular photoacoustic imaging and ultrasonic elastography. Then, the two magnetic resonance images after registration are fused to obtain the final registration image of the magnetic resonance image , and the loss is used to ensure that the gray registration result has the second highest gray similarity with the original microvascular photoacoustic imaging and ultrasonic elastography in the second registration substructure, so as to achieve the highest quality feature registration result in the second registration substructure.

[0127] The feature registration based neural network is connected after the gray registration based neural network, and is used to perform feature registration on the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance image on the basis of the completed gray registration. Then, the gray registration image of the magnetic resonance image is registered with the microvascular photoacoustic imaging and ultrasonic elastography. Then, the two magnetic resonance images after registration are fused to obtain the final registration image of the magnetic resonance image , and the loss is used to ensure that the feature registration result on the basis of the gray registration The feature with the highest feature similarity in the second registration substructure is similar to the feature with the highest feature similarity in the original microvascular photoacoustic imaging and ultrasonic elastography, so as to achieve the feature registration result with the highest quality in the second registration substructure.

[0128] The neural network based on gray-scale registration and the neural network based on feature registration in the second registration substructure are trained as losses, and the gray-scale registration and the feature registration with the highest performance are expected to be self-evolved to generate a self-supervised local optimization process, so as to ensure that the performance of the second registration substructure is continuously self-optimized, thereby continuously generating a registration result with higher quality than the first registration substructure, for the first registration substructure to catch up and realize the performance improvement process of supervising and guiding the first registration substructure. and

[0129] The loss function between the first registration substructure and the second registration substructure is:

[0130] ;

[0131] In the formula, is the constraint loss between the first registration substructure and the second registration substructure, is a feature gray-scale registration image of magnetic resonance imaging, is a gray-scale feature registration image of magnetic resonance imaging.

[0132] Existing segmentation networks such as U-Net network, FastR-CNN network, etc. can be used, and a custom segmentation network can also be used. The same neural network Existing deformation field positioning network GAN network can be used, and a custom deformation field positioning network can also be used. For example, the deformation field positioning network is similar to the encoding-decoding structure of the segmentation network, and includes an encoder, a decoder, a skip connection, and a bottleneck module. The encoder layer is used to extract the features of the prostate image input into the network, the decoder layer is used to establish a spatial deformation field according to the corresponding relationship between the extracted features, the encoding-decoding layer uses a residual convolution block, and the down-sampling is performed through a convolution with a stride of 2, and the up-sampling is performed through a transposed convolution with a stride of 2. The skip connection integrated with the residual convolution block and the channel attention mechanism module transmits more information extracted in the encoder to the decoder layer, while avoiding too large semantic difference between the low-level semantic features of the encoder and the high-level semantic features of the decoder. The bottleneck module integrates a double attention mechanism to enhance the ability of the network to map the features between images, and finally outputs a spatial deformation field used for registration.

[0133] ​The difference between the registration results generated by the first registration substructure and the second registration substructure is taken as a loss, so that in the training process of the two substructures, the registration results of the two substructures are constantly close to each other, the low-quality registration results are close to the high-quality registration results, forming a progressive process of you chasing me, and finally the performance of the two substructures reaches the highest to obtain high-quality registration results, generating a mutual supervision global optimization process.

[0134] The application combines the local optimization process in the global optimization process, optimizes the performance of the two substructures, and obtains high-quality photoacoustic magnetic multi-modal registration results.

[0135] The above-mentioned two-way supervised registration network can obtain high-quality registration results, but the structure is relatively complex, the requirements for model deployment environment and hardware are relatively high, which reduces the universality, therefore, in order to enhance the model universality, the application adopts transfer learning to perform volume lightweight processing on the registration model, forms a fast registration network, and achieves the advantages of reducing model volume, reducing calculation complexity, improving running efficiency, saving storage space and optimizing user experience.

[0136] As shown in Figure 3 The fast registration network includes two neural networks for image modal conversion, which can use existing segmentation networks, such as a generative adversarial network (GAN), or a custom modal conversion network, respectively, as a first modal conversion network and a second modal conversion network, wherein the first modal conversion network is used to convert microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging into a multi-modal fusion image.

[0137] The second modal conversion network is used to convert the multi-modal fusion image into microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging.

[0138] The application uses the high-quality registration result output by the two-way supervised registration network as a data sample for training the fast registration network, so that the fast registration network can directly convert microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging into high-quality registration results of magnetic resonance imaging through modal conversion, or in other words, the gray feature registration image of the magnetic resonance image output by the two-way supervised registration network, the feature gray registration image.

[0139] The application takes microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging as input, takes the multimodal fusion image (that is, the gray feature registration image of the magnetic resonance image or the feature gray registration image of the magnetic resonance image) output by the bidirectional supervised registration network as output, establishes the modal mapping relationship between the two, realizes the multimodal fusion image obtained directly according to the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging, lightens the high-quality registration process of the bidirectional supervised registration network to a high-quality registration process realized by a modal conversion network, does not need to perform detailed feature registration and gray registration processes, and accordingly does not need to store a large amount of data generated by the feature registration and gray registration processes, has the advantages of reducing the model volume, reducing the calculation complexity, improving the operation efficiency, saving the storage space and optimizing the user experience, etc., but the lightening process will inevitably sacrifice certain precision performance, the application learns the high-quality registration performance as much as possible through lightening, and meanwhile meets the registration timeliness demand in a specific scene, and the precision and efficiency are considered.

[0140] Further, in order to avoid random mapping in the process of establishing the modal mapping relationship between the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging and the multimodal fusion image, so that the modal conversion is invalid, that is, the registration result is invalid, two mapping processes are simultaneously trained to establish the forward mapping relationship between the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging and the multimodal fusion image, and the reverse mapping relationship between the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging and the multimodal fusion image, the mutual constraint of the forward mapping and the reverse mapping relationship avoids the generation of mapping randomness to the greatest extent, thereby providing a solid guarantee for the lightening of the high-quality registration process of the bidirectional supervised registration network through the modal conversion network.

[0141] The first modal conversion network takes the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging as input, and takes the multimodal fusion image as output,

[0142] The structure expression of the first modal conversion network is:

[0143] ;

[0144] In the formula, is the multimodal fusion image output by the first modal conversion network, respectively, the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging, is a neural network for image modal conversion;

[0145] The loss function of the first modal conversion network is:

[0146] ;

[0147] wherein, is a conversion loss of the first modality conversion network, is a gray-scale feature registration image of magnetic resonance imaging, is a feature gray-scale registration image of magnetic resonance imaging, is an L1 norm formula.

[0148] In the present application, a forward mapping relationship between microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging and a multi-modal fusion image is established by a first modality conversion network, so that is a loss, which ensures that the difference between the multi-modal fusion image output by the first modality conversion network and the two high-quality registration results output by the bidirectional supervision registration network and is minimized, so that the output of the first modality conversion network can approach and can also approach , the output is not single-selectable, so that the model is allowed to output a fusion image between the two or close to either one, generating a more robust and more reliable fusion result, increasing the prediction fault tolerance of the model, requiring the output to approach or , compared with the requirement that the output must be strictly equal to one of them, it is a more relaxed, easier to learn and less prone to overfitting target. The model has more freedom to find a solution space that satisfies "approaching either high-quality registration" during training, which helps to learn more generalized feature representations. This design can be regarded as an implicit "perturbation" or "interpolation" of high-quality registration results during the training phase, encouraging the model to learn a continuous solution space of high-quality registration represented by and , rather than just two discrete points, which helps the model to better generalize to new data that may be between the training samples. Therefore, the loss can provide a more relaxed training target and an implicit data augmentation effect, which helps to improve the generalization ability and training stability of the model.

[0149] The second modality conversion network takes the multi-modal fusion image as input and outputs microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging,

[0150] The structural expression of the second modality conversion network is:

[0151] ;

[0152] ;

[0153] wherein, respectively, are the second modality conversion network according to The output microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging, respectively, according to the second modal conversion network The output microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging, is a neural network for image modal conversion, is a gray-scale feature registration image of magnetic resonance imaging, is a feature gray-scale registration image of magnetic resonance imaging;

[0154] The loss function of the second modal conversion network is:

[0155] ;

[0156] In the formula, is the conversion loss of the second modal conversion network, X is a variable identifier of, is the imaging corresponding to X output by the second modal conversion network according to is the imaging corresponding to X output by the second modal conversion network according to is the imaging corresponding to X, is the imaging corresponding to X, is the imaging corresponding to X, is the L1 norm formula.

[0157] In the present application, the reverse mapping relationship between the microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging and the multi-modal fusion image is established by the second modal conversion network, so that is the loss, to ensure that the and output by the second modal conversion network are closest to the original microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging, so that the two high-quality registration results and output by the bidirectional supervised registration network and are close to the original microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging, that is, and can be modal converted into the original microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging, and the randomness of the forward conversion of the first modal conversion network is constrained.

[0158] It is also ensured that the difference between and is minimal, so that and conversion results are similar, and the data relationship of and after conversion is inherited from and The similarity between the two modalities forms a fixed data relationship, guarantees the effectiveness of the second modality conversion network, and restricts the randomness of the self-modality conversion.

[0159] Total loss of training fast registration network For:

[0160] ;

[0161] In the formula, is the conversion loss of the first modality conversion network, is the conversion loss of the second modality conversion network.

[0162] After training, the first modality conversion network is used as a fast registration network and embedded in the medical image processing software 3D Slicer for registration operation, so as to realize the high-quality registration process realized by the modality conversion network on the medical image processing software 3D Slicer, compared with directly using the bidirectional supervised registration network, the model volume is reduced, the calculation complexity is reduced, the running efficiency is improved, the storage space is saved, and the user experience is optimized.

[0163] The present application first obtains a high-precision registration result of magnetic resonance, photoacoustic and ultrasound images through a complex model structure, and then forms a small model structure through transfer learning to learn the high-precision performance of the complex model structure, realizes the combination of simple structure and high-precision performance of the model, reduces the requirement of the deployment environment, improves the universality, and meets the timeliness requirement of the specific scene.

[0164] The above examples are only exemplary embodiments of the present application and are not used to limit the present application, and the protection scope of the present application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements to the present application within the spirit and protection scope of the present application, and such modifications or equivalent replacements are also regarded as falling within the protection scope of the present application.

Claims

1. A multi-modal image fusion method based on prostate cancer tissue morphology, characterized by, The method comprises the following steps: obtaining microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging of the same prostate part; taking the microvascular photoacoustic imaging and the ultrasonic elastography as reference images and taking the magnetic resonance imaging as a floating image, constructing a bidirectional supervised registration network for feature and grayscale registration fusion of the microvascular photoacoustic imaging, the ultrasonic elastography and the magnetic resonance imaging; performing lightweight processing on the bidirectional supervised registration network through transfer learning, and establishing a fast registration network for modality conversion between the microvascular photoacoustic imaging, the ultrasonic elastography and the magnetic resonance imaging and a multi-modal fusion image output by the bidirectional supervised registration network; the bidirectional supervised registration network comprises a first registration substructure and a second registration substructure, and the first registration substructure and the second registration substructure are both composed of a feature registration-based neural network and a grayscale registration-based neural network; a loss function of mutual constraint between the first registration substructure and the second registration substructure is: ； wherein is a constraint loss between the first and second registration substructures, is a feature intensity registration image for magnetic resonance imaging, is a feature intensity registration image for magnetic resonance imaging; the feature registration-based neural network in the first registration substructure is arranged in front of the grayscale registration-based neural network, wherein the feature registration-based neural network in the first registration substructure takes the magnetic resonance imaging as a floating image and takes the microvascular photoacoustic imaging and the ultrasonic elastography as reference images to generate a feature registration image of the magnetic resonance imaging; an expression of the feature registration-based neural network in the first registration substructure is: ； ； ； wherein is a magnetic resonance imaging, is a microvascular photoacoustic imaging, is an ultrasound elastography, is is a deformation field, is a deformation field, is is a deformation field, is a deformation field, is a feature registration image for magnetic resonance imaging, is a neural network for predicting a deformation field, is a spatial transformation function; a loss function of the feature registration-based neural network in the first registration substructure is: ； In the formula, is a feature loss of the first registration substructure, is a feature extraction network, is an L1 norm formula; the grayscale registration-based neural network in the first registration substructure takes the feature registration image of the magnetic resonance imaging as a floating image and takes the microvascular photoacoustic imaging and the ultrasonic elastography as reference images to generate a feature grayscale registration image of the magnetic resonance imaging; an expression of the grayscale registration-based neural network in the first registration substructure is: ； ； ； wherein is between a deformation field for gray-scale registration between is between a deformation field for gray-scale registration between is a feature gray-scale registration image for magnetic resonance imaging; a loss function of the grayscale registration-based neural network in the first registration substructure is: ； wherein is the loss of gray levels for the first registration substructure.

2. The multi-modal image fusion method based on prostate cancer tissue morphology according to claim 1, characterized in that: the grayscale registration-based neural network in the second registration substructure is arranged in front of the feature registration-based neural network, wherein the grayscale registration-based neural network in the second registration substructure takes the magnetic resonance imaging as a floating image and takes the microvascular photoacoustic imaging and the ultrasonic elastography as reference images to generate a grayscale registration image of the magnetic resonance imaging; an expression of the grayscale registration-based neural network in the second registration substructure is: ； ； ； wherein is magnetic resonance imaging, is microvascular photoacoustic imaging, is ultrasound elastography, is and is a deformation field for gray-scale registration between is and is a deformation field for gray-scale registration between is a gray-scale registered image for magnetic resonance imaging; a loss function of the grayscale registration-based neural network in the second registration substructure is: ； wherein a loss of gray scale for the second registration substructure, is an L1 norm the feature registration-based neural network in the second registration substructure takes the grayscale registration image of the magnetic resonance imaging as a floating image and takes the microvascular photoacoustic imaging and the ultrasonic elastography as reference images to generate a grayscale feature registration image of the magnetic resonance imaging; an expression of the feature registration-based neural network in the second registration substructure is: ； ； ； wherein is between a deformation field that is registered to features between is between a deformation field that is registered to features between is a gray-scale registered image of magnetic resonance imaging; a loss function of the feature registration-based neural network in the second registration substructure is: ； In the formula, is a loss of features of the second registration substructure, is a feature extraction network.

3. The multi-modal image fusion method based on prostate cancer tissue morphology according to claim 2, characterized in that: Registering gray level features of magnetic resonance images or magnetic resonance images Multi-modal fused images output as a bidirectional supervised registration network.

4. The multi-modal image fusion method based on prostate cancer tissue morphology according to claim 3, characterized in that: the fast registration network comprises two neural networks for image modality conversion, namely a first modality conversion network and a second modality conversion network, wherein the first modality conversion network is used for converting the microvascular photoacoustic imaging, the ultrasonic elastography and the magnetic resonance imaging into a multi-modal fusion image; The second modality conversion network is used for converting the multi-modal fusion image into microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging.

5. The multi-modal image fusion method based on prostate cancer tissue morphology according to claim 4, characterized in that: The first modality conversion network takes microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging as input and takes a multi-modal fusion image as output, The structural expression of the first modality conversion network is: ； In the formula, a multimodal fusion image output by the first modal conversion network, respectively magnetic resonance imaging, ultrasonic elastography and microvascular photoacoustic imaging, is a neural network for image modal conversion; The loss function of the first modality conversion network is: ； wherein is a conversion loss of the first modality conversion network, is a gray-scale feature registration image of the magnetic resonance imaging, is a feature gray-scale registration image of the magnetic resonance imaging, is an L1 norm 6. The multi-modal image fusion method based on prostate cancer tissue morphology according to claim 5, characterized in that: The second modality conversion network takes a multi-modal fusion image as input and takes microvascular photoacoustic imaging, ultrasonic elastography and magnetic resonance imaging as output, The structural expression of the second modality conversion network is: ； ； wherein are second modality conversion networks according to outputted magnetic resonance imaging, ultrasound elastography and microvascular photoacoustic imaging, are second modality conversion networks according to outputted magnetic resonance imaging, ultrasound elastography and microvascular photoacoustic imaging, is a neural network for image modality conversion, is a gray-scale feature registration image of magnetic resonance imaging, is a feature gray-scale registration image of magnetic resonance imaging; The loss function of the second modality conversion network is: ； wherein, is a conversion loss of the second modality conversion network, X is is a variable identifier of X, is a conversion loss of the second modality conversion network according to is an imaging corresponding to X output by the second modality conversion network, is a conversion loss of the second modality conversion network according to is an imaging corresponding to X output by the second modality conversion network, is an imaging corresponding to X, is an L1 norm.

7. The multi-modal image fusion method based on prostate cancer tissue morphology according to claim 6, characterized in that: Total loss for training the fast registration network is: ； In the formula, is a conversion loss of the first modal conversion network, is a conversion loss of the second modal conversion network.

8. The multi-modal image fusion method based on prostate cancer tissue morphology according to claim 7, characterized in that: After the training is completed, the first modality conversion network is taken as the fast registration network.

Images