Multimode image real-time registration method and system based on neural network

Abstract

The invention relates to the technical field of image registration, in particular to a multi-mode image real-time registration method and system based on a neural network, and the method comprises the following steps: carrying out the edge detection and principal component analysis of an MR image and an ultrasonic image in sequence, and obtaining an MR main structure feature map and an ultrasonic main structure feature map; measuring and calculating a registration deformation field according to the MR and the ultrasonic main structure feature map by using a generator G in a GAN network; performing spatial transformation on the corresponding deformation fields of the MR main structure feature map and the ultrasonic main structure feature map to generate an MR registration image and an ultrasonic registration image; and repeatedly registering the MR registration image and the ultrasonic registration image which are identified by the discriminator D and have the optimal registration effect until the optimal registration image is obtained. According to the method, multimode image registration in two directions is carried out by utilizing the GAN network, registration in the two directions is mutually constrained, registration randomness is eliminated, registration precision is improved, similarity measurement of an image main structure is adopted for registration, information processing amount is reduced, and the requirement of real-time registration for timeliness can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and specifically relates to a method and system for real-time registration of multimodal images based on a neural network. Background Art

[0002] Performing deformation registration on MR images and ultrasound images, and registering and aligning the two brain images in the same space can better master and analyze the detailed information of the two images, which has important research value.

[0003] Currently, for the deformation registration between multimodal images, a registration algorithm for images based on deep learning has been proposed, which automatically extracts image features or analyzes spatial deformation through a deep network. For example, a registration network is constructed using a fully convolutional network FCN, and a global dense deformation field is predicted at one time from the two input images to be registered. Then, the registration and fusion of the two images are achieved through the spatial transformation of the deformation field.

[0004] Although the prior art can complete the deformation registration of multimodal images by constructing a deep learning model, however, in the process of constructing the deep learning model, only a single deformation field is obtained and single-direction registration is performed. The unidirectional information similarity used in the registration will have a certain degree of unidirectional randomness and is not restricted, thus affecting the registration accuracy. Moreover, in the registration process, the similarity of all information of the two images is considered to complete the registration process of multimodal images. Therefore, all information of multiple modal images needs to be processed in the registration process, with a large amount of information processing and a long time, making it difficult to meet the real-time requirement of real-time registration. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for real-time registration of multimodal images based on a neural network, so as to solve the technical problems in the prior art that a single deformation field is obtained and single-direction registration is performed, the unidirectional information similarity used in the registration has a certain degree of unidirectional randomness, which affects the registration accuracy, and all information of multiple modal images needs to be processed in the registration process, making it difficult to meet the real-time requirement of real-time registration.

[0006] To solve the above technical problems, the present invention specifically provides the following technical solutions: A method for real-time registration of multimodal images based on a neural network, comprising the following steps: Step 1, obtaining MR images and ultrasound images for registration; Step 2, respectively performing edge detection and principal component analysis on the MR images and ultrasound images in sequence to obtain an MR main structure feature map and an ultrasound main structure feature map; Step 3, using the generator G in the GAN network to respectively calculate an MR registration deformation field and an ultrasound registration deformation field from the MR main structure feature map and the ultrasound main structure feature map; Step 4: Respectively perform spatial transformation on the MR main structural feature map and the ultrasound main structural feature map by using the MR registration deformation field and the ultrasound registration deformation field to generate an MR registered image and an ultrasound registered image; Step 5: Use the discriminator D in the GAN network to perform registration discrimination on the MR registered image and the ultrasound registered image compared with the MR image and the ultrasound image, and identify the MR registered image and the ultrasound registered image with the best registration effect; Step 6: Replace the MR image and the ultrasound image in Step 2 with the MR registered image and the ultrasound registered image with the best registration effect identified by the discriminator D, and repeat Steps 2 - 6 until the structural feature similarity between the MR registered image and the ultrasound registered image with the best registration effect identified by the discriminator D is maximally optimized, so as to obtain the optimal registered images of the MR image and the ultrasound image.

[0007] As a preferred embodiment of the present invention, the extraction method of the MR main structural feature map and the ultrasound main structural feature map includes: Respectively use the Canny operator to perform edge detection on the MR image and the ultrasound image in sequence to obtain an MR structural feature map and an ultrasound structural feature map; Respectively perform principal component analysis on the MR structural feature map and the ultrasound structural feature map to obtain the MR main structural feature map and the ultrasound main structural feature map.

[0008] As a preferred embodiment of the present invention, the method for performing principal component analysis on the MR structural feature map and the ultrasound structural feature map includes: Use the first convolutional neural network and the second convolutional neural network in the principal component analysis network to perform convolutional processing on the MR structural feature map respectively, and fuse the output results of the first convolutional neural network and the second convolutional neural network through an exponential function to obtain the MR main structural feature map PSR_f; Use the first convolutional neural network and the second convolutional neural network in the principal component analysis network to perform convolutional processing on the ultrasound structural feature map respectively, and fuse the output results of the first convolutional neural network and the second convolutional neural network through an exponential function to obtain the ultrasound main structural feature map PSR_r.

[0009] As a preferred embodiment of the present invention, the setting method of the first convolutional neural network and the second convolutional neural network in the principal component analysis network includes: Select the image patches centered on each voxel in the MR structural feature map, perform de - mean and vectorization processing, then form a matrix with the vectorized results for PCA processing, matrixize the eigenvectors corresponding to the obtained eigenvalues, and obtain the convolutional kernels of the first convolutional neural network in the principal component analysis network for the principal component analysis of the MR structural feature map; For each voxel-centered image patch in the output result of the first-layer convolutional neural network, perform mean subtraction and vectorization, then form a matrix from the vectorized result for PCA processing, and matrixize the eigenvectors corresponding to the obtained eigenvalues to obtain the convolutional kernels of the second-layer convolutional neural network in the principal component analysis network for principal component analysis of the MR structural feature map; Select image patches centered on each voxel in the ultrasound structural feature map, perform mean subtraction and vectorization, then form a matrix from the vectorized result for PCA processing, and matrixize the eigenvectors corresponding to the obtained eigenvalues to obtain the convolutional kernels of the first-layer convolutional neural network in the principal component analysis network for principal component analysis of the ultrasound structural feature map; For each voxel-centered image patch in the output result of the first-layer convolutional neural network, perform mean subtraction and vectorization, then form a matrix from the vectorized result for PCA processing, and matrixize the eigenvectors corresponding to the obtained eigenvalues to obtain the convolutional kernels of the second-layer convolutional neural network in the principal component analysis network for principal component analysis of the ultrasound structural feature map.

[0010] As a preferred embodiment of the present invention, the method for generating the MR registration deformation field and the ultrasound registration deformation field includes: Input the MR main structural feature map and the ultrasound main structural feature map into the generator G together to obtain the MR registration deformation field and the ultrasound registration deformation field ; The generation process of the registration deformation field is: ; In the formula, and are the MR registration deformation field and the ultrasound registration deformation field respectively, and are the MR main structural feature map and the ultrasound main structural feature map respectively, and G is the generator; The generator G is formed by the Transformer model structure combined with graph processing, and the generator G includes two input channels; The loss function for training the generator G is: ; Among them, ; ; ; In the formula, is the generator loss function, , and are respectively , and Penalty coefficient is the adversarial loss is the registration structure loss is the deformation field smoothing loss, D is the output discrimination probability of discriminator D is the discrimination probability of is the discrimination probability of is the main structure feature map of the MR registration image is the main structure feature map of the ultrasound registration image and are the main structure feature map of the MR and the main structure feature map of the ultrasound respectively denotes and the gradients of, E represents expectation, F represents Frobenius norm

[0011] As a preferred embodiment of the present invention, the recognition method of the MR registration image and the ultrasound registration image with the best registration effect includes: respectively obtain the main structure feature map of the MR registration image by using edge detection and principal component analysis and the main structure feature map of the ultrasound registration image ; input the main structure feature map of the MR registration image into the first discriminator D, and obtain the MR registration image corresponding to the discrimination result of being discriminated as as the MR registration image with the best registration effect; input the main structure feature map of the ultrasound registration image into the second discriminator D, and obtain the ultrasound registration image corresponding to the discrimination result of being discriminated as as the ultrasound registration image with the best registration effect; Both the first discriminator D and the second discriminator D are formed by the Transformer model structure combined with graph processing, and both include an input channel; The loss function for training the first discriminator D and the second discriminator D is: ; In the formula, is the loss function of discriminator D is the discrimination probability of is the discrimination probability of is the discrimination probability of is Discrimination probability, is the main structure feature map of the MR registration image, is the main structure feature map of the ultrasound registration image, and are the MR main structure feature map and the ultrasound main structure feature map respectively, E represents expectation, and F represents the Frobenius norm.

[0012] As a preferred embodiment of the present invention, obtaining the main structure feature map of the MR registration image and the main structure feature map of the ultrasound registration image The edge detection and principal component analysis used in the process are the same as those used in the process of obtaining the MR main structure feature map and the ultrasound main structure feature map.

[0013] As a preferred embodiment of the present invention, the method for generating the optimal registration images of the MR image and the ultrasound image includes: Maximizing the structural feature similarity between the MR registration image and the ultrasound registration image with the best registration effect is set as the optimization goal, and the optimization objective function is: ; In the formula, M is the optimization goal identifier, is the MR registration image with the best registration effect, is the ultrasound registration image with the best registration effect, E represents expectation, and F represents the Frobenius norm; Based on the optimization goal M, the GAN network that has completed the training of the generator G and the first discriminator D and the second discriminator D is repeatedly used for the registration process until the optimization goal M is achieved. The MR registration image and the ultrasound registration image with the best registration effect generated by the generator G in the GAN network at this time are used as the optimal registration images of the MR image and the ultrasound image.

[0014] As a preferred embodiment of the present invention, the structures of the first discriminator D and the second discriminator D are the same.

[0015] As a preferred embodiment of the present invention, the present invention provides a multi-modal image real-time registration system based on a neural network, which is applied to a multi-modal image real-time registration method based on a neural network. The system includes: A data acquisition unit for acquiring MR images and ultrasound images for registration; A feature preprocessing unit for sequentially performing edge detection and principal component analysis on the MR image and the ultrasound image respectively to obtain the MR main structure feature map and the ultrasound main structure feature map; A deep learning unit is used to calculate the MR registration deformation field and the ultrasound registration deformation field respectively from the MR main structure feature map and the ultrasound main structure feature map by using the generator G in the GAN network; the MR main structure feature map and the ultrasound main structure feature map are respectively subjected to spatial transformation using the MR registration deformation field and the ultrasound registration deformation field to generate an MR registered image and an ultrasound registered image; the MR registered image and the ultrasound registered image are compared with the MR image and the ultrasound image by using the discriminator D in the GAN network for registration discrimination, and the MR registered image and the ultrasound registered image with the best registration effect are identified; A registration optimization unit is used to replace the MR image and the ultrasound image in the feature preprocessing unit with the MR registered image and the ultrasound registered image with the best registration effect identified by the discriminator D, and repeat the processing processes of the feature preprocessing unit and the deep learning unit until the structural feature similarity between the MR registered image and the ultrasound registered image with the best registration effect identified by the discriminator D is maximally optimized, and the optimal registered images of the MR image and the ultrasound image are obtained.

[0016] The present invention has the following beneficial effects compared with the prior art: The present invention uses the GAN network for multi-modal image registration in two directions. The two-direction registrations are mutually constrained, eliminating the randomness of registration, improving the registration accuracy. During the registration process, the similarity measure of the main structure of the image is adopted, only involving the main structure information, reducing the amount of information processing, and helping to meet the real-time requirement of registration for timeliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary, and for those of ordinary skill in the art, other implementation drawings can be obtained according to the provided drawings without creative efforts.

[0018] Figure 1 It is a flowchart of a multi-modal image real-time registration method based on a neural network provided by an embodiment of the present invention; Figure 2 It is a flowchart of a multi-modal image real-time registration system based on a neural network provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of the multi-modal image real-time registration process provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the model structures of the generator and the discriminator provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0020] As Figure 1 and Figure 3 shown, the present invention provides a multi-modal image real-time registration method based on a neural network, including the following steps: Step 1: Obtain MR images and ultrasound images for registration; Step 2: Perform edge detection and principal component analysis on the MR image and the ultrasound image in sequence to obtain an MR main structure feature map and an ultrasound main structure feature map; Step 3: Use the generator G in the GAN network to calculate the MR registration deformation field and the ultrasound registration deformation field for the MR main structure feature map and the ultrasound main structure feature map respectively; Step 4: Perform spatial transformation on the MR main structure feature map and the ultrasound main structure feature map using the MR registration deformation field and the ultrasound registration deformation field respectively to generate an MR registered image and an ultrasound registered image; Step 5: Use the discriminator D in the GAN network to perform registration discrimination on the MR registered image and the ultrasound registered image compared with the MR image and the ultrasound image, and identify the MR registered image and the ultrasound registered image with the best registration effect; Step 6: Replace the MR image and the ultrasound image in Step 2 with the MR registered image and the ultrasound registered image with the best registration effect identified by the discriminator D, and repeat Steps 2 - 6 until the structural feature similarity between the MR registered image and the ultrasound registered image with the best registration effect identified by the discriminator D is maximally optimized, and the optimal registered images of the MR image and the ultrasound image are obtained.

[0021] In the registration of MR images and ultrasound images, the present invention uses structural features as the information for registration processing, enabling the alignment of the same physiological structure on the two images after registration. The purpose of registration is to discover the changes presented by the physiological structure in different modalities. The important structure, or the main structure, can reflect most of the characteristics or the main characteristics of the disease lesion, while the detailed structure contains the remaining small part or secondary characteristics. In order to adapt to the real-time registration of MR images and ultrasound images and improve the timeliness or efficiency of registration, the present invention takes the main features in multi-modal images as the focus of registration, weakening or even eliminating the registration attention to the detailed features. Compared with registering all structures, the amount of information processing is reduced, and the corresponding information processing time is also reduced, thus improving the efficiency. Of course, a certain accuracy advantage of registering all structures is sacrificed. Therefore, the present invention takes the main features as the focus of registration, sacrificing part of the accuracy advantage to improve the efficiency advantage, meeting the timeliness requirement of the real-time registration of MR images and ultrasound images.

[0022] The present invention first extracts all structural information from MR images and ultrasound images through an edge detection algorithm (such as the Canny operator or other algorithms with equivalent effects), and then extracts the main structural information from all the structural information through two-layer convolution operations of the principal component analysis network, which is used as the focus of registration, so as to align the main structure during the registration process, quickly grasp the changes of the main structure in different modalities, and correspondingly quickly grasp the characteristics that reflect most of the disease lesions or the main characteristics, achieving the expectation of real-time registration to assist in the rapid diagnosis of diseases.

[0023] The present invention uses the GAN network to achieve registration in two directions for the main structural information. In the GAN network, the generator G generates the deformation fields for registration in two directions. One direction is to generate the registration deformation field for registering the MR image to the ultrasound image, which is used to achieve the spatial transformation of the MR image towards the ultrasound image to complete the registration of the main structure in the two modalities. The other direction is to generate the registration deformation field for registering the ultrasound image to the MR image, which is used to achieve the spatial transformation of the ultrasound image towards the MR image to complete the registration of the main structure in the two modalities. The two discriminators D in the GAN network are used to discriminate the registration results of the main structure formed in the two directions, respectively ensuring that the registration of the main structure output in the two registration directions reaches the best registration result. Therefore, after the GAN network is trained, the best MR registration image and ultrasound registration image in the two registration directions can be obtained, that is, the best registration of the main structure from the MR image to the ultrasound image is completed, and the best registration of the main structure from the ultrasound image to the MR image is completed, ensuring high-precision registration that is independent in the two directions.

[0024] Furthermore, in order to mutually constrain two independent high-precision registrations of the present invention and limit the randomness generated in their respective independent registration directions, an optimization objective is established, and the GAN network registration process is repeatedly cycled. The optimal main structure registration results obtained in the two registration directions are further constrained and optimized. Specifically, the main structure registration results in the two registration directions are re-used as two images to be registered and input into the GAN network for registration in the two directions, and this process is repeated until the maximum similarity is achieved between the main structure registration results in the two registration directions, ensuring that the registration processes in the two registration directions attract each other. That is, the registration result of the MR image to the ultrasound image approaches the registration result of the ultrasound image to the MR image during the process of achieving the optimization objective, thereby pulling the registration direction of the MR image to the ultrasound image towards the registration of the ultrasound image to the MR image. Similarly, the registration result of the ultrasound image to the MR image approaches the registration result of the MR image to the ultrasound image during the process of achieving the optimization objective, thereby pulling the registration direction of the ultrasound image to the MR image towards the registration of the MR image to the ultrasound image. The two registration directions mutually restrict and constrain each other, ensuring that the obtained main structure information registration result can integrate the high precision of the two registration directions and obtain a higher-precision registration result of the MR image and the ultrasound image.

[0025] The present invention first extracts all structure information in the MR image and the ultrasound image through an edge detection algorithm (such as the Canny operator or other algorithms with equivalent effects), and then extracts the main structure information from all the structure information through two-layer convolution operations of the principal component analysis network, which is used as the registration focus point, so as to align the main structure during the registration process, quickly grasp the changes of the main structure in different modalities, and correspondingly quickly grasp the majority of the features / main features reflecting the disease lesions, achieving the expectation of real-time registration. Specifically as follows: The extraction methods of the MR main structure feature map and the ultrasound main structure feature map include: Respectively use the Canny operator to perform edge detection on the MR image and the ultrasound image in sequence to obtain the MR structure feature map and the ultrasound structure feature map; Respectively perform principal component analysis on the MR structure feature map and the ultrasound structure feature map to obtain the MR main structure feature map and the ultrasound main structure feature map.

[0026] The method for performing principal component analysis on the MR structure feature map and the ultrasound structure feature map includes: Use the first convolutional neural network and the second convolutional neural network in the principal component analysis network to perform convolutional processing on the MR structure feature map respectively, and fuse the output results of the first convolutional neural network and the second convolutional neural network through an exponential function to obtain the MR main structure feature map PSR_f; The first convolutional neural network and the second convolutional neural network in the principal component analysis network are used to perform convolutional processing on the ultrasonic structure feature map respectively, and the output results of the first convolutional neural network and the second convolutional neural network are fused through an exponential function to obtain the ultrasonic principal structure feature map PSR_r.

[0027] The setting methods of the first convolutional neural network and the second convolutional neural network in the principal component analysis network include: Select image patches centered on each voxel in the MR structure feature map, perform mean removal and vectorization processing, then form a matrix with the vectorized results for PCA processing, matrixize the eigenvectors corresponding to the obtained eigenvalues, and obtain the convolutional kernels of the first convolutional neural network in the principal component analysis network for the principal component analysis of the MR structure feature map; Perform mean removal and vectorization processing on the image patches centered on each voxel in the output result of the first convolutional neural network, then form a matrix with the vectorized results for PCA processing, matrixize the eigenvectors corresponding to the obtained eigenvalues, and obtain the convolutional kernels of the second convolutional neural network in the principal component analysis network for the principal component analysis of the MR structure feature map; Select image patches centered on each voxel in the ultrasonic structure feature map, perform mean removal and vectorization processing, then form a matrix with the vectorized results for PCA processing, matrixize the eigenvectors corresponding to the obtained eigenvalues, and obtain the convolutional kernels of the first convolutional neural network in the principal component analysis network for the principal component analysis of the ultrasonic structure feature map; Perform mean removal and vectorization processing on the image patches centered on each voxel in the output result of the first convolutional neural network, then form a matrix with the vectorized results for PCA processing, matrixize the eigenvectors corresponding to the obtained eigenvalues, and obtain the convolutional kernels of the second convolutional neural network in the principal component analysis network for the principal component analysis of the ultrasonic structure feature map.

[0028] In the present invention, the GAN network is used to achieve registration in two directions for the principal structure information. Among them, the generator G in the GAN network generates deformation fields for registration in two directions. One direction is to generate a registration deformation field for registering the MR image to the ultrasonic image, which is used to achieve spatial transformation of the MR image towards the ultrasonic image to complete the registration of the principal structure in the two modalities. The other direction is to generate a registration deformation field for registering the ultrasonic image to the MR image, which is used to achieve spatial transformation of the ultrasonic image towards the MR image to complete the registration of the principal structure in the two modalities. Specifically as follows: The generation methods of the MR registration deformation field and the ultrasonic registration deformation field include: Input the MR principal structure feature map and the ultrasonic principal structure feature map into the generator G together to obtain the MR registration deformation field and the ultrasound registration deformation field ; The generation process of the registration deformation field is as follows: ; In the formula, and are the MR registration deformation field and the ultrasound registration deformation field respectively, and are the MR main structure feature map and the ultrasound main structure feature map respectively, and G is the generator; The generator G is formed by the Transformer model structure combined with graph processing, and the generator G includes two input channels; The loss function for training the generator G is: ; Among them, ; ; ; In the formula, is the generator loss function, , and are respectively , and penalty coefficients, is the adversarial loss, is the registration structure loss, is the deformation field smoothing loss, D is the output discrimination probability of the discriminator D, is 's discrimination probability, is 's discrimination probability, is the main structure feature map of the MR registration image, is the main structure feature map of the ultrasound registration image, and are the MR main structure feature map and the ultrasound main structure feature map respectively, represents and 's gradient, E represents expectation, and F represents the Frobenius norm.

[0029] Taking the three loss functions of adversarial loss, registration structure loss, and deformation field smoothing loss as the loss of the generator G can ensure that the generator generates a smooth and accurate main structure registration result.

[0030] In addition, the training objective of the generator G is to generate registered images that the discriminator D cannot distinguish. Therefore, the optimization of the generator G is transformed into maximization. Thus, the training of the GAN network of the present invention is a minimax game process between the generator G and the discriminator D, which is an adversarial learning strategy. The present invention speeds up the convergence rate of the generator G by minimizing the loss between the discrimination probability of the registered image and 1. Therefore, the learning objective of this adversarial loss term is as follows: for the registered image generated by the generator G, the discrimination value of the discriminator D is close to 1, that is, the discriminator D is misled to misjudge the registered image as the reference image (i.e., the main structure feature map of the MR image and the main structure feature map of the ultrasound image in the present invention). This adversarial loss term penalizes the difference between the registered image and the input image of the discriminator D, so as to urge the registered image generated by the GAN network to match / align with the main structure of the input image, that is, the main structure in the MR registered image matches / aligns with the main structure in the ultrasound image, and the main structure in the ultrasound registered image matches / aligns with the main structure in the MR image, completing the main structure registration in two registration directions.

[0031] In the GAN network of the present invention, two discriminators D are used to discriminate the main structure registration results formed by registration in two directions, respectively to ensure that the main structure registrations output in both registration directions reach the best registration results, specifically as follows: The recognition methods for the MR registered image and the ultrasound registered image with the best registration effect include: Respectively use edge detection and principal component analysis to obtain the main structure feature map of the MR registered image and the main structure feature map of the ultrasound registered image ; Input the main structure feature map of the MR registered image into the first discriminator D, and obtain the MR registered image corresponding to the discrimination result of being discriminated as as the MR registered image with the best registration effect; Input the main structure feature map of the ultrasound registered image into the second discriminator D, and obtain the ultrasound registered image corresponding to the discrimination result of being discriminated as as the ultrasound registered image with the best registration effect; Both the first discriminator D and the second discriminator D are formed by the Transformer model structure combined with graph processing, and both include an input channel; The loss functions for training the first discriminator D and the second discriminator D are: ; In the formula, is the loss function of the discriminator D, is The discrimination probability, is The discrimination probability, is The discrimination probability, is The discrimination probability, is the main structure feature map of the MR registered image, is the main structure feature map of the ultrasound registered image, and are the main structure feature map of the MR and the main structure feature map of the ultrasound respectively. E represents expectation, and F represents the Frobenius norm.

[0032] In the GAN network setting, D() is the output of the discriminator D, representing the probability that the input image of the discriminator D is discriminated as the reference image (i.e., the main structure feature maps of the MR image and the ultrasound image in the present invention). That is, the closer the output probability is to 1, the more likely the discriminator network D is to discriminate the input image as the reference image; the closer the output probability is to 0, the more likely the discriminator network D is to discriminate the input image as the generated registered image.

[0033] Obtain the main structure feature map of the MR registered image and the main structure feature map of the ultrasound registered image The edge detection and principal component analysis adopted in the process are consistent with those adopted in the process of obtaining the main structure feature maps of the MR and the ultrasound.

[0034] Therefore, after the GAN network is trained, it can obtain the best MR registered image and ultrasound registered image in two registration directions, that is, complete the best main structure registration of the MR image to the ultrasound image, and complete the best main structure registration of the ultrasound image to the MR image, ensuring high-precision registration that is independent of each other in two directions.

[0035] In order to make the two independent high-precision registrations restrict each other and limit the randomness generated in their own independent registration directions, the present invention establishes an optimization objective, repeatedly cycles the GAN network registration process, and performs secondary constraint optimization on the optimal main structure registration results obtained in the two registration directions respectively, as follows: The generation method of the optimal registered images of the MR image and the ultrasound image includes: Set the maximization of the structural feature similarity between the MR registered image and the ultrasound registered image with the best registration effect as the optimization objective, and the optimization objective function is: ; In the formula, M is the optimization objective identifier, is the MR registered image with the best registration effect, The ultrasound registration image with the best registration effect, where E represents expectation and F represents the Frobenius norm; Based on the optimization objective M, the registration process of the GAN network that has completed the training of the generator G, the first discriminator D, and the second discriminator D is repeated multiple times until the optimization objective M is achieved. The MR registration image and the ultrasound registration image with the best registration effect generated by the generator G in the GAN network at this time are used as the optimal registration images of the MR image and the ultrasound image.

[0036] In the present invention, the registration results of the main structures in the two registration directions are used as two images to be registered and then input into the GAN network for registration in two directions, and this process is repeated until the maximum similarity is obtained for the registration results of the main structures in the two registration directions, ensuring that the registration processes in the two registration directions attract each other. That is, the registration result of the MR image to the ultrasound image approaches the registration result of the ultrasound image to the MR image during the process of achieving the optimization objective, thereby pulling the registration direction of the MR image to the ultrasound image towards the registration of the ultrasound image to the MR image. Similarly, the registration result of the ultrasound image to the MR image approaches the registration result of the MR image to the ultrasound image during the process of achieving the optimization objective, thereby pulling the registration direction of the ultrasound image to the MR image towards the registration of the MR image to the ultrasound image. The two registration directions restrict and constrain each other, ensuring that the obtained registration result of the main structure information can integrate the high precision of the two registration directions and obtain a higher-precision registration result of the MR image and the ultrasound image.

[0037] The structures of the first discriminator D and the second discriminator D are the same.

[0038] Both its generator G and discriminator D adopt the Transformer model combined with graph processing (as Figure 4 shown). Among them, the G network has two input channels and the output is the generated deformation field, while the D network has only one channel for judging whether the registration is completed. The graph-based Transformer model is the core of the registration network, and its implementation idea is as follows: by dividing the image into blocks, each block corresponds to a node in the graph, and each node searches for the node with the closest distance to form an edge. Then, graph processing and the Transformer model are used to complete feature representation. The Transformer model uses the self-attention mechanism to calculate the correlation between blocks, and reduces the sizes of the key-value feature K and the content feature V through convolutional downsampling to reduce the computational amount and ensure real-time requirements, while the query feature maintains the feature size unchanged, and spatial encoding and position encoding are added to improve the robustness of the algorithm.

[0039] Such as Figure 2As shown in the figure, the present invention provides a real-time multimodal image registration system based on a neural network, which is applied to a real-time multimodal image registration method based on a neural network. The system includes: A data acquisition unit for acquiring MR images and ultrasound images for registration; A feature preprocessing unit for performing edge detection and principal component analysis on the MR image and the ultrasound image in sequence to obtain an MR main structure feature map and an ultrasound main structure feature map; A deep learning unit for using the generator G in the GAN network to respectively calculate the MR registration deformation field and the ultrasound registration deformation field from the MR main structure feature map and the ultrasound main structure feature map; respectively performing spatial transformation on the MR main structure feature map and the ultrasound main structure feature map using the MR registration deformation field and the ultrasound registration deformation field to generate an MR registered image and an ultrasound registered image; using the discriminator D in the GAN network to perform registration discrimination on the MR registered image and the ultrasound registered image compared with the MR image and the ultrasound image, and identifying the MR registered image and the ultrasound registered image with the best registration effect; A registration optimization unit for replacing the MR image and the ultrasound image in the feature preprocessing unit with the MR registered image and the ultrasound registered image with the best registration effect identified by the discriminator D, and repeating the processing procedures of the feature preprocessing unit and the deep learning unit until the structural feature similarity between the MR registered image and the ultrasound registered image with the best registration effect identified by the discriminator D is maximally optimized, so as to obtain the optimal registered images of the MR image and the ultrasound image.

[0040] The present invention uses the GAN network to perform multimodal image registration in two directions. The two-direction registration is mutually constrained, eliminating the randomness of registration, improving the registration accuracy, and adopting the similarity measure of the main structure of the image during the registration process, only involving the main structure information, reducing the amount of information processing, and helping to meet the real-time requirement of real-time registration.

[0041] The above embodiments are only exemplary embodiments of the present application and are not used to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements within the essence and protection scope of the present application, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present application.

Claims

1. A multimodal image real-time registration method based on neural network, characterized in that: The following steps are involved: Step 1, obtaining MR images and ultrasound images for registration; Step 2, performing edge detection and principal component analysis on the MR image and the ultrasound image respectively, to obtain an MR main structure feature map and an ultrasound main structure feature map; Step 3, using the MR main structure feature map and the ultrasound main structure feature map to respectively calculate the MR registration deformation field and the ultrasound registration deformation field using the generator G in the GAN network; Step 4: spatially transform the MR main structure feature map and the ultrasound main structure feature map using the MR registration deformation field and the ultrasound registration deformation field to generate an MR registration image and an ultrasound registration image; Step 5: Use the discriminator D in the GAN network to compare the MR registration image and the ultrasound registration image with the MR image and the ultrasound image for registration discrimination, and identify the MR registration image and the ultrasound registration image with the best registration effect; Step 6: Replace the MR image and ultrasound image in step 2 with the MR registration image and ultrasound registration image with the best registration effect identified by the discriminator D, and repeat steps 2 to 6 until the similarity of structural features between the MR registration image and ultrasound registration image with the best registration effect identified by the discriminator D is maximized and optimized, thereby obtaining the optimal registration image of the MR image and ultrasound image.

2. The method for real-time multimodal image registration based on a neural network according to claim 1, characterized in that: The method for extracting the MR main structure characteristic map and the ultrasound main structure characteristic map comprises: The Canny operator is used to perform edge detection on the MR image and the ultrasound image in sequence to obtain the MR structure feature map and the ultrasound structure feature map; The principal component analysis is performed on the MR structure feature map and the ultrasound structure feature map respectively to obtain the MR main structure feature map and the ultrasound main structure feature map.

3. The method for real-time multimodal image registration based on a neural network according to claim 2, characterized in that: The principal component analysis method for the MR structural characteristic map and the ultrasound structural characteristic map includes: The MR structure feature map is convolved by using the first convolution neural network and the second convolution neural network in the principal component analysis network, and the output results of the first convolution neural network and the second convolution neural network are fused through an exponential function to obtain the MR main structure feature map PSR_f; The first convolutional neural network and the second convolutional neural network in the principal component analysis network are used to perform convolution processing on the ultrasonic structure feature map respectively, and the output results of the first convolutional neural network and the second convolutional neural network are fused through an exponential function to obtain the ultrasonic main structure feature map PSR_r.

4. The method for real-time multimodal image registration based on a neural network according to claim 3, characterized in that: The setting method of the first layer of convolutional neural network and the second layer of convolutional neural network in the principal component analysis network includes: An image block centered on each voxel in the MR structural feature map is selected and de-meaned and vectorized, and the vectorized result is formed into a matrix for PCA processing, and the eigenvector corresponding to the obtained eigenvalue is matrixed to obtain the convolution kernel of the first layer of the convolutional neural network in the principal component analysis network for principal component analysis of the MR structural feature map; The image blocks centered on each voxel in the output results of the first layer of the convolutional neural network are de-averaged and vectorized, and the vectorized results are then formed into a matrix for PCA processing, and the eigenvectors corresponding to the obtained eigenvalues ​​are matrixed to obtain the convolution kernel of the second layer of the convolutional neural network in the principal component analysis network for principal component analysis of the MR structural feature map; An image block centered on each voxel in the ultrasonic structural feature map is selected and de-meaned and vectorized, and the vectorized result is formed into a matrix for PCA processing, and the eigenvector corresponding to the obtained eigenvalue is matrixed to obtain the convolution kernel of the first layer of the convolutional neural network in the principal component analysis network for principal component analysis of the ultrasonic structural feature map; The image blocks centered on each voxel in the output results of the first layer of the convolutional neural network are de-averaged and vectorized, and the vectorized results are formed into a matrix for PCA processing. The eigenvectors corresponding to the obtained eigenvalues ​​are matrixed to obtain the convolution kernel of the second layer of the convolutional neural network in the principal component analysis network for principal component analysis of the ultrasonic structure feature map.

5. The method for real-time multimodal image registration based on a neural network according to claim 4, characterized in that: The method for generating the MR registration deformation field and the ultrasound registration deformation field comprises: MR main structure feature map and ultrasound main structure feature map Input them into the generator G together to obtain the MR registration deformation field and ultrasound registration deformation field ; The generation process of the registration deformation field is: ; In the formula, and are the MR registration deformation field and the ultrasound registration deformation field, and They are MR main structure feature map and ultrasound main structure feature map respectively, and G is the generator; The generator G is formed by a Transformer model structure combined with graph processing, and the generator G includes two input channels; The loss function for training the generator G is: ; in, ; ； ； In the formula, is the generator loss function, , and They are , and Penalty coefficient, To combat losses, is the registration structure loss, is the deformation field smoothing loss, D is the output discrimination probability of the discriminator D, for The probability of discrimination, for The probability of discrimination, is the main structural feature map of the MR registration image, is the main structural feature map of the ultrasound registration image, and They are MR main structure feature map and ultrasound main structure feature map, express and The gradient of , E stands for expectation, and F stands for Frobenius norm.

6. The method for real-time multimodal image registration based on a neural network according to claim 5, characterized in that: The identification method of the MR registration image and the ultrasound registration image with the best registration effect includes: The main structural feature maps of MR registration images are obtained by edge detection and principal component analysis respectively. and the main structural feature map of the ultrasound registration image ; The main structural feature map of the MR registered image Input to the first discriminator D, and get Identify as The MR registration image corresponding to the discrimination result is taken as the MR registration image with the best registration effect; The main structural feature map of the ultrasound registration image Input to the second discriminator D, and get Identify as The ultrasound registration image corresponding to the discrimination result is taken as the ultrasound registration image with the best registration effect; The first discriminator D and the second discriminator D are both formed by a Transformer model structure combined with graph processing, and each includes an input channel; The loss function for training the first discriminator D and the second discriminator D is: ; In the formula, is the loss function of the discriminator D, for The probability of discrimination, for The probability of discrimination, for The probability of discrimination, for The probability of discrimination, is the main structural feature map of the MR registration image, is the main structural feature map of the ultrasound registration image, and They are MR main structure feature map and ultrasound main structure feature map, E represents expectation, and F represents Frobenius norm.

7. The method for real-time multimodal image registration based on a neural network according to claim 6, characterized in that: Obtaining the main structural feature map of MR registration images and the main structural feature map of the ultrasound registration image The edge detection and principal component analysis used in the process are consistent with the edge detection and principal component analysis used in the process of obtaining the MR main structure feature map and the ultrasound main structure feature map.

8. The method for real-time multimodal image registration based on a neural network according to claim 7, characterized in that: The method for generating the optimal registration image of the MR image and the ultrasound image includes: The optimization objective is to maximize the similarity of the structural features between the MR registration image and the ultrasound registration image with the best registration effect. The optimization objective function is: ; Where M is the optimization target identifier, For the MR registration image with the best registration effect, is the ultrasound registration image with the best registration effect, E represents expectation, and F represents Frobenius norm; Based on the optimization target M, the registration process of the GAN network that has completed the training of the generator G and the first discriminator D and the second discriminator D is repeated multiple times until the optimization target M is achieved. The MR registration image and the ultrasound registration image with the best registration effect generated by the generator G in the GAN network at this time are used as the optimal registration images of the MR image and the ultrasound image.

9. The method for real-time multimodal image registration based on a neural network according to claim 8, characterized in that: The first discriminator D and the second discriminator D have the same structure.

10. A multi-modal image real-time registration system based on neural network, characterized in that: A multi-modal image real-time registration method based on a neural network as described in any one of claims 1 to 9, the system comprising: A data acquisition unit, used for acquiring an MR image and an ultrasound image for registration; A feature preprocessing unit is used to perform edge detection and principal component analysis on the MR image and the ultrasound image in sequence to obtain an MR main structure feature map and an ultrasound main structure feature map; A deep learning unit is used to use the generator G in the GAN network to respectively calculate the MR registration deformation field and the ultrasound registration deformation field of the MR main structure feature map and the ultrasound main structure feature map; respectively use the MR registration deformation field and the ultrasound registration deformation field to perform spatial transformation on the MR main structure feature map and the ultrasound main structure feature map to generate an MR registration image and an ultrasound registration image; use the discriminator D in the GAN network to compare the MR registration image and the ultrasound registration image with the MR image and the ultrasound image for registration discrimination, and identify the MR registration image and the ultrasound registration image with the best registration effect; The registration optimization unit is used to replace the MR image and the ultrasound image in the feature preprocessing unit with the MR registration image and the ultrasound registration image with the best registration effect identified by the discriminator D, and repeat the processing of the feature preprocessing unit and the deep learning unit until the similarity of structural features between the MR registration image and the ultrasound registration image with the best registration effect identified by the discriminator D is maximized and optimized, so as to obtain the optimal registration image of the MR image and the ultrasound image.

Images