Multi-modal medical image registration method and device, equipment and storage medium

Abstract

The invention provides a multi-modal medical image registration method and device, equipment and a storage medium, and relates to the technical field of medical instruments, and the method comprises the steps: obtaining an initial medical image set, carrying out the initialization training of the initial medical image set through an initialization registration network, so as to generate first deformation field data and second deformation field data, adjusting the second initial medical image based on the first deformation field data to generate a first initial registration image, adjusting the third initial medical image based on the second deformation field data to generate a second initial registration image, and fusing the first initial registration image, the second initial registration image and the first initial medical image to obtain a fused medical image, and performing segmentation training on the fused medical image by using the segmentation neural network to obtain a segmentation result, calculating loss data according to the segmentation result to optimize the first deformation field data and the second deformation field data, and optimizing the segmentation neural network and the registration network by using the loss data while training the segmentation neural network.

Description

Technical Field

[0001] The present invention relates to the field of medical device technology, and in particular to a multimodal medical image configuration method, device, equipment and storage medium. Background Art

[0002] Multimodal nuclear magnetic resonance imaging (MRI) medical images play an important role in the diagnosis and evaluation of lesions. Common modalities in MRI medical images include diffusion-weighted imaging (DWI), apparent diffusion coefficient (ADC), T1-weighted imaging (T1W1) and T2-weighted imaging (T2W1). MRI medical images of different modalities provide different tissue information. Comprehensive analysis of MRI medical images of multiple modalities can improve the accuracy and reliability of lesion diagnosis.

[0003] For some MRI medical images that take a long time to shoot and are easily affected by the patient's physiological movements such as breathing and heartbeat, such as medical images with DWI modality, there is a problem of misalignment between the image space and pixels. This misalignment problem will seriously affect the comprehensive lesion detection analysis based on multi-modal MRI medical images, thereby reducing the lesion detection accuracy based on multi-modal MRI medical images.

[0004] Based on this, it is necessary to propose a solution to the misalignment problem of MRI medical images with multi-modality. Summary of the invention

[0005] In response to the above problems, the present application provides a multimodal medical image registration method, apparatus, device, storage medium and program product.

[0006] To achieve the above objectives: In a first aspect, an embodiment of the present application provides a multimodal medical image registration method, comprising: Acquire an initial medical image set, wherein the initial medical image set includes a first initial medical image, a second initial medical image, and a third initial medical image of different modalities; Performing initialization training on the initial medical image set using a registration network, and generating first deformation field data and second deformation field data according to the initialization training, wherein the first deformation field data is the deformation field data between the first initial medical image and the second initial medical image, and the second deformation field data is the deformation field data between the first initial medical image and the third initial medical image; Adjusting the second initial medical image based on the first deformation field data to generate a first initial registered image, and adjusting the third initial medical image based on the second deformation field data to generate a second initial registered image; fusing the first initial registered image, the second initial registered image, and the first initial medical image to obtain a fused medical image; The fused medical image is segmented and trained by using a segmentation neural network, loss data is calculated according to a segmentation result obtained by the segmentation training, and the first deformation field data and the second deformation field data are optimized according to the loss data.

[0007] Furthermore, before the step of initializing the training of the initial medical image set using the registration network, the method further comprises: Normalization processing is performed on the first initial medical image, the second initial medical image, and the third initial medical image respectively.

[0008] Further, the step of adjusting the second initial medical image based on the first deformation field data to generate a first initial registered image includes: Acquire a first standard sampling grid of the second initial medical image; Obtaining a first reference sampling grid according to the first standard sampling grid and the first deformation field data; Normalizing the first reference sampling grid to obtain a first target sampling grid; The second initial medical image is sampled according to the first target sampling grid and a preset sampling interpolation function to obtain the first initial registered image.

[0009] Furthermore, the step of adjusting the third initial medical image based on the second deformation field data to generate a second initial registered image includes: Acquire a second standard sampling grid of the third initial medical image; Obtaining a second reference sampling grid according to the second standard sampling grid and the second deformation field data; Normalizing the second reference sampling grid to obtain a second target sampling grid; The third initial medical image is sampled according to the second target sampling grid and a preset sampling interpolation function to obtain the second initial registered image.

[0010] Furthermore, the step of calculating loss data according to the segmentation result obtained by the segmentation training includes: A preset loss function is obtained, and the loss data is calculated according to the loss function, the segmentation result, and the real annotation of the fused medical image.

[0011] Further, the step of optimizing the first deformation field data and the second deformation field data according to the loss data comprises: Determining whether the loss data converges to a first preset condition; If not, calculate the gradient of the loss function with respect to the first deformation field data and calculate the gradient of the loss function with respect to the second deformation field data, update the first deformation field data according to the gradient of the first deformation field data and a preset global learning rate back propagation, and update the second deformation field data according to the gradient of the second deformation field data and the global learning rate back propagation.

[0012] Furthermore, the method further comprises: Determining whether the loss data converges to a first preset condition; If not, the gradient of the loss function with respect to the segmentation result is calculated, and the segmentation result is updated according to the gradient of the segmentation result by back propagation.

[0013] Furthermore, the method further comprises: Determining whether the loss data converges to a first preset condition; If not, the gradient of the loss function with respect to the segmentation parameters of the segmentation neural network is calculated, and the segmentation parameters are updated by back propagation according to the gradient of the segmentation parameters and a preset global learning rate.

[0014] Further, the first initial medical image is a T2-weighted imaging medical image, one of the second initial medical image and the third initial medical image is an apparent diffusion coefficient medical image, and the other of the second initial medical image and the third initial medical image is a diffusion-weighted imaging medical image; Or, the first initial medical image is the apparent diffusion coefficient medical image, one of the second initial medical image and the third initial medical image is the diffusion weighted imaging medical image, and the other of the second initial medical image and the third initial medical image is the T2 weighted imaging medical image; Or, the first initial medical image is a diffusion weighted imaging medical image, one of the second initial medical image and the third initial medical image is the T2 weighted imaging medical image, and the other of the second initial medical image and the third initial medical image is the apparent diffusion coefficient medical image.

[0015] In a second aspect, an embodiment of the present application provides a multimodal medical image registration device, comprising: An initial medical image set acquisition module is configured to acquire an initial medical image set, wherein the initial medical image set includes a first initial medical image, a second initial medical image, and a third initial medical image of different modalities; a deformation field data generating module, configured to perform initialization training on the initial medical image set using a registration network, and generate first deformation field data and second deformation field data according to the initialization training, wherein the first deformation field data is the deformation field data between the first initial medical image and the second initial medical image, and the second deformation field data is the deformation field data between the first initial medical image and the third initial medical image; an initial registration image generation module, configured to adjust the second initial medical image based on the first deformation field data to generate a first initial registration image, and to adjust the third initial medical image based on the second deformation field data to generate a second initial registration image; an image fusion module, configured to fuse the first initial registered image, the second initial registered image, and the first initial medical image to obtain a fused medical image; The deformation field data optimization module is configured to perform segmentation training on the fused medical image using a segmentation neural network, calculate loss data according to the segmentation results obtained from the segmentation training, and optimize the first deformation field data and the second deformation field data according to the loss data.

[0016] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the methods provided in the first aspect above.

[0019] In the multimodal medical image registration method, apparatus, equipment, storage medium and program product provided in the present application, the first initial medical image of the initial medical image set is used as a reference, the second initial medical image is corrected by the first deformation field data of self-supervised learning, and the third initial medical image is corrected by the second deformation field data of automatic supervised learning, so as to realize the automatic alignment of the first initial registration image obtained based on the second initial medical image with the first initial medical image, and realize the automatic alignment of the second initial registration image obtained based on the third initial medical image with the first initial medical image. Then, a segmentation neural network is used to segment the fused medical image obtained by fusing the first initial registration image, the second initial registration image and the first initial medical image, and the loss data is calculated according to the segmentation result to optimize the first deformation field data and the second deformation field data, so as to realize the training of the segmentation neural network while using the loss data to optimize the segmentation neural network and the registration network, thereby improving the accuracy of lesion detection based on the fused medical image, and facilitating accurate quantitative analysis and research on medical images including those fused with multiple modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a process flow of a multimodal medical image registration method provided in one embodiment of the present application; Figure 2 A schematic flow chart of a multimodal medical image registration method provided in another embodiment of the present application; Figure 3 Based on Figure 1 or Figure 2 Schematic diagram of multimodal medical image registration shown; Figure 4 A block diagram of a multimodal medical image registration device provided in one embodiment of the present application; Figure 5 A schematic diagram of the structure of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below according to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0022] In addition, some of the processes described in the specification, claims and the above-mentioned figures of the present application include multiple operations that appear in a specific order, and these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0023] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] The inventor of the present application has found through research that in MRI medical images, the shooting time of MRI medical images with modalities of DWI, ADC and T2W1 is relatively long, and they are easily affected by physiological movements such as breathing and heartbeat of patients, and are prone to image space and pixel misalignment problems, which will seriously affect the comprehensive analysis of MRI medical images based on multi-modality and reduce the detection accuracy of lesions. Especially for deep learning models, the design of multi-channel input can provide rich feature representations for the fusion of medical images of different modalities, but misplaced pixels may cause the fusion of medical images of different modalities to lose some information or erroneously superimpose some information, and misplaced pixels may also cause the boundaries of medical images of different modalities to be blurred, resulting in the problem of inaccurate lesion detection in the subsequent fusion of medical images of different modalities due to the confusion of features at different positions.

[0025] In order to improve the accuracy of lesion detection in MRI medical images that are prone to pixel space and pixel misalignment problems when multimodal image fusion is input, the inventor of the present application proposes to introduce a medical image registration method when multimodal MRI medical image data is fused and input to improve the accuracy of lesion detection in MRI medical images that are fused with multiple modalities.

[0026] In the first aspect, in one embodiment of the present application, Figure 1 As shown, a multimodal medical image registration method is provided, and the method 100 comprises: Step S10, obtaining an initial medical image set.

[0027] The initial medical image set includes a first initial medical image, a second initial medical image and a third initial medical image of different modalities.

[0028] In some embodiments, different MRI imaging devices may be used to collect medical images of the same target object to obtain initial medical images of different modalities of the same target object. Correspondingly, different modalities of MRI medical images may also be collected by the same MRI imaging device to obtain initial medical images of different modalities of the same target object.

[0029] In some other embodiments, the initial medical image set may be stored in a storage device, so that when executing step S10, the initial medical image set is obtained from the storage device in a data reading manner.

[0030] Step S30: Initialize the training of the initial medical image set using the registration network, and generate first deformation field data and second deformation field data according to the initialization training.

[0031] The first deformation field data is the deformation field data between the first initial medical image and the second initial medical image, and the second deformation field data is the deformation field data between the first initial medical image and the third initial medical image.

[0032] That is, in this step S30, the first initial medical image is used as a fixed image, and the second initial medical image and the third initial medical image are used as floating images respectively, and the first deformation field data corresponding to the second initial medical image and the second deformation field data corresponding to the third initial medical image are output.

[0033] Further, in one embodiment, the first initial medical image is a T2-weighted imaging (T2W1) medical image, the second initial medical image is an apparent diffusion coefficient (ADC) medical image, and the third initial medical image is a diffusion-weighted imaging (DWI) medical image.

[0034] Since T2W1 medical images, ADC medical images and DWI medical images are all taken over a long period of time in MRI medical images and are easily affected by physiological movements such as breathing and heartbeat of the patient, any of the three is prone to image space and pixel misalignment. The three play an important role in the diagnosis and evaluation of prostate tumors for prostate multimodal magnetic resonance imaging. In this embodiment, the T2W1 medical image is exemplarily used as a fixed image, and the ADC medical image and the DWI medical image are respectively used as floating images to introduce an efficient and autonomous multimodal image registration method to improve the detection accuracy of prostate tumors.

[0035] It is worth mentioning that in another embodiment of the present application, the ADC medical image can also be used as a fixed image, and the T2W1 medical image and the DWI medical image can be used as floating images respectively. In another embodiment of the present application, the DWI medical image can also be used as a fixed image, and the ADC medical image and the T2W1 medical image can be used as floating images respectively.

[0036] Furthermore, in one embodiment, the "trained registration network" refers to a three-dimensional codec model (also known as a 3D codec model) that has been trained with data. Specifically, the 3D codec model includes a three-dimensional data encoder and a three-dimensional data decoder. The three-dimensional data encoder is used to extract three-dimensional features from a fixed image and a floating image, and the three-dimensional data decoder is used to upsample the three-dimensional features extracted by the three-dimensional data encoder to obtain deformation field data between the fixed image and the floating image. The deformation field data includes a deformation field and a variation parameter.

[0037] Exemplarily, the “extracting first deformation field data” in step S30 may specifically include: Step S311, based on the convolution block and the pooling layer of the three-dimensional data encoder, the size of the second initial medical image is gradually reduced and the features of the second initial medical image are extracted by down-sampling.

[0038] In step S311, taking the input second initial medical image with a size of 1×C×H×W×D as an example, 1 represents the batch size, C represents the number of channels, and H, W and D represent the height, width and depth of the second initial medical image, respectively.

[0039] The three-dimensional data encoder gradually reduces the size of the second initial medical image and extracts its features through multiple convolution blocks and pooling layers. Each convolution block includes multiple three-dimensional convolution layers, each of which is followed by a ReLU activation function for introducing nonlinearity and a batch normalization layer. In this way, the new size of the output of the size of 1×C×H×W×D after passing through the convolution block is 1×2C×H×W×D, that is, the number of channels of the second initial medical image is doubled, while the height, width and depth of the second initial medical image remain unchanged.

[0040] Furthermore, the pooling layer of the three-dimensional data encoder is used to reduce the size of the second initial medical image. Generally speaking, the pooling layer uses a maximum pooling with a step size of 2, so that after each passing through the pooling layer, the output second initial medical image is: .

[0041] That is, in the size of the second initial medical image after each time passing through the pooling layer, the number of channels of the second initial medical image remains unchanged, while the height, width and depth of the second initial medical image are all reduced by one. The second initial medical image after N times of downsampling feature extraction operations is output to the bridge layer of the three-dimensional data encoder.

[0042] Step S312: Based on the bridge layer of the three-dimensional data encoder, directly transmit the extracted features of the second initial medical image to the corresponding layer of the three-dimensional data decoder.

[0043] Step S313: gradually restore the size of the second initial medical image based on the upsampling layer and the convolution block of the three-dimensional data decoder.

[0044] Corresponding to the N times downsampling feature extraction operation in step S311, in step S313, N times upsampling decoding is required, and then the second initial medical image that has undergone N times upsampling decoding is output to the output layer of the three-dimensional data decoder.

[0045] Wherein, N is a preset value, and N is an integer not equal to 0. When the floating image is an image of low image quality, if the floating image is downsampled too many times, the main features thereof will be lost. By limiting the number of downsampling times of the floating image to a set number, the three-dimensional data encoder can reduce the loss of features when downsampling the floating image, thereby improving the accuracy of the first deformation field data between the first initial medical image and the second initial medical image, and further improving the accuracy of the subsequent registration result.

[0046] Step S314: Generate learnable first deformation field data based on the output layer of the three-dimensional data decoder.

[0047] The first deformation field data includes a first deformation field and a first deformation field parameter.

[0048] In step S314, the output layer is passed through a 3-channel 3D convolutional layer to generate a first deformation field. The first deformation field is a 3D vector field, which represents the displacement of each voxel in the second initial medical image in space.

[0049] As described above, according to steps S311 to S314, the second initial medical image generates a learnable first deformation field flow_1, and the first deformation field flow_1 represents a five-dimensional tensor. The dimensions of the first deformation field flow_1 are (N, 1, 3, H, W), where N is the batch size, 1 is the number of channels, 3 represents the components of the displacement vector of each voxel in three dimensions, and H and W are the height and width of the second initial medical image, respectively.

[0050] The “extracting the second deformation field data” in step S30 may be implemented by referring to the aforementioned steps S311 to S314 , which will not be described in detail herein.

[0051] It is worth mentioning that, in one embodiment, Figure 2 As shown, before step S30, the method 100 may further include: Step S20 , normalizing the first initial medical image, the second initial medical image, and the third initial medical image respectively.

[0052] That is, in this step S20, each initial medical image in the initial medical image set is preprocessed, the intensity value of each initial medical image in the initial medical image set is mapped to the range of 0 to 1 using linear normalization, and the size of each initial medical image in the initial medical image set is adjusted to a size of 1×C×H×W×D.

[0053] Exemplarily, in one embodiment, minimum-maximum normalization is used to scale each initial medical image in the initial medical image set to between 0 and 1. For the same initial medical image, the minimum intensity value of the initial image is mapped to 0, and the maximum intensity value of the initial image is mapped to 1.

[0054] In this embodiment, when preprocessing the initial medical image set, each initial medical image in the initial medical image set is normalized, thereby reducing the difference between any two initial medical images, and further reducing the difference between the fixed image and any floating image, so as to facilitate the registration network to quickly obtain the first deformation field data and the second deformation field data, thereby improving the work efficiency of multiple registrations of medical images of different modalities.

[0055] Please continue reading Figure 1 as well as Figure 2 After step S30, the method 100 includes: Step S40: adjusting the second initial medical image based on the first deformation field data to generate a first initial registered image, and adjusting the third initial medical image based on the second deformation field data to generate a second initial registered image.

[0056] In step S40, by aligning the second initial medical image with the first initial medical image and aligning the third initial medical image with the first initial medical image so as to facilitate the subsequent steps of synchronously fusing all the initial medical images in the initial medical image set, the work efficiency of fusing medical images of three or more modalities is improved.

[0057] Specifically, in one embodiment, the step S40 of “adjusting the second initial medical image based on the first deformation field data to generate the first initial registered image” specifically includes: Step S411, obtaining a first standard sampling grid of a second initial medical image.

[0058] The first standard sampling grid is a standard sampling grid of the second initial medical image generated when the first deformation field is initialized. In medical image registration, a "standard sampling grid" refers to a regular three-dimensional grid used to define the initial position of each voxel in the medical image. During the forward propagation of the deformation field, the standard sampling grid is combined with the deformation field to calculate a new sampling position to achieve the deformation of the medical image. Based on the introduction of the standard sampling grid, the deformation process of the medical image can be carried out accurately.

[0059] Therefore, in step S411, the first standard sampling grid provides an initial spatial position for each voxel in the second initial medical image, and the first standard sampling grid is evenly distributed and covers the entire spatial range of the second initial medical image.

[0060] Step S412: obtaining a first reference sampling grid according to the first standard sampling grid and the first deformation field data.

[0061] The first reference sampling grid is a new sampling position of the second initial medical image obtained based on the first standard sampling grid and the first deformation field, so as to achieve deformation of the second initial image.

[0062] Specifically, during the forward propagation of the first deformation field, the first standard sampling grid is added to the first deformation field. Since the first deformation field is a vector field, that is, the first deformation field indicates the displacement direction and distance of each voxel in the second initial medical image, a new sampling position of the second initial medical image can be obtained by adding the vector of the first deformation field to each point of the first standard sampling grid.

[0063] Step S412 can be understood by the first preset relationship: new_locs1=grid1+flow1(1); In the first preset relational expression (1), new_locs1 is the first reference sampling grid, grid1 is the first standard sampling grid, and flow1 is the first deformation field.

[0064] Step S413: normalize the first reference sampling grid to obtain a first target sampling grid.

[0065] Exemplarily, step S413 specifically includes normalizing the value of each point of the first parameter sampling grid to a range of [-1, 1]. In the first parameter sampling grid, "-1" and "1" represent displacement vectors with different directions but the same vector value.

[0066] Since the deformation vectors of the first deformation field may have different dimensions and orders of magnitude, normalizing the first reference sampling grid can eliminate the differences between dimensions and orders of magnitude, so that deformations in different directions can be compared and processed fairly, thereby simplifying the subsequent calculation process based on the first target sampling grid. For medical image registration, the first target sampling grid can be directly used for coordinate transformation without additional scaling operations.

[0067] Step S413 can be understood by the second preset relationship: (2); In the second preset relationship (2), flow_shape1[i] is the size of the first deformation field in dimension i, wherein dimension i specifically refers to the first deformation field in three-dimensional space along one of the x-axis, y-axis and z-axis, the x-axis is perpendicular to the plane where the y-axis and the z-axis are located, the y-axis is perpendicular to the plane where the x-axis and the z-axis are located, and the z-axis is perpendicular to the plane where the x-axis and the y-axis are located.

[0068] new_locs1[:, i, …] is the position in dimension i after deformation by the first deformation field.

[0069] Step S414: sampling the second initial medical image according to the first target sampling grid and a preset sampling interpolation function to obtain a first initial registered image.

[0070] Exemplarily, the sampling interpolation function is a trilinear interpolation function, and the trilinear interpolation function is used to interpolate the value of each point of the first target sampling grid to obtain a new sampling position of the second initial medical image. Therefore, based on the new sampling position, the voxels in the second initial medical image are sampled again to obtain the first initial registration image.

[0071] Specifically, in one embodiment, the step S40 of “adjusting the third initial medical image based on the second deformation field data to generate the second initial registered image” specifically includes: Step S421, obtaining a second standard sampling grid of the third initial medical image.

[0072] The second standard sampling grid is the standard sampling grid of the third initial medical image generated during initialization.

[0073] Step S422: obtaining a second reference sampling grid according to the second standard sampling grid and the second deformation field data.

[0074] Step S423: normalize the second reference sampling grid to obtain a second target sampling grid.

[0075] Step S424: sampling the third initial medical image according to the second target sampling grid and a preset sampling interpolation function to obtain a second initial registered image.

[0076] The specific principles of steps S421 to S424 can be found in steps S411 to S414, and this application will not elaborate on them.

[0077] Please continue reading Figures 1 to 3 After step S40, the method 100 further includes: Step S50: fusing the first initial registered image, the second initial registered image, and the first initial medical image to obtain a fused medical image.

[0078] Exemplarily, the first initial medical image is a T2W1 medical image, the second initial medical image is an ADC medical image, and the third initial medical image is a DWI medical image.

[0079] The first initial registration image is transformed into T(ADC; θ) according to the first deformation field parameter matrix of the first deformation field data. ADC ), where θ ADC is the parameter weight of the first deformation field parameter. The second initial registration image is transformed into T(DWI; θ) according to the second deformation field parameter matrix of the second deformation field data. DWI ), where θ DWI is the parameter weight of the second deformation field parameter. Then, according to step S50, a fused medical image I can be obtained based on the fusion function, the first initial registered image, the second initial registered image and the first initial medical image. f =Fuse(T(ADC);θ ADC ), T(DWI;θ DWI ), T2W1).

[0080] It is worth mentioning that this embodiment does not specifically limit the fusion function. The fusion function may be, but is not limited to, a weighted average fusion function, a feature-based fusion function, and a deep learning model-based fusion function.

[0081] In step S50, by integrating the different medical images of the three modalities, the advantages of each modality can be brought into play, the shortcomings of the medical images of a single modality can be compensated, and more accurate detection results can be provided for lesion detection.

[0082] Step S60, performing segmentation training on the fused medical image using a segmentation neural network, calculating loss data according to the segmentation results obtained from the segmentation training, and optimizing the first deformation field data and the second deformation field data according to the loss data.

[0083] In this step, the segmentation neural network is a deep learning model for image segmentation tasks. Its main goal is to divide the fused medical image into multiple regions, each representing a different object or a specific category, and perform feature extraction to determine the segmentation result output from the segmentation neural network.

[0084] It is worth mentioning that this embodiment does not specifically limit the segmentation neural network. The segmentation neural network can be, but is not limited to, a SegNet model, a full convolutional network model, a Mask R-CNN model, a Faster R-CNN model, and a DeepLab series model.

[0085] Taking the SegNet model as an example, the segmentation result can be expressed by the following third preset relationship: S=SegmentationNet(I f )(3); In the third preset relational expression (3), I f is the fusion medical image, and S is the segmentation result.

[0086] Specifically, the “calculating loss data according to the segmentation result” in step S60 specifically includes: Get the preset loss function, and calculate the loss data based on the loss function, segmentation results, and real annotations of the fused medical image.

[0087] In this embodiment, the loss function is an indicator for measuring the difference between the segmentation result output by the segmentation neural network and the true annotation of the fused medical image.

[0088] Furthermore, in one embodiment, the Dice loss function is used as the loss function. Thus, the Dice loss function is used to evaluate the similarity between the segmentation result and the ground truth (GT), and the value range of the Dice loss function is between 0 and 1. When the segmentation result and the ground truth are completely consistent, the Dice loss function takes the minimum value of 0. The ground truth is the image segmentation result obtained by manual delineation by an expert or other reliable method, which represents the "gold standard" of the segmentation task.

[0089] The Dice Loss function is specifically expressed by the following fourth preset relational expression: (4); In the fourth preset relational expression (4), For loss data, is the segmentation result obtained by segmenting the fused medical image using the segmentation neural network, where: , is the first initial registration image, is the second initial registration image, is the first initial medical image.

[0090] Further, in one embodiment, the step S60 of “optimizing the first deformation field data and the second deformation field data according to the loss data” specifically includes: Determine whether the loss data converges to the first preset condition. If not, calculate the gradient of the loss function with respect to the first deformation field data and the gradient of the loss function with respect to the second deformation field data, update the first deformation field data according to the gradient of the first deformation field data and the global learning rate back propagation, and update the second deformation field data according to the gradient of the second deformation field data and the global learning rate back propagation. On the contrary, if the loss data converges to the first preset condition, the training of the segmentation neural network is completed.

[0091] The first preset condition can be set according to actual needs, and can be determined by setting a loss function threshold or a set number of iterations according to the loss function called. For example, still taking the Dice loss function as an example, if the loss data The closer the value is to 0, the closer the segmentation result obtained by the segmentation neural network for the fused medical image is to the true annotation.

[0092] Among them, the gradient of the first deformation field parameter is: ,θ ADC is the first deformation field parameter, D ADC is the first deformation field. Back propagation updates the first deformation field parameters: , α is the global learning rate.

[0093] Among them, the gradient of the second deformation field parameter is: ,θ DWI is the second deformation field parameter, D DWI is the second deformation field. Back propagation updates the second deformation field parameters: , α is the global learning rate.

[0094] In the multimodal medical image registration method provided in this embodiment, the first initial medical image of the initial medical image set is used as a reference, the second initial medical image is corrected by the first deformation field data of self-supervised learning, and the third initial medical image is corrected by the second deformation field data of automatic supervised learning, so as to realize the automatic alignment of the first initial registered image obtained based on the second initial medical image with the first initial medical image, and realize the automatic alignment of the second initial registered image obtained based on the third initial medical image with the first initial medical image. Then, the trained segmentation neural network is used to segment the fused medical image obtained by fusing the first initial registered image, the second initial registered image and the first initial medical image, and the loss data is calculated according to the segmentation result to optimize the first deformation field data and the second deformation field data, so as to improve the lesion detection accuracy based on the fused medical image, and facilitate accurate quantitative analysis and research on medical images including multi-modal fusion.

[0095] Furthermore, in one embodiment, in the method 100, after step S60, it also includes: step S70, determining whether the loss data converges to a first preset condition, if not, calculating the gradient of the loss function with respect to the segmentation result, and updating the segmentation result by backpropagation according to the gradient of the segmentation result.

[0096] Specifically, the gradient of the segmentation result is: .

[0097] Similar to the above-mentioned embodiment, when the gradient value of the segmentation result converges to the corresponding gradient threshold or the back propagation update reaches the corresponding number of iterations, the training of the segmentation neural network is completed, and the segmentation result of the fused medical image achieves satisfactory segmentation accuracy.

[0098] Furthermore, in one embodiment, in the method 100, after step S60, it also includes: step S80, determining whether the loss data converges to a first preset condition, if not, calculating the gradient of the loss function with respect to the segmentation parameters of the segmentation neural network, and updating the segmentation parameters according to the gradient of the segmentation parameters and the global learning rate back propagation.

[0099] Specifically, the parameter gradient of the segmentation neural network is: . Back-propagation updates the segmentation parameters of the segmentation neural network: , α is the global learning rate, and θs is the segmentation parameter of the segmentation neural network.

[0100] Similar to the aforementioned embodiment, when the value of the parameter gradient of the segmentation neural network converges to the corresponding gradient threshold or the back propagation update reaches the corresponding number of iterations, the training of the segmentation neural network is completed, and the segmentation result of the fused medical image achieves satisfactory segmentation accuracy.

[0101] In the second aspect, in an embodiment provided in the present application, if Figure 4 As shown, a multimodal medical image registration device 200 is also provided, comprising: The initial medical image set acquisition module 201 is configured to acquire an initial medical image set, wherein the initial medical image set includes a first initial medical image, a second initial medical image, and a third initial medical image of different modalities.

[0102] The deformation field data generating module 202 is configured to perform initialization training on the initial medical image set using the registration network, and generate first deformation field data and second deformation field data according to the initialization training.

[0103] The first deformation field data is the deformation field data between the first initial medical image and the second initial medical image, and the second deformation field data is the deformation field data between the first initial medical image and the third initial medical image.

[0104] The initial registration image generation module 203 is configured to adjust the second initial medical image based on the first deformation field data to generate a first initial registration image and to adjust the third initial medical image based on the second deformation field data to generate a second initial registration image.

[0105] The image fusion module 204 is configured to fuse the first initial registered image, the second initial registered image and the first initial medical image to obtain a fused medical image.

[0106] The deformation field data optimization module 205 is configured to perform segmentation training on the fused medical image using a segmentation neural network, calculate loss data according to the segmentation results obtained from the segmentation training, and optimize the first deformation field data and the second deformation field data according to the loss data.

[0107] For the specific definition of the multimodal medical image registration device, please refer to the definition of the multimodal medical image registration method above, which will not be repeated here. Each module in the above-mentioned multimodal medical image registration device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0108] Thirdly, Figure 5 FIG. 1 is a schematic diagram showing the structure of a computer device provided by an embodiment of the present application. Figure 5 As shown, the computer device 300 may include: a processor, a memory, a communication module, a display screen, an input module, a power supply module, and an audio module connected via a system bus.

[0109] Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication module of the computer device is used to communicate with an external computer device or server in a wired or wireless manner, and the wireless manner can be realized by WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the multimodal medical image registration method provided in the present application is realized. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input module of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device, or an external keyboard, touchpad or mouse. The audio module of the computer device is used to convert digital audio information into an analog audio signal output, and is also used to convert analog audio input into a digital audio signal. The audio module can also be used to encode and decode audio signals. In some embodiments, the audio module can be arranged in the processor, or some functional modules of the audio module can be arranged in the processor. Figure 5 Only some components are shown schematically, which does not mean that the computer device 300 only includes Figure 5 Components shown.

[0110] Correspondingly, in a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the multimodal medical image registration method provided in the above-mentioned method embodiments.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the description of this application.

[0113] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal medical image registration method, characterized in that: The method comprises: Acquire an initial medical image set, wherein the initial medical image set includes a first initial medical image, a second initial medical image, and a third initial medical image of different modalities; Performing initialization training on the initial medical image set using a registration network, and generating first deformation field data and second deformation field data according to the initialization training, wherein the first deformation field data is the deformation field data between the first initial medical image and the second initial medical image, and the second deformation field data is the deformation field data between the first initial medical image and the third initial medical image; Adjusting the second initial medical image based on the first deformation field data to generate a first initial registered image, and adjusting the third initial medical image based on the second deformation field data to generate a second initial registered image; fusing the first initial registered image, the second initial registered image, and the first initial medical image to obtain a fused medical image; The fused medical image is segmented and trained by using a segmentation neural network, loss data is calculated according to a segmentation result obtained by the segmentation training, and the first deformation field data and the second deformation field data are optimized according to the loss data.

2. The method according to claim 1, characterized in that Before the step of initializing the training of the initial medical image set using the registration network, the method further comprises: Normalization processing is performed on the first initial medical image, the second initial medical image, and the third initial medical image respectively.

3. The method according to claim 1, characterized in that The step of adjusting the second initial medical image based on the first deformation field data to generate a first initial registration image comprises: Acquire a first standard sampling grid of the second initial medical image; Obtaining a first reference sampling grid according to the first standard sampling grid and the first deformation field data; Normalizing the first reference sampling grid to obtain a first target sampling grid; The second initial medical image is sampled according to the first target sampling grid and a preset sampling interpolation function to obtain the first initial registered image.

4. The method according to claim 1, characterized in that: The step of adjusting the third initial medical image based on the second deformation field data to generate a second initial registration image comprises: Acquire a second standard sampling grid of the third initial medical image; Obtaining a second reference sampling grid according to the second standard sampling grid and the second deformation field data; Normalizing the second reference sampling grid to obtain a second target sampling grid; The third initial medical image is sampled according to the second target sampling grid and a preset sampling interpolation function to obtain the second initial registered image.

5. The method according to claim 1, characterized in that The step of calculating loss data according to the segmentation result obtained by the segmentation training comprises: A preset loss function is obtained, and the loss data is calculated according to the loss function, the segmentation result, and the real annotation of the fused medical image.

6. The method according to claim 5, characterized in that The step of optimizing the first deformation field data and the second deformation field data according to the loss data comprises: Determining whether the loss data converges to a first preset condition; If not, calculate the gradient of the loss function with respect to the first deformation field data and calculate the gradient of the loss function with respect to the second deformation field data, update the first deformation field data according to the gradient of the first deformation field data and a preset global learning rate back propagation, and update the second deformation field data according to the gradient of the second deformation field data and the global learning rate back propagation.

7. The method according to claim 5, characterized in that The method further comprises: Determining whether the loss data converges to a first preset condition; If not, the gradient of the loss function with respect to the segmentation result is calculated, and the segmentation result is updated according to the gradient of the segmentation result by back propagation.

8. The method according to claim 5, characterized in that The method further comprises: Determining whether the loss data converges to a first preset condition; If not, the gradient of the loss function with respect to the segmentation parameters of the segmentation neural network is calculated, and the segmentation parameters are updated by back propagation according to the gradient of the segmentation parameters and a preset global learning rate.

9. The method according to claim 1, characterized in that: The first initial medical image is a T2-weighted imaging medical image, one of the second initial medical image and the third initial medical image is an apparent diffusion coefficient medical image, and the other of the second initial medical image and the third initial medical image is a diffusion-weighted imaging medical image; Or, the first initial medical image is the apparent diffusion coefficient medical image, one of the second initial medical image and the third initial medical image is the diffusion weighted imaging medical image, and the other of the second initial medical image and the third initial medical image is the T2 weighted imaging medical image; Or, the first initial medical image is a diffusion weighted imaging medical image, one of the second initial medical image and the third initial medical image is the T2 weighted imaging medical image, and the other of the second initial medical image and the third initial medical image is the apparent diffusion coefficient medical image.

10. A multimodal medical image registration device, characterized in that: include: An initial medical image set acquisition module is configured to acquire an initial medical image set, wherein the initial medical image set includes a first initial medical image, a second initial medical image, and a third initial medical image of different modalities; a deformation field data generating module, configured to perform initialization training on the initial medical image set using a registration network, and generate first deformation field data and second deformation field data according to the initialization training, wherein the first deformation field data is the deformation field data between the first initial medical image and the second initial medical image, and the second deformation field data is the deformation field data between the first initial medical image and the third initial medical image; an initial registration image generation module, configured to adjust the second initial medical image based on the first deformation field data to generate a first initial registration image, and to adjust the third initial medical image based on the second deformation field data to generate a second initial registration image; an image fusion module, configured to fuse the first initial registered image, the second initial registered image, and the first initial medical image to obtain a fused medical image; The deformation field data optimization module is configured to perform segmentation training on the fused medical image using a segmentation neural network, calculate loss data according to the segmentation results obtained from the segmentation training, and optimize the first deformation field data and the second deformation field data according to the loss data.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images