Data processing method and device based on multi-modal data knowledge alignment fusion

Abstract

The invention provides a data processing method and device based on multi-modal data knowledge alignment fusion, and the method comprises the steps: carrying out the preprocessing of an original MR scanning image and an original PET scanning image in a to-be-fused image pair, and obtaining an aligned to-be-fused image pair; inputting the aligned to-be-fused image pair into the trained twin neural network to obtain a weight map matched with the to-be-fused image pair; performing MST decomposition on the aligned to-be-fused image pair to obtain a low-frequency coefficient matrix and a high-frequency coefficient matrix; and fusing the low-frequency coefficient matrix and the high-frequency coefficient matrix based on the weight map, and performing MST reconstruction on the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix to obtain a target fused image, so that the precision of the fused image can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method and device based on multimodal data knowledge alignment and fusion. Background Art

[0002] In the medical field, MR (Magnetic Resonance Imaging) scans can obtain stereoscopic images of the brain and spinal cord, and are mainly used to diagnose diseases of the brain nervous system (such as common meningiomas). PET (Positron Emission Tomography) scans, as an imaging technology that can image at the molecular level, are mainly used for the diagnosis of malignant tumors and the differential diagnosis of benign and malignant lesions. In order to ensure the accuracy of the diagnosis of brain diseases, existing technologies usually obtain blood circulation information through cerebral arteries and veins through PET scan images, obtain physical anatomy or structural information of the brain through MR scan images, and fuse the images of the two modalities to obtain better information and better visual appearance, avoiding the influence of blurred or uncertain areas in single-modality scan images on the accuracy of subsequent diagnosis.

[0003] However, existing image fusion methods often apply simple fusion rules (such as selecting the maximum value) to obtain the fusion coefficient, resulting in the loss of some spectral information, details and contrast in the fused image. The low-precision fused image with information loss will directly affect the accuracy of subsequent disease diagnosis. Summary of the invention

[0004] The present invention provides a data processing method and device based on multimodal data knowledge alignment and fusion, which is used to solve the problem of low precision of fused images obtained by fusing PET scan images and MR scan images in the prior art.

[0005] On the one hand, the present invention provides a data processing method based on multimodal data knowledge alignment and fusion, comprising: Preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images; Converting the target PET scan image into a target PET scan grayscale image, and inputting the target MR scan image and the target PET scan grayscale image into a trained twin neural network to obtain a weight map for matching the image pair to be fused; Performing MST decomposition on the target MR scan image and the target PET scan grayscale image to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; Based on the weight map, the low-frequency coefficient matrix and the high-frequency coefficient matrix are fused respectively, and the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set are reconstructed by MST to obtain a target fused image corresponding to the pair of images to be fused.

[0006] In an optional embodiment of the present application, the twin neural network includes: a first input branch and a second input branch with the same structure, a fully connected layer and a softmax layer; The first input branch and the second input branch both include a first convolutional layer, a second convolutional layer, a maximum pooling layer and a third convolutional layer connected in sequence; The third convolutional layers of the first input branch and the second input branch are both connected to a fully connected layer, and the fully connected layer is connected to the softmax layer; The size of the input patch images of the first input branch and the second input branch is 16×16 pixels.

[0007] In an optional embodiment of the present application, the twin neural network is obtained after training based on a medical image sample set and predetermined sample labels; The medical image sample set includes a plurality of medical image sample pairs, and any target medical image sample pair includes a clear patch image sample and a fuzzy patch image sample from the same region of the same standard medical image; in the target medical image sample pair, when the clear patch image sample is used as the input patch image of the first input branch, and the fuzzy patch image sample is used as the input patch image of the second input branch, the sample label of the target medical image sample pair is 1; otherwise, the sample label of the target medical image sample pair is 0; The standard medical image is a standard MR scan image or a standard PET scan grayscale image.

[0008] In an optional embodiment of the present application, preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain the aligned image pair to be fused specifically includes: Extracting feature points of the original MR scan image and the original PET scan image respectively, and determining associated descriptors of the feature points; Determine matching feature points of the original MR scan image and the original PET scan image based on the associated descriptors of the feature points of the original MR scan image and the original PET scan image; Determine a transformation model based on matching feature points of the original MR scan image and the original PET scan image; The original MR scan image or the original PET scan image is transformed based on the transformation model to obtain an aligned pair of images to be fused.

[0009] In an optional embodiment of the present application, converting the target PET scan image into a target PET scan grayscale image specifically includes: The target PET scan image is converted from the RGB color space to the IHS color space, and the intensity component of the target PET scan image in the IHS color space is extracted as the target PET scan grayscale image.

[0010] In an optional embodiment of the present application, the fusing of the low-frequency coefficient matrix and the high-frequency coefficient matrix based on the weight map specifically includes: Performing MST decomposition on the weight graph to obtain a high-frequency weight matrix set and a low-frequency weight matrix; Based on the high-frequency weight matrix set, the first high-frequency coefficient matrix set and the second high-frequency coefficient matrix set are fused to obtain a fused high-frequency coefficient matrix set; based on the low-frequency weight matrix, the first low-frequency coefficient matrix and the second low-frequency coefficient matrix are fused to obtain a fused low-frequency coefficient matrix; Among them, the number of high-frequency coefficient matrices in the first high-frequency coefficient matrix set, the second high-frequency coefficient matrix set and the fused high-frequency coefficient matrix set, and the number of high-frequency weight matrices in the high-frequency weight matrix set are the sum of the number of directions at each level corresponding to the MST decomposition.

[0011] In an optional embodiment of the present application, the target fused image is a grayscale image, and the method further includes: Based on the hue component and the saturation component in the IHS color space of the target fused image and the target PET scan image, an HIS inverse transform is performed to obtain a target fused RGB image.

[0012] In a second aspect, the present invention further provides a data processing device based on multimodal data knowledge alignment and fusion, comprising: an alignment module, used for preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images; A weight map acquisition module, used to convert the target PET scan image into a target PET scan grayscale image, and input the target MR scan image and the target PET scan grayscale image into a trained twin neural network to obtain a weight map for matching the image pair to be fused; An MST decomposition module is used to perform MST decomposition on the target MR scan image and the target PET scan grayscale image to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; A fusion module is used to fuse the low-frequency coefficient matrix and the high-frequency coefficient matrix respectively based on the weight map, and perform MST reconstruction on the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set to obtain a target fused image corresponding to the pair of images to be fused.

[0013] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method based on multimodal data knowledge alignment and fusion as described above is implemented.

[0014] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method based on multimodal data knowledge alignment and fusion as described above.

[0015] The present invention provides a data processing method and device based on multimodal data knowledge alignment and fusion, which obtains an aligned image pair to be fused by preprocessing the original MR scan image and the original PET scan image in the image pair to be fused; the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images; the target PET scan image is converted into a target PET scan grayscale image, and the target MR scan image and the target PET scan grayscale image are input into a trained twin neural network to obtain a weight map for matching the image pair to be fused; The target MR scan image and the target PET scan grayscale image are subjected to MST decomposition to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; the low-frequency coefficient matrix and the high-frequency coefficient matrix are fused respectively based on the weight map, and the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set are reconstructed by MST to obtain the target fused image corresponding to the pair of images to be fused. The weight map of the image pair to be fused by the twin neural network can be used, and the high-frequency coefficients and the low-frequency coefficients are fused based on the weight map, so as to avoid the problem of loss of spectral information, details and contrast in the fused image, and improve the accuracy of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of a data processing method based on multimodal data knowledge alignment and fusion provided by the present invention; Figure 2 A schematic diagram of the alignment process of the image pair to be fused provided by the present invention; Figure 3 A schematic diagram of the structure of a twin neural network provided by the present invention; Figure 4 A schematic diagram of the fusion process of the low-frequency coefficient matrix and the high-frequency coefficient matrix provided by the present invention; Figure 5 A structural block diagram of a data processing device based on multimodal data knowledge alignment and fusion provided by the present invention; Figure 6It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The specific implementation process of the data processing method and device based on multimodal data knowledge alignment and fusion in an embodiment of the present invention is described below.

[0020] Figure 1 A flowchart of a data processing method based on multimodal data knowledge alignment and fusion provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method may include: Step 101, preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images.

[0021] Specifically, the original MR scan image and the original PET scan image in the image pair to be fused are obtained by scanning the same part of the human brain and other organs. It is understandable that the original MR scan image and the original PET scan image have spatial deviations due to different acquisition methods, so the two images need to be aligned to ensure that the same anatomical structure is in the same position, and then ensure that the information from the two images can accurately correspond during fusion, thereby ensuring the accuracy of the fused image. At the same time, the alignment operation can also avoid image artifacts and further improve the accuracy of the fused image. More specifically, Figure 2 The schematic diagram of the alignment process of the image pair to be fused provided by the present invention is as follows: Figure 2 As shown, the preprocessing of the original MR scan image and the original PET scan image in the image pair to be fused to obtain the aligned image pair to be fused specifically includes: Step 1011, extracting feature points of the original MR scan image and the original PET scan image respectively, and determining associated descriptors of the feature points; Step 1012, determining matching feature points of the original MR scan image and the original PET scan image based on the associated descriptors of the feature points of the original MR scan image and the original PET scan image; Step 1013, determining a transformation model based on matching feature points of the original MR scan image and the original PET scan image; Step 1014: transform the original MR scan image or the original PET scan image based on the transformation model to obtain an aligned pair of images to be fused.

[0022] It is understandable that the feature point may be a corner point, an edge, etc., which is not specifically limited in the embodiment of the present invention. The embodiment of the present invention preferably uses a SIFT (Scale-invariant feature transform) algorithm to realize feature point extraction and automatic calculation of associated descriptors of feature points. After determining the associated descriptors of feature points, the matching feature points of the original MR scan image and the original PET scan image can be determined based on the similarity of the associated descriptors of the feature points of the original MR scan image and the original PET scan image. Specifically, the matching can be performed based on the Euclidean distance of the associated descriptors of the feature points of the original MR scan image and the original PET scan image. Of course, other feasible matching methods can also be used, which is not specifically limited in the embodiment of the present invention. Afterwards, the transformation model can be determined based on the matching feature points (i.e., matching results) of the original MR scan image and the original PET scan image. Specifically, the transformation model can use an affine transformation model or a nonlinear transformation model as needed, which is not specifically limited in the embodiment of the present invention. After determining the transformation model, the original MR scan image or the original PET scan image can be transformed based on the transformation model to obtain an aligned image pair to be fused.

[0023] It is worth noting that the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, the original PET scan image and the target PET scan image are both RGB images, and the target MR scan image and the target PET scan image have the same size.

[0024] Step 102, converting the target PET scan image into a target PET scan grayscale image, and inputting the target MR scan image and the target PET scan grayscale image into a trained twin neural network to obtain a weight map for matching the image pair to be fused; Specifically, based on the above content, it can be known that since the target MR scan image is a grayscale image, the target PET scan image is an RGB image, and the number of channels of the grayscale image and the RGB image is different (the grayscale image is a single channel, and the RGB image is a three-channel), the shapes of the two inputs of the twin neural network will be inconsistent, which will have a negative impact on the effectiveness and stability of the twin neural network. Therefore, before obtaining the weight map based on the twin neural network, the embodiment of the present invention first converts the target PET scan image into a target PET scan grayscale image. Specifically, the conversion of the target PET scan image into the target PET scan grayscale image specifically includes: The target PET scan image is converted from the RGB color space to the HIS (Intensity-Hue-Saturation) color space, and the intensity component of the target PET scan image in the IHS color space is extracted as the target PET scan grayscale image.

[0025] It can be understood that the intensity component of the target PET scan image in the IHS color space is the target PET scan grayscale image.

[0026] More specifically, Figure 3 A schematic diagram of the structure of the twin neural network provided by the present invention is shown in FIG. Figure 3 As shown, the twin neural network includes: a first input branch and a second input branch with the same structure, a fully connected layer and a softmax layer; The first input branch and the second input branch both include a first convolutional layer, a second convolutional layer, a maximum pooling layer and a third convolutional layer connected in sequence; The third convolutional layers of the first input branch and the second input branch are both connected to a fully connected layer, and the fully connected layer is connected to the softmax layer; The size of the input patch images of the first input branch and the second input branch is 16×16 pixels.

[0027] A key issue in the design of twin neural networks is the selection of the size of the input patch image. When the size of the input patch image is 32×32 pixels, the number of maximum pooling layers is not easy to determine. When the input patch image size is set to 8×8 pixels, the input patch image used to train the twin neural network is too small and the accuracy cannot be guaranteed. Therefore, the preferred input patch image size of the embodiment of the present invention is 16×16 pixels. At the same time, the kernel size of each convolution layer is preferably 3×3 pixels, and the step size is preferably 1 pixel. The size of the convolution kernel determines the size of the neuron receptive field. If it is too small, the twin neural network will not be able to accurately obtain the useful local features of the input image, and if it is too large, the extracted features will be too complex, exceeding the feature extraction capability of the convolution kernel. Since the scaling factor is too large, the extracted image features will not be fine enough, thereby losing a large number of image detail features, and the scaling factor is too small to affect the downsampling effect, therefore, the scaling factor and span of the maximum pooling layer in the embodiment of the present invention are preferably set to 2×2 pixels and 2 pixels respectively, where the scaling is exponential scaling. The 256 feature maps obtained from the first input branch and the second input branch are cascaded, and weights are assigned through the fully connected layer. By assigning the values ​​of all pixels and averaging the overlapping pixels, the weight map W for matching the image pair to be fused can be obtained. CNN It is worth noting that the size of the weight map is the same as the size of the target MR scan image and the target PET scan grayscale image.

[0028] The twin neural network is obtained after training based on a medical image sample set and predetermined sample labels; The medical image sample set includes a plurality of medical image sample pairs, and any target medical image sample pair includes a clear patch image sample and a fuzzy patch image sample from the same region of the same standard medical image; in the target medical image sample pair, when the clear patch image sample is used as the input patch image of the first input branch, and the fuzzy patch image sample is used as the input patch image of the second input branch, the sample label of the target medical image sample pair is 1; otherwise, the sample label of the target medical image sample pair is 0; The standard medical image is a standard MR scan image or a standard PET scan grayscale image.

[0029] It can be understood that the standard MR scan image refers to a high-quality MR scan image, and the standard PET scan grayscale image refers to a high-quality PET scan grayscale image. Based on this, the accuracy of the trained twin neural network can be ensured. For the fuzzy patch image samples from the same area of ​​the same standard medical image in the target medical image sample pair, they can be obtained by fuzzifying the clear patch image samples. For the specific method of fuzzification, conventional methods in the field such as Gaussian filters can be used, and the embodiments of the present invention do not specifically limit this.

[0030] Step 103, performing MST (Multi-Scale Transform) decomposition on the target MR scan image (hereinafter referred to as image A) and the target PET scan grayscale image (hereinafter referred to as image B) to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image.

[0031] Specifically, it can be understood that an image can generally be regarded as a two-dimensional piecewise smooth signal, in which most of the energy is contained in the low-frequency coefficients, while the image edge and contour information is contained in the high-frequency coefficients. Based on this, an embodiment of the present invention performs MST decomposition on the target MR scan image and the target PET scan grayscale image to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image. Based on this, it is possible to take into account both the low-frequency band information and the high-frequency short information of the image, thereby avoiding the problem of loss of spectral information, details and contrast of the target MR scan image and the target PET scan grayscale image.

[0032] Step 104 , based on the weight map, respectively fuse the low-frequency coefficient matrix and the high-frequency coefficient matrix, and perform MST reconstruction on the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set to obtain a target fused image corresponding to the pair of images to be fused.

[0033] Specifically, to ensure the accuracy of the fused image, the setting of the fusion rule is also crucial. Conventional fusion methods usually use average fusion rules for low-frequency domains, which usually leads to contrast loss; the maximum absolute fusion rule is usually used in high-frequency domains, but this method results in less fine-scale details being transmitted between the target MR scan image and the target PET scan grayscale image. To address these problems, the embodiment of the present invention extracts a weight map W based on a twin neural network. CNN The low-frequency coefficient matrix and the high-frequency coefficient matrix are fused to enhance the detail information of the target fused image. More specifically, Figure 4A schematic diagram of the fusion process of the low-frequency coefficient matrix and the high-frequency coefficient matrix provided by the present invention is shown in FIG. Figure 4 As shown, the low-frequency coefficient matrix and the high-frequency coefficient matrix are respectively fused based on the weight map, specifically including: Step 1041, performing MST decomposition on the weight graph to obtain a high-frequency weight matrix set and a low-frequency weight matrix; Step 1042, based on the high-frequency weight matrix set, the first high-frequency coefficient matrix set and the second high-frequency coefficient matrix set are fused to obtain a fused high-frequency coefficient matrix set, and based on the low-frequency weight matrix, the first low-frequency coefficient matrix and the second low-frequency coefficient matrix are fused to obtain a fused low-frequency coefficient matrix; Among them, the number of high-frequency coefficient matrices in the first high-frequency coefficient matrix set, the second high-frequency coefficient matrix set and the fused high-frequency coefficient matrix set, and the number of high-frequency weight matrices in the high-frequency weight matrix set are the sum of the number of directions at each level corresponding to the MST decomposition.

[0034] It can be understood that since the MST decomposition usually includes multiple levels, and each level usually involves multiple directions, assuming that the number of levels of the MST decomposition is L, and the number of directions of level l is k (l), then after the image is decomposed by MST, a low-frequency coefficient matrix and k (1) + k (2) + ... k (L) high-frequency coefficient matrices will be obtained. Based on this, the number of high-frequency coefficient matrices in the first high-frequency coefficient matrix set, the second high-frequency coefficient matrix set, and the fused high-frequency coefficient matrix set are all the sum of the number of directions of each level corresponding to the MST decomposition. It is worth noting that since the target MR scan image and the target PET scan grayscale image are both converted into frequency domain information by MST decomposition, in order to improve the accuracy of subsequent fusion, the embodiment of the present invention also performs MST decomposition on the weight map to obtain a high-frequency weight matrix set and a low-frequency weight matrix (that is, converting the weight map to the frequency domain). Based on this, it can be understood that the number of high-frequency weight matrices in the high-frequency weight matrix set is also the sum of the number of directions of each level corresponding to the MST decomposition. Based on the foregoing, it can also be understood that each element value in any of the foregoing matrices corresponds to a pixel value in the target MR scan image and the target PET scan grayscale image.

[0035] Based on the above embodiment, the high-frequency weight matrix corresponding to the layer number l and direction k in the high-frequency weight matrix concentration is recorded as ; The low-frequency weight matrix is ​​recorded as The first high-frequency coefficient matrix has a concentration of layer l and a high-frequency coefficient matrix corresponding to direction k, which is recorded as The second high-frequency coefficient matrix has a concentration of layer l and a high-frequency coefficient matrix corresponding to direction k, which is recorded as ; The first low-frequency coefficient matrix is ​​recorded as ; The second low frequency coefficient matrix ; The high-frequency coefficient matrix corresponding to the layer number l and direction k of the fused high-frequency coefficient matrix is ​​recorded as ; The fused low-frequency coefficient matrix is ​​recorded as .

[0036] For the fusion rule corresponding to the high-frequency coefficient matrix corresponding to the layer number l and direction k in the fusion high-frequency coefficient matrix concentration, the following formula is used to represent it: ; Among them, i is the matrix row number and j is the matrix column number.

[0037] For the fusion rule corresponding to the fusion low-frequency coefficient matrix, the activity level metric WLE is introduced. The calculation formula of WLE is: ; Where S is the source image, S∈{A,B}; W is a (2r+1)×(2r+1) weighting matrix with radius r, where r determines the size of the neighborhood considered when calculating the activity level; the value of any element in W is 2 2r-d , d is the distance from the four neighbors of the element to the center position; Represents the pixel value of the source image S at position (i+m,j+n), which is the neighborhood pixel value required to calculate the activity level; m and n are cumulative variables used to traverse each element in the weighting matrix, and m and n are iterated from -r to r to cover the entire (2r+1)×(2r+1) neighborhood. On this basis, the fusion rule corresponding to the fused low-frequency coefficient matrix is ​​characterized by the following formula: ; Through the above fusion rules, the embodiment of the present invention can avoid the problem of loss of spectral information, details and contrast in the fused image to the greatest extent, thereby improving the accuracy of the fused image.

[0038] It can be understood that the target fused image is a grayscale image. On this basis, the method further includes: Based on the hue component and the saturation component in the IHS color space of the target fused image and the target PET scan image, an HIS inverse transform is performed to obtain a target fused RGB image.

[0039] Based on this, the embodiment of the present application can obtain the target fused RGB image, thereby further ensuring the efficiency and accuracy of the subsequent diagnosis process.

[0040] The solution provided by the present invention obtains an aligned image pair to be fused by preprocessing the original MR scan image and the original PET scan image in the image pair to be fused; the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images; the target PET scan image is converted into a target PET scan grayscale image, and the target MR scan image and the target PET scan grayscale image are input into a trained twin neural network to obtain a weight map for matching the image pair to be fused; the target MR scan image and the target PET scan grayscale image are matched. The target PET scan grayscale image is subjected to MST decomposition to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; the low-frequency coefficient matrix and the high-frequency coefficient matrix are fused respectively based on the weight map, and the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set are reconstructed by MST to obtain the target fused image corresponding to the image pair to be fused, and the weight map of the image pair to be fused by the twin neural network can be used, and the high-frequency coefficients and the low-frequency coefficients are fused based on the weight map, so as to avoid the problem of loss of spectral information, details and contrast in the fused image, and improve the accuracy of the fused image.

[0041] Figure 5 A structural block diagram of a data processing device based on multimodal data knowledge alignment and fusion is provided for an embodiment of the present invention, such as Figure 5 As shown, the system includes: The alignment module 201 is used to pre-process the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images; A weight map acquisition module 202 is used to convert the target PET scan image into a target PET scan grayscale image, and input the target MR scan image and the target PET scan grayscale image into a trained twin neural network to obtain a weight map for matching the image pair to be fused; An MST decomposition module 203 is used to perform MST decomposition on the target MR scan image and the target PET scan grayscale image to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; The fusion module 204 is used to fuse the low-frequency coefficient matrix and the high-frequency coefficient matrix respectively based on the weight map, and perform MST reconstruction on the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set to obtain a target fused image corresponding to the pair of images to be fused.

[0042] The solution provided by the present invention pre-processes the original MR scan image and the original PET scan image in the image pair to be fused by an alignment module 201 to obtain an aligned image pair to be fused; the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images; the weight map acquisition module 202 converts the target PET scan image into a target PET scan grayscale image, and inputs the target MR scan image and the target PET scan grayscale image into a trained twin neural network to obtain a weight map for matching the image pair to be fused; the MST decomposition module 203 performs weight map acquisition on the target PET scan image and the target PET scan grayscale image; The target MR scan image and the target PET scan grayscale image are subjected to MST decomposition to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; the fusion module 204 fuses the low-frequency coefficient matrix and the high-frequency coefficient matrix respectively based on the weight map, and performs MST reconstruction on the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set to obtain the target fused image corresponding to the pair of images to be fused, and can utilize the weight map of the twin neural network to match the pair of images to be fused, and fuse the high-frequency coefficients and the low-frequency coefficients based on the weight map, so as to avoid the problem of loss of spectral information, details and contrast in the fused image, and improve the accuracy of the fused image.

[0043] In an optional embodiment of the present application, the twin neural network includes: a first input branch and a second input branch with the same structure, a fully connected layer and a softmax layer; The first input branch and the second input branch both include a first convolutional layer, a second convolutional layer, a maximum pooling layer and a third convolutional layer connected in sequence; The third convolutional layers of the first input branch and the second input branch are both connected to a fully connected layer, and the fully connected layer is connected to the softmax layer; The size of the input patch images of the first input branch and the second input branch is 16×16 pixels.

[0044] In an optional embodiment of the present application, the twin neural network is obtained after training based on a medical image sample set and predetermined sample labels; The medical image sample set includes a plurality of medical image sample pairs, and any target medical image sample pair includes a clear patch image sample and a fuzzy patch image sample from the same region of the same standard medical image; in the target medical image sample pair, when the clear patch image sample is used as the input patch image of the first input branch, and the fuzzy patch image sample is used as the input patch image of the second input branch, the sample label of the target medical image sample pair is 1; otherwise, the sample label of the target medical image sample pair is 0; The standard medical image is a standard MR scan image or a standard PET scan grayscale image.

[0045] In an optional embodiment of the present application, preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain the aligned image pair to be fused specifically includes: Extracting feature points of the original MR scan image and the original PET scan image respectively, and determining associated descriptors of the feature points; Determine matching feature points of the original MR scan image and the original PET scan image based on the associated descriptors of the feature points of the original MR scan image and the original PET scan image; Determine a transformation model based on matching feature points of the original MR scan image and the original PET scan image; The original MR scan image or the original PET scan image is transformed based on the transformation model to obtain an aligned pair of images to be fused.

[0046] In an optional embodiment of the present application, converting the target PET scan image into a target PET scan grayscale image specifically includes: The target PET scan image is converted from the RGB color space to the IHS color space, and the intensity component of the target PET scan image in the IHS color space is extracted as the target PET scan grayscale image.

[0047] In an optional embodiment of the present application, the fusing of the low-frequency coefficient matrix and the high-frequency coefficient matrix based on the weight map specifically includes: Performing MST decomposition on the weight graph to obtain a high-frequency weight matrix set and a low-frequency weight matrix; Based on the high-frequency weight matrix set, the first high-frequency coefficient matrix set and the second high-frequency coefficient matrix set are fused to obtain a fused high-frequency coefficient matrix set; based on the low-frequency weight matrix, the first low-frequency coefficient matrix and the second low-frequency coefficient matrix are fused to obtain a fused low-frequency coefficient matrix; Among them, the number of high-frequency coefficient matrices in the first high-frequency coefficient matrix set, the second high-frequency coefficient matrix set and the fused high-frequency coefficient matrix set, and the number of high-frequency weight matrices in the high-frequency weight matrix set are the sum of the number of directions at each level corresponding to the MST decomposition.

[0048] In an optional embodiment of the present application, the target fused image is a grayscale image, and the fusion module 204 is further configured to perform the following operations: Based on the hue component and the saturation component in the IHS color space of the target fused image and the target PET scan image, an HIS inverse transform is performed to obtain a target fused RGB image.

[0049] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 301, a communications interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communications interface 302 and the memory 303 communicate with each other via the communication bus 304. The processor 301 may call the logic instructions in the memory 303 to execute a data processing method based on multimodal data knowledge alignment and fusion, the method comprising: preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; the aligned image pair to be fused comprises: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, the original PET scan image and the target PET scan image are both RGB images; converting the target PET scan image into a target PET scan grayscale image, and converting the target The MR scan image and the target PET scan grayscale image are input into the trained twin neural network to obtain a weight map for matching the image pair to be fused; the target MR scan image and the target PET scan grayscale image are subjected to MST decomposition to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; the low-frequency coefficient matrix and the high-frequency coefficient matrix are fused respectively based on the weight map, and the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set are reconstructed by MST to obtain the target fused image corresponding to the image pair to be fused.

[0050] In addition, the logic instructions in the above-mentioned memory 303 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0051] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the data processing method based on multimodal data knowledge alignment and fusion provided by the above methods, the method includes: preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; the aligned image pair to be fused includes: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images; The target PET scan image is converted into a target PET scan grayscale image, and the target MR scan image and the target PET scan grayscale image are input into a trained twin neural network to obtain a weight map for matching the image pair to be fused; the target MR scan image and the target PET scan grayscale image are subjected to MST decomposition to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; the low-frequency coefficient matrix and the high-frequency coefficient matrix are respectively fused based on the weight map, and the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set are subjected to MST reconstruction to obtain a target fused image corresponding to the image pair to be fused.

[0052] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the data processing method based on multimodal data knowledge alignment and fusion provided by the above-mentioned methods, the method comprising: preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; the aligned image pair to be fused comprises: a target MR scan image and a target PET scan image, the original MR scan image and the target MR scan image are both grayscale images, and the original PET scan image and the target PET scan image are both RGB images; converting the target PET scan image into A target PET scan grayscale image is obtained, and the target MR scan image and the target PET scan grayscale image are input into a trained twin neural network to obtain a weight map for matching the image pair to be fused; the target MR scan image and the target PET scan grayscale image are subjected to MST decomposition to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; based on the weight map, the low-frequency coefficient matrix and the high-frequency coefficient matrix are respectively fused, and the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set are subjected to MST reconstruction to obtain a target fused image corresponding to the image pair to be fused.

[0053] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0054] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the data processing method based on multimodal data knowledge alignment and fusion described in each embodiment or some parts of the embodiment.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on multimodal data knowledge alignment and fusion, characterized in that: include: Preprocessing the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; The aligned image pair to be fused includes: a target MR scan image and a target PET scan image, wherein both the original MR scan image and the target MR scan image are grayscale images, and both the original PET scan image and the target PET scan image are RGB images; Converting the target PET scan image into a target PET scan grayscale image, and inputting the target MR scan image and the target PET scan grayscale image into a trained twin neural network to obtain a weight map for matching the image pair to be fused; Performing MST decomposition on the target MR scan image and the target PET scan grayscale image to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; Based on the weight map, the low-frequency coefficient matrix and the high-frequency coefficient matrix are fused respectively, and the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set are reconstructed by MST to obtain a target fused image corresponding to the pair of images to be fused.

2. The data processing method based on multimodal data knowledge alignment and fusion according to claim 1 is characterized in that: The twin neural network comprises: a first input branch and a second input branch with the same structure, a fully connected layer and a softmax layer; The first input branch and the second input branch both include a first convolutional layer, a second convolutional layer, a maximum pooling layer and a third convolutional layer connected in sequence; The third convolutional layers of the first input branch and the second input branch are both connected to a fully connected layer, and the fully connected layer is connected to the softmax layer; The size of the input patch images of the first input branch and the second input branch is 16×16 pixels.

3. The data processing method based on multimodal data knowledge alignment and fusion according to claim 2 is characterized in that: The twin neural network is obtained after training based on a medical image sample set and predetermined sample labels; The medical image sample set includes a plurality of medical image sample pairs, and any target medical image sample pair includes a clear patch image sample and a fuzzy patch image sample from the same region of the same standard medical image; in the target medical image sample pair, when the clear patch image sample is used as the input patch image of the first input branch, and the fuzzy patch image sample is used as the input patch image of the second input branch, the sample label of the target medical image sample pair is 1; otherwise, the sample label of the target medical image sample pair is 0; The standard medical image is a standard MR scan image or a standard PET scan grayscale image.

4. The data processing method based on multimodal data knowledge alignment and fusion according to claim 1 is characterized in that: The preprocessing of the original MR scan image and the original PET scan image in the image pair to be fused to obtain the aligned image pair to be fused specifically includes: Extracting feature points of the original MR scan image and the original PET scan image respectively, and determining associated descriptors of the feature points; Determine matching feature points of the original MR scan image and the original PET scan image based on the associated descriptors of the feature points of the original MR scan image and the original PET scan image; Determine a transformation model based on matching feature points of the original MR scan image and the original PET scan image; The original MR scan image or the original PET scan image is transformed based on the transformation model to obtain an aligned pair of images to be fused.

5. The data processing method based on multimodal data knowledge alignment and fusion according to claim 4 is characterized in that: The step of converting the target PET scan image into a target PET scan grayscale image specifically includes: The target PET scan image is converted from the RGB color space to the IHS color space, and the intensity component of the target PET scan image in the IHS color space is extracted as the target PET scan grayscale image.

6. The data processing method based on multimodal data knowledge alignment and fusion according to claim 5 is characterized in that: The fusing of the low-frequency coefficient matrix and the high-frequency coefficient matrix based on the weight map specifically includes: Performing MST decomposition on the weight graph to obtain a high-frequency weight matrix set and a low-frequency weight matrix; Based on the high-frequency weight matrix set, the first high-frequency coefficient matrix set and the second high-frequency coefficient matrix set are fused to obtain a fused high-frequency coefficient matrix set; based on the low-frequency weight matrix, the first low-frequency coefficient matrix and the second low-frequency coefficient matrix are fused to obtain a fused low-frequency coefficient matrix; Among them, the number of high-frequency coefficient matrices in the first high-frequency coefficient matrix set, the second high-frequency coefficient matrix set and the fused high-frequency coefficient matrix set, and the number of high-frequency weight matrices in the high-frequency weight matrix set are the sum of the number of directions at each level corresponding to the MST decomposition.

7. The data processing method based on multimodal data knowledge alignment and fusion according to claim 6 is characterized in that: The target fused image is a grayscale image, and the method further includes: Based on the hue component and the saturation component in the IHS color space of the target fused image and the target PET scan image, an HIS inverse transform is performed to obtain a target fused RGB image.

8. A data processing device based on multimodal data knowledge alignment and fusion, characterized in that: include: An alignment module is used to pre-process the original MR scan image and the original PET scan image in the image pair to be fused to obtain an aligned image pair to be fused; The aligned image pair to be fused includes: a target MR scan image and a target PET scan image, wherein both the original MR scan image and the target MR scan image are grayscale images, and both the original PET scan image and the target PET scan image are RGB images; A weight map acquisition module, used for converting the target PET scan image into a target PET scan grayscale image, and inputting the target MR scan image and the target PET scan grayscale image into a trained twin neural network to obtain a weight map for matching the image pair to be fused; An MST decomposition module is used to perform MST decomposition on the target MR scan image and the target PET scan grayscale image to obtain a first low-frequency coefficient matrix and a first high-frequency coefficient matrix set corresponding to the target MR scan image and a second low-frequency coefficient matrix and a second high-frequency coefficient matrix set corresponding to the target PET scan grayscale image; A fusion module is used to fuse the low-frequency coefficient matrix and the high-frequency coefficient matrix respectively based on the weight map, and perform MST reconstruction on the fused low-frequency coefficient matrix and the fused high-frequency coefficient matrix set to obtain a target fused image corresponding to the pair of images to be fused.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data processing method based on multimodal data knowledge alignment and fusion as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method based on multimodal data knowledge alignment and fusion as described in any one of claims 1 to 7 is implemented.

Images