Medical Image Fusion Method and Device Based on Edge Texture Enhancement

Abstract

The present invention provides a medical image fusion method and device based on edge texture enhancement. The method includes: obtaining an edge texture image and a tissue image of a computed tomography (CT) image, and an edge texture image and a tissue image of a magnetic resonance imaging (MRI) image; inputting the edge texture image features of the CT image and the edge texture image features of the MRI image into an image construction network to obtain respective edge texture construction images, inputting the edge texture construction images and the actual edge texture images into an image recognition network for recognition, and if the objective function and activation function of the image recognition network meet corresponding preset conditions, then performing an inverse wavelet transform to finally fuse the edge texture construction image and the fused tissue image to obtain a final fused image. The present invention can obtain a fused medical image with clearer edge texture.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of medical image signal processing, and in particular, to a medical image fusion method and device based on edge texture enhancement. Background Art

[0002] At present, fusing computed tomography (CT) images and magnetic resonance imaging (MRI) images is the most common mode in modern medical image signal processing. In the field of medical imaging, CT images have a large tissue density and high resolution, and can clearly show human bones; while MRI images have better soft tissue density resolution and can more clearly display the soft tissue structure information of the human body. How to make good use of the advantages of both types of images to help doctors make quick diagnoses is one of the current research hotspots in the medical field. Although there are also methods for fusing the two types of image signals in related technologies, they are all simple image feature extraction followed by fusion, and the clarity of the fused edge texture is poor, which has an adverse impact on the correct determination of the actual situation of human tissues. Therefore, developing a medical image fusion method and device based on edge texture enhancement to effectively overcome the above defects in related technologies has become an urgent technical problem in the industry. Summary of the Invention

[0003] In view of the above problems existing in the prior art, the embodiments of the present invention provide a medical image fusion method and device based on edge texture enhancement.

[0004] In a first aspect, an embodiment of the present invention provides a medical image fusion method based on edge texture enhancement, including: decomposing a computed tomography image and a magnetic resonance imaging image respectively by using wavelet transform to obtain an edge texture image and a tissue image of the computed tomography image, and an edge texture image and a tissue image of the magnetic resonance imaging image; fusing the tissue image of the computed tomography image and the tissue image of the magnetic resonance imaging image to obtain a fused tissue image; respectively extracting features of the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance imaging image by using a feature extraction network, and inputting the extracted features into an image construction network to obtain an edge texture construction image of the computed tomography image and an edge texture construction image of the magnetic resonance imaging image; inputting the edge texture construction image of the computed tomography image, the edge texture construction image of the magnetic resonance imaging image, the edge texture image of the computed tomography image, and the edge texture image of the magnetic resonance imaging image into an image recognition network for recognition; if the objective function and activation function of the image recognition network meet corresponding preset conditions, then perform final fusion on the edge texture construction image and the fused tissue image by using inverse wavelet transform to obtain a final fused image.

[0005] Based on the content of the above method embodiments, the medical image fusion method based on edge texture enhancement provided in the embodiments of the present invention, the fusion of the tissue image of the computed tomography image and the tissue image of the magnetic resonance image includes: inputting the tissue image of the computed tomography image and the tissue image of the magnetic resonance image into a fusion convolutional neural network for fusion, and the fusion convolutional neural network includes: the output end of the first convolutional layer 7*7 is connected to the input end of the first max pooling layer; the output end of the first max pooling layer is connected to the input end of the second convolutional layer 7*7; the output end of the second convolutional layer 7*7 is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the first convolutional layer 5*5.

[0006] Based on the content of the above method embodiments, in the medical image fusion method based on edge texture enhancement provided in the embodiments of the present invention, the feature extraction network includes: the output end of the ReLU activation function + convolutional layer 5*5 is connected to the input end of the second max pooling layer and the input end of the fourth ReLU activation function + convolutional layer 3*3; the output end of the second max pooling layer is connected to the input end of the first batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the first batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the first transition layer and the input end of the third ReLU activation function + convolutional layer 3*3; the output end of the first transition layer is connected to the input end of the second batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the second batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the second transition layer and the input end of the second ReLU activation function + convolutional layer 3*3; the output end of the second transition layer is connected to the input end of the third batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the third batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the third transition layer; the output end of the third transition layer is connected to the input end of the fourth batch normalization + ReLU activation function + convolutional layer 4*4 and the input end of the second upsampling layer; the output end of the fourth batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the first upsampling layer; the output end of the first upsampling layer is connected to the input end of the first ReLU activation function + convolutional layer 3*3; the output end of the first ReLU activation function + convolutional layer 3*3 is connected to the input end of the second upsampling layer; the output end of the second upsampling layer is connected to the input end of the second ReLU activation function + convolutional layer 3*3; the output end of the second ReLU activation function + convolutional layer 3*3 is connected to the input end of the third upsampling layer; the output end of the third upsampling layer is connected to the input end of the third ReLU activation function + convolutional layer 3*3; the output end of the third ReLU activation function + convolutional layer 3*3 is connected to the input end of the fourth upsampling layer; the output end of the fourth upsampling layer is connected to the input end of the fourth ReLU activation function + convolutional layer 3*3; the output end of the fourth ReLU activation function + convolutional layer 3*3 is connected to the input end of the transposed convolutional layer.

[0007] Based on the content of the above method embodiments, in the medical image fusion method based on edge texture enhancement provided in the embodiments of the present invention, the image construction network includes: the output end of the first convolutional layer of 3×3 is connected to the input end of the first batch normalization layer; the output end of the first batch normalization layer is connected to the input end of the first SELU activation function; the output end of the first SELU activation function is connected to the input end of the second convolutional layer of 3×3; the output end of the second convolutional layer of 3×3 is connected to the input end of the second batch normalization layer; the output end of the second batch normalization layer is connected to the input end of the second SELU activation function; the output end of the second SELU activation function is connected to the input end of the third max pooling layer; the output end of the third max pooling layer is connected to the input end of the third convolutional layer of 3×3; the output end of the third convolutional layer of 3×3 is connected to the input end of the third batch normalization layer; the output end of the third batch normalization layer is connected to the input end of the third SELU activation function; the output end of the third SELU activation function is connected to the input end of the fourth convolutional layer of 3×3.

[0008] Based on the content of the above method embodiments, in the medical image fusion method based on edge texture enhancement provided in the embodiments of the present invention, the image recognition network includes: the output end of the fifth convolutional layer of 3×3 is connected to the input end of the fourth batch normalization layer; the output end of the fourth batch normalization layer is connected to the input end of the fourth SELU activation function; the output end of the fourth SELU activation function is connected to the input end of the fifth upsampling layer; the output end of the fifth upsampling layer is connected to the input end of the sixth convolutional layer of 3×3; the output end of the sixth convolutional layer of 3×3 is connected to the input end of the fifth batch normalization layer; the output end of the fifth batch normalization layer is connected to the input end of the fifth SELU activation function; the output end of the fifth SELU activation function is connected to the input end of the seventh convolutional layer of 3×3; the output end of the seventh convolutional layer of 3×3 is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the tanh activation function layer; the output end of the tanh activation function layer is connected to the input end of the objective function layer.

[0009] Based on the content of the above method embodiments, the medical image fusion method based on edge texture enhancement provided in the embodiments of the present invention, if the objective function and activation function of the image recognition network meet the corresponding preset conditions, includes: first input the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image into the image recognition network for recognition. By adjusting the image recognition network, ensure that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image conforms to the objective function. Then input the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image into the adjusted image recognition network for recognition. If the output value of the activation function after recognition is less than a predetermined threshold, adjust the image construction network to reconstruct the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image until the output value of the activation function is greater than the predetermined threshold.

[0010] Based on the content of the above method embodiments, the medical image fusion method based on edge texture enhancement provided in the embodiments of the present invention, ensuring that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image conforms to the objective function, includes:

[0011]

[0012] Among them, is the objective function; max is the symbol for taking the maximum value; is the i-th element of the final output vector of the image recognition network; tanh is the symbol of the hyperbolic tangent function; N is the total number of elements of the final output vector of the image recognition network.

[0013] In a second aspect, an embodiment of the present invention provides a medical image fusion device based on edge texture enhancement, including: a first main module for decomposing a computed tomography (CT) image and a magnetic resonance imaging (MRI) image respectively by using wavelet transform to obtain an edge texture image and a tissue image of the CT image, and an edge texture image and a tissue image of the MRI image; a second main module for fusing the tissue image of the CT image and the tissue image of the MRI image to obtain a fused tissue image; a third main module for respectively extracting features from the edge texture image of the CT image and the edge texture image of the MRI image by using a feature extraction network, and inputting the extracted features into an image construction network to obtain an edge texture construction image of the CT image and an edge texture construction image of the MRI image; a fourth main module for inputting the edge texture construction image of the CT image, the edge texture construction image of the MRI image, the edge texture image of the CT image and the edge texture image of the MRI image into an image recognition network for recognition; a fifth main module for, if the objective function and the activation function of the image recognition network meet corresponding preset conditions, finally fusing the edge texture construction image and the fused tissue image by using inverse wavelet transform to obtain a final fused image.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0015] at least one processor, at least one memory and a communication interface; wherein,

[0016] the processor, the memory and the communication interface communicate with each other;

[0017] the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the medical image fusion method based on edge texture enhancement provided by any one of the various implementation manners in the first aspect.

[0018] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, and the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the medical image fusion method based on edge texture enhancement provided by any one of the various implementation manners in the first aspect.

[0019] The medical image fusion method and device based on edge texture enhancement provided by the embodiments of the present invention fuse the tissue images of computed tomography (CT) images obtained by wavelet transform decomposition with the tissue images of magnetic resonance imaging (MRI) images to obtain fused tissue images, input the edge texture image features of the extracted CT images and the edge texture image features of the MRI images into an image construction network to obtain their respective edge texture construction images, input the edge texture construction images and the actual edge texture images into an image recognition network for recognition. If the objective function and activation function of the image recognition network meet the corresponding preset conditions, the edge texture construction images and the fused tissue images are fused by inverse wavelet transform to obtain the final fused image, and a fused medical image with clearer edge texture can be obtained, providing a more solid technical support for medical clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a schematic flowchart of the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention;

[0022] Figure 2 It is a schematic structural diagram of the medical image fusion device based on edge texture enhancement provided by the embodiments of the present invention;

[0023] Figure 3 It is a schematic physical structure diagram of the electronic device provided by the embodiments of the present invention;

[0024] Figure 4 It is a schematic overall network structure diagram for implementing the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention;

[0025] Figure 5 It is a schematic structure diagram of the fusion convolutional neural network provided by the embodiments of the present invention;

[0026] Figure 6 It is a schematic structure diagram of the feature extraction network provided by the embodiments of the present invention;

[0027] Figure 7 It is a schematic structure diagram of the image construction network provided by the embodiments of the present invention;

[0028] Figure 8 It is a schematic structure diagram of the image recognition network provided by the embodiments of the present invention;

[0029] Figure 9(a) is a schematic diagram of the original computed tomography image effect provided by the embodiment of the present invention;

[0030] Figure 9(b) is a schematic diagram of the original magnetic resonance imaging effect provided by the embodiment of the present invention;

[0031] Figure 9(c) is a schematic diagram of the medical image fusion effect provided by the related art;

[0032] Figure 9(d) is a schematic diagram of the medical image fusion effect provided by the embodiment of the present invention. Detailed implementation manners

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. In addition, the technical features in the various embodiments or individual embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the order of steps and / or the structural composition mode, but must be based on the fact that those of ordinary skill in the art can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention. If there are step numbers in the following embodiments, they are only set for the convenience of explanation and illustration, and no limitation is imposed on the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0034] The embodiment of the present invention provides a medical image fusion method based on edge texture enhancement. Refer to Figure 1, the method includes: decomposing the computed tomography (CT) image and the magnetic resonance imaging (MRI) image respectively by using wavelet transform to obtain the edge texture image and the tissue image of the CT image, and the edge texture image and the tissue image of the MRI image; fusing the tissue image of the CT image and the tissue image of the MRI image to obtain a fused tissue image; using a feature extraction network to extract features from the edge texture image of the CT image and the edge texture image of the MRI image respectively, and inputting the extracted features into an image construction network to obtain the edge texture construction image of the CT image and the edge texture construction image of the MRI image; inputting the edge texture construction image of the CT image, the edge texture construction image of the MRI image, the edge texture image of the CT image and the edge texture image of the MRI image into an image recognition network for recognition; if the objective function and the activation function of the image recognition network meet the corresponding preset conditions, then using inverse wavelet transform to finally fuse the edge texture construction image and the fused tissue image to obtain a final fused image.

[0035] Specifically, the network structure implemented by the medical image fusion method based on edge texture enhancement can be referred to Figure 4 . Decompose the CT image by using the first wavelet transform (the same as the second wavelet transform, with different names only for distinction) to obtain the edge texture image of the CT image and the tissue image of the CT image (such as the brain tissue), and similarly obtain the edge texture image of the MRI image and the tissue image of the MRI image. Input the tissue image of the CT image and the tissue image of the MRI image into a fusion convolutional neural network (the specific structure can be referred to Figure 5 ), and after fusion, obtain a fused tissue image; input the edge texture image of the CT image and the edge texture image of the MRI image into an image recognition network, and train the image recognition network until the objective function of the image recognition network is satisfied, then the training of the image recognition network is completed. Then, input the edge texture image of the CT image and the edge texture image of the MRI image into a feature extraction network, input the extracted features into an image construction network to obtain the edge texture construction image of the CT image and the edge texture construction image of the MRI image, and input the two into the trained image recognition network. If the output of the activation function of the image recognition network meets a predetermined threshold, it means that the edge texture image constructed by the image construction network already meets the requirements. At this time, Figure 4 the image recognition network in it can be removed to obtain the final network structure for implementing the medical image fusion method based on edge texture enhancement; use inverse wavelet transform to finally fuse the edge texture construction image and the fused tissue image to obtain a final fused image.

[0036] Based on the content of the above method embodiments, as an alternative embodiment, in the method for medical image fusion based on edge texture enhancement provided by the embodiments of the present invention, the fusion of the tissue image of the computed tomography image and the tissue image of the magnetic resonance image includes: inputting the tissue image of the computed tomography image and the tissue image of the magnetic resonance image into a fusion convolutional neural network for fusion. The fusion convolutional neural network includes: the output end of the first convolutional layer with a kernel size of 7×7 is connected to the input end of the first max pooling layer; the output end of the first max pooling layer is connected to the input end of the second convolutional layer with a kernel size of 7×7; the output end of the second convolutional layer with a kernel size of 7×7 is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the first convolutional layer with a kernel size of 5×5. It should be noted that all descriptions like "first" added in the first convolutional layer with a kernel size of 7×7 in this specification are only for differentiating each convolutional layer in name. The basic structures of each convolutional layer are similar, and the only difference lies in the different convolutional kernels (i.e., specific convolutional kernel matrices). The meanings expressed by the remaining similar descriptions are the same and will not be elaborated further.

[0037] Based on the content of the above method embodiments, as an alternative embodiment, in the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention, the feature extraction network includes: the output end of the ReLU activation function + convolutional layer 5*5 is connected to the input end of the second max pooling layer and the input end of the fourth ReLU activation function + convolutional layer 3*3; the output end of the second max pooling layer is connected to the input end of the first batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the first batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the first transition layer and the input end of the third ReLU activation function + convolutional layer 3*3; the output end of the first transition layer is connected to the input end of the second batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the second batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the second transition layer and the input end of the second ReLU activation function + convolutional layer 3*3; the output end of the second transition layer is connected to the input end of the third batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the third batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the third transition layer; the output end of the third transition layer is connected to the input end of the fourth batch normalization + ReLU activation function + convolutional layer 4*4 and the input end of the second upsampling layer; the output end of the fourth batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the first upsampling layer; the output end of the first upsampling layer is connected to the input end of the first ReLU activation function + convolutional layer 3*3; the output end of the first ReLU activation function + convolutional layer 3*3 is connected to the input end of the second upsampling layer; the output end of the second upsampling layer is connected to the input end of the second ReLU activation function + convolutional layer 3*3; the output end of the second ReLU activation function + convolutional layer 3*3 is connected to the input end of the third upsampling layer; the output end of the third upsampling layer is connected to the input end of the third ReLU activation function + convolutional layer 3*3; the output end of the third ReLU activation function + convolutional layer 3*3 is connected to the input end of the fourth upsampling layer; the output end of the fourth upsampling layer is connected to the input end of the fourth ReLU activation function + convolutional layer 3*3; the output end of the fourth ReLU activation function + convolutional layer 3*3 is connected to the input end of the transposed convolutional layer. It should be noted that the feature extraction network constructed through this embodiment can obtain and transfer richer features during training, and has a better gradient enhancement effect, and can locate the global optimal solution at the same time.

[0038] Based on the content of the above method embodiments, as an alternative embodiment, in the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention, the image construction network includes: the output end of the first convolutional layer 3*3 is connected to the input end of the first batch normalization layer; the output end of the first batch normalization layer is connected to the input end of the first SELU activation function; the output end of the first SELU activation function is connected to the input end of the second convolutional layer 3*3; the output end of the second convolutional layer 3*3 is connected to the input end of the second batch normalization layer; the output end of the second batch normalization layer is connected to the input end of the second SELU activation function; the output end of the second SELU activation function is connected to the input end of the third max pooling layer; the output end of the third max pooling layer is connected to the input end of the third convolutional layer 3*3; the output end of the third convolutional layer 3*3 is connected to the input end of the third batch normalization layer; the output end of the third batch normalization layer is connected to the input end of the third SELU activation function; the output end of the third SELU activation function is connected to the input end of the fourth convolutional layer 3*3.

[0039] Specifically, after adopting the design structure of this image construction network, a good balance is achieved between the network capacity and the feature extraction ability, which can further alleviate the problem of gradient disappearance when the image resolution is not very high and the network width and depth are not very large, while strengthening the utilization and effective transmission of features.

[0040] Based on the content of the above method embodiments, as an alternative embodiment, in the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention, the image recognition network includes: the output end of the fifth convolutional layer 3*3 is connected to the input end of the fourth batch normalization layer (e.g., using the softmax function); the output end of the fourth batch normalization layer is connected to the input end of the fourth SELU activation function; the output end of the fourth SELU activation function is connected to the input end of the fifth upsampling layer; the output end of the fifth upsampling layer is connected to the input end of the sixth convolutional layer 3*3; the output end of the sixth convolutional layer 3*3 is connected to the input end of the fifth batch normalization layer; the output end of the fifth batch normalization layer is connected to the input end of the fifth SELU activation function; the output end of the fifth SELU activation function is connected to the input end of the seventh convolutional layer 3*3; the output end of the seventh convolutional layer 3*3 is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the tanh activation function layer; the output end of the tanh activation function layer is connected to the input end of the objective function layer. It should be noted that the objective function layer is a layer retained during the continuous adjustment of the image recognition network. When the adjustment of the image recognition network is completed (i.e., the training of the image recognition network is completed), the function of this layer has been completed, and subsequent determination of the edge texture construction image can be completed only by detecting the output of the tanh activation function.

[0041] Based on the content of the above method embodiments, as an alternative embodiment, in the medical image fusion method based on edge texture enhancement provided in the embodiments of the present invention, if the objective function and activation function of the image recognition network meet the corresponding preset conditions, it includes: first inputting the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image into the image recognition network for recognition, and by adjusting the image recognition network, ensuring that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image conforms to the objective function, and then inputting the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image into the adjusted image recognition network for recognition. If the output value of the activation function after recognition is less than a predetermined threshold, then adjust the image construction network to reconstruct the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image until the output value of the activation function is greater than the predetermined threshold. In another embodiment, the predetermined threshold may be 90% or 80% of the objective function of the corresponding element of the output vector.

[0042] Based on the content of the above method embodiments, as an alternative embodiment, in the medical image fusion method based on edge texture enhancement provided in the embodiments of the present invention, the ensuring that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image conforms to the objective function includes:

[0043]

[0044] Wherein, is the objective function; max is the symbol for taking the maximum value; is the i-th element of the final output vector of the image recognition network; tanh is the symbol of the hyperbolic tangent function; N is the total number of elements of the final output vector of the image recognition network.

[0045] Specifically, after determining the corresponding objective function value of each element (bit) of the output vector, the predetermined threshold for determining whether each element of the output vector of the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image is qualified can be determined as or .

[0046] The fusion effect of the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention can be seen in FIGS. 9(a) to 9(d). By comparing with the original computed tomography (CT) image shown in FIG. 9(a) and the original magnetic resonance imaging (MRI) image shown in FIG. 9(b) (the white circle part), it can be seen that the edge texture of the image fused by the related technology shown in FIG. 9(c) is relatively blurred (the white circle part) and is basically difficult to distinguish. By comparing with the image fusion effect of the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention shown in FIG. 9(d) (the white circle part), whether it is the bone edge feature in the CT image, the clarity of the contour and structure in the MRI image, or the contrast and minute gradient changes inside the texture, the fusion effect of the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention is clearer. Specifically, the edge texture details of the fused image are significantly clearer, and the edge texture of the image fusion result is more consistent with that of the MRI image in FIG. 9(b), which proves that the medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention has obvious advantages.

[0047] The medical image fusion method based on edge texture enhancement provided by the embodiments of the present invention fuses the tissue image of the CT image obtained by wavelet transform decomposition with the tissue image of the MRI image to obtain a fused tissue image, inputs the extracted edge texture image features of the CT image and the edge texture image features of the MRI image into an image construction network to obtain their respective edge texture construction images, inputs the edge texture construction images and the actual edge texture images into an image recognition network for recognition. If the objective function and activation function of the image recognition network meet the corresponding preset conditions, then the edge texture construction image and the fused tissue image are fused by inverse wavelet transform to obtain a final fused image, and a fused medical image with clearer edge texture can be obtained, providing a more solid technical support for medical clinical diagnosis.

[0048] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with a processor function. Therefore, in engineering practice, the technical solutions and their functions of each embodiment of the present invention can be encapsulated into various modules. Based on this actual situation, on the basis of the above embodiments, the embodiments of the present invention provide a medical image fusion device based on edge texture enhancement, and this device is used to execute the medical image fusion method based on edge texture enhancement in the above method embodiments. See Figure 2, the device includes: a first main module, configured to perform decomposition on a computed tomography (CT) image and a magnetic resonance imaging (MRI) image respectively by using wavelet transform to obtain an edge texture image and a tissue image of the CT image, and an edge texture image and a tissue image of the MRI image; a second main module, configured to fuse the tissue image of the CT image and the tissue image of the MRI image to obtain a fused tissue image; a third main module, configured to perform feature extraction on the edge texture image of the CT image and the edge texture image of the MRI image respectively by using a feature extraction network, and input the extracted features into an image construction network to obtain an edge texture construction image of the CT image and an edge texture construction image of the MRI image; a fourth main module, configured to input the edge texture construction image of the CT image, the edge texture construction image of the MRI image, the edge texture image of the CT image, and the edge texture image of the MRI image into an image recognition network for recognition; a fifth main module, configured to, if the objective function and the activation function of the image recognition network meet corresponding preset conditions, perform final fusion on the edge texture construction image and the fused tissue image by using inverse wavelet transform to obtain a final fused image.

[0049] The medical image fusion device based on edge texture enhancement provided by an embodiment of the present invention uses Figure 2 several modules therein. By fusing the tissue image of the CT image and the tissue image of the MRI image obtained by wavelet transform decomposition to obtain a fused tissue image, inputting the extracted edge texture image features of the CT image and the edge texture image features of the MRI image into an image construction network to obtain respective edge texture construction images, inputting the edge texture construction images and the actual edge texture images into an image recognition network for recognition, and if the objective function and the activation function of the image recognition network meet corresponding preset conditions, performing fusion on the edge texture construction image and the fused tissue image by using inverse wavelet transform to obtain a final fused image, a fused medical image with clearer edge texture can be obtained, providing a more solid technical support for medical clinical diagnosis.

[0050] It should be noted that the device in the device embodiment provided by the present invention can be used not only to implement the method in the above method embodiment, but also to implement the methods in other method embodiments provided by the present invention. The difference lies only in setting corresponding functional modules, and its principle is basically the same as that of the above device embodiment provided by the present invention. As long as those skilled in the art, on the basis of the above device embodiment, refer to the specific technical solutions in other method embodiments, obtain corresponding technical means by combining technical features, and the technical solutions constituted by these technical means, and on the premise of ensuring the practicability of the technical solutions, the device in the above device embodiment can be improved, so as to obtain corresponding device type embodiments for implementing the methods in other method type embodiments. For example:

[0051] Based on the content of the above device embodiment, as an optional embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiment of the present invention further includes: a first sub-module for implementing the fusion of the tissue image of the computed tomography image and the tissue image of the magnetic resonance image, including: inputting the tissue image of the computed tomography image and the tissue image of the magnetic resonance image into a fusion convolutional neural network for fusion, and the fusion convolutional neural network includes: the output end of the first convolutional layer 7*7 is connected to the input end of the first max pooling layer; the output end of the first max pooling layer is connected to the input end of the second convolutional layer 7*7; the output end of the second convolutional layer 7*7 is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the first convolutional layer 5*5.

[0052] Based on the content of the above device embodiments, as an alternative embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiments of the present invention further includes: a second sub-module for implementing the feature extraction network, including: the output end of the ReLU activation function + convolutional layer 5*5 is connected to the input end of the second max pooling layer and the input end of the fourth ReLU activation function + convolutional layer 3*3; the output end of the second max pooling layer is connected to the input end of the first batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the first batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the first transition layer and the input end of the third ReLU activation function + convolutional layer 3*3; the output end of the first transition layer is connected to the input end of the second batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the second batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the second transition layer and the input end of the second ReLU activation function + convolutional layer 3*3; the output end of the second transition layer is connected to the input end of the third batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the third batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the third transition layer; the output end of the third transition layer is connected to the input end of the fourth batch normalization + ReLU activation function + convolutional layer 4*4 and the input end of the second upsampling layer; the output end of the fourth batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the first upsampling layer; the output end of the first upsampling layer is connected to the input end of the first ReLU activation function + convolutional layer 3*3; the output end of the first ReLU activation function + convolutional layer 3*3 is connected to the input end of the second upsampling layer; the output end of the second upsampling layer is connected to the input end of the second ReLU activation function + convolutional layer 3*3; the output end of the second ReLU activation function + convolutional layer 3*3 is connected to the input end of the third upsampling layer; the output end of the third upsampling layer is connected to the input end of the third ReLU activation function + convolutional layer 3*3; the output end of the third ReLU activation function + convolutional layer 3*3 is connected to the input end of the fourth upsampling layer; the output end of the fourth upsampling layer is connected to the input end of the fourth ReLU activation function + convolutional layer 3*3; the output end of the fourth ReLU activation function + convolutional layer 3*3 is connected to the input end of the transposed convolutional layer.

[0053] Based on the content of the above device embodiments, as an alternative embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiments of the present invention further includes: a third sub-module for implementing the image construction network, including: the output end of the first convolutional layer 3*3 is connected to the input end of the first batch normalization layer; the output end of the first batch normalization layer is connected to the input end of the first SELU activation function; the output end of the first SELU activation function is connected to the input end of the second convolutional layer 3*3; the output end of the second convolutional layer 3*3 is connected to the input end of the second batch normalization layer; the output end of the second batch normalization layer is connected to the input end of the second SELU activation function; the output end of the second SELU activation function is connected to the input end of the third max pooling layer; the output end of the third max pooling layer is connected to the input end of the third convolutional layer 3*3; the output end of the third convolutional layer 3*3 is connected to the input end of the third batch normalization layer; the output end of the third batch normalization layer is connected to the input end of the third SELU activation function; the output end of the third SELU activation function is connected to the input end of the fourth convolutional layer 3*3.

[0054] Based on the content of the above device embodiments, as an alternative embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiments of the present invention further includes: a fourth sub-module for implementing the image recognition network, including: the output end of the fifth convolutional layer 3*3 is connected to the input end of the fourth batch normalization layer; the output end of the fourth batch normalization layer is connected to the input end of the fourth SELU activation function; the output end of the fourth SELU activation function is connected to the input end of the fifth upsampling layer; the output end of the fifth upsampling layer is connected to the input end of the sixth convolutional layer 3*3; the output end of the sixth convolutional layer 3*3 is connected to the input end of the fifth batch normalization layer; the output end of the fifth batch normalization layer is connected to the input end of the fifth SELU activation function; the output end of the fifth SELU activation function is connected to the input end of the seventh convolutional layer 3*3; the output end of the seventh convolutional layer 3*3 is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the tanh activation function layer; the output end of the tanh activation function layer is connected to the input end of the objective function layer.

[0055] Based on the content of the above device embodiments, as an alternative embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiments of the present invention further includes: a fifth sub-module, configured to implement that if the objective function and activation function of the image recognition network meet corresponding preset conditions, including: first inputting the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image into the image recognition network for recognition, adjusting the image recognition network to ensure that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image conforms to the objective function, and then inputting the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image into the adjusted image recognition network for recognition. If the output value of the activation function after recognition is less than a predetermined threshold, adjust the image construction network to reconstruct the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image until the output value of the activation function is greater than the predetermined threshold.

[0056] Based on the content of the above device embodiments, as an alternative embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiments of the present invention further includes: a sixth sub-module, configured to implement ensuring that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image conforms to the objective function, including:

[0057]

[0058] wherein, is the objective function; max is the symbol for taking the maximum value; is the i-th element of the final output vector of the image recognition network; tanh is the symbol of the hyperbolic tangent function; N is the total number of elements of the final output vector of the image recognition network.

[0059] The method of the embodiments of the present invention is implemented relying on an electronic device. Therefore, it is necessary to introduce the relevant electronic device. For this purpose, the embodiments of the present invention provide an electronic device, as Figure 3 shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus. Among them, at least one processor, the communication interface, and at least one memory complete mutual communication through the communication bus. The at least one processor can call the logical instructions in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0060] In addition, when the logical instructions in at least one of the above-mentioned memories can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various method embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0061] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0062] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0063] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. Based on this understanding, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0064] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. Any "predetermined threshold", "preset threshold" and other similar expressions that do not indicate specific values ​​can be determined by a person of ordinary skill in the art through simple experiments or corresponding debugging.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image fusion method based on edge texture enhancement, characterized in that: include: The computer tomography image and the magnetic resonance image are decomposed by wavelet transform respectively to obtain the edge texture image and tissue image of the computer tomography image, and the edge texture image and tissue image of the magnetic resonance image; the tissue image of the computer tomography image and the tissue image of the magnetic resonance image are fused to obtain a fused tissue image; the edge texture image of the computer tomography image and the edge texture image of the magnetic resonance image are extracted by feature extraction network respectively, and the extracted features are input into image construction network to obtain the edge texture construction image of the computer tomography image and the edge texture construction image of the magnetic resonance image; the edge texture construction image of the computer tomography image, the edge texture construction image of the magnetic resonance image, the edge texture image of the computer tomography image and the edge texture image of the magnetic resonance image are input into image recognition network for recognition; if the objective function and activation function of the image recognition network meet the corresponding preset conditions, the edge texture construction image and the fused tissue image are finally fused by inverse wavelet transform to obtain a final fused image.

2. The medical image fusion method based on edge texture enhancement according to claim 1, characterized in that: The method of fusing the tissue image of the computed tomography image with the tissue image of the magnetic resonance image includes: inputting the tissue image of the computed tomography image and the tissue image of the magnetic resonance image into a fused convolutional neural network for fusing, wherein the fused convolutional neural network includes: connecting the output end of the first convolutional layer 7*7 to the input end of the first maximum pooling layer; connecting the output end of the first maximum pooling layer to the input end of the second convolutional layer 7*7; connecting the output end of the second convolutional layer 7*7 to the input end of the first fully connected layer; and connecting the output end of the first fully connected layer to the input end of the first convolutional layer 5*5.

3. The medical image fusion method based on edge texture enhancement according to claim 2, characterized in that: The feature extraction network includes: the output end of the ReLU activation function + convolution layer 5*5 is connected to the input end of the second maximum pooling layer and the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the second maximum pooling layer is connected to the input end of the first batch normalization + ReLU activation function + convolution layer 4*4; the output end of the first batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input end of the first transition layer and the input end of the third ReLU activation function + convolution layer 3*3; the output end of the first transition layer is connected to the input end of the first batch normalization + ReLU activation function + convolution layer 4*4; The output of the second batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input of the second transition layer and the input of the second ReLU activation function + convolutional layer 3*3; the output of the second transition layer is connected to the input of the third batch normalization + ReLU activation function + convolutional layer 4*4; the output of the third batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input of the third transition layer; the output of the third transition layer is connected to the input of the third batch normalization + ReLU activation function + convolutional layer 4*4; The first end is connected to the input end of the fourth batch normalization + ReLU activation function + convolution layer 4*4 and the input end of the second upsampling layer; the output end of the fourth batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input end of the first upsampling layer; the output end of the first upsampling layer is connected to the input end of the first ReLU activation function + convolution layer 3*3; the output end of the first ReLU activation function + convolution layer 3*3 is connected to the input end of the second upsampling layer; the output end of the second upsampling layer is connected to the second ReLU activation function + convolution layer 3* 3; the output end of the second ReLU activation function + convolution layer 3*3 is connected to the input end of the third upsampling layer; the output end of the third upsampling layer is connected to the input end of the third ReLU activation function + convolution layer 3*3; the output end of the third ReLU activation function + convolution layer 3*3 is connected to the input end of the fourth upsampling layer; the output end of the fourth upsampling layer is connected to the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the fourth ReLU activation function + convolution layer 3*3 is connected to the input end of the transposed convolution layer.

4. The medical image fusion method based on edge texture enhancement according to claim 3, characterized in that: The image construction network includes: the output end of the first convolutional layer 3*3 is connected to the input end of the first batch normalization layer; the output end of the first batch normalization layer is connected to the input end of the first SELU activation function; the output end of the first SELU activation function is connected to the input end of the second convolutional layer 3*3; the output end of the second convolutional layer 3*3 is connected to the input end of the second batch normalization layer; the output end of the second batch normalization layer is connected to the input end of the second SELU activation function; the output end of the second SELU activation function is connected to the input end of the third maximum pooling layer; the output end of the third maximum pooling layer is connected to the input end of the third convolutional layer 3*3; the output end of the third convolutional layer 3*3 is connected to the input end of the third batch normalization layer; the output end of the third batch normalization layer is connected to the input end of the third SELU activation function; the output end of the third SELU activation function is connected to the input end of the fourth convolutional layer 3*3.

5. The medical image fusion method based on edge texture enhancement according to claim 4, characterized in that: The image recognition network includes: the output end of the fifth convolutional layer 3*3 is connected to the input end of the fourth batch normalization layer; the output end of the fourth batch normalization layer is connected to the input end of the fourth SELU activation function; the output end of the fourth SELU activation function is connected to the input end of the fifth upsampling layer; the output end of the fifth upsampling layer is connected to the input end of the sixth convolutional layer 3*3; the output end of the sixth convolutional layer 3*3 is connected to the input end of the fifth batch normalization layer; the output end of the fifth batch normalization layer is connected to the input end of the fifth SELU activation function; the output end of the fifth SELU activation function is connected to the input end of the seventh convolutional layer 3*3; the output end of the seventh convolutional layer 3*3 is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the tanh activation function layer; the output end of the tanh activation function layer is connected to the input end of the target function layer.

6. The medical image fusion method based on edge texture enhancement according to claim 5, characterized in that: If the objective function and activation function of the image recognition network meet the corresponding preset conditions, it includes: first inputting the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image into the image recognition network for recognition, adjusting the image recognition network to ensure that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image meets the objective function, and then inputting the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image into the adjusted image recognition network for recognition, and if the output value of the activation function after recognition is less than a predetermined threshold value, adjusting the image construction network to reconstruct the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image until the output value of the activation function is greater than the predetermined threshold value.

7. The medical image fusion method based on edge texture enhancement according to claim 6, characterized in that: The method of ensuring that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image meets the objective function includes: ； in, is the objective function; max is the maximum value symbol; is the i-th element of the final output vector of the image recognition network; tanh is the symbol of the hyperbolic tangent function; N is the total number of elements of the final output vector of the image recognition network.

8. A medical image fusion device based on edge texture enhancement, characterized in that: include: The first main module is used to realize the decomposition of the computed tomography image and the nuclear magnetic resonance image by wavelet transform, so as to obtain the edge texture image and tissue image of the computed tomography image, and the edge texture image and tissue image of the nuclear magnetic resonance image; the second main module is used to realize the fusion of the tissue image of the computed tomography image and the tissue image of the nuclear magnetic resonance image, so as to obtain the fused tissue image; the third main module is used to realize the feature extraction of the edge texture image of the computed tomography image and the edge texture image of the nuclear magnetic resonance image by feature extraction network, and input the extracted features into the image construction network to obtain the edge texture construction image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image; the fourth main module is used to realize the input of the edge texture construction image of the computed tomography image, the edge texture construction image of the nuclear magnetic resonance image, the edge texture image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image into the image recognition network for recognition; the fifth main module is used to realize the final fusion of the edge texture construction image and the fused tissue image by wavelet inverse transform if the objective function and activation function of the image recognition network meet the corresponding preset conditions, so as to obtain the final fused image.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause a computer to execute the method of any one of claims 1 to 7.

Images