Medical image fusion method and equipment based on edge texture enhancement

Abstract

The invention provides a medical image fusion method and equipment based on edge texture enhancement. The method comprises the following steps: acquiring an edge texture image and a tissue image of a computed tomography image and an edge texture image and a tissue image of a nuclear magnetic resonance image; the extracted edge texture image features of the computed tomography image and the extracted edge texture image features of the nuclear magnetic resonance image are input into an image construction network to obtain respective edge texture construction images, and the edge texture construction images and the actual edge texture image are input into an image recognition network for recognition; and if the target function and the activation function of the image recognition network meet corresponding preset conditions, performing final fusion on the edge texture construction image and the fusion tissue image by adopting wavelet inverse transformation to obtain a final fusion image. According to the invention, a fused medical image with clearer edge textures can be obtained.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of medical image signal processing, and in particular to a medical image fusion method and device based on edge texture enhancement. Background Art

[0002] At present, the fusion of computer tomography images and magnetic resonance imaging is the most common mode in modern medical image signal processing. In the field of medical imaging, computer tomography images have high tissue density and high resolution, showing human bones more clearly; while magnetic resonance imaging has better soft tissue density resolution, showing human soft tissue structure information more clearly. How to make good use of the advantages of both images to help doctors make quick diagnoses is one of the current research hotspots in the medical community. Although there are methods for fusing two image signals in related technologies, they are all simple image feature extraction and then fusion. The clarity of the edge texture after fusion is poor, which has an adverse effect on the correct judgment of the actual situation of human tissue. Therefore, the development of a medical image fusion method and device based on edge texture enhancement can effectively overcome the defects in the above-mentioned related technologies, which has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0003] In view of the above problems existing in the prior art, an embodiment of the present invention provides a medical image fusion method and device based on edge texture enhancement.

[0004] In a first aspect, an embodiment of the present invention provides a medical image fusion method based on edge texture enhancement, comprising: using wavelet transform to decompose a computed tomography image and a magnetic resonance image respectively to obtain an edge texture image and a tissue image of the computed tomography image, as well as an edge texture image and a tissue image of the magnetic resonance image; fusing the tissue image of the computed tomography image with the tissue image of the magnetic resonance image to obtain a fused tissue image; using a feature extraction network to extract features from the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image respectively, and inputting the extracted features into an image construction network to obtain an edge texture construction image of the computed tomography image and an edge texture construction image of the magnetic resonance image; inputting the edge texture construction image of the computed tomography image, the edge texture construction image of the magnetic resonance image, the edge texture image of the computed tomography image, and the edge texture image of the magnetic resonance image into an image recognition network for recognition; if the objective function and activation function of the image recognition network meet corresponding preset conditions, using inverse wavelet transform to finally fuse the edge texture construction image with the fused tissue image to obtain a final fused image.

[0005] Based on the content of the above method embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, wherein the tissue image of the computed tomography image and the tissue image of the magnetic resonance image are fused, comprising: inputting the tissue image of the computed tomography image and the tissue image of the magnetic resonance image into a fused convolutional neural network for fusion, wherein the fused convolutional neural network comprises: the output end of the first convolutional layer 7*7 is connected to the input end of the first maximum pooling layer; the output end of the first maximum pooling layer is connected to the input end of the second convolutional layer 7*7; the output end of the second convolutional layer 7*7 is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the first convolutional layer 5*5.

[0006] Based on the content of the above method embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, the feature extraction network includes: the output end of the ReLU activation function + convolution layer 5*5 is connected to the input end of the second maximum pooling layer and the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the second maximum pooling layer is connected to the input end of the first batch normalization + ReLU activation function + convolution layer 4*4; the output end of the first batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input end of the first transition ... The output of the first transition layer is connected to the input of the second batch normalization + ReLU activation function + convolution layer 4*4; the output of the second batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input of the second transition layer and the input of the second ReLU activation function + convolution layer 3*3; the output of the second transition layer is connected to the input of the third batch normalization + ReLU activation function + convolution layer 4*4; the output of the third batch normalization + ReLU activation function + convolution layer 4*4 is connected The output of the third transition layer is connected to the input of the fourth batch normalization + ReLU activation function + convolution layer 4*4 and the input of the second upsampling layer; the output of the fourth batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input of the first upsampling layer; the output of the first upsampling layer is connected to the input of the first ReLU activation function + convolution layer 3*3; the output of the first ReLU activation function + convolution layer 3*3 is connected to the input of the second upsampling layer; the output of the second upsampling layer is connected to the second ReLU The output end of the second ReLU activation function + convolution layer 3*3 is connected to the input end of the third upsampling layer; the output end of the third upsampling layer is connected to the input end of the third ReLU activation function + convolution layer 3*3; the output end of the third ReLU activation function + convolution layer 3*3 is connected to the input end of the fourth upsampling layer; the output end of the fourth upsampling layer is connected to the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the fourth ReLU activation function + convolution layer 3*3 is connected to the input end of the transposed convolution layer.

[0007] Based on the content of the above method embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, the image construction network includes: the output end of the first convolutional layer 3*3 is connected to the input end of the first batch normalization layer; the output end of the first batch normalization layer is connected to the input end of the first SELU activation function; the output end of the first SELU activation function is connected to the input end of the second convolutional layer 3*3; the output end of the second convolutional layer 3*3 is connected to the input end of the second batch normalization layer; the output end of the second batch normalization layer is connected to the input end of the second SELU activation function; the output end of the second SELU activation function is connected to the input end of the third maximum pooling layer; the output end of the third maximum pooling layer is connected to the input end of the third convolutional layer 3*3; the output end of the third convolutional layer 3*3 is connected to the input end of the third batch normalization layer; the output end of the third batch normalization layer is connected to the input end of the third SELU activation function; the output end of the third SELU activation function is connected to the input end of the fourth convolutional layer 3*3.

[0008] Based on the content of the above method embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, the image recognition network includes: the output end of the fifth convolutional layer 3*3 is connected to the input end of the fourth batch normalization layer; the output end of the fourth batch normalization layer is connected to the input end of the fourth SELU activation function; the output end of the fourth SELU activation function is connected to the input end of the fifth upsampling layer; the output end of the fifth upsampling layer is connected to the input end of the sixth convolutional layer 3*3; the output end of the sixth convolutional layer 3*3 is connected to the input end of the fifth batch normalization layer; the output end of the fifth batch normalization layer is connected to the input end of the fifth SELU activation function; the output end of the fifth SELU activation function is connected to the input end of the seventh convolutional layer 3*3; the output end of the seventh convolutional layer 3*3 is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the tanh activation function layer; the output end of the tanh activation function layer is connected to the input end of the target function layer.

[0009] Based on the content of the above method embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, if the objective function and activation function of the image recognition network meet the corresponding preset conditions, it includes: first inputting the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image into the image recognition network for recognition, adjusting the image recognition network to ensure that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image meets the objective function, and then inputting the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image into the adjusted image recognition network for recognition, if the output value of the activation function after recognition is less than a predetermined threshold value, adjusting the image construction network to reconstruct the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image until the output value of the activation function is greater than the predetermined threshold value.

[0010] Based on the content of the above method embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention ensures that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image meets the objective function, including: ; in, is the objective function; max is the maximum value symbol; is the i-th element of the final output vector of the image recognition network; tanh is the symbol of the hyperbolic tangent function; N is the total number of elements of the final output vector of the image recognition network.

[0011] In a second aspect, an embodiment of the present invention provides a medical image fusion device based on edge texture enhancement, comprising: a first main module, used to implement the use of wavelet transform to decompose the computed tomography image and the magnetic resonance image respectively, to obtain the edge texture image and tissue image of the computed tomography image, and the edge texture image and tissue image of the magnetic resonance image; a second main module, used to implement the fusion of the tissue image of the computed tomography image and the tissue image of the magnetic resonance image to obtain a fused tissue image; a third main module, used to implement the use of a feature extraction network to respectively decompose the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image Perform feature extraction, input the extracted features into the image construction network to obtain the edge texture construction image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image; the fourth main module is used to input the edge texture construction image of the computed tomography image, the edge texture construction image of the nuclear magnetic resonance image, the edge texture image of the computed tomography image and the edge texture image of the nuclear magnetic resonance image into the image recognition network for recognition; the fifth main module is used to realize that if the objective function and activation function of the image recognition network meet the corresponding preset conditions, the edge texture construction image and the fused tissue image are finally fused by using inverse wavelet transform to obtain the final fused image.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, including: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the medical image fusion method based on edge texture enhancement provided by any one of the various implementations of the first aspect.

[0013] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute a medical image fusion method based on edge texture enhancement provided by any one of the various implementation methods of the first aspect.

[0014] The medical image fusion method and device based on edge texture enhancement provided by the embodiment of the present invention obtain a fused tissue image by fusing the tissue image of the computed tomography image obtained by wavelet transform decomposition with the tissue image of the magnetic resonance image, and input the extracted edge texture image features of the computed tomography image and the edge texture image features of the magnetic resonance image into an image construction network to obtain respective edge texture construction images, and input the edge texture construction image and the actual edge texture image into an image recognition network for recognition. If the objective function and activation function of the image recognition network meet the corresponding preset conditions, the edge texture construction image and the fused tissue image are fused by wavelet inverse transform to obtain the final fused image, so as to obtain a fused medical image with clearer edge texture, thereby providing a more solid technical support for medical clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a medical image fusion method based on edge texture enhancement provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a medical image fusion device based on edge texture enhancement provided by an embodiment of the present invention; Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention; Figure 4 A schematic diagram of the overall network structure of a medical image fusion method based on edge texture enhancement provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a fused convolutional neural network provided in an embodiment of the present invention; Figure 6 A schematic diagram of a feature extraction network structure provided by an embodiment of the present invention; Figure 7 A schematic diagram of an image construction network structure provided by an embodiment of the present invention; Figure 8 A schematic diagram of an image recognition network structure provided by an embodiment of the present invention; FIG9 (a) is a schematic diagram of an original CT image effect provided by an embodiment of the present invention; FIG9( b ) is a schematic diagram of an original MRI image effect provided by an embodiment of the present invention; FIG9 (c) is a schematic diagram of medical image fusion effects provided by related technologies; FIG9( d ) is a schematic diagram of the medical image fusion effect provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantages of the embodiment of the present invention clearer, the technical scheme in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that the combination of such technical solutions does not exist and is not within the scope of protection required by the present invention. If there are step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0018] The embodiment of the present invention provides a medical image fusion method based on edge texture enhancement, see Figure 1 The method comprises: using wavelet transform to decompose the computed tomography image and the nuclear magnetic resonance image respectively to obtain the edge texture image and tissue image of the computed tomography image, and the edge texture image and tissue image of the nuclear magnetic resonance image; fusing the tissue image of the computed tomography image with the tissue image of the nuclear magnetic resonance image to obtain a fused tissue image; using a feature extraction network to extract features of the edge texture image of the computed tomography image and the edge texture image of the nuclear magnetic resonance image respectively, inputting the extracted features into an image construction network to obtain the edge texture construction image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image; inputting the edge texture construction image of the computed tomography image, the edge texture construction image of the nuclear magnetic resonance image, the edge texture image of the computed tomography image, and the edge texture image of the nuclear magnetic resonance image into an image recognition network for recognition; if the objective function and the activation function of the image recognition network meet the corresponding preset conditions, using inverse wavelet transform to finally fuse the edge texture construction image with the fused tissue image to obtain a final fused image.

[0019] Specifically, the network structure of the medical image fusion method based on edge texture enhancement can be found in Figure 4The first wavelet transform (which is the same as the second wavelet transform, the names are different only for distinction) is used to decompose the CT image to obtain the edge texture image of the CT image and the tissue image of the CT image (such as brain tissue). Similarly, the edge texture image of the MRI image and the tissue image of the MRI image are obtained. The tissue image of the CT image and the tissue image of the MRI image are input into the fusion convolutional neural network (for the specific structure, see Figure 5 ), after fusion, a fused tissue image is obtained; the edge texture image of the CT image and the edge texture image of the MRI image are input into the image recognition network, and the image recognition network is trained until the objective function of the image recognition network is met, and the image recognition network training is completed. Then, the edge texture image of the CT image and the edge texture image of the MRI image are input into the feature extraction network, and the extracted features are input into the image construction network to obtain the edge texture construction image of the CT image and the edge texture construction image of the MRI image, and both are input into the trained image recognition network. If the output of the activation function of the image recognition network meets the predetermined threshold, it means that the edge texture image constructed by the image construction network can meet the requirements, and at this time, Figure 4 By removing the image recognition network in the image recognition network, the final network structure for realizing the medical image fusion method based on edge texture enhancement can be obtained; the edge texture structure image and the fused tissue image are finally fused by using inverse wavelet transform to obtain the final fused image.

[0020] Based on the content of the above method embodiment, as an optional embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, the fusion of the tissue image of the computed tomography image and the tissue image of the nuclear magnetic resonance image, includes: the tissue image of the computed tomography image and the tissue image of the nuclear magnetic resonance image are input into the fusion convolutional neural network for fusion, and the fusion convolutional neural network includes: the output end of the first convolutional layer 7*7 is connected to the input end of the first maximum pooling layer; the output end of the first maximum pooling layer is connected to the input end of the second convolutional layer 7*7; the output end of the second convolutional layer 7*7 is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the first convolutional layer 5*5. It should be noted that all descriptions such as adding "first" to the first convolutional layer 7*7 in this specification are only to distinguish the various convolutional layers in terms of name, and the basic structure of each convolutional layer is similar, and the only difference is the difference in the convolution kernel (that is, the specific convolution kernel matrix is ​​different). The meanings of other similar descriptions are the same as this and will not be repeated.

[0021] Based on the content of the above method embodiment, as an optional embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, the feature extraction network includes: the output end of the ReLU activation function + convolution layer 5*5 is connected to the input end of the second maximum pooling layer and the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the second maximum pooling layer is connected to the input end of the first batch normalization + ReLU activation function + convolution layer 4*4; the output end of the first batch normalization + ReLU activation function + convolution layer 4*4 is connected to the first transition The input end of the layer is connected to the input end of the third ReLU activation function + convolutional layer 3*3; the output end of the first transition layer is connected to the input end of the second batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the second batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the second transition layer and the input end of the second ReLU activation function + convolutional layer 3*3; the output end of the second transition layer is connected to the input end of the third batch normalization + ReLU activation function + convolutional layer 4*4; the third batch normalization + ReLU activation function + convolutional layer 4* The output end of the third transition layer is connected to the input end of the fourth batch normalization + ReLU activation function + convolution layer 4*4 and the input end of the second upsampling layer; the output end of the fourth batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input end of the first upsampling layer; the output end of the first upsampling layer is connected to the input end of the first ReLU activation function + convolution layer 3*3; the output end of the first ReLU activation function + convolution layer 3*3 is connected to the input end of the second upsampling layer; the output end of the second upsampling layer is connected to the second R The output of the second ReLU activation function + convolution layer 3*3 is connected to the input of the third upsampling layer; the output of the third upsampling layer is connected to the input of the third ReLU activation function + convolution layer 3*3; the output of the third ReLU activation function + convolution layer 3*3 is connected to the input of the fourth upsampling layer; the output of the fourth upsampling layer is connected to the input of the fourth ReLU activation function + convolution layer 3*3; the output of the fourth ReLU activation function + convolution layer 3*3 is connected to the input of the transposed convolution layer. It should be noted that the feature extraction network constructed by this embodiment can obtain and transmit richer features in training, has a better gradient enhancement effect, and can locate the global optimal solution.

[0022] Based on the content of the above method embodiment, as an optional embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, the image construction network includes: the output end of the first convolutional layer 3*3 is connected to the input end of the first batch normalization layer; the output end of the first batch normalization layer is connected to the input end of the first SELU activation function; the output end of the first SELU activation function is connected to the input end of the second convolutional layer 3*3; the output end of the second convolutional layer 3*3 is connected to the input end of the second batch normalization layer; the output end of the second batch normalization layer is connected to the input end of the second SELU activation function; the output end of the second SELU activation function is connected to the input end of the third maximum pooling layer; the output end of the third maximum pooling layer is connected to the input end of the third convolutional layer 3*3; the output end of the third convolutional layer 3*3 is connected to the input end of the third batch normalization layer; the output end of the third batch normalization layer is connected to the input end of the third SELU activation function; the output end of the third SELU activation function is connected to the input end of the fourth convolutional layer 3*3.

[0023] Specifically, after adopting the design structure of the image construction network, the network capacity and feature extraction capabilities are well balanced. When the image resolution is not very high and the network width and depth are not very large, the gradient vanishing problem can be further alleviated, while the utilization and effective transmission of features are enhanced.

[0024] Based on the content of the above method embodiment, as an optional embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, the image recognition network includes: the output end of the fifth convolutional layer 3*3 is connected to the input end of the fourth batch normalization layer (for example, using the softmax function); the output end of the fourth batch normalization layer is connected to the input end of the fourth SELU activation function; the output end of the fourth SELU activation function is connected to the input end of the fifth upsampling layer; the output end of the fifth upsampling layer is connected to the input end of the sixth convolutional layer 3*3; the output end of the sixth convolutional layer 3*3 is connected to the input end of the fifth batch normalization layer; the output end of the fifth batch normalization layer is connected to the input end of the fifth SELU activation function; the output end of the fifth SELU activation function is connected to the input end of the seventh convolutional layer 3*3; the output end of the seventh convolutional layer 3*3 is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the tanh activation function layer; the output end of the tanh activation function layer is connected to the input end of the target function layer. It should be noted that the objective function layer is the layer retained during the continuous adjustment of the image recognition network. When the adjustment of the image recognition network is completed (that is, the training of the image recognition network is completed), the function of this layer is completed. Subsequently, it is only necessary to detect the output of the tanh activation function to complete the determination of the edge texture structure image.

[0025] Based on the content of the above method embodiment, as an optional embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, if the objective function and activation function of the image recognition network meet the corresponding preset conditions, it includes: first inputting the edge texture image of the computed tomography image and the edge texture image of the nuclear magnetic resonance image into the image recognition network for recognition, adjusting the image recognition network to ensure that the activation function output of the edge texture image of the computed tomography image and the edge texture image of the nuclear magnetic resonance image meets the objective function, and then inputting the edge texture construction image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image into the adjusted image recognition network for recognition, if the output value of the activation function after recognition is less than a predetermined threshold, then adjusting the image construction network to reconstruct the edge texture construction image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image until the output value of the activation function is greater than the predetermined threshold. In another embodiment, the predetermined threshold can be 90% or 80% of the objective function of the corresponding element of the output vector.

[0026] Based on the content of the above method embodiment, as an optional embodiment, the medical image fusion method based on edge texture enhancement provided in the embodiment of the present invention, the method of ensuring that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image meets the objective function, includes: ; in, is the objective function; max is the maximum value symbol; is the i-th element of the final output vector of the image recognition network; tanh is the symbol of the hyperbolic tangent function; N is the total number of elements of the final output vector of the image recognition network.

[0027] Specifically, after determining the corresponding objective function value of each element (bit) of the output vector, the predetermined threshold for judging whether each element of the output vector of the edge texture structure image of the computed tomography image and the edge texture structure image of the magnetic resonance image is qualified can be determined as or .

[0028] The fusion effect of the medical image fusion method based on edge texture enhancement provided by the embodiment of the present invention can be seen in Figures 9 (a) to 9 (d). Compared with the original CT image shown in Figure 9 (a) and the original MRI image shown in Figure 9 (b) (white circle part), it can be seen that the edge texture of the image fused using the relevant technology shown in Figure 9 (c) is relatively blurred (white circle part), which is basically difficult to distinguish. Compared with the image fusion effect of the medical image fusion method based on edge texture enhancement provided by the embodiment of the present invention shown in Figure 9 (d), it can be seen (white circle part) that no matter it is the bone edge feature in the CT image, or the clarity of the contour and structure in the MRI image, as well as the contrast and slight gradient changes within the texture, the fusion effect of the medical image fusion method based on edge texture enhancement provided by the embodiment of the present invention is clearer, which is specifically reflected in that the edge texture details of the fused image are obviously clearer, and the image fusion result is more consistent with the edge texture of the MRI image in Figure 9 (b), which proves that the medical image fusion method based on edge texture enhancement provided by the embodiment of the present invention has obvious advantages.

[0029] The medical image fusion method based on edge texture enhancement provided by the embodiment of the present invention obtains a fused tissue image by fusing the tissue image of the computed tomography image obtained by wavelet transform decomposition with the tissue image of the magnetic resonance image, and inputs the extracted edge texture image features of the computed tomography image and the edge texture image features of the magnetic resonance image into an image construction network to obtain respective edge texture construction images, and inputs the edge texture construction image and the actual edge texture image into an image recognition network for recognition. If the objective function and activation function of the image recognition network meet corresponding preset conditions, the edge texture construction image and the fused tissue image are fused by inverse wavelet transform to obtain the final fused image, so as to obtain a fused medical image with clearer edge texture, thereby providing a more solid technical support for medical clinical diagnosis.

[0030] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention can be encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a medical image fusion device based on edge texture enhancement, which is used to execute the medical image fusion method based on edge texture enhancement in the above method embodiment. Figure 2The device comprises: a first main module, which is used to realize the decomposition of the computed tomography image and the nuclear magnetic resonance image by wavelet transform, so as to obtain the edge texture image and tissue image of the computed tomography image, and the edge texture image and tissue image of the nuclear magnetic resonance image; a second main module, which is used to realize the fusion of the tissue image of the computed tomography image and the tissue image of the nuclear magnetic resonance image, so as to obtain the fused tissue image; a third main module, which is used to realize the feature extraction of the edge texture image of the computed tomography image and the edge texture image of the nuclear magnetic resonance image by feature extraction network, and input the extracted features into the image construction network to obtain the edge texture construction image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image; a fourth main module, which is used to realize the input of the edge texture construction image of the computed tomography image, the edge texture construction image of the nuclear magnetic resonance image, the edge texture image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image into the image recognition network for recognition; a fifth main module, which is used to realize the final fusion of the edge texture construction image and the fused tissue image by wavelet inverse transform if the objective function and activation function of the image recognition network meet the corresponding preset conditions, so as to obtain the final fused image.

[0031] The medical image fusion device based on edge texture enhancement provided by the embodiment of the present invention adopts Figure 2 Several modules in it, by fusing the tissue image of the computed tomography image obtained by wavelet transform decomposition with the tissue image of the magnetic resonance image, a fused tissue image is obtained, the edge texture image features of the extracted computed tomography image and the edge texture image features of the magnetic resonance image are input into the image construction network to obtain respective edge texture construction images, the edge texture construction image and the actual edge texture image are input into the image recognition network for recognition, if the objective function and the activation function of the image recognition network meet the corresponding preset conditions, the edge texture construction image and the fused tissue image are fused by inverse wavelet transform to obtain the final fused image, so that a fused medical image with clearer edge texture can be obtained, which provides a more solid technical support for medical clinical diagnosis.

[0032] It should be noted that the device in the device embodiment provided by the present invention can be used to implement the method in the above method embodiment as well as the method in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set, and the principle is basically the same as the principle of the above device embodiment provided by the present invention. As long as the technical personnel in the field refer to the specific technical solutions in other method embodiments on the basis of the above device embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining technical features, the device in the above device embodiment can be improved on the premise of ensuring the practicality of the technical solution, thereby obtaining the corresponding device class embodiment, which is used to implement the methods in other method class embodiments. For example: Based on the content of the above-mentioned device embodiment, as an optional embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiment of the present invention further includes: a first sub-module, used to realize the fusion of the tissue image of the computed tomography image and the tissue image of the magnetic resonance image, including: inputting the tissue image of the computed tomography image and the tissue image of the magnetic resonance image into a fused convolutional neural network for fusion, and the fused convolutional neural network includes: the output end of the first convolutional layer 7*7 is connected to the input end of the first maximum pooling layer; the output end of the first maximum pooling layer is connected to the input end of the second convolutional layer 7*7; the output end of the second convolutional layer 7*7 is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the first convolutional layer 5*5.

[0033] Based on the content of the above device embodiment, as an optional embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiment of the present invention further includes: a second submodule, which is used to implement the feature extraction network, including: the output end of the ReLU activation function + convolution layer 5*5 is connected to the input end of the second maximum pooling layer and the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the second maximum pooling layer is connected to the input end of the first batch normalization + ReLU activation function + convolution layer 4*4; the first batch normalization + ReLU activation function + convolution layer 4 The output end of *4 is connected to the input end of the first transition layer and the input end of the third ReLU activation function + convolutional layer 3*3; the output end of the first transition layer is connected to the input end of the second batch normalization + ReLU activation function + convolutional layer 4*4; the output end of the second batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input end of the second transition layer and the input end of the second ReLU activation function + convolutional layer 3*3; the output end of the second transition layer is connected to the input end of the third batch normalization + ReLU activation function + convolutional layer 4*4; the third batch normalization + ReLU activation function The output end of the number + convolution layer 4*4 is connected to the input end of the third transition layer; the output end of the third transition layer is connected to the input end of the fourth batch normalization + ReLU activation function + convolution layer 4*4 and the input end of the second upsampling layer; the output end of the fourth batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input end of the first upsampling layer; the output end of the first upsampling layer is connected to the input end of the first ReLU activation function + convolution layer 3*3; the output end of the first ReLU activation function + convolution layer 3*3 is connected to the input end of the second upsampling layer; the output end of the second upsampling layer is connected The input end of the second ReLU activation function + convolution layer 3*3 is connected; the output end of the second ReLU activation function + convolution layer 3*3 is connected to the input end of the third upsampling layer; the output end of the third upsampling layer is connected to the input end of the third ReLU activation function + convolution layer 3*3; the output end of the third ReLU activation function + convolution layer 3*3 is connected to the input end of the fourth upsampling layer; the output end of the fourth upsampling layer is connected to the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the fourth ReLU activation function + convolution layer 3*3 is connected to the input end of the transposed convolution layer.

[0034] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiment of the present invention further includes: a third submodule, used to implement the image construction network, including: the output end of the first convolutional layer 3*3 is connected to the input end of the first batch normalization layer; the output end of the first batch normalization layer is connected to the input end of the first SELU activation function; the output end of the first SELU activation function is connected to the input end of the second convolutional layer 3*3; the output end of the second convolutional layer 3*3 is connected to the input end of the second batch normalization layer; the output end of the second batch normalization layer is connected to the input end of the second SELU activation function; the output end of the second SELU activation function is connected to the input end of the third maximum pooling layer; the output end of the third maximum pooling layer is connected to the input end of the third convolutional layer 3*3; the output end of the third convolutional layer 3*3 is connected to the input end of the third batch normalization layer; the output end of the third batch normalization layer is connected to the input end of the third SELU activation function; the output end of the third SELU activation function is connected to the input end of the fourth convolutional layer 3*3.

[0035] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiment of the present invention also includes: a fourth submodule, used to implement the image recognition network, including: the output end of the fifth convolutional layer 3*3 is connected to the input end of the fourth batch normalization layer; the output end of the fourth batch normalization layer is connected to the input end of the fourth SELU activation function; the output end of the fourth SELU activation function is connected to the input end of the fifth upsampling layer; the output end of the fifth upsampling layer is connected to the input end of the sixth convolutional layer 3*3; the output end of the sixth convolutional layer 3*3 is connected to the input end of the fifth batch normalization layer; the output end of the fifth batch normalization layer is connected to the input end of the fifth SELU activation function; the output end of the fifth SELU activation function is connected to the input end of the seventh convolutional layer 3*3; the output end of the seventh convolutional layer 3*3 is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the tanh activation function layer; the output end of the tanh activation function layer is connected to the input end of the target function layer.

[0036] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiment of the present invention further includes: a fifth sub-module, which is used to realize that if the objective function and activation function of the image recognition network meet the corresponding preset conditions, it includes: first inputting the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image into the image recognition network for recognition, adjusting the image recognition network to ensure that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image meets the objective function, and then inputting the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image into the adjusted image recognition network for recognition, if the output value of the activation function after recognition is less than a predetermined threshold value, then adjusting the image construction network to reconstruct the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image until the output value of the activation function is greater than the predetermined threshold value.

[0037] Based on the content of the above device embodiment, as an optional embodiment, the medical image fusion device based on edge texture enhancement provided in the embodiment of the present invention further includes: a sixth submodule, which is used to implement the activation function output of the image recognition network for edge texture images of computed tomography images and edge texture images of magnetic resonance images to meet the objective function, including: ; in, is the objective function; max is the maximum value symbol; is the i-th element of the final output vector of the image recognition network; tanh is the symbol of the hyperbolic tangent function; N is the total number of elements of the final output vector of the image recognition network.

[0038] The method of the embodiment of the present invention is implemented by relying on electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 3 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. At least one processor can call the logic instructions in at least one memory to execute all or part of the steps of the method provided by the aforementioned various method embodiments.

[0039] In addition, the logic instructions in the at least one memory mentioned above can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0040] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0041] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0042] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. Based on this understanding, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0043] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. Any "predetermined threshold", "preset threshold" and other similar expressions that do not indicate specific values ​​can be determined by a person of ordinary skill in the art through simple experiments or corresponding debugging.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image fusion method based on edge texture enhancement, characterized in that: include: The computer tomography image and the magnetic resonance image are decomposed by wavelet transform respectively to obtain the edge texture image and tissue image of the computer tomography image, and the edge texture image and tissue image of the magnetic resonance image; the tissue image of the computer tomography image and the tissue image of the magnetic resonance image are fused to obtain a fused tissue image; the edge texture image of the computer tomography image and the edge texture image of the magnetic resonance image are extracted by feature extraction network respectively, and the extracted features are input into image construction network to obtain the edge texture construction image of the computer tomography image and the edge texture construction image of the magnetic resonance image; the edge texture construction image of the computer tomography image, the edge texture construction image of the magnetic resonance image, the edge texture image of the computer tomography image and the edge texture image of the magnetic resonance image are input into image recognition network for recognition; if the objective function and activation function of the image recognition network meet the corresponding preset conditions, the edge texture construction image and the fused tissue image are finally fused by inverse wavelet transform to obtain a final fused image.

2. The medical image fusion method based on edge texture enhancement according to claim 1, characterized in that: The method of fusing the tissue image of the computed tomography image with the tissue image of the magnetic resonance image includes: inputting the tissue image of the computed tomography image and the tissue image of the magnetic resonance image into a fused convolutional neural network for fusing, wherein the fused convolutional neural network includes: connecting the output end of the first convolutional layer 7*7 to the input end of the first maximum pooling layer; connecting the output end of the first maximum pooling layer to the input end of the second convolutional layer 7*7; connecting the output end of the second convolutional layer 7*7 to the input end of the first fully connected layer; and connecting the output end of the first fully connected layer to the input end of the first convolutional layer 5*5.

3. The medical image fusion method based on edge texture enhancement according to claim 2, characterized in that: The feature extraction network includes: the output end of the ReLU activation function + convolution layer 5*5 is connected to the input end of the second maximum pooling layer and the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the second maximum pooling layer is connected to the input end of the first batch normalization + ReLU activation function + convolution layer 4*4; the output end of the first batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input end of the first transition layer and the input end of the third ReLU activation function + convolution layer 3*3; the output end of the first transition layer is connected to the input end of the first batch normalization + ReLU activation function + convolution layer 4*4; The output of the second batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input of the second transition layer and the input of the second ReLU activation function + convolutional layer 3*3; the output of the second transition layer is connected to the input of the third batch normalization + ReLU activation function + convolutional layer 4*4; the output of the third batch normalization + ReLU activation function + convolutional layer 4*4 is connected to the input of the third transition layer; the output of the third transition layer is connected to the input of the third batch normalization + ReLU activation function + convolutional layer 4*4; The first end is connected to the input end of the fourth batch normalization + ReLU activation function + convolution layer 4*4 and the input end of the second upsampling layer; the output end of the fourth batch normalization + ReLU activation function + convolution layer 4*4 is connected to the input end of the first upsampling layer; the output end of the first upsampling layer is connected to the input end of the first ReLU activation function + convolution layer 3*3; the output end of the first ReLU activation function + convolution layer 3*3 is connected to the input end of the second upsampling layer; the output end of the second upsampling layer is connected to the second ReLU activation function + convolution layer 3* 3; the output end of the second ReLU activation function + convolution layer 3*3 is connected to the input end of the third upsampling layer; the output end of the third upsampling layer is connected to the input end of the third ReLU activation function + convolution layer 3*3; the output end of the third ReLU activation function + convolution layer 3*3 is connected to the input end of the fourth upsampling layer; the output end of the fourth upsampling layer is connected to the input end of the fourth ReLU activation function + convolution layer 3*3; the output end of the fourth ReLU activation function + convolution layer 3*3 is connected to the input end of the transposed convolution layer.

4. The medical image fusion method based on edge texture enhancement according to claim 3, characterized in that: The image construction network includes: the output end of the first convolutional layer 3*3 is connected to the input end of the first batch normalization layer; the output end of the first batch normalization layer is connected to the input end of the first SELU activation function; the output end of the first SELU activation function is connected to the input end of the second convolutional layer 3*3; the output end of the second convolutional layer 3*3 is connected to the input end of the second batch normalization layer; the output end of the second batch normalization layer is connected to the input end of the second SELU activation function; the output end of the second SELU activation function is connected to the input end of the third maximum pooling layer; the output end of the third maximum pooling layer is connected to the input end of the third convolutional layer 3*3; the output end of the third convolutional layer 3*3 is connected to the input end of the third batch normalization layer; the output end of the third batch normalization layer is connected to the input end of the third SELU activation function; the output end of the third SELU activation function is connected to the input end of the fourth convolutional layer 3*3.

5. The medical image fusion method based on edge texture enhancement according to claim 4, characterized in that: The image recognition network includes: the output end of the fifth convolutional layer 3*3 is connected to the input end of the fourth batch normalization layer; the output end of the fourth batch normalization layer is connected to the input end of the fourth SELU activation function; the output end of the fourth SELU activation function is connected to the input end of the fifth upsampling layer; the output end of the fifth upsampling layer is connected to the input end of the sixth convolutional layer 3*3; the output end of the sixth convolutional layer 3*3 is connected to the input end of the fifth batch normalization layer; the output end of the fifth batch normalization layer is connected to the input end of the fifth SELU activation function; the output end of the fifth SELU activation function is connected to the input end of the seventh convolutional layer 3*3; the output end of the seventh convolutional layer 3*3 is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the tanh activation function layer; the output end of the tanh activation function layer is connected to the input end of the target function layer.

6. The medical image fusion method based on edge texture enhancement according to claim 5, characterized in that: If the objective function and activation function of the image recognition network meet the corresponding preset conditions, it includes: first inputting the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image into the image recognition network for recognition, adjusting the image recognition network to ensure that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image meets the objective function, and then inputting the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image into the adjusted image recognition network for recognition, and if the output value of the activation function after recognition is less than a predetermined threshold value, adjusting the image construction network to reconstruct the edge texture construction image of the computed tomography image and the edge texture construction image of the magnetic resonance image until the output value of the activation function is greater than the predetermined threshold value.

7. The medical image fusion method based on edge texture enhancement according to claim 6, characterized in that: The method of ensuring that the activation function output of the image recognition network for the edge texture image of the computed tomography image and the edge texture image of the magnetic resonance image meets the objective function includes: ; in, is the objective function; max is the maximum value symbol; is the i-th element of the final output vector of the image recognition network; tanh is the symbol of the hyperbolic tangent function; N is the total number of elements of the final output vector of the image recognition network.

8. A medical image fusion device based on edge texture enhancement, characterized in that: include: The first main module is used to realize the decomposition of the computed tomography image and the nuclear magnetic resonance image by wavelet transform, so as to obtain the edge texture image and tissue image of the computed tomography image, and the edge texture image and tissue image of the nuclear magnetic resonance image; the second main module is used to realize the fusion of the tissue image of the computed tomography image and the tissue image of the nuclear magnetic resonance image, so as to obtain the fused tissue image; the third main module is used to realize the feature extraction of the edge texture image of the computed tomography image and the edge texture image of the nuclear magnetic resonance image by feature extraction network, and input the extracted features into the image construction network to obtain the edge texture construction image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image; the fourth main module is used to realize the input of the edge texture construction image of the computed tomography image, the edge texture construction image of the nuclear magnetic resonance image, the edge texture image of the computed tomography image and the edge texture construction image of the nuclear magnetic resonance image into the image recognition network for recognition; the fifth main module is used to realize the final fusion of the edge texture construction image and the fused tissue image by wavelet inverse transform if the objective function and activation function of the image recognition network meet the corresponding preset conditions, so as to obtain the final fused image.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause a computer to execute the method of any one of claims 1 to 7.

Images