Deep semantic fusion-based brain trauma segmentation method
Through the deep semantic fusion method, the semantic coding module, the semantic segmentation module and the image fusion module are designed, which solves the problem of lack of trauma segmentation methods in the prior art, and achieves high accuracy and low subjectivity of trauma penumbra evaluation.
Patent Information
- Application Number
- CN202411748399.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-05-09
AI Technical Summary
The lack of deep learning-based trauma (trauma penumbra and injury core area) segmentation method is in the prior art, which leads to the time-consuming, labor-intensive and extremely subjective evaluation of the trauma penumbra.
The brain trauma segmentation method based on deep semantic fusion is adopted, including designing a semantic coding module to extract semantic features, the semantic segmentation module completes the segmentation task, and helps improve segmentation accuracy through the image fusion module.
High accuracy segmentation of brain trauma areas is achieved, subjectivity is reduced, and evaluation efficiency is improved.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a brain trauma segmentation method based on deep semantic fusion. Background Art
[0002] Traditional methods for assessing the penumbra of trauma are often qualitative assessments made by doctors after analyzing magnetic resonance images based on their own experience. This method is not only time-consuming and labor-intensive, but also highly subjective. Advances in computer science have made it possible to quantitatively assess the penumbra of trauma using computers. In the field of medical image analysis, artificial intelligence has received widespread attention due to its outstanding performance. Deep learning is a subset of artificial intelligence. It captures complex data patterns and features by simulating the neural network structure of the human brain, learns complex nonlinear relationships in the data, and provides strong data support for the work of clinicians. For the problem of medical image segmentation, some deep learning-based methods have been proposed. They model the segmentation problem as a patch-based classification problem, and use convolutional neural networks to predict the category of the central voxel, thereby representing the category of the entire patch area. This solution ignores the correlation between different patches in the receptive field, resulting in poor capture of global context information. In order to solve this problem, people have proposed a solution based on semantic segmentation, among which the U-Net network has become the mainstream of medical image segmentation research. In addition, in order to improve the feature learning ability of the model, some segmentation methods introduce a multi-task learning framework into the segmentation network to guide the feature learning of the model through knowledge sharing between tasks. However, to the best of our knowledge, there is currently no deep learning-based brain trauma (traumatic penumbra and injury core) segmentation method. Summary of the invention
[0003] In view of the shortcomings of the prior art mentioned above, the object of the present invention is to provide a brain trauma segmentation method based on deep semantic fusion, so as to solve the problem that there is no brain trauma (traumatic penumbra and injury core area) segmentation method based on deep learning in the prior art.
[0004] To achieve the above objectives and other related objectives, the present invention provides a brain injury segmentation method based on deep semantic fusion, comprising:
[0005] Collecting traumatic brain injury tissue data, and annotating the image to obtain a target data set of the traumatic brain injury tissue image;
[0006] Design a semantic encoding module to extract semantic features of the sample data;
[0007] Design a semantic segmentation module to complete the semantic segmentation task, and design an image fusion module to assist semantic segmentation and improve segmentation accuracy;
[0008] Segmenting the area to be segmented;
[0009] Optionally, before the step of extracting the semantic features of the sample data by using the semantic encoding module, the step further includes:
[0010] The data was preprocessed to improve the image quality through contrast enhancement technology, and the image was smoothed by combining Gaussian filtering. In order to increase the diversity of the data set, the mirror flip data enhancement method was applied. The program decided whether to perform horizontal or vertical mirroring based on random numbers.
[0011] Optionally, a semantic encoding module is designed, and the step of extracting the semantic features of the sample data includes:
[0012] The multimodal data is fed into a three-layer residual network block for feature extraction. Downsampling is a frequently used operation in deep learning.
[0013] Optionally, a semantic encoding module is designed, and the step of extracting the semantic features of the sample data includes:
[0014] The data output by the downsampling layer is sent to the feature extraction layer to continue extracting deep semantic features.
[0015] Optionally, a semantic encoding module is designed, and the step of extracting the semantic features of the sample data further includes:
[0016] The feature extraction layer is mainly divided into three steps: normalization, activation, and convolution. In deep learning, there are many normalization methods. Therefore, it is necessary to choose a suitable method according to the specific task scenario.
[0017] Optionally, we use group normalization. The mathematical expression of the operation process of the feature extraction layer is:
[0018]
[0019] in, is the final output of the feature extraction layer, GN is the group normalization operation, and y i Data for all channels.
[0020] Optionally, the structure of the semantic segmentation module and the encoder module is symmetrical, and also includes four blocks, and the steps include:
[0021] The first three blocks with skip connections are used to restore the image resolution layer by layer. When data flows into this module, it is first upsampled and then sent to the ResNet-like block to connect the high-level semantic features extracted by the decoder and the low-level semantic features extracted by the encoder in the same dimension.
[0022] Optionally, bilinear interpolation is used to restore the image resolution. The mathematical expression of the output feature after three layers of upsampling is:
[0023]
[0024] Among them, up is upsampling, H is height, and W is width.
[0025] Optionally, the image fusion module is used to fuse the semantic features of different modalities. It fuses the semantic features of two modalities, namely the penumbra feature information from the DWI modality and the lesion core area information from the SWI modality. The network structure of IFM is similar to that of SSM, and there are also three upsampling blocks for restoring resolution and a block for adjusting the number of channels. The final output of IFM is a weight matrix whose value is between 0 and 1. It represents the weight of the DWI modality and the SWI modality in the fused image. Therefore, to obtain the fusion result, a fusion operation should also be performed. The mathematical expression of the fusion operation process is:
[0026] F fusion =(F weight ⊙S1)+(1-F weight )⊙S2
[0027] Among them, F weight is the weight matrix output by the IFM module. S1 and S2 are the images of DWI and SWI modalities respectively, and X is the element-by-element multiplication between tensors.
[0028] Optionally, the loss is determined by a loss function. The mathematical expression for the training loss function of the segmentation network is:
[0029] L=L seg +ψL fusin
[0030] Among them, ψ is the trade-off parameter variable, L seg is the loss of the semantic segmentation module for learning semantic features, L fusin is the loss of the image fusion module.
[0031] Optionally, the segmentation loss function is a combination of binary cross entropy loss and Dice loss, which provides better gradient information and can measure the overlap between the predicted area and the true value area. The loss of the semantic segmentation module that learns semantic features is mathematically expressed as:
[0032] L seg =0.5×L BCE +L Dice
[0033] Among them, L BCE is the binary cross entropy loss, LDice is the dice loss.
[0034] Optionally, the mathematical expression of the loss of the image fusion module for auxiliary semantic segmentation is:
[0035]
[0036] Where N is the number of pixels in the image, C is the number of classes, is the output probability of the pixel set of class j, y ij is the ground truth label of the pixel in class i, and ε is a small constant.
[0037] Optionally, the image fusion module assists semantic segmentation by fusing semantic feature information of different modalities. The loss function of image fusion is defined as:
[0038] L fusion =L pixel +L ssim
[0039] Among them, L pixel To measure the pixel-level difference between the fused image and the target image, L ssim is the brightness, contrast and structural similarity of the image.
[0040] Optional, through L ssim Extract the structural information of the source image. The mathematical expressions of Lpixel and Lssim are:
[0041]
[0042] L ssin =η(1-SSIM(F,S1))+(1-SSIM(F,S2)
[0043] Where S1 is the DWI source image, S2 is the SWI source image, F is the fused image, and N is the number of pixels in the fused image. ||·|| and SSIM represent the tensor Frobenius norm and structural similarity measure, respectively, and ξ and η are weight parameters.
[0044] As described above, the brain trauma resection method based on deep semantic fusion of the present invention has the following beneficial effects:
[0045] We propose a semantic encoder module to extract semantic features of traumatic brain injury tissue. In particular, to avoid the degradation of deep neural networks, we introduce ResNet-like blocks into the semantic encoder module. We propose a semantic segmentation module to accomplish the semantic segmentation task. In this module, we use the skip connection operation to fuse the shallow semantic features extracted by the semantic encoder and the deep semantic features extracted by the semantic segmentation decoder to help the neural network retain more detailed information. In order to learn useful features from data of different modalities, we introduce a novel multimodal image fusion module based on a weighted average strategy to learn relevant features. At the same time, image fusion is used as an auxiliary task to improve the segmentation accuracy. To improve the performance of the model, we introduce a method based on homoscedastic uncertainty to dynamically adjust the weights of different modules. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 Shown is a schematic diagram of a brain injury image according to an embodiment of the present invention.
[0047] Figure 2 Shown is a flowchart of a brain injury segmentation method based on deep semantic fusion according to an embodiment of the present invention.
[0048] Figure 3 A schematic diagram showing the architecture of the proposed semantic encoder module according to an embodiment of the present invention is shown.
[0049] Figure 4 A schematic diagram showing the architecture of the proposed semantic segmentation decoder according to an embodiment of the present invention. DETAILED DESCRIPTION
[0050] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.
[0051] It should be noted that the diagram provided in the present embodiment only illustrates the basic concept of the present invention in a schematic manner, so the diagram only shows the components related to the present invention rather than drawing according to the number, shape and size of the components during actual implementation. The type, quantity and ratio of each component during actual implementation can be a random change, and the component layout type may also be more complicated. The structure, ratio, size, etc. illustrated in the drawings of the present specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that the present invention can implement, so they have no technical substantive significance. Any modification of the structure, change of the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the effect that the present invention can produce and the purpose that can be achieved. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of narration, and are not used to limit the scope of the present invention. The change or adjustment of its relative relationship should also be regarded as the scope of the present invention without substantially changing the technical content.
[0052] See also Figure 1 and Figure 2 The present invention provides a brain injury segmentation method based on deep semantic fusion, comprising:
[0053] S1: collecting traumatic brain injury tissue data, and annotating the image to obtain a target data set of the traumatic brain injury tissue image;
[0054] S2: Design a semantic encoding module to extract the semantic features of the sample data;
[0055] S3: Design a semantic segmentation module to complete the semantic segmentation task, and design an image fusion module to assist semantic segmentation and improve the accuracy of segmentation;
[0056] S4: Segment the area to be segmented;
[0057] Before the step of extracting the semantic features of the sample data by using the semantic coding module, the method further includes:
[0058] The data was preprocessed to improve the image quality through contrast enhancement technology, and the image was smoothed by combining Gaussian filtering. In order to increase the diversity of the data set, the mirror flip data enhancement method was applied, and the program decided whether to perform horizontal or vertical mirroring based on a random number. A semantic encoding module was designed, and the steps of extracting the semantic features of the sample data included:
[0059] The multimodal data is fed into a three-layer residual network block for feature extraction. Downsampling is a frequently used operation in deep learning.
[0060] Designing a semantic encoding module, the steps of extracting the semantic features of the sample data include:
[0061] The data output by the downsampling layer is sent to the feature extraction layer to continue extracting deep semantic features.
[0062] Designing a semantic encoding module, the step of extracting the semantic features of the sample data also includes:
[0063] The feature extraction layer is mainly divided into three steps: normalization, activation, and convolution. In deep learning, there are many normalization methods. Therefore, it is necessary to choose a suitable method according to the specific task scenario.
[0064] We use the group normalization method. The mathematical expression of the operation process of the feature extraction layer is:
[0065]
[0066] in, is the final output of the feature extraction layer, GN is the group normalization operation, and y i Data for all channels.
[0067] The structure of the semantic segmentation module is symmetrical to that of the encoder module, and also includes four blocks. Its steps include:
[0068] The first three blocks with skip connections are used to restore the image resolution layer by layer. When data flows into this module, it is first upsampled and then sent to the ResNet-like block to connect the high-level semantic features extracted by the decoder and the low-level semantic features extracted by the encoder in the same dimension.
[0069] The bilinear interpolation method is used to restore the image resolution. The mathematical expression of the output feature after three layers of upsampling operation is:
[0070]
[0071] Among them, up is upsampling, H is height, and W is width.
[0072] The image fusion module is used to fuse the semantic features of different modalities. It fuses the semantic features of two modalities, namely the penumbra feature information from the DWI modality and the damaged core area information from the SWI modality. The network structure of IFM is similar to that of SSM. There are also three upsampling blocks for restoring resolution and a block for adjusting the number of channels. The final output of IFM is a weight matrix whose value is between 0 and 1. It represents the weight of the DWI modality and the SWI modality in the fused image. Therefore, in order to obtain the fusion result, a fusion operation should also be performed. The mathematical expression of the fusion operation process is:
[0073] F fusion =(F weight ⊙S1)+(1-F weight )⊙S2
[0074] where F weight is the weight matrix output by the IFM module. S1 and S2 are the images of DWI and SWI modalities respectively, and X represents the element-by-element multiplication between tensors.
[0075] The loss is determined by the loss function. The mathematical expression of the training loss function of the segmentation network is defined as:
[0076] L=L seg +ψL fusin
[0077] Among them, ψ is the trade-off parameter variable, L seg is the loss of the semantic segmentation module for learning semantic features, L fusin is the loss of the image fusion module.
[0078] The segmentation loss function is a combination of binary cross entropy loss and Dice loss, which provides better gradient information and can measure the overlap between the predicted area and the true value area. The loss of the semantic segmentation module that learns semantic features is mathematically expressed as:
[0079] L seg =0.5×L BCE +L Dice
[0080] Among them, L BCE is the binary cross entropy loss, L Dice is the dice loss.
[0081] The mathematical expression of the loss of the image fusion module for auxiliary semantic segmentation is:
[0082]
[0083] Where N is the number of pixels in the image, C is the number of classes, is the output probability of the pixel set of class j, y ij is the ground truth label of the pixel in class i, and ε is a small constant.
[0084] The image fusion module assists semantic segmentation and fuses semantic feature information of different modalities. The loss function of image fusion is defined as:
[0085] L fusion =L pixel +L ssim
[0086] Among them, L pixel To measure the pixel-level difference between the fused image and the target image, L ssim is the brightness, contrast and structural similarity of the image.
[0087] By L ssim Extract the structural information of the source image. The mathematical expressions of Lpixel and Lssim are:
[0088]
[0089] L ssin =η(1-SSIM(F,S1))+(1-SSIM(F,S2)
[0090] Where S1 is the DWI source image, S2 is the SWI source image, F is the fused image, and N is the number of pixels in the fused image. ||·|| and SSIM represent the tensor Frobenius norm and structural similarity measure, respectively, and ξ and η are weight parameters.
[0091] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.
[0092] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.
[0093] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0094] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0095] In the above embodiments, the description's reference to "this embodiment" indicates that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily all refer to the same embodiment.
[0096] In the above embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications and variations that fall within the broad scope of the appended claims.
[0097] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0098] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.
[0099] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0100] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A brain trauma segmentation method based on deep semantic fusion, characterized in that: include: Collecting traumatic brain injury tissue data, and annotating the image to obtain a target data set of the traumatic brain injury tissue image; Design a semantic encoding module to extract semantic features of the sample data; Design a semantic segmentation module to complete the semantic segmentation task, and design an image fusion module to assist semantic segmentation and improve segmentation accuracy; The area to be segmented is segmented.
2. The brain injury segmentation method based on deep semantic fusion according to claim 1, characterized in that: Before the step of extracting the semantic features of the sample data by using the semantic coding module, the method further includes: The data was preprocessed to improve the image quality through contrast enhancement technology, and the image was smoothed by combining Gaussian filtering. In order to increase the diversity of the data set, the mirror flip data enhancement method was applied. The program decided whether to perform horizontal or vertical mirroring based on random numbers.
3. The brain injury segmentation method based on deep semantic fusion according to claim 1, characterized in that: Designing a semantic encoding module, the steps of extracting the semantic features of the sample data include: The multimodal data is fed into a three-layer residual network block for feature extraction.
4. A brain injury segmentation method based on deep semantic fusion according to claim 1 or 3, characterized in that: Designing a semantic encoding module, the steps of extracting the semantic features of the sample data include: The data output by the downsampling layer is sent to the feature extraction layer to continue extracting deep semantic features.
5. The brain injury segmentation method based on deep semantic fusion according to claim 1, characterized in that: Designing a semantic encoding module, the step of extracting the semantic features of the sample data also includes: The feature extraction layer is mainly divided into three steps: normalization, activation, and convolution. In deep learning, there are many normalization methods. Therefore, it is necessary to choose a suitable method according to the specific task scenario.
6. The brain injury segmentation method based on deep semantic fusion according to claim 2, characterized in that: We use the group normalization method, and the mathematical expression of the operation process of the feature extraction layer is: in, is the final output of the feature extraction layer, GN is the group normalization operation, and y i Data for all channels.
7. The brain injury segmentation method based on deep semantic fusion according to claim 4, characterized in that: The structure of the semantic segmentation module is symmetrical to that of the encoder module, and also includes four blocks. Its steps include: The first three blocks with skip connections are used to restore the image resolution layer by layer. When data flows into this module, it is first upsampled and then sent to the ResNet-like block to connect the high-level semantic features extracted by the decoder and the low-level semantic features extracted by the encoder in the same dimension.
8. The semantic segmentation module according to claim 4, characterized in that The bilinear interpolation method is used to restore the image resolution. The mathematical expression of the output feature after three layers of upsampling operation is: Among them, up is upsampling, H is height, and W is width.
9. The image fusion module according to claim 1, characterized in that: The image fusion module is used to fuse the semantic features of different modalities. It fuses the semantic features of two modalities, namely the penumbra feature information from the DWI modality and the lesion core area information from the SWI modality. The network structure of IFM is similar to that of SSM. There are also three upsampling blocks for restoring resolution and a block for adjusting the number of channels. The final output of IFM is a weight matrix with a value between 0 and 1. It represents the weight of the DWI modality and the SWI modality in the fused image. Therefore, to obtain the fusion result, the fusion operation should be performed. The mathematical expression of the fusion operation process is: F fusion =(F weight ⊙S1)+(1-F weight )⊙S2 Among them, F weight is the weight matrix output by the IFM module, S1 and S2 are the images of DWI and SWI modalities respectively, and X is the element-by-element multiplication between tensors.
10. The semantic segmentation module and image fusion module according to claim 7 or 9, characterized in that: The loss is determined by the loss function. The mathematical expression of the training loss function of the segmentation network is defined as: L=L seg +ψL fusin Among them, ψ is the trade-off parameter variable, L seg is the loss of the semantic segmentation module for learning semantic features, L fusin is the loss of the image fusion module.
11. The training loss function of the segmentation network according to claim 10, characterized in that The segmentation loss function is a combination of binary cross entropy loss and Dice loss, which provides better gradient information and can measure the overlap between the predicted area and the true value area. The loss of the semantic segmentation module that learns semantic features is mathematically expressed as: L seg =0.5×L BCE +L Dice Among them, L BCE is the binary cross entropy loss, L Dice is the dice loss.
12. The training loss function of the segmentation network according to claim 11, characterized in that The mathematical expression of the loss of the image fusion module for auxiliary semantic segmentation is: Where N is the number of pixels in the image, C is the number of classes, is the output probability of the pixel set of class j, y ij is the ground truth label of the pixel in class i, and ε is a small constant.
13. The image fusion module according to claim 9, characterized in that: The image fusion module assists semantic segmentation and fuses semantic feature information of different modalities. The loss function of image fusion is defined as: L fusion =L pixel +L ssim Among them, L pixel To measure the pixel-level difference between the fused image and the target image, L ssim is image brightness, contrast and structural similarity.
14. The image fusion module according to claim 13, characterized in that: We can use L ssim Extract the structural information of the source image. The mathematical expressions of Lpixel and Lssim are: L ssin =η(1-SSIM(F,S1))+(1-SSIM(F,S2) Where S1 is the DWI modality source image, S2 is the SWI modality source image, F is the fused image, N is the number of pixels of the fused image, ||·|| and SSIM represent the tensor Frobenius norm and structural similarity measure, respectively, and ξ and η are weight parameters.