Image segmentation method, device, equipment and storage medium

Abstract

The present invention provides an image segmentation method, apparatus, device, and storage medium, comprising: obtaining a three-dimensional image to be segmented; performing pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence; inputting the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model; the image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image. The present invention uses the pulse model to perform segmentation prediction on the pulse sequence of the three-dimensional image to be segmented. Since the neurons in the pulse model are only active when receiving or emitting spike signals, the time consumption of deep learning neural networks can be reduced. The image segmentation pulse model obtained based on multi-scale feature fusion training can accurately segment small targets in feature maps of different scales, thereby improving the efficiency and accuracy of three-dimensional image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, apparatus, device and storage medium. Background Art

[0002] Three-dimensional volume image segmentation in medical images is mainly used to segment organs, tumors, blood vessels and other areas in three-dimensional images, which is of great help in disease diagnosis, monitoring and specifying corresponding treatment plans.

[0003] Currently, three-dimensional volume image segmentation in medical images is mainly performed through manual segmentation and image segmentation based on CNN deep learning network models. However, the manual segmentation process is cumbersome, time-consuming and labor-intensive, and is easily mixed with human segmentation errors, resulting in low image segmentation accuracy. In addition, due to the redundancy of deep learning networks, the efficiency of semantic segmentation of three-dimensional images using CNN deep learning network models is low. Summary of the Invention

[0004] The present invention provides an image segmentation method, apparatus, device and storage medium, aiming to solve the technical problem of low efficiency and accuracy in three-dimensional image segmentation.

[0005] The present invention provides an image segmentation method, comprising:

[0006] Acquire a three-dimensional image to be segmented;

[0007] performing pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence;

[0008] Inputting the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model;

[0009] The image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

[0010] Optionally, according to an image segmentation method provided by the present invention, the image segmentation impulse model includes an encoding module, a decoding module and a segmentation output module, wherein:

[0011] The encoding module includes a plurality of cascaded encoding units; each encoding unit except the last encoding unit includes a first pulse convolution layer and a pulse downsampling layer, and the last encoding unit includes a first pulse convolution layer;

[0012] The decoding module includes several cascaded decoding units, each of which includes a pulse upsampling layer, a multi-scale feature fusion layer, a feature connection layer and a second pulse convolution layer;

[0013] The output of the last decoding unit serves as the input of the segmentation output module.

[0014] Optionally, according to an image segmentation method provided by the present invention, inputting the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model includes:

[0015] Inputting the pulse sequence into the first pulse convolution layer in the first encoding unit to obtain a coding feature map output by the first pulse convolution layer;

[0016] Downsampling the encoding feature map through the pulse downsampling layer in the first encoding unit to obtain the downsampled feature map, and using the downsampled feature map as the input of the next encoding unit until the encoding feature map output by the first pulse convolution layer in the last encoding unit is obtained;

[0017] fusing the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit through the multi-scale feature fusion layer in the first decoding unit to obtain a first fused feature map;

[0018] Upsampling the encoding feature map output by the last encoding unit through the pulse upsampling layer in the first decoding unit to obtain a first upsampled feature map;

[0019] Performing feature splicing on the first fused feature map and the first upsampled feature map through a feature connection layer in the first decoding unit to obtain a first spliced ​​feature map;

[0020] Performing convolution processing on the first spliced ​​feature map through the second pulse convolution layer in the first decoding unit to obtain a decoding feature map output by the first decoding unit, and using the decoding feature map output by the first decoding unit as the input of the next decoding unit;

[0021] fusing the decoded feature map output by the first decoding unit and the encoded feature map output by the encoding unit at the same depth level as the next decoding unit through a multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map;

[0022] Upsampling the decoded feature map output by the first decoding unit through the pulse upsampling layer in the next decoding unit to obtain a second upsampled feature map;

[0023] Performing feature splicing on the second fused feature map and the second up-sampled feature map through a feature connection layer in the next decoding unit to obtain a second spliced ​​feature map;

[0024] Performing convolution processing on the second splicing feature map through a second pulse convolution layer in the next decoding unit to obtain a decoding feature map output by the next decoding unit;

[0025] Returning to the step of fusing the decoding feature map output by the first decoding unit and the encoding feature map output by the encoding unit at the same depth level as the next decoding unit through the multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map, until a decoding feature map output by the last decoding module is obtained;

[0026] The decoding feature map of the last decoding module is input into the segmentation output module to obtain the image segmentation result output by the segmentation output module.

[0027] Optionally, according to an image segmentation method provided by the present invention, the first pulse convolution layer includes a cascaded first convolution layer, a first normalization layer and a first pulse emission layer;

[0028] The pulse downsampling layer includes a cascaded second convolutional layer and a second pulse emission layer;

[0029] The pulse upsampling layer includes a cascaded deconvolution layer and a third pulse emission layer;

[0030] The second pulse convolution layer includes a cascaded third convolution layer, a second normalization layer and a fourth pulse emission layer.

[0031] Optionally, according to an image segmentation method provided by the present invention, the multi-scale feature fusion layer includes a fourth convolution layer, a fifth convolution layer, a feature fusion layer, a third pulse convolution layer and a feature point product layer;

[0032] The step of fusing the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit through the multi-scale feature fusion layer in the first decoding unit to obtain a fused feature map includes:

[0033] Performing convolution processing on the encoding feature map output by the last encoding unit through the fourth convolution layer to obtain a first convolution feature map; and performing convolution processing on the encoding feature map output by the encoding unit at the same depth level through the fifth convolution layer to obtain a second convolution feature map;

[0034] The first convolution feature map and the second convolution feature map are summed up by the feature fusion layer to obtain a target feature map;

[0035] Performing convolution processing on the target feature map through the third pulse convolution layer to obtain a focus feature map;

[0036] The feature dot product layer performs dot product processing on the focus feature map and the encoded feature map output by the last encoding unit to obtain the fused feature map.

[0037] Optionally, according to an image segmentation method provided by the present invention, the image segmentation impulse model is trained based on the following steps:

[0038] Obtain several original three-dimensional images after data enhancement processing;

[0039] Performing pulse coding processing on each of the original three-dimensional images to obtain a plurality of pulse sequence samples corresponding to a preset time step;

[0040] Based on each of the pulse sequence samples and the image segmentation label of the original three-dimensional image, the pulse model to be trained is iteratively trained to obtain the image segmentation pulse model.

[0041] Optionally, according to an image segmentation method provided by the present invention, the multi-scale feature fusion iterative training is performed on the pulse model to be trained based on each of the pulse sequence samples and the image segmentation label of the original three-dimensional image to obtain the image segmentation pulse model, including:

[0042] Inputting the pulse sequence samples into the pulse model to be trained to obtain a predicted segmentation result output by the pulse model to be trained;

[0043] Calculating a model loss value based on the predicted segmentation result and the image segmentation label;

[0044] Based on the model loss value obtained in each iteration, the parameters of the pulse model to be trained are updated using a gradient substitution algorithm to obtain the image segmentation pulse model.

[0045] The present invention also provides an image segmentation device, comprising:

[0046] An acquisition module, used for acquiring a three-dimensional image to be segmented;

[0047] a pulse coding module, configured to perform pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence;

[0048] An image segmentation module, configured to input the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model;

[0049] The image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

[0050] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described image segmentation methods when executing the program.

[0051] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned image segmentation methods when executed by a processor.

[0052] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned image segmentation methods.

[0053] The image segmentation method, apparatus, device and storage medium provided by the present invention encode the three-dimensional image to be segmented into a pulse sequence, and use the pulse model to perform segmentation prediction on the pulse sequence. Since the neurons in the pulse model are only active when receiving or sending spike signals, the time consumption of ordinary deep learning neural networks is greatly reduced, thereby improving the efficiency of three-dimensional image segmentation. In addition, the image segmentation pulse model obtained through multi-scale feature fusion training can accurately segment small targets in feature maps of different scales, thereby improving the accuracy of three-dimensional image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 It is a flowchart of the image segmentation method provided by the present invention;

[0056] Figure 2 It is a structural diagram of the image segmentation impulse model provided by the present invention;

[0057] Figure 3 It is a schematic diagram of the structure of the multi-scale feature fusion layer in the image segmentation impulse model provided by the present invention;

[0058] Figure 4 It is a structural schematic diagram of the image segmentation device provided by the present invention;

[0059] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] The terms used in one or more embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present invention. The singular forms "a", "the" and "the" used in one or more embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0062] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present invention, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present invention, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0063] The following combination Figure 1-Figure 3 Exemplary embodiments of the present invention are described in detail.

[0064] Figure 1 It is a flow chart of the image segmentation method provided by the present invention, such as Figure 1 As shown, the image segmentation method includes:

[0065] Step 11, obtaining a three-dimensional image to be segmented;

[0066] It should be noted that the three-dimensional image to be segmented is a 3D medical image, and the dimensions of the three-dimensional image to be segmented include the depth, length, width and number of channels of the image.

[0067] Step 12: performing pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence;

[0068] It should be noted that the pulse coding process includes Poisson coding, which is a rate-based coding that can encode images into discrete pulse sequences. The embodiment of the present invention introduces a time step variable T, thereby applying a Poisson coding process at each time step to convert static pixel data in the three-dimensional image to be segmented into a mapping of the pulse sequence at each preset time step, so that each pixel generates T pulse sequences within T preset time steps.

[0069] Step 13, inputting the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model;

[0070] The image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

[0071] Specifically, a pulse sequence is input into an image segmentation pulse model, and an image segmentation result is obtained based on the output of the image segmentation pulse model. The image segmentation pulse model is trained using multi-scale feature fusion based on pulse sequence samples generated by pulse encoding of the original three-dimensional image and the image segmentation labels of the original three-dimensional image. It is understood that after training, the image segmentation pulse model can effectively segment the three-dimensional image to be segmented, thereby obtaining an image segmentation result for the three-dimensional image to be segmented.

[0072] It should be noted that the image segmentation pulse model is a model coupled with an encoder-decoder architecture based on a 3D U-net model, and the image segmentation pulse model is a pulse neural network model in the form of biological neurons. The image segmentation pulse model includes an encoding module, a decoding module and a segmentation output module.

[0073] The encoding module includes several cascaded encoding units; each encoding unit except the last encoding unit includes a first pulse convolution layer and a pulse downsampling layer, and the last encoding unit includes a first pulse convolution layer; the decoding module includes several cascaded decoding units, each decoding unit includes a pulse upsampling layer, a multi-scale feature fusion layer, a feature connection layer and a second pulse convolution layer; the output of the last decoding unit serves as the input of the segmentation output module.

[0074] like Figure 2As shown in the figure, the encoding module includes 4 encoding units, the decoding module includes 3 decoding units, the encoding feature map output by the fourth encoding unit is directly used as an input of the first decoding unit, and the encoding unit is connected to the multi-scale feature fusion layer in the decoding unit at the same depth level, wherein the third encoding unit and the first decoding unit are at the same depth level, the second encoding unit and the second decoding unit are at the same depth level, and the first encoding unit and the third decoding unit are at the same depth level.

[0075] Specifically, the pulse sequence is input into the first pulse convolution layer in the first encoding unit, so as to use the first pulse convolution layer to perform convolution processing on the pulse sequence to obtain a convolution feature map, and then the convolution feature map is input into the pulse downsampling layer in the first encoding unit to obtain a downsampling feature map output by the pulse downsampling layer, wherein the downsampling feature map output by the previous encoding unit is used as the input of the next encoding unit until the encoding feature map output by the last encoding unit is obtained. Furthermore, in the first decoding unit, the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit are used as the input of the first decoding unit, so as to fusion the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit through the multi-scale feature fusion layer in the first decoding unit. The coding feature map output by the unit is fused to obtain a first fused feature map, and the coding feature map output by the last coding unit is upsampled through the pulse upsampling layer in the first decoding unit to obtain a first upsampled feature map, and then the first fused feature map and the first upsampled feature map are feature spliced ​​through the feature connection layer in the first decoding unit to obtain a first spliced ​​feature map. After obtaining the spliced ​​feature map output by the feature connection layer, the spliced ​​feature map is convolved through the second pulse convolution layer in the first decoding unit to obtain a decoding feature map, wherein the decoding feature map output by the previous decoding unit is used as the input of the next decoding unit, and the cycle is repeated until the decoding feature map output by the last decoding unit is obtained. Furthermore, the decoding feature map output by the last decoding unit is input into the segmentation output module to obtain the image segmentation result output by the segmentation output module.

[0076] Through the above scheme, this embodiment encodes the three-dimensional image to be segmented into a pulse sequence, and uses a pulse model to perform segmentation prediction on the pulse sequence. Since the neurons in the pulse model are only active when receiving or sending spike signals, the time consumption of ordinary deep learning neural networks is greatly reduced, thereby improving the efficiency of three-dimensional image segmentation. In addition, the image segmentation pulse model obtained through multi-scale feature fusion training can accurately segment small targets in feature maps of different scales, thereby improving the accuracy of three-dimensional image segmentation.

[0077] In one embodiment, the above step 13: inputting the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model includes:

[0078] Step 131: input the pulse sequence into the first pulse convolution layer in the first encoding unit to obtain a coding feature map output by the first pulse convolution layer;

[0079] Specifically, the pulse sequence is input into the first pulse convolution layer in the first encoding unit, and the pulse sequence is convolved by the first pulse convolution layer to obtain a coding feature map output by the first pulse convolution layer.

[0080] It should be noted that the first pulse convolution layer includes a cascaded first convolution layer, a first normalization layer, and a first pulse discharge layer. Preferably, the number of the first pulse convolution layers is 2. The first convolution layer is a 3D convolution layer, the first normalization layer is a BatchNorm normalization layer, and the first pulse discharge layer is a parameterized LIF neuron model based on the neuron dynamics equation. The LIF neuron model accumulates the membrane potential in the form of integration and decays exponentially with time when there is no input. When the charge accumulates to a certain level, that is, the membrane potential reaches a preset threshold, the neuron generates and discharges a pulse, and then resets the membrane potential.

[0081] Its neural dynamics equation is specifically:

[0082]

[0083] in, w is a learnable parameter, V[t] represents the membrane potential of the LIF neuron model at time t, X[t] represents the input of the LIF neuron model at time t, V reset Indicates resetting the membrane potential.

[0084] Step 132: downsample the encoding feature map through the pulse downsampling layer in the first encoding unit to obtain the downsampled feature map, and use the downsampled feature map as the input of the next encoding unit until the encoding feature map output by the first pulse convolution layer in the last encoding unit is obtained;

[0085] It should be noted that the pulse downsampling layer includes a cascaded second convolution layer and a second pulse emission layer, the second convolution layer is a 3D convolution layer, and the second pulse emission layer has basically the same structure and function as the first pulse emission layer in step 131, which will not be repeated here.

[0086] It can be understood that if the size of the encoding feature map is 32×128×128×128, the encoding feature map is downsampled by the pulse downsampling layer in the first encoding unit to obtain a downsampled feature map of 32×64×64×64 to reduce the length, width and depth of the feature map, and then the 32×64×64×64 feature map is used as the input of the first pulse convolution layer in the next encoding unit, and so on, until the encoding feature map output by the first pulse convolution layer in the last encoding unit is obtained.

[0087] Step 133: fusing the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit through the multi-scale feature fusion layer in the first decoding unit to obtain a first fused feature map;

[0088] Step 134: upsampling the encoding feature map output by the last encoding unit through the pulse upsampling layer in the first decoding unit to obtain a first upsampled feature map;

[0089] It should be noted that, since the feature map obtained by upsampling the coded feature map output by the last coding unit is different in size from the coded feature map output by the coding unit at the same depth level as the first decoding unit, direct feature splicing cannot be performed. In this embodiment, a multi-scale feature fusion layer is provided to perform feature fusion processing on the coded feature map output by the last coding unit and the coded feature map output by the coding unit at the same depth level as the first decoding unit.

[0090] Specifically, the multi-scale feature fusion layer includes a fourth convolution layer, a fifth convolution layer, a feature fusion layer, a third pulse convolution layer and a feature point convolution layer, wherein the third pulse convolution layer includes a cascaded sixth convolution layer and a fifth pulse discharge layer, and the fourth convolution layer, the fifth convolution layer and the sixth convolution layer are all 3D convolution layers. The encoding feature map output by the last encoding unit is convolved through the fourth convolution layer to obtain a first convolution feature map, and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit is convolved through the fifth convolution layer to obtain a second convolution feature map, and then the first convolution feature map and the second convolution feature map are added through the feature fusion layer to obtain a target feature map. Further, the target feature map is convolved through the third pulse convolution layer to obtain a focus feature map, and then the focus feature map and the encoding feature map output by the last encoding unit are dot-product processed through the feature point convolution layer to obtain the fusion feature map, thereby realizing the fusion of feature maps of different scales.

[0091] In addition, the coding feature map output by the last coding unit is input to the pulse upsampling layer in the first decoding unit, so as to perform upsampling processing on the coding feature map output by the last coding unit through the pulse upsampling layer to obtain a first upsampling feature map, wherein the pulse upsampling layer includes a cascaded deconvolution layer and a third pulse discharge layer, wherein the deconvolution layer is a 3D convolution layer, and the third pulse discharge layer and the first pulse discharge layer in step 131 have basically the same structure and function, which will not be repeated here.

[0092] In addition, the order of step 133 and step 134 can be to execute step 133 first and then execute step 134, or to execute step 134 first and then execute step 133, which is not specifically limited here.

[0093] Step 135: performing feature splicing on the first fused feature map and the first upsampled feature map through a feature connection layer in the first decoding unit to obtain a first spliced ​​feature map;

[0094] It can be understood that if the first fused feature map with a size of 128×32×32×32 is obtained by fusion processing through the multi-scale feature fusion layer, and the first up-sampled feature map with a size of 128×32×32×32 is obtained by upsampling processing through the pulse upsampling layer, the first feature map with a size of 256×32×32×32 is obtained after feature splicing.

[0095] Step 136: performing convolution processing on the first spliced ​​feature map through the second pulse convolution layer in the first decoding unit to obtain a decoding feature map output by the first decoding unit, and using the decoding feature map output by the first decoding unit as the input of the next decoding unit;

[0096] It should be noted that the second pulse convolution layer includes a cascaded third convolution layer, a second normalization layer, and a fourth pulse discharge layer. Preferably, in the model network structure, the number of the second pulse convolution layer is 2, wherein the third convolution layer is a 3D convolution layer, and the structure and function of the fourth pulse discharge layer are basically the same as those of the first pulse discharge layer in step 131, which will not be repeated here. Specifically, the first spliced ​​feature map is input into the second pulse convolution layer in the first decoding unit, so that the first spliced ​​feature map is convolved using the second pulse convolution layer to obtain a decoding feature map output by the first decoding unit.

[0097] Step 137: Fusing the decoded feature map output by the first decoding unit and the encoded feature map output by the encoding unit at the same depth level as the next decoding unit through a multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map;

[0098] Step 138: upsampling the decoded feature map output by the first decoding unit through the pulse upsampling layer in the next decoding unit to obtain a second upsampled feature map;

[0099] Step 139: performing feature splicing on the second fused feature map and the second up-sampled feature map through a feature connection layer in the next decoding unit to obtain a second spliced ​​feature map;

[0100] Step 1310: performing convolution processing on the second spliced ​​feature map through a second pulse convolution layer in the next decoding unit to obtain a decoding feature map output by the next decoding unit;

[0101] Returning to the step of fusing the decoding feature map output by the first decoding unit and the encoding feature map output by the encoding unit at the same depth level as the next decoding unit through the multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map, until a decoding feature map output by the last decoding module is obtained;

[0102] Specifically, the decoding feature map output by the first decoding unit is input into the next decoding unit, so as to fuse the decoding feature map and the encoding feature map output by the encoding unit at the same depth level as the next decoding unit through the multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map, and upsample the decoding feature map through the pulse upsampling layer in the next decoding unit. It should be noted that the specific implementation process of steps 137 to 1310 in this embodiment is basically the same as the implementation process of steps 133 to 136 above, that is, the decoding process of each decoding unit is basically the same, which will not be repeated here until the decoding feature map output by the last decoding module is obtained.

[0103] Step 1311: Input the decoding feature map of the last decoding module into the segmentation output module to obtain the image segmentation result output by the segmentation output module.

[0104] Specifically, the decoding feature map of the last decoding module is input into the segmentation output module to obtain the image segmentation result output by the segmentation output module.

[0105] Through the above-mentioned scheme, the embodiment of the present invention realizes the segmentation of three-dimensional images based on biological neurons, and neurons are only active when receiving or sending spike signals, which effectively reduces the time consumption of ordinary deep learning neural networks. In addition, multi-scale feature fusion is introduced in the decoding process, which can accurately segment small targets in feature maps of different scales, thereby improving the accuracy of three-dimensional image segmentation.

[0106] In one embodiment, the multi-scale feature fusion layer in the first decoding unit fuses the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit to obtain a fused feature map, including:

[0107] The coding feature map output by the last coding unit is convolved by the fourth convolution layer to obtain a first convolution feature map; and the coding feature map output by the coding unit at the same depth level is convolved by the fifth convolution layer to obtain a second convolution feature map; the first convolution feature map and the second convolution feature map are added together by the feature fusion layer to obtain a target feature map; the target feature map is convolved by the third pulse convolution layer to obtain a focus feature map; the focus feature map and the coding feature map output by the last coding unit are dot-producted by the feature dot product layer to obtain the fusion feature map.

[0108] It should be noted that Figure 3This is a schematic diagram of the structure of the multi-scale feature fusion layer in the image segmentation impulse model provided by the present invention. Figure 3 As shown, the multi-scale feature fusion layer includes a fourth convolution layer, a fifth convolution layer, a feature fusion layer, a third pulse convolution layer and a feature point convolution layer. Specifically, the coding feature map output by the last coding unit is input into the fourth convolution layer, so as to use the fourth convolution layer to perform convolution processing on the coding feature map output by the last coding unit to obtain a first convolution feature map; and the coding feature map output by the coding unit at the same depth level is input into the fifth convolution layer, so as to use the fifth convolution layer to perform convolution processing on the coding feature map output by the coding unit at the same depth level. Convolution processing is performed on the first convolution feature map to obtain a second convolution feature map. Further, the first convolution feature map and the second convolution feature map are input into the feature fusion layer, so as to use the feature fusion layer to sum the first convolution feature map and the second convolution feature map to obtain a target feature map, and then the target feature map is convolved by the third pulse convolution layer to obtain an attention feature map, and then the attention feature map and the encoding feature map output by the last encoding unit are dot-product processed by the attention feature map to re-weight the image features to obtain the fusion feature map.

[0109] Through the above scheme, the embodiment of the present invention realizes the fusion of encoding feature maps and decoding feature maps of different scales through the multi-scale feature fusion layer during the decoding process, thereby accurately learning and segmenting small targets of feature maps of different scales, thereby improving the accuracy of image segmentation.

[0110] In one embodiment, the image segmentation pulse model is trained based on the following steps: obtaining several original three-dimensional images after data enhancement processing; performing pulse coding processing on each of the original three-dimensional images to obtain several pulse sequence samples corresponding to preset time steps; based on each of the pulse sequence samples and the image segmentation labels of the original three-dimensional images, performing multi-scale feature fusion iterative training on the pulse model to be trained to obtain the image segmentation pulse model.

[0111] Among them, the pulse model to be trained is iteratively trained based on each of the pulse sequence samples and the image segmentation label of the original three-dimensional image to obtain the image segmentation pulse model, including: inputting the pulse sequence samples into the pulse model to be trained to obtain the predicted segmentation result output by the pulse model to be trained; calculating the model loss value based on the predicted segmentation result and the image segmentation label; based on the model loss value obtained in each iteration, using the gradient substitution algorithm to update the parameters of the pulse model to be trained to obtain the image segmentation pulse model.

[0112] Specifically, a number of sample three-dimensional images are obtained, and data enhancement processing is performed on the sample three-dimensional images to obtain each of the original three-dimensional images, wherein the dimensions of the original three-dimensional images can be expressed as [C, D, H, W], C represents the number of channels of the original three-dimensional image, D represents the depth of the original three-dimensional image, H represents the length of the original three-dimensional image, and W represents the width of the original three-dimensional image. The data enhancement processing includes processing methods such as flipping the image up and down, flipping the image left and right, and randomly cropping the image. Then, the following steps are performed for each original three-dimensional image: pulse coding processing is performed on the original three-dimensional image to obtain a pulse sequence with spatiotemporal information within a number of preset time steps, for example, a pulse sequence within T preset time steps is obtained. At this time, the dimensions of the pulse sequence can be expressed as [T, C, D, H, W].

[0113] Furthermore, each of the pulse sequences is input into the pulse model to be trained to obtain a predicted segmentation result output by the pulse model to be trained, and then a model loss value is calculated based on the predicted segmentation result and the image segmentation label, wherein the model loss value calculation formula is as follows:

[0114]

[0115] Among them, A represents the predicted segmentation result, B represents the image segmentation label, and L DICE (A, B) represents the model loss value. In another embodiment, the pulse sequences corresponding to the batch of original 3D images are input into the pulse model to be trained for iterative training. In this case, the input dimensions can be expressed as [T, N, C, D, H, W], where N represents the number of original 3D image samples input into the pulse model to be trained each time.

[0116] On this basis, in other embodiments, the model loss value can be calculated by setting the loss function according to actual needs, and no specific limitation is made here. After the model loss value is calculated, the training process ends. Taking into account the non-differentiability of the pulse emission function, the gradient substitution algorithm is used to update the model parameters in the pulse model to be trained during the back propagation process. For example, the gradient of the sigmoid, tanh and other functions replaces the gradient value at the corresponding position, and then the next training is carried out. During the training process, it is determined whether the updated pulse model to be trained meets the preset training end conditions. If so, the updated pulse model to be trained is used as the image segmentation pulse model. If not, the model training continues, wherein the preset training end conditions include loss convergence and reaching the maximum iteration number threshold.

[0117] The present invention realizes, through the above scheme, training the image segmentation pulse model, which is conducive to controlling the loss value of the image segmentation pulse model within a preset range, thereby facilitating improving the accuracy of image segmentation performed by the image segmentation pulse model.

[0118] The image segmentation device provided by the present invention is described below. The image segmentation device described below and the image segmentation method described above can be referred to in correspondence with each other.

[0119] Figure 4 Schematic diagram of the structure of the image segmentation device provided by the present invention, such as Figure 4 As shown, the device includes:

[0120] An acquisition module 41 is used to acquire a three-dimensional image to be segmented;

[0121] a pulse coding module 42 for performing pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence;

[0122] An image segmentation module 43 is configured to input the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model;

[0123] The image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

[0124] The image segmentation device further includes:

[0125] The image segmentation impulse model includes an encoding module, a decoding module and a segmentation output module, wherein:

[0126] The encoding module includes a plurality of cascaded encoding units; each encoding unit except the last encoding unit includes a first pulse convolution layer and a pulse downsampling layer, and the last encoding unit includes a first pulse convolution layer;

[0127] The decoding module includes several cascaded decoding units, each of which includes a pulse upsampling layer, a multi-scale feature fusion layer, a feature connection layer and a second pulse convolution layer;

[0128] The output of the last decoding unit serves as the input of the segmentation output module.

[0129] The image segmentation module 43 is further configured to:

[0130] Inputting the pulse sequence into the first pulse convolution layer in the first encoding unit to obtain a coding feature map output by the first pulse convolution layer;

[0131] Downsampling the encoding feature map through the pulse downsampling layer in the first encoding unit to obtain the downsampled feature map, and using the downsampled feature map as the input of the next encoding unit until the encoding feature map output by the first pulse convolution layer in the last encoding unit is obtained;

[0132] fusing the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit through the multi-scale feature fusion layer in the first decoding unit to obtain a first fused feature map;

[0133] Upsampling the encoding feature map output by the last encoding unit through the pulse upsampling layer in the first decoding unit to obtain a first upsampled feature map;

[0134] Performing feature splicing on the first fused feature map and the first upsampled feature map through a feature connection layer in the first decoding unit to obtain a first spliced ​​feature map;

[0135] Performing convolution processing on the first spliced ​​feature map through the second pulse convolution layer in the first decoding unit to obtain a decoding feature map output by the first decoding unit, and using the decoding feature map output by the first decoding unit as the input of the next decoding unit;

[0136] fusing the decoded feature map output by the first decoding unit and the encoded feature map output by the encoding unit at the same depth level as the next decoding unit through a multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map;

[0137] Upsampling the decoded feature map output by the first decoding unit through the pulse upsampling layer in the next decoding unit to obtain a second upsampled feature map;

[0138] Performing feature splicing on the second fused feature map and the second up-sampled feature map through a feature connection layer in the next decoding unit to obtain a second spliced ​​feature map;

[0139] Performing convolution processing on the second splicing feature map through a second pulse convolution layer in the next decoding unit to obtain a decoding feature map output by the next decoding unit;

[0140] Returning to the step of fusing the decoding feature map output by the first decoding unit and the encoding feature map output by the encoding unit at the same depth level as the next decoding unit through the multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map, until a decoding feature map output by the last decoding module is obtained;

[0141] The decoding feature map of the last decoding module is input into the segmentation output module to obtain the image segmentation result output by the segmentation output module.

[0142] The image segmentation device further includes:

[0143] The multi-scale feature fusion layer includes a fourth convolution layer, a fifth convolution layer, a feature fusion layer, a third pulse convolution layer and a feature point product layer.

[0144] The image segmentation module 43 is further configured to:

[0145] Performing convolution processing on the encoding feature map output by the last encoding unit through the fourth convolution layer to obtain a first convolution feature map; and performing convolution processing on the encoding feature map output by the encoding unit at the same depth level through the fifth convolution layer to obtain a second convolution feature map;

[0146] The first convolution feature map and the second convolution feature map are added together by the feature fusion layer to obtain a target feature map;

[0147] Performing convolution processing on the target feature map through the third pulse convolution layer to obtain a focus feature map;

[0148] The feature dot product layer performs dot product processing on the focus feature map and the encoded feature map output by the last encoding unit to obtain the fused feature map.

[0149] The image segmentation device further includes:

[0150] The first pulse convolution layer includes a cascaded first convolution layer, a first normalization layer, and a first pulse emission layer;

[0151] The pulse downsampling layer includes a cascaded second convolutional layer and a second pulse emission layer;

[0152] The pulse upsampling layer includes a cascaded deconvolution layer and a third pulse emission layer;

[0153] The second pulse convolution layer includes a cascaded third convolution layer, a second normalization layer and a fourth pulse emission layer.

[0154] The image segmentation device further includes:

[0155] Obtain several original three-dimensional images after data enhancement processing;

[0156] Performing pulse coding processing on each of the original three-dimensional images to obtain a plurality of pulse sequence samples corresponding to a preset time step;

[0157] Based on each of the pulse sequence samples and the image segmentation label of the original three-dimensional image, multi-scale feature fusion iterative training is performed on the pulse model to be trained to obtain the image segmentation pulse model.

[0158] The image segmentation device further includes:

[0159] Inputting the pulse sequence samples into the pulse model to be trained to obtain a predicted segmentation result output by the pulse model to be trained;

[0160] Calculating a model loss value based on the predicted segmentation result and the image segmentation label;

[0161] Based on the model loss value obtained in each iteration, the parameters of the pulse model to be trained are updated using a gradient substitution algorithm to obtain the image segmentation pulse model.

[0162] It should be noted here that the above-mentioned device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0163] Figure 5 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a memory 520, a communication interface 530 and a communication bus 540, wherein the processor 510, the memory 520, and the communication interface 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 520 to execute the image segmentation method, which includes: obtaining a three-dimensional image to be segmented; performing pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence; inputting the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model; wherein the image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

[0164] In addition, the logic instructions in the above-mentioned memory 520 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0165] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image segmentation method provided by the above-mentioned methods, the method comprising: obtaining a three-dimensional image to be segmented; performing pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence; inputting the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model; wherein the image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image segmentation method provided by the above methods, which includes: obtaining a three-dimensional image to be segmented; performing pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence; inputting the pulse sequence into an image segmentation pulse model to obtain an image segmentation result output by the image segmentation pulse model; wherein the image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0168] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image segmentation method, characterized in that: include: Acquire a three-dimensional image to be segmented; performing pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence; Inputting the pulse sequence into the first pulse convolution layer in the first encoding unit to obtain an encoding feature map output by the first pulse convolution layer; Downsampling the encoding feature map through the pulse downsampling layer in the first encoding unit to obtain a downsampled feature map, and using the downsampled feature map as the input of the next encoding unit until the encoding feature map output by the first pulse convolution layer in the last encoding unit is obtained; fusing the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit through the multi-scale feature fusion layer in the first decoding unit to obtain a first fused feature map; Upsampling the encoding feature map output by the last encoding unit through the pulse upsampling layer in the first decoding unit to obtain a first upsampled feature map; Performing feature splicing on the first fused feature map and the first upsampled feature map through a feature connection layer in the first decoding unit to obtain a first spliced ​​feature map; Performing convolution processing on the first spliced ​​feature map through the second pulse convolution layer in the first decoding unit to obtain a decoding feature map output by the first decoding unit, and using the decoding feature map output by the first decoding unit as the input of the next decoding unit; fusing the decoded feature map output by the first decoding unit and the encoded feature map output by the encoding unit at the same depth level as the next decoding unit through a multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map; Upsampling the decoded feature map output by the first decoding unit through the pulse upsampling layer in the next decoding unit to obtain a second upsampled feature map; Performing feature splicing on the second fused feature map and the second up-sampled feature map through a feature connection layer in the next decoding unit to obtain a second spliced ​​feature map; Performing convolution processing on the second splicing feature map through a second pulse convolution layer in the next decoding unit to obtain a decoding feature map output by the next decoding unit; Returning to the step of fusing the decoding feature map output by the first decoding unit and the encoding feature map output by the encoding unit at the same depth level as the next decoding unit through the multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map, until a decoding feature map output by the last decoding module in the image segmentation impulse model is obtained; Inputting the decoding feature map of the last decoding module into the segmentation output module in the image segmentation impulse model to obtain the image segmentation result output by the segmentation output module; The image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

2. The image segmentation method according to claim 1, wherein: The image segmentation impulse model includes an encoding module, a decoding module and a segmentation output module, wherein: The encoding module includes a plurality of cascaded encoding units; each encoding unit except the last encoding unit includes a first pulse convolution layer and a pulse downsampling layer, and the last encoding unit includes a first pulse convolution layer; The decoding module includes several cascaded decoding units, each of which includes a pulse upsampling layer, a multi-scale feature fusion layer, a feature connection layer and a second pulse convolution layer; The output of the last decoding unit serves as the input of the segmentation output module.

3. The image segmentation method according to claim 1, wherein: The multi-scale feature fusion layer includes a fourth convolution layer, a fifth convolution layer, a feature fusion layer, a third pulse convolution layer and a feature point product layer; The step of fusing the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit through the multi-scale feature fusion layer in the first decoding unit to obtain a fused feature map includes: Performing convolution processing on the encoding feature map output by the last encoding unit through the fourth convolution layer to obtain a first convolution feature map; and performing convolution processing on the encoding feature map output by the encoding unit at the same depth level through the fifth convolution layer to obtain a second convolution feature map; The first convolution feature map and the second convolution feature map are summed up by the feature fusion layer to obtain a target feature map; Performing convolution processing on the target feature map through the third pulse convolution layer to obtain a focus feature map; The feature dot product layer performs dot product processing on the focus feature map and the encoded feature map output by the last encoding unit to obtain the fused feature map.

4. The image segmentation method according to claim 2, wherein: The first pulse convolution layer includes a cascaded first convolution layer, a first normalization layer and a first pulse emission layer; The pulse downsampling layer includes a cascaded second convolutional layer and a second pulse emission layer; The pulse upsampling layer includes a cascaded deconvolution layer and a third pulse emission layer; The second pulse convolution layer includes a cascaded third convolution layer, a second normalization layer and a fourth pulse emission layer.

5. The image segmentation method according to claim 2, wherein: The image segmentation impulse model is trained based on the following steps: Obtain several original three-dimensional images after data enhancement processing; Performing pulse coding processing on each of the original three-dimensional images to obtain a plurality of pulse sequence samples corresponding to a preset time step; Based on each of the pulse sequence samples and the image segmentation label of the original three-dimensional image, multi-scale feature fusion iterative training is performed on the pulse model to be trained to obtain the image segmentation pulse model.

6. The image segmentation method according to claim 5, characterized in that: The iterative training of the pulse model to be trained based on each of the pulse sequence samples and the image segmentation label of the original three-dimensional image to obtain the image segmentation pulse model includes: Inputting the pulse sequence samples into the pulse model to be trained to obtain a predicted segmentation result output by the pulse model to be trained; Calculating a model loss value based on the predicted segmentation result and the image segmentation label; Based on the model loss value obtained in each iteration, the parameters of the pulse model to be trained are updated using a gradient substitution algorithm to obtain the image segmentation pulse model.

7. An image segmentation device, characterized in that: include: An acquisition module, used for acquiring a three-dimensional image to be segmented; a pulse coding module, configured to perform pulse coding processing on the three-dimensional image to be segmented to obtain a pulse sequence; An image segmentation module, configured to input the pulse sequence into a first pulse convolution layer in a first encoding unit to obtain a coding feature map output by the first pulse convolution layer; Downsampling the encoding feature map through the pulse downsampling layer in the first encoding unit to obtain a downsampled feature map, and using the downsampled feature map as the input of the next encoding unit until the encoding feature map output by the first pulse convolution layer in the last encoding unit is obtained; fusing the encoding feature map output by the last encoding unit and the encoding feature map output by the encoding unit at the same depth level as the first decoding unit through the multi-scale feature fusion layer in the first decoding unit to obtain a first fused feature map; Upsampling the encoding feature map output by the last encoding unit through the pulse upsampling layer in the first decoding unit to obtain a first upsampled feature map; Performing feature splicing on the first fused feature map and the first upsampled feature map through a feature connection layer in the first decoding unit to obtain a first spliced ​​feature map; Performing convolution processing on the first spliced ​​feature map through the second pulse convolution layer in the first decoding unit to obtain a decoding feature map output by the first decoding unit, and using the decoding feature map output by the first decoding unit as the input of the next decoding unit; fusing the decoded feature map output by the first decoding unit and the encoded feature map output by the encoding unit at the same depth level as the next decoding unit through a multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map; Upsampling the decoded feature map output by the first decoding unit through the pulse upsampling layer in the next decoding unit to obtain a second upsampled feature map; Performing feature splicing on the second fused feature map and the second up-sampled feature map through a feature connection layer in the next decoding unit to obtain a second spliced ​​feature map; Performing convolution processing on the second splicing feature map through a second pulse convolution layer in the next decoding unit to obtain a decoding feature map output by the next decoding unit; Returning to the step of fusing the decoding feature map output by the first decoding unit and the encoding feature map output by the encoding unit at the same depth level as the next decoding unit through the multi-scale feature fusion layer in the next decoding unit to obtain a second fused feature map, until a decoding feature map output by the last decoding module in the image segmentation impulse model is obtained; Inputting the decoding feature map of the last decoding module into the segmentation output module in the image segmentation impulse model to obtain the image segmentation result output by the segmentation output module; The image segmentation pulse model is obtained by multi-scale feature fusion training based on pulse sequence samples formed by pulse coding of the original three-dimensional image and the image segmentation label of the original three-dimensional image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image segmentation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image segmentation method according to any one of claims 1 to 6 is implemented.

Images