An image segmentation method, device, apparatus and storage medium

Abstract

Embodiments of the present application disclose an image segmentation method, device and equipment and a storage medium, which can be applied to the field of digital medicine. The method comprises: obtaining a target image to be segmented; determining a local image block in the target image and a global image block matching the size of the local image block; inputting the local image block and the global image block into a preset segmentation network model for image segmentation processing, wherein the preset segmentation network model is configured to: perform feature fusion on the input local image block and global image block, and perform segmentation processing based on the fused target feature information; and determining a segmentation result corresponding to the target image according to the output of the preset segmentation network model. The technical solution of the embodiments of the present application can ensure the accuracy of image segmentation.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and more particularly to an image segmentation method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer technology and deep learning, neural network models can be used to segment high-resolution images. For example, segmenting 3D medical images can reduce the workload of doctors and provide them with a quantitative reference for treatment planning and surgical guidance.

[0003] Currently, local image patches can be obtained by cropping high-resolution images and then segmented based on a convolutional neural network model.

[0004] However, in the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0005] Because the receptive field of the convolution operator in a convolutional neural network model is limited, the convolutional neural network model only focuses on a small number of local features in the image and cannot obtain global features. Especially when dealing with target structures that have large differences in texture, shape and size, existing convolutional neural network models cannot effectively guarantee the accuracy of image segmentation. Summary of the Invention

[0006] This invention provides an image segmentation method, apparatus, device, and storage medium to ensure the accuracy of image segmentation.

[0007] In a first aspect, embodiments of the present invention provide an image segmentation method, comprising:

[0008] Obtain the target image to be segmented;

[0009] Identify local image patches in the target image and global image patches whose scale matches that of the local image patches;

[0010] The local image patch and the global image patch are input into a preset segmentation network model for image segmentation processing. The preset segmentation network model is used to: perform feature fusion on the input local image patch and the global image patch, and perform segmentation processing based on the fused target feature information;

[0011] Based on the output of the preset segmentation network model, the segmentation result corresponding to the target image is determined.

[0012] Secondly, embodiments of the present invention also provide an image segmentation apparatus, comprising:

[0013] The target image acquisition module is used to acquire the target image to be segmented.

[0014] An image patch determination module is used to determine local image patches in the target image and global image patches whose scale matches that of the local image patches;

[0015] An image patch input module is used to input the local image patch and the global image patch into a preset segmentation network model for image segmentation processing. The preset segmentation network model is used to: perform feature fusion on the input local image patch and the global image patch, and perform segmentation processing based on the fused target feature information.

[0016] The segmentation result determination module is used to determine the segmentation result corresponding to the target image based on the output of the preset segmentation network model.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0018] One or more processors;

[0019] Memory, used to store one or more programs;

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the image segmentation method provided in any embodiment of the present invention.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the image segmentation method as provided in any embodiment of the present invention.

[0022] The embodiments of the above invention have the following advantages or beneficial effects:

[0023] By identifying local image patches and global image patches that match the scale of the local image patches in the target image to be segmented, local image patches with complete detail information and global image patches with spatial context information and neighborhood dependencies are obtained. The local image patches and global image patches are then input into a preset segmentation network model for image segmentation. This preset segmentation network model performs feature fusion on the input local image patches and global image patches, so that the spatial context information and neighborhood dependencies missing in the local image patches can be supplemented by the global image patches, resulting in target feature information with stronger expressive power. Therefore, based on the fused target feature information, segmentation can be performed more accurately, effectively ensuring the accuracy of image segmentation. Attached Figure Description

[0024] Figure 1This is a flowchart of an image segmentation method provided in an embodiment of the present invention;

[0025] Figure 2 This is an example of determining local image blocks and global image blocks according to an embodiment of the present invention;

[0026] Figure 3 This is a flowchart of an image segmentation method provided in an embodiment of the present invention;

[0027] Figure 4 This is a structural example of a preset segmentation network model involved in the embodiments of the present invention;

[0028] Figure 5 This is a structural example of a local feature encoding sub-model and a feature decoding sub-model involved in the embodiments of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of an image segmentation device provided in an embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0031] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0032] Figure 1 This is a flowchart illustrating an image segmentation method provided by an embodiment of the present invention. This embodiment is applicable to the segmentation of high-resolution images, particularly for the segmentation of three-dimensional medical images. The method can be executed by an image segmentation device, which can be implemented in software and / or hardware and integrated into an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0033] S110. Obtain the target image to be segmented.

[0034] The target image can refer to a high-resolution two-dimensional or three-dimensional image, such as a gigapixel resolution. For example, the target image can be a three-dimensional medical image. Three-dimensional medical images can include, but are not limited to, magnetic resonance imaging (MRI), positron emission tomography (PET), computer tomography (CT), cone-beam CT, or three-dimensional ultrasound imaging.

[0035] S120. Determine the local image patches in the target image and the global image patches that match the scale of the local image patches.

[0036] In this context, a local image patch refers to a local image region within the target image. The resolution of the local image patch is the same as the original resolution of the target image to preserve its complete detail information. A global image patch refers to an image patch that contains the spatial context information and neighborhood dependencies of the local image patches. In this embodiment, the target image region corresponding to the global image patch is larger than the target image region corresponding to the local image patch, so that the global image patch can contain the spatial context information and neighborhood dependencies of the local image patch. The scale of the global image patch can be greater than or equal to the scale of the local image patch. This embodiment can reduce the scale of the global image patch through downsampling, making the scale of the global image patch equal to the scale of the local image patch, thus avoiding increased computational resource consumption. Furthermore, the loss of local detail information caused by downsampling can be compensated for by the local image patch.

[0037] Specifically, due to the high resolution of the target image, it cannot be directly input into the network model for target image segmentation. Therefore, a sampling method can be used to obtain local image patches in the target image, which can then be used for target image segmentation. For example, a sliding window approach can be used to determine each local image patch in the target image, and a global image patch matching the scale of the local image patch can be determined by directly downsampling the target image. For example, a global image patch with a scale greater than or equal to that of the local image patch.

[0038] For example, S120 may include: cropping the target image to determine a reference image block; cropping the reference image block again to determine a local image block; and downsampling the reference image block to determine a global image block that matches the scale of the local image block.

[0039] Among them, the reference image patch can refer to a local image patch in the target image that retains the original resolution. The scale of the reference image patch in this embodiment is larger than that of the local image patch, so that the reference image patch can contain the spatial context information and neighborhood dependence relationship information of the local image patch.

[0040] Specifically, the target image can be cropped by means of a larger sliding window to obtain each reference image patch, and the reference image patch at this time maintains the original resolution. For each reference image patch, the reference image patch is cropped again to obtain each local image patch corresponding to the reference image patch, so that each local image retains the original resolution, that is, retains complete detail information. By downsampling the reference image patch, a global image patch matching the scale of the local image patch can be obtained to reduce the consumption of computing resources. Figure 2 An example of determining local image patches and global patches is given. As Figure 2 shown, the target image is an abdominal CT image. The abdominal CT image can be cropped to obtain a reference image patch with a scale of D×H×W. The reference image patch is cropped again to obtain a local image patch with a scale of nD×nH×nW, where n is the downsampling ratio and 0 < n < 1. To avoid increasing the consumption of computing resources during training the model, the reference image patch can be downsampled at this time, and the scale of the reference image patch is reduced from D×H×W to nD×nH×nW, so as to obtain a global image patch with a scale of nD×nH×nW. Thus, the local image patch can be input into the model at the original resolution, retaining complete detail information, and the missing spatial context information and neighborhood dependence relationship of the local image patch can be supplemented in the global image patch.

[0041] S130. Input the local image patch and the global image patch into a preset segmentation network model for image segmentation processing. The preset segmentation network model is used to: perform feature fusion on the input local image patch and global image patch, and perform segmentation processing based on the fused target feature information.

[0042] Among them, the preset segmentation network model can be obtained by pre-training based on sample data to ensure the accuracy of image segmentation.

[0043] Specifically, local image patches with detailed information and global image patches with more semantic information can be input into a pre-trained pre-defined segmentation network model. The pre-defined segmentation network model can extract features from the input local and global image patches, and fuse the extracted local and global feature information. This allows the spatial context information and neighborhood dependencies missing in the local image patches to be supplemented by the global image patches, resulting in enhanced target feature information, i.e., target feature information with stronger expressive power. Thus, the segmentation mask, i.e., the segmentation result corresponding to the local image patch, can be decoded more accurately from the target feature information and output, thereby effectively ensuring the accuracy of image segmentation.

[0044] S140. Determine the segmentation result corresponding to the target image based on the output of the preset segmentation network model.

[0045] Specifically, by inputting each determined local image block and the corresponding global image block into a preset segmentation network model, and based on the output of the preset segmentation network model, the segmentation result corresponding to each local image block can be obtained. Furthermore, the segmentation results can be spliced ​​together based on the cropping order of the local image blocks to obtain the segmentation result corresponding to the entire target image, i.e., the segmentation mask.

[0046] The technical solution of this embodiment determines local image patches in the target image to be segmented and global image patches that match the scale of the local image patches, thereby obtaining local image patches that retain complete detail information and global image patches with spatial context information and neighborhood dependencies. The local image patches and global image patches are then input into a preset segmentation network model for image segmentation processing. The preset segmentation network model performs feature fusion on the input local image patches and global image patches, so that the spatial context information and neighborhood dependencies missing in the local image patches can be supplemented by the global image patches, obtaining target feature information with stronger expressive power. Therefore, based on the fused target feature information, segmentation processing can be performed more accurately, effectively ensuring the accuracy of image segmentation.

[0047] Figure 3 This is a flowchart of an image segmentation method provided in an embodiment of the present invention. Based on the above embodiments, the preset segmentation network model includes: a local feature encoding sub-model, a global feature encoding sub-model, a feature fusion sub-model, and a feature decoding sub-model. Furthermore, the step of "inputting local image blocks and global image blocks into the preset segmentation network model for image segmentation processing" is further optimized. The explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0048] See Figure 3 The image segmentation method provided in this embodiment specifically includes the following steps:

[0049] S310. Obtain the target image to be segmented.

[0050] S320. Determine local image patches in the target image and global image patches that match the scale of the local image patches.

[0051] S330. Input the local image patch into the local feature encoding sub-model and extract the local feature information corresponding to the local image patch.

[0052] The local feature encoding sub-model can be an encoder used to extract features from local image patches. In this embodiment, the local feature information can be represented in the form of feature vectors.

[0053] Specifically, Figure 4 An example of the structure of a pre-defined segmentation network model is given. Figure 4 As shown, local image patches are input into the local feature encoding sub-model for feature extraction to obtain local feature information.

[0054] For example, the local feature encoding sub-model may include a preset number of local feature encoding layers. Accordingly, S330 may include: inputting the local image patch into the first local feature encoding layer in the local feature encoding sub-model for feature encoding, and inputting the local feature information output by the first local feature encoding layer into the next local feature encoding layer; and using the local feature information output by the last local feature encoding layer as the local feature information corresponding to the local image patch.

[0055] Specifically, Figure 5 A structural example of a local feature encoding sub-model and a feature decoding sub-model is given. For example... Figure 5 As shown, the backbone network of the preset segmentation network model can be a U-shaped structure. Figure 5 The local feature coding submodel can include 5 local feature coding layers (i.e. Figure 5 (The five dashed boxes in the image) The first local feature encoding layer can consist of a convolutional layer, and the subsequent four local feature encoding layers can consist of a convolutional layer and a pooling layer followed by a convolutional layer. Each convolutional layer is followed by an activation function and an instance normalization operation. For the segmentation of 3D medical images, the convolutional layer can be a 3D convolution with a stride of 1, and the pooling layer can be a 3D convolution with a stride of 2, used to downsample the features and increase the number of feature channels. Figure 5 The numbers above or below the features in the image represent the number of channels for that feature. For example... Figure 5 As shown, by inputting local image patches layer by layer into each local feature encoding layer, a local image patch with a scale of nD×nH×nW can be mapped to a local feature encoding layer with a dimension of nD×nH×nW. The high-dimensional feature information refers to local feature information containing complete details. Here, c is the number of channels in the last convolutional layer of the last local feature encoding layer, for example, c = 288.

[0056] S340. Input the global image patch into the global feature encoding sub-model and extract the global feature information corresponding to the global image patch.

[0057] The global feature encoding sub-model can be an encoder used to extract global image patch features.

[0058] Specifically, such as Figure 4 As shown, the global image patch is input into the global feature encoding sub-model for feature extraction to obtain global feature information. This embodiment does not limit the execution order of steps S330 and S340. For example, steps S330 and S340 can be executed simultaneously, or step S330 can be executed first and then step S340, or step S340 can be executed first and then step S3330.

[0059] For example, the global feature encoding sub-model may include a preset number of global feature encoding layers. Accordingly, S340 may include: inputting a global image patch into the first global feature encoding layer in the global feature encoding sub-model for feature encoding, and inputting the global feature information output by the first global feature encoding layer into the next global feature encoding layer; and using the global feature information output by the last global feature encoding layer as the global feature information corresponding to the global image patch.

[0060] Specifically, the global feature encoding sub-model and the local feature encoding sub-model in this embodiment have the same network structure, as can be found in the relevant description of the local feature encoding sub-model above.

[0061] S350. Input local and global feature information into the feature fusion sub-model. Based on the self-attention mechanism, perform feature fusion on the local and global feature information to determine the fused target feature information.

[0062] Among them, the feature fusion sub-model can refer to a network model that uses a self-attention mechanism to fuse local feature information and global feature information.

[0063] Specifically, such as Figure 4As shown, local and global feature information is input into the feature fusion sub-model. This sub-model can extract the long-range correlation between the input local and global feature information through a self-attention mechanism, effectively mitigating the induction bias present in the convolutional architecture. This allows local feature information to utilize global feature information to enhance its expressive power, resulting in enhanced target feature information. The inherent long-range correlation characteristic of the self-attention mechanism enables the model to utilize effective global information, and the self-attention mechanism ensures that the model pays attention to multiple distinctive parts of the image, thereby further guaranteeing the accuracy of image segmentation.

[0064] For example, the feature fusion sub-model may include a feature splicing module and a self-attention module. Accordingly, S350 may include: inputting local feature information and global feature information into the feature splicing module, performing dimensional expansion and splicing processing on the local feature information and global feature information to determine the spliced ​​feature information; and inputting the spliced ​​feature information into the self-attention module for attention processing to determine the target feature information.

[0065] Specifically, all scales are Local feature information z l and global feature information z n The input is fed into the feature concatenation module, and z l and z n Expanding along the last three dimensions (length, width, and height) yields scales of [missing information]. of and And and The scale is obtained by splicing in the first dimension. splicing feature information z c The splicing feature information z c The input undergoes attention processing within the self-attention module, resulting in a feature fusion scale of [scale value missing]. Target feature information z t .

[0066] For example, the self-attention module may include a self-attention layer and a feedforward layer. Accordingly, inputting the concatenated feature information into the self-attention module for attention processing to determine the target feature information may include: inputting the concatenated feature information into the self-attention layer, mapping the input concatenated feature information to determine a query feature vector and a key-value pair feature vector, and performing attention processing based on the query feature vector and the key-value pair feature vector to obtain the attention-processed feature information; and inputting the attention-processed feature information into the feedforward layer to introduce non-linear information to determine the target feature information.

[0067] Specifically, the self-attention layer can use a fully connected layer to concatenate the feature information z. c The mapping is represented by a query feature vector Q and a set of key-value pair feature vectors (including a key feature vector K and a value feature vector V). By performing attention processing on the query feature vector Q, the key feature vector K, and the value feature vector V, the attention-processed feature information Attention(Q,K,V) can be represented as follows:

[0068]

[0069] Where d refers to the splicing feature information z c The number of channels, i.e., 2c. The self-attention layer can filter out a small amount of important information from a large amount of data and focus on this important information, ignoring most of the unimportant information. The larger the weight, the more it focuses on its corresponding value feature vector V; that is, the weight represents the importance of the information. The specific process of the self-attention mechanism can be summarized into two processes: the first process determines the weight coefficients based on the query feature vector Q and the key feature vector K; the second process performs a weighted summation of the value feature vector V based on the weight coefficients. The first process can be further subdivided into two stages: the first stage determines the similarity or relevance between the query feature vector Q and the key feature vector K; the second stage normalizes the original scores from the first stage to obtain the normalized weight coefficients.

[0070] The feedforward layer can be composed of a stacked structure of a first fully connected layer, an activation function, a second fully connected layer, and a layer normalization function. The processing flow of the feedforward layer can be represented as follows:

[0071] FFN(x) = max(0, xW1+b1)W2+b2

[0072] Where FFN(x) represents the target feature information output by the feedforward layer, W1, W2, b1, and b2 are the weights and biases of the first and second fully connected layers, respectively, and x is the input of the feedforward layer, i.e., the feature information Attention(Q,K,V) after attention processing. The self-attention layer can be used to extract... and Long-range related information between them makes Can be used The self-attention module uses spatial context information to enhance itself, and the feedforward layer is used to introduce nonlinear information into the self-attention module to further improve the accuracy of image segmentation.

[0073] S360. Input the target feature information into the feature decoding sub-model for feature decoding to determine the segmentation result corresponding to the local image block.

[0074] Specifically, such as Figure 4 As shown, the scale is Target feature information z t The re-expansion as The target feature information is input into the feature decoding sub-model for feature decoding. This feature decoding sub-model can decode the segmentation mask result from the target feature information.

[0075] For example, the feature decoding sub-model has a similar structure to the local feature encoding sub-model. For instance, the feature decoding sub-model may include a predetermined number of feature decoding layers. The number of feature decoding layers is equal to the number of local feature encoding layers. Each local feature encoding layer corresponds one-to-one with a feature decoding layer, the encoding order of the local feature encoding layers is the reverse of the decoding order of the feature decoding layers, and the local feature encoding layers are skipped connections to their corresponding feature decoding layers.

[0076] For example, S360 may include: inputting target feature information and local feature information obtained by a local feature encoding layer that is skip-connected to the first feature decoding layer in the feature decoding sub-model into the first feature decoding layer for feature decoding operation, and inputting the decoded feature information output by the first feature decoding layer and local feature information obtained by a local feature encoding layer that is skip-connected to the next feature decoding layer into the next feature decoding layer; and determining the segmentation result corresponding to the local image patch based on the decoded feature information output by the last feature encoding layer.

[0077] Specifically, such as Figure 5 As shown, the feature decoding sub-model includes 5 feature decoding layers (i.e. Figure 5(The five dashed boxes in the image). The first four feature decoding layers can consist of an upsampling layer, a convolutional layer, and a single convolutional layer. The last feature decoding layer can consist of a single convolutional layer. Each of the first four convolutional layers is followed by an activation function and instance normalization. For 3D medical image segmentation, the upsampling layer can be a 3D deconvolutional layer with a stride of 1. The last four local feature encoding layers can use skip connections to pass the local feature information obtained after passing through the convolutional layers in the local feature encoding layers to the corresponding feature decoding layers. This allows detailed information to be added to the corresponding feature decoding layers for decoding, further improving the accuracy of segmentation. For example, the target feature information and the local feature information obtained after passing through the convolutional layer in the last local feature encoding layer can be input into the first feature decoding layer for feature decoding. The first feature decoding layer upsamples the input target feature information and concatenates the upsampled features with the local feature information passed through the skip connections before inputting them into subsequent convolutional layers, thus obtaining the decoded feature information output by the first feature decoding layer. Similarly, the decoding operation is performed sequentially until the decoded feature information output by the fourth feature decoding layer is obtained. The decoded feature information output by the fourth feature decoding layer is then input into the fifth feature decoding layer to decode and obtain the final segmentation result, which is then output.

[0078] S370. Determine the segmentation result corresponding to the target image based on the output of the preset segmentation network model.

[0079] The technical solution of this embodiment inputs local and global feature information into the feature fusion sub-model, and extracts the long-range correlation between the input local and global feature information based on the self-attention mechanism. This effectively alleviates the induction bias in the convolutional architecture, allowing local feature information to use global feature information to enhance its expressive power and obtain enhanced target feature information. This enables the model to focus on multiple distinctive parts in the image, thereby further ensuring the accuracy of image segmentation.

[0080] Based on the above technical solution, the training process of the pre-defined segmentation network model may include the following steps S301-S305:

[0081] S301. Based on the preset segmentation network model to be trained, determine the output segmentation result corresponding to the sample image.

[0082] The pre-defined segmentation network model to be trained may include a local feature encoding sub-model, a global feature encoding sub-model, a feature fusion sub-model, and a feature decoding sub-model.

[0083] Specifically, local and global image blocks in a sample image block can be determined by identifying the same local and global image blocks in the target image. These local and global image blocks are then input into a pre-defined segmentation network model to be trained. Based on the output of the pre-defined segmentation network model, the output segmentation result corresponding to the sample image is determined.

[0084] S302. Based on the preset loss function, the output segmentation result, and the label segmentation result corresponding to the sample image, determine the training error.

[0085] The preset loss function can consist of one type of loss function or a combination of multiple loss functions. For example, the preset loss function can be composed of the cross-entropy loss function. and Dice loss function Composition. Cross-entropy loss function and Dice loss function These can be represented as follows:

[0086]

[0087]

[0088] Where N is the semantic category of the segmentation target, L is the number of voxels in each semantic channel, s is the segmentation label encoded by a one-hot code, and s′ is the output prediction result determined by the flexible maxima. l,n The semantic value of the l-th voxel is the standard probability of the n-th semantic category, which is obtained based on the label segmentation results; s′ l,n The semantic probability of the l-th voxel is the predicted probability of the n-th semantic category, which is obtained based on the output segmentation result.

[0089] S303. Backpropagate the training error to the local feature encoding sub-model, feature fusion sub-model, and feature decoding sub-model in the preset segmentation network model to be trained, and iteratively update the weights in the local feature encoding sub-model, feature fusion sub-model, and feature decoding sub-model.

[0090] Specifically, the weights in the local feature encoding sub-model, feature fusion sub-model, and feature decoding sub-model to be trained are adjusted through backpropagation to minimize the preset loss function.

[0091] S304. Based on the iteratively updated weights in the local feature encoding sub-model, iteratively update the weights in the global feature encoding sub-model of the preset segmentation network model to be trained.

[0092] Specifically, during each training iteration, the exponential moving average (EMA) method can be used to update the weights θ in the local feature encoding sub-model based on the iteratively updated weights θ. l Update the weights θ in the global feature encoding submodel. n This eliminates the need for backpropagation to update, allowing global feature information to be acquired without increasing computational load, thereby reducing GPU memory overhead for maintaining the global feature encoding sub-model.

[0093] For example, S304 may include: determining the weights of the global feature encoding sub-model after the current iteration based on the weights of the local feature encoding sub-model after the current iteration and the weights of the global feature encoding sub-model after the previous iteration in the preset segmentation network model to be trained, using an exponential moving average method.

[0094] Specifically, during each training iteration, after updating the weights in the local feature encoding sub-model, feature fusion sub-model, and feature decoding sub-model through backpropagation, the training can be based on the weights in the local feature encoding sub-model after the current iteration. The weights from the previous iteration in the global feature encoding submodel of the pre-defined segmentation network model to be trained. Determine the weights in the global feature encoding sub-model after the current iteration. Right now Where i is the iteration number and α is a hyperparameter used to control the speed of weight updates. Using the EMA method, θ is adjusted at each iteration of training. n Gradient backpropagation is not required; instead, a moving average θ is used. l This method updates the model, thereby reducing the GPU memory consumption by nearly half during the feature extraction stage and greatly improving the utilization of GPU memory.

[0095] S305. When the preset convergence condition is met, the training of the preset segmentation network model to be trained ends, and the preset segmentation network model is obtained.

[0096] Specifically, when a preset convergence condition is met, such as when the number of iterations equals a preset number, or when the training error stabilizes, the training of the preset segmentation network model is considered complete, and the trained preset segmentation network model can be obtained. This embodiment uses an exponential moving average method to update the weights in the global feature encoding sub-model, thereby saving computational overhead during model training while acquiring global features. This allows the model to possess both the relatively low computational cost of a convolutional neural network model and the long-range correlation characteristics of a self-attention model, further improving the accuracy of image segmentation.

[0097] The following are embodiments of the image segmentation apparatus provided in this invention. This apparatus and the image segmentation methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the image segmentation apparatus, please refer to the embodiments of the above image segmentation methods.

[0098] Figure 6 This is a schematic diagram of an image segmentation device provided in an embodiment of the present invention. This embodiment is applicable to the segmentation of high-resolution images, especially for the segmentation of three-dimensional medical images. The device specifically includes: a target image acquisition module 610, an image block determination module 620, an image block input module 630, and a segmentation result determination module 640.

[0099] The system includes a target image acquisition module 610 for acquiring the target image to be segmented; an image patch determination module 620 for determining local image patches and global image patches that match the scale of the local image patches in the target image; an image patch input module 630 for inputting the local image patches and global image patches into a preset segmentation network model for image segmentation processing, wherein the preset segmentation network model is used to: fuse the features of the input local image patches and global image patches, and perform segmentation processing based on the fused target feature information; and a segmentation result determination module 640 for determining the segmentation result corresponding to the target image based on the output of the preset segmentation network model.

[0100] Optionally, the image patch determination module 620 is specifically used for:

[0101] The target image is cropped to determine a reference image block; the reference image block is cropped again to determine a local image block; the reference image block is downsampled to determine a global image block that matches the scale of the local image block.

[0102] Optionally, the preset segmentation network model includes: a local feature encoding sub-model, a global feature encoding sub-model, a feature fusion sub-model, and a feature decoding sub-model;

[0103] Image block input module 630 includes:

[0104] The local feature extraction unit is used to input local image patches into the local feature encoding sub-model and extract the local feature information corresponding to the local image patches;

[0105] The global feature extraction unit is used to input global image patches into the global feature encoding sub-model and extract global feature information corresponding to the global image patches;

[0106] The feature fusion unit is used to input local and global feature information into the feature fusion sub-model. Based on the self-attention mechanism, it performs feature fusion on the local and global feature information to determine the fused target feature information.

[0107] The feature decoding unit is used to input the target feature information into the feature decoding sub-model for feature decoding to determine the segmentation result corresponding to the local image patch.

[0108] Optionally, the local feature coding sub-model includes: a preset number of local feature coding layers;

[0109] The local feature extraction unit is specifically used to: input local image patches into the first local feature encoding layer in the local feature encoding sub-model for feature encoding operations, and input the local feature information output by the first local feature encoding layer into the next local feature encoding layer; and use the local feature information output by the last local feature encoding layer as the local feature information corresponding to the local image patch.

[0110] Optionally, the feature fusion sub-model includes: a feature concatenation module and a self-attention module;

[0111] Feature fusion unit, including:

[0112] The feature splicing subunit is used to input local and global feature information into the feature splicing module, perform dimensional expansion and splicing processing on the local and global feature information, and determine the spliced ​​feature information;

[0113] The attention processing subunit is used to input the spliced ​​feature information into the self-attention module for attention processing to determine the target feature information.

[0114] Optionally, the self-attention module includes: a self-attention layer and a feedforward layer;

[0115] The attention processing subunit is specifically used for: inputting the concatenated feature information into the self-attention layer, mapping the input concatenated feature information, determining the query feature vector and key-value pair feature vector, and performing attention processing based on the query feature vector and key-value pair feature vector to obtain the attention-processed feature information; inputting the attention-processed feature information into the feedforward layer to introduce nonlinear information and determine the target feature information.

[0116] Optionally, the feature decoding sub-model includes: a preset number of feature decoding layers, wherein the local feature encoding layer corresponds one-to-one with the feature decoding layer, the encoding order of the local feature encoding layer is the reverse of the decoding order of the feature decoding layer, and the local feature encoding layer is skipped to the corresponding feature decoding layer.

[0117] The feature decoding unit is specifically used to: input the target feature information and the local feature information obtained by the local feature encoding layer that is skip-connected to the first feature decoding layer in the feature decoding sub-model into the first feature decoding layer for feature decoding operation; input the decoded feature information output by the first feature decoding layer and the local feature information obtained by the local feature encoding layer that is skip-connected to the next feature decoding layer into the next feature decoding layer; and determine the segmentation result corresponding to the local image patch based on the decoded feature information output by the last feature encoding layer.

[0118] Optionally, the device further includes: a preset segmentation network model training module, used for:

[0119] Based on the pre-defined segmentation network model to be trained, the output segmentation result corresponding to the sample image is determined; based on the pre-defined loss function, the output segmentation result, and the label segmentation result corresponding to the sample image, the training error is determined; the training error is backpropagated to the local feature encoding sub-model, feature fusion sub-model, and feature decoding sub-model in the pre-defined segmentation network model to be trained, and the weights in the local feature encoding sub-model, feature fusion sub-model, and feature decoding sub-model are iteratively updated; based on the iteratively updated weights in the local feature encoding sub-model, the weights in the global feature encoding sub-model in the pre-defined segmentation network model to be trained are iteratively updated; when the pre-defined convergence condition is reached, the training of the pre-defined segmentation network model to be trained ends, and the pre-defined segmentation network model is obtained.

[0120] Optionally, a pre-defined segmentation network model training module is provided, specifically for:

[0121] Based on the exponential moving average method, the weights of the global feature encoding sub-model after the current iteration are determined according to the weights of the local feature encoding sub-model after the current iteration and the weights of the global feature encoding sub-model after the previous iteration in the preset segmentation network model to be trained.

[0122] Optionally, the target image is a three-dimensional medical image.

[0123] The image segmentation apparatus provided in the embodiments of the present invention can execute the image segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the image segmentation method.

[0124] It is worth noting that in the above-described embodiments of the image segmentation device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0125] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 7 A block diagram is shown of an exemplary electronic device 12 suitable for implementing embodiments of the present invention. Figure 7 The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0126] like Figure 7 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0127] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0128] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0129] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0130] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0131] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0132] Processing unit 16 executes various functional applications and data processing by running programs stored in system memory 28, such as implementing the steps of an image segmentation method provided in this embodiment, the method including:

[0133] Obtain the target image to be segmented;

[0134] Identify local image patches in the target image and global image patches that match the scale of the local image patches;

[0135] Local and global image patches are input into a preset segmentation network model for image segmentation. The preset segmentation network model is used to: fuse features of the input local and global image patches, and perform segmentation based on the fused target feature information.

[0136] Based on the output of the preset segmentation network model, the segmentation result corresponding to the target image is determined.

[0137] Of course, those skilled in the art will understand that the processor can also implement the technical solutions of the image segmentation method provided in any embodiment of the present invention.

[0138] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image segmentation method steps provided in any embodiment of the present invention, the method comprising:

[0139] Obtain the target image to be segmented;

[0140] Identify local image patches in the target image and global image patches that match the scale of the local image patches;

[0141] Local and global image patches are input into a preset segmentation network model for image segmentation. The preset segmentation network model is used to: fuse features of the input local and global image patches, and perform segmentation based on the fused target feature information.

[0142] Based on the output of the preset segmentation network model, the segmentation result corresponding to the target image is determined.

[0143] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0144] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0145] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0146] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0148] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. An image segmentation method characterized by, The method comprises: acquiring a target image to be segmented; determining a local image block in the target image and a global image block matching the size of the local image block; inputting the local image block and the global image block into a preset segmentation network model for image segmentation processing, wherein the preset segmentation network model is configured to perform feature fusion on the input local image block and global image block, and perform segmentation processing based on the fused target feature information; determining a segmentation result corresponding to the target image according to the output of the preset segmentation network model; the training process of the preset segmentation network model comprises: determining an output segmentation result corresponding to a sample image based on a preset segmentation network model to be trained; wherein the preset segmentation network model to be trained comprises a local feature encoding sub-model, a global feature encoding sub-model, a feature fusion sub-model and a feature decoding sub-model; determining a training error based on a preset loss function, the output segmentation result and a label segmentation result corresponding to the sample image; propagating the training error to the local feature encoding sub-model, the feature fusion sub-model and the feature decoding sub-model in the preset segmentation network model to be trained, and iteratively updating the weights in the local feature encoding sub-model, the feature fusion sub-model and the feature decoding sub-model; iteratively updating the weights in the global feature encoding sub-model in the preset segmentation network model to be trained based on the iteratively updated weights in the local feature encoding sub-model; when a preset convergence condition is reached, the training of the preset segmentation network model to be trained is completed, and the preset segmentation network model is obtained; wherein the iteratively updating the weights in the global feature encoding sub-model in the preset segmentation network model to be trained based on the iteratively updated weights in the local feature encoding sub-model comprises: determining the weights in the global feature encoding sub-model after the current iteration based on the weights in the local feature encoding sub-model after the current iteration and the weights in the global feature encoding sub-model after the last iteration in the preset segmentation network model to be trained according to an exponential moving average method.

2. The method of claim 1, wherein, The determination of the local image block in the target image and the global image block matching the size of the local image block comprises: cropping the target image to determine a reference image block; re-cropping the reference image block to determine a local image block; down-sampling the reference image block to determine a global image block matching the size of the local image block.

3. The method of claim 1, wherein, The preset segmentation network model comprises a local feature encoding sub-model, a global feature encoding sub-model, a feature fusion sub-model and a feature decoding sub-model. The inputting the local image block and the global image block into the preset segmentation network model for image segmentation processing comprises: inputting the local image block into the local feature encoding sub-model to extract local feature information corresponding to the local image block; inputting the global image block into the global feature encoding sub-model to extract global feature information corresponding to the global image block; The local feature information and the global feature information are input into the feature fusion sub-model, feature fusion is performed on the local feature information and the global feature information based on a self-attention mechanism, and target feature information after fusion is determined. The target feature information is input into the feature decoding sub-model for feature decoding, and a segmentation result corresponding to the local image block is determined.

4. The method of claim 3, wherein, The local feature encoding sub-model includes a preset number of local feature encoding layers. The local image block is input into the local feature encoding sub-model, and local feature information corresponding to the local image block is extracted, including: The local image block is input into the first local feature encoding layer of the local feature encoding sub-model for feature encoding operation, and the local feature information output by the first local feature encoding layer is input into the next local feature encoding layer. The local feature information output by the last local feature encoding layer is taken as the local feature information corresponding to the local image block.

5. The method of claim 3, wherein, The feature fusion sub-model includes a feature splicing module and a self-attention module. The local feature information and the global feature information are input into the feature fusion sub-model, feature fusion is performed on the local feature information and the global feature information based on a self-attention mechanism, and target feature information after fusion is determined, including: The local feature information and the global feature information are input into the feature splicing module, and the local feature information and the global feature information are dimensionally expanded and spliced to determine spliced feature information; The spliced feature information is input into the self-attention module for attention processing to determine target feature information.

6. The method of claim 5, wherein, The self-attention module includes a self-attention layer and a feedforward layer. The spliced feature information is input into the self-attention module for attention processing to determine target feature information, including: The spliced feature information is input into the self-attention layer, the input spliced feature information is mapped to determine a query feature vector and a key-value pair feature vector, and attention processing is performed based on the query feature vector and the key-value pair feature vector to obtain feature information after attention processing; The feature information after attention processing is input into the feedforward layer to introduce nonlinear information, and target feature information is determined.

7. The method of claim 4, wherein, The feature decoding sub-model includes a preset number of feature decoding layers, wherein the local feature encoding layers and the feature decoding layers one-to-one correspond, the encoding order of the local feature encoding layers is opposite to the decoding order of the feature decoding layers, and the local feature encoding layers and the corresponding feature decoding layers are connected by jumping; The target feature information is input into the feature decoding sub-model for feature decoding, and a segmentation result corresponding to the local image block is determined, including: input the target feature information and local feature information obtained by the local feature encoding layer connected to the first feature decoding layer in the feature decoding sub-model into the first feature decoding layer for feature decoding operation, and input the decoding feature information output by the first feature decoding layer and the local feature information obtained by the local feature encoding layer connected to the next feature decoding layer into the next feature decoding layer; determine the segmentation result corresponding to the local image block based on the decoding feature information output by the last feature encoding layer.

8. The method according to any one of claims 1 to 7, characterized in that, The target image is a three-dimensional medical image.

9. An image segmentation apparatus characterized by comprising: The method comprises: an image block determination module configured to determine a local image block in the target image and a global image block matching the size of the local image block; an image block input module configured to input the local image block and the global image block into a preset segmentation network model for image segmentation processing, wherein the preset segmentation network model is configured to perform feature fusion on the input local image block and global image block, and perform segmentation processing based on the fused target feature information; a segmentation result determination module configured to determine a segmentation result corresponding to the target image according to the output of the preset segmentation network model; a preset segmentation network model training module configured to: determine an output segmentation result corresponding to a sample image based on a preset segmentation network model to be trained; wherein the preset segmentation network model to be trained comprises a local feature encoding sub-model, a global feature encoding sub-model, a feature fusion sub-model, and a feature decoding sub-model; determine a training error based on a preset loss function, the output segmentation result, and a label segmentation result corresponding to the sample image; propagate the training error to the local feature encoding sub-model, the feature fusion sub-model, and the feature decoding sub-model in the preset segmentation network model to be trained, and iteratively update the weights in the local feature encoding sub-model, the feature fusion sub-model, and the feature decoding sub-model; iteratively update the weights in the global feature encoding sub-model in the preset segmentation network model to be trained based on the iteratively updated weights in the local feature encoding sub-model; when a preset convergence condition is reached, the training of the preset segmentation network model to be trained is completed, and a preset segmentation network model is obtained; the preset segmentation network model training module is specifically configured to: determine the weights in the global feature encoding sub-model after the current iteration based on the weights in the local feature encoding sub-model after the current iteration and the weights in the global feature encoding sub-model after the last iteration in the preset segmentation network model to be trained according to an exponential moving average method. The electronic device comprises:

10. An electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the image segmentation method of any one of claims 1-8. The program is executed by the processor to implement the image segmentation method of any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, ​

Images