Image segmentation method, image segmentation device, electronic device and storage medium

Abstract

The present invention relates to the field of image processing technology, and provides an image segmentation method, an image segmentation device, an electronic device, and a storage medium. The image segmentation method includes: extracting features from an image to be segmented to obtain multiple semantic feature maps of different scales, sorting the multiple semantic feature maps from high to low according to scale, and using the first semantic feature map as the first segmentation semantic; performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantic to obtain the current segmentation semantic, wherein the semantic feature map of the current scale includes other semantic feature maps except the first semantic feature map, and the target segmentation semantic is the segmentation semantic corresponding to the semantic feature map of the previous scale; and determining the segmentation semantic corresponding to the last semantic feature map as the image segmentation result. The image segmentation method of the present invention can make full use of semantic context, and then combine semantic embedding and attention conversion to perform coarse-to-fine iterative segmentation across layers, which helps to improve segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, an image segmentation device, an electronic device and a storage medium. Background Art

[0002] Image segmentation is a crucial preprocessing in image recognition and computer vision tasks, and is widely used in various fields, including object recognition, object tracking, and image classification.

[0003] Related technologies for image segmentation often focus on the boundaries of local features, with little consideration given to the correlations between features. To address this, some techniques use semantic context to guide the representation of specific feature maps. However, these techniques ignore the global representation of semantic context, resulting in less than ideal segmentation results. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an image segmentation method to improve the accuracy of the segmentation result.

[0005] The present invention also provides an image segmentation device.

[0006] The present invention also provides an electronic device.

[0007] The present invention also provides a non-transitory computer-readable storage medium.

[0008] The present invention also provides a computer program product.

[0009] The image segmentation method according to the first embodiment of the present invention includes:

[0010] Perform feature extraction on the image to be segmented to obtain multiple semantic feature maps of different scales, wherein the multiple semantic feature maps are sorted from high to low according to the scale, and the first semantic feature map is used as the first segmentation semantic;

[0011] Performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics, wherein the semantic feature map of the current scale includes other semantic feature maps except the first semantic feature map, and the target segmentation semantics is the segmentation semantics corresponding to the semantic feature map of the previous scale;

[0012] Determine the segmentation semantics corresponding to the last semantic feature map as the image segmentation result.

[0013] According to the image segmentation method of an embodiment of the present invention, by extracting semantic feature maps at multiple scales, the semantic context can be fully utilized, and then combined with semantic embedding and attention conversion to perform coarse-to-fine iterative segmentation across layers, which helps to improve the accuracy of the segmentation results.

[0014] According to one embodiment of the present invention, performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics includes:

[0015] Perform semantic embedding on the semantic feature map of the current scale to obtain the embedded feature map of the current scale;

[0016] Performing attention conversion on the embedded feature map of the current scale through target segmentation semantics to obtain semantic representation features of the current scale;

[0017] Obtaining image features at the current scale based on the semantic representation features at the current scale;

[0018] The image features of the current scale are fused with the target segmentation semantics to obtain the current segmentation semantics.

[0019] According to one embodiment of the present invention, performing semantic embedding on the semantic feature map of the current scale to obtain the embedded feature map of the current scale includes:

[0020] Embed the semantic feature map of the current scale into the memory tensor to obtain the embedded feature map of the current scale.

[0021] According to one embodiment of the present invention, embedding the semantic feature map of the current scale into the memory tensor to obtain the embedded feature map of the current scale includes:

[0022] Determine the target memory feature that has the greatest similarity to the first feature of the target in the semantic feature map of the current scale from the memory tensor;

[0023] The target first feature is embedded into the target memory feature to obtain the target embedded feature in the embedded feature map of the current scale.

[0024] According to one embodiment of the present invention, performing attention conversion on the embedded feature map of the current scale by using target segmentation semantics to obtain semantic representation features of the current scale includes:

[0025] Perform softmax operation on the target segmentation semantics to obtain a probability map;

[0026] Attention conversion is performed on the embedded feature map of the current scale through the probability map to obtain the semantic representation feature of the current scale.

[0027] According to one embodiment of the present invention, obtaining the image features at the current scale based on the semantic representation features at the current scale includes:

[0028] The semantic representation features of the current scale are connected with the semantic feature map of the current scale and features are extracted to obtain image features of the current scale.

[0029] According to one embodiment of the present invention, performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics includes:

[0030] The semantic feature map of the current scale is input into the current segmentation layer of the image segmentation model to obtain the segmentation semantics output by the current segmentation layer.

[0031] According to one embodiment of the present invention, the loss function of the image segmentation model is a cross entropy loss function, and the cross entropy loss function is used to characterize the segmentation loss of each segmentation layer.

[0032] An image segmentation device according to an embodiment of the second aspect of the present invention includes:

[0033] An extraction module is used to extract features from the image to be segmented to obtain multiple semantic feature maps of different scales, wherein the multiple semantic feature maps are sorted from high to low according to the scale, and the first semantic feature map is used as the first segmentation semantic;

[0034] A semantic segmentation module is configured to perform semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics, wherein the semantic feature map of the current scale includes other semantic feature maps except the first semantic feature map, and the target segmentation semantics is the segmentation semantics corresponding to the semantic feature map of the previous scale;

[0035] The determination module is used to determine the segmentation semantics corresponding to the last semantic feature map as the image segmentation result.

[0036] According to the image segmentation device of an embodiment of the present invention, by extracting semantic feature maps of multiple scales, it can make full use of semantic context, and then combine semantic embedding and attention conversion to perform coarse-to-fine iterative segmentation across layers, which helps to improve the accuracy of the segmentation results.

[0037] According to an embodiment of the third aspect of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the above-mentioned image segmentation methods are implemented.

[0038] According to the non-transitory computer-readable storage medium of the fourth aspect of the present invention, a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the above-mentioned image segmentation methods are implemented.

[0039] A computer program product according to an embodiment of the fifth aspect of the present invention comprises a computer program, which, when executed by a processor, implements the steps of any one of the above-described image segmentation methods.

[0040] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0041] By extracting semantic feature maps at multiple scales, we can make full use of the semantic context, and then combine semantic embedding and attention conversion to perform coarse-to-fine iterative segmentation across layers, which helps to improve the accuracy of the segmentation results.

[0042] Furthermore, through the inter-layer attention mechanism, the model can focus on feature learning during residual learning, thereby further improving the accuracy and reliability of feature extraction.

[0043] Furthermore, by concatenating the semantic representation features of the current scale with the semantic feature map of the current scale, the recovered image features are combined with the original semantic feature map and the semantic representation features obtained through semantic embedding and attention, which can focus on the connection between semantic contexts.

[0044] Furthermore, through cross-layer iterative segmentation, the number of segmentation layers can be reduced while ensuring segmentation accuracy, thereby helping to streamline the network. In this way, this method can be directly used for independent operation on various terminals without relying on server calculations.

[0045] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is one of the flowcharts of the image segmentation method provided by an embodiment of the present invention;

[0048] Figure 2 This is the second flow chart of the image segmentation method provided by an embodiment of the present invention;

[0049] Figure 3 This is the third flow chart of the image segmentation method provided by an embodiment of the present invention;

[0050] Figure 4This is a fourth flowchart of the image segmentation method provided by an embodiment of the present invention;

[0051] Figure 5 This is the fifth flowchart of the image segmentation method provided by the embodiment of the present invention;

[0052] Figure 6 This is the sixth flowchart of the image segmentation method provided by an embodiment of the present invention;

[0053] Figure 7 1 is a schematic structural diagram of an image segmentation device provided by an embodiment of the present invention;

[0054] Figure 8 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0056] In the description of the embodiments of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as limiting the embodiments of the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0057] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "connected" and "connection" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may be connected wirelessly or electrically; they may be connected directly or indirectly through an intermediary. Those skilled in the art will understand the specific meanings of the above terms in the embodiments of the present invention.

[0058] In the embodiments of the present invention, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Furthermore, a first feature being "above," "above," or "above" a second feature may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. A first feature being "below," "below," or "below" a second feature may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0059] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0060] The following combination Figures 1-6 An image segmentation method according to an embodiment of the present invention is described.

[0061] like Figure 1 As shown, the image segmentation method according to the embodiment of the present invention includes steps 110 to 130.

[0062] Step 110: extract features from the image to be segmented to obtain multiple semantic feature maps of different scales. The multiple semantic feature maps are sorted from high to low according to scale, and the first semantic feature map is used as the first segmentation semantic map;

[0063] In this step, the resolutions of multiple semantic feature maps obtained through feature extraction are the same.

[0064] like Figure 3 As shown, this step may include: performing context semantic capture on the segmented image to obtain multiple first semantic features (Res-1, Res-2, Res-3 and Res-4), each of which has the same dimension; decoding the first semantic feature of the current layer based on the second semantic feature of the previous layer to obtain the second semantic feature of the current layer (F 2 、F 3 and F 4 ); Extract the second semantic feature of the current layer and obtain the semantic feature map of the current layer (X 1 、X2 、X 3 and X 4 ); Among them, the second semantic feature F 1 Obtained based on decoding of the first semantic feature Res-4 at the highest level.

[0065] The image segmentation method can be implemented by an image segmentation model. Correspondingly, step 110 can be implemented by a feature extraction network of the image segmentation model. The feature extraction network includes feature extraction layers at multiple scales, where each scale corresponds to a feature extraction layer.

[0066] Here, the features of the sampling areas at different scales can reflect the information of the image to be segmented at different scales. The larger / higher the scale (for example, the semantic feature map X 1 ), the corresponding features can better reflect the description of the overall information of the segmented image, thereby capturing the semantic context information; the smaller / lower the scale (such as the semantic feature map X 4 ), the corresponding features are more able to reflect the description of the detailed information of the image to be segmented, and the semantic feature map provides local text details.

[0067] In actual implementation, this step may include: inputting the image to be segmented into feature extraction layers at multiple scales of the image segmentation model to obtain multiple semantic feature maps, where the scales of the multiple semantic feature maps are different.

[0068] The feature extraction network may include an encoder-decoder architecture with a ResNet backbone, where the encoder-decoder architecture is used to capture multi-scale feature information, and the encoder-decoder architecture may be implemented by dilated convolution.

[0069] like Figure 3 As shown, the image to be segmented is input into the encoding layer (backbone) to obtain multiple first semantic features (Res-1, Res-2, Res-3 and Res-4), and the first semantic features (Res-1, Res-2, Res-3 and Res-4) are decoded to obtain the second semantic features (F 1 、F 2 、F 3 and F 4 ); for the second semantic feature (F 1 、F 2 、F 3 and F 4 ) to extract features and obtain multiple semantic feature maps of different scales (X 1 、X 2 、X 3 and X 4 ).

[0070] The semantic feature map can be represented as a three-dimensional matrix HW×D, where H represents height, W represents width, and D represents the number of channels. This feature extraction network can be implemented using dilated convolution.

[0071] The image segmentation method of the embodiment of the present invention needs to implement segmentation through inter-layer iteration. For the first semantic feature map X 1 , which does not exist in the previous layer, so if Figure 3 As shown, directly convert the first semantic feature map X 1 As the first segmentation semantic L 1 .

[0072] Step 120: Perform semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics.

[0073] It can be understood that in this step, each time semantic segmentation is performed, a global semantic representation of the semantic context can be obtained through semantic embedding and attention conversion.

[0074] In related technologies, feature representation is generally performed through simple aggregation methods such as class center features, which may confuse different features, provide unreasonable feature information and mislead feature representation.

[0075] In the present invention, segmentation semantics are generated by combining embedded features with the attention mechanism for aggregation. The connection between feature information and semantic context can be established through global semantic representation, making segmentation more accurate.

[0076] like Figure 3 As shown, the first semantic feature map X 1 As the first segmentation semantic L 1 .

[0077] The semantic feature map of the current scale in step 120 includes other semantic feature maps except the first semantic feature map, and the target segmentation semantics is the segmentation semantics corresponding to the semantic feature map of the previous scale.

[0078] From the second semantic feature map X 2 First, the semantic feature map X 2 and target segmentation semantics L 1 Perform semantic embedding and attention conversion to obtain the current segmentation semantics L 2 .

[0079] The semantic feature map X 3 and target segmentation semantics L 2 Perform semantic embedding and attention conversion to obtain the current segmentation semantics L 3 .

[0080] The semantic feature map X4 and target segmentation semantics L 3 Perform semantic embedding and attention conversion to obtain the current segmentation semantics L 4 .

[0081] If there are more layers, continue to follow this rule to perform semantic embedding and attention transfer between layers.

[0082] This step provides a way to iteratively segment the semantics from coarse to fine, and ultimately iteratively refine the semantic segmentation results.

[0083] Step 130: Determine the segmentation semantics corresponding to the last semantic feature map as the image segmentation result.

[0084] It is understandable that after executing step 110 and executing step 120 multiple times, multiple segmentation semantics from coarse to fine are provided. Figure 3 For example, we get the segmentation semantics L 1 , L 2 , L 3 and L 4 , where segmentation semantics L 4 For the segmentation semantics corresponding to the last semantic feature map, the segmentation semantics L 4 It can be used as the image segmentation result.

[0085] The image segmentation method provided by the embodiment of the present invention can fully utilize the semantic context by extracting semantic feature maps at multiple scales, and then combine semantic embedding and attention conversion to perform coarse-to-fine iterative segmentation across layers, which helps to improve the accuracy of the segmentation results.

[0086] In addition, through cross-layer iterative segmentation, the number of segmentation layers can be reduced while ensuring segmentation accuracy, thereby helping to streamline the network. In this way, this method can be directly used for independent operation of various terminals without relying on server calculations.

[0087] For example, when it is loaded in the refrigerator, the food image can be segmented to identify whether the food is spoiled; or when it is loaded on the smoke extractor, the food image can be segmented to identify whether the food is done.

[0088] In some embodiments, step 120 , performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics, may include: step 121 , step 122 , step 123 and step 124 .

[0089] Step 121: semantically embed the semantic feature map of the current scale to obtain an embedded feature map of the current scale;

[0090] It should be noted that the semantic feature map of the current scale does not include the semantic feature map of the highest scale, that is, it does not include Figure 3 X in 1 .

[0091] In the process of embedding and encoding the semantic feature map, the feature representation of each category can be recorded, and the embedded features corresponding to each category can be distinguished through similarity calculation.

[0092] like Figure 5 As shown, by semantically embedding the semantic feature map X, the embedded feature map Y can be obtained. c .

[0093] Semantic embedding does not change the dimension of the semantic feature map, such as Figure 5 As shown, the semantic feature map X and the embedded feature map Y c They can be represented as a three-dimensional matrix HW×D, where H represents height, W represents width, and D represents the number of channels.

[0094] In some embodiments, step 121, semantically embedding the semantic feature map of the current scale to obtain the embedded feature map of the current scale, may include: embedding the semantic feature map X of the current scale into the memory tensor M to obtain the embedded feature map Y of the current scale. c .

[0095] The memory tensor M can record the global representation of the semantic context. The memory tensor M=[M1,M2,…,M C ]. Each element Mi in the memory tensor M = [m i1 ,m i2 ,…,m iN ] record the semantic representation of category i, i = 1, 2, ..., c, m ij ∈R D is a single memory feature, j = 1, 2, ..., N, N is the memory capacity, and D is the channel dimension.

[0096] like Figure 5 As shown, for each category c, given a single input first feature x∈R in the semantic feature map X D , the semantic embedding process is to embed the input first feature x into the semantic representation, and use the memory feature Mc to construct the embedded feature y∈R D :

[0097]

[0098] Among them, m cN ∈R D is a weight vector of non-negative entries summing to 1, w cn Represents the input feature x to feature M cn The embedding weights of .

[0099] In some embodiments, the above steps embed the semantic feature map X of the current scale into the memory tensor M to obtain the embedded feature map Y of the current scale. c ,include:

[0100] Determine the target memory feature Mc with the greatest similarity to the target first feature x in the semantic feature map X of the current scale from the memory tensor M; embed the target first feature x into the target memory feature Mc to obtain the embedded feature map Y of the current scale c The target embedding feature y in .

[0101] Perform the above steps for each first feature x in the semantic feature map X to obtain the embedded feature map Y c .

[0102] In the above method, the first input feature x is reconstructed to obtain the embedded feature y represented by the memory feature Mc. In actual implementation, the memory feature that is most similar to the first input feature x is selected to construct the embedded feature y.

[0103] Specifically, the weight vector w c The memory similarity in

[0104]

[0105] Among them, m cn' Refer to the above m ij The definition of is the n'th memory feature of category c, and d(,) is the similarity measure:

[0106] d(x,m cn )=x T m cn ,

[0107] It can be understood that the above semantic embedding process constructs the corresponding embedded feature y by extracting the memory feature Mc that is most similar to the first input feature x. For different features of the same category, since their most similar memory features are different, different embedded features can be obtained through similarity calculation. In addition, multiple memory features can represent category features with complex distributions. Assuming that a category has multiple feature centers, the semantic embedding process can record their feature representations and distinguish their corresponding embedded features through similarity calculation.

[0108] like Figure 4 and Figure 5 As shown, in the actual execution process, the above semantic embedding process can be completed through the semantic embedding module SE (Semantic Embedding Block).

[0109] The semantic embedding process based on a single first feature x, input semantic feature map X = [x1, x2, ..., x HW ], where x i ∈R D is the input feature of pixel i, i = 1, 2, ..., HW. HW is the size of the feature map (height times width).

[0110] like Figure 5 As shown, for each category c, we apply the semantic embedding process to each first feature x i And obtain the feature map Y c =[y1,y2,…,y HW ], y i ∈R D is with x i The corresponding embedded features, i = 1, 2, ..., HW. HW is the size of the feature map (height multiplied by width). By calculating the embedded features between all categories, we get the embedded feature map Y = [Y1, Y2, ..., Y C ].

[0111] Step 122: Perform attention conversion on the embedded feature map of the current scale through the target segmentation semantics to obtain the semantic representation features of the current scale;

[0112] like Figure 3 、 Figure 4 and Figure 6 As shown, by segmenting the semantic L 1 Perform attention conversion on the embedded feature map Y2 to obtain the semantic representation feature Z2; by segmenting the semantic L 2 Perform attention conversion on the embedded feature map Y3 to obtain the semantic representation feature Z3; by segmenting the semantic L 3 Perform attention transformation on the embedded feature map Y4 to obtain the semantic representation feature Z4.

[0113] like Figure 4 and Figure 6 As shown, in the actual execution process, the attention conversion process can be completed by the attention module SA (SemanticAttention Block).

[0114] Through the inter-layer attention mechanism, the model can focus on feature learning during residual learning, thereby further improving the accuracy and reliability of feature extraction.

[0115] In some embodiments, as Figure 6 As shown, step 122, performing attention conversion on the embedded feature map of the current scale through the target segmentation semantics to obtain the semantic representation features of the current scale, including:

[0116] Perform softmax operation on the target segmentation semantics L to obtain the probability map P;

[0117] The probability map P is used to perform attention conversion on the embedded feature map Y of the current scale to obtain the semantic representation feature Z of the current scale.

[0118] In actual implementation, the softmax operation is applied to the segmentation semantics L and a probability map P is obtained, P = [p1, p2, ..., p HW ], p i ∈R C is the probability vector of pixel i, i = 1, 2, ..., HW. HW is the size of the feature map (height times width). The probability map P is used as an attention map to aggregate the embedded features Y and obtain the semantic representation features Z = [z1, z2, ..., z HW ],z i ∈R D is the semantic representation feature of pixel i, i = 1, 2, ..., HW. HW is the size of the feature map (height times width).

[0119] For each pixel i, the semantic attention conversion is implemented by the following formula:

[0120]

[0121] Step 123: obtaining image features at the current scale based on the semantic representation features at the current scale;

[0122] The semantic representation feature Z of the current scale is connected with the semantic feature map X of the current scale and features are extracted to obtain the image features of the current scale.

[0123] like Figure 4 As shown in FIG, the semantic representation features of the current scale are connected to the semantic feature map X of the current scale (Concat), and then the connected map is subjected to feature extraction (Conv) to obtain the image features of the current scale.

[0124] The image features retrieved in this way are combined with the original semantic feature map X and the semantic representation features obtained by semantic embedding and attention, which can focus on the connection between semantic contexts.

[0125] like Figure 3 and Figure 4 As shown, the above steps 121 to 123 can be performed by the semantic representation module SR (Semantic Representation Module). Figure 4As shown in Figure 2, the semantic representation module SR combines the semantic embedding module SE and the attention module SA. The semantic representation module SR establishes a connection between feature information and semantic context through global semantic representation. For each single feature x, the feature is embedded into the semantic memory according to the semantic context to obtain the reconstructed semantic representation feature Z.

[0126] The features output by the semantic representation module SR are the image features of the current scale.

[0127] In other words, the above method selects corresponding memory features and embedding weights based on semantic context and feature information, respectively. Therefore, the semantic representation module SR of the present invention can adaptively learn the global representation of all categories.

[0128] Step 124: Fuse the image features of the current scale with the target segmentation semantics to obtain the current segmentation semantics.

[0129] like Figure 3 As shown, the image features output by the semantic representation module SR of the current layer are consistent with the segmentation semantics L of the previous scale. i-1 By fusion, we can get the current segmentation semantics L i .

[0130] The above image segmentation method, through semantic embedding and attention, as well as multiple fusions across layers, can fully utilize the semantic context and perform iterative segmentation from coarse to fine, with high accuracy of the segmentation results.

[0131] In some embodiments, step 120, semantic embedding and attention conversion are performed on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics, including: inputting the semantic feature map of the current scale into the current segmentation layer of the image segmentation model to obtain the segmentation semantics output by the current segmentation layer.

[0132] In other words, this step can be implemented by a neural network, wherein step 110 can be implemented by a feature extraction network, which can include an encoder-decoder architecture with a ResNet backbone, and steps 121-124 can be implemented by multiple segmentation layers, each segmentation layer can include the above-mentioned semantic representation module SR.

[0133] The image segmentation model can be trained using sample input images as samples and using sample segmentation results corresponding to the sample segmented images as sample labels. In the actual training process, the feature extraction network and the segmentation layer can be trained jointly or independently. For example, the feature extraction network can be a pre-trained network.

[0134] The loss function of the image segmentation model is the cross entropy loss function, and the cross entropy loss function is used to characterize the segmentation loss of each segmentation layer.

[0135] The image segmentation model includes a feature extraction network ResNet and a semantic representation segmentation network SRRNet (Semantic Rep-resentation Refinement Network). The input of the feature extraction network ResNet is the image to be segmented, and the output is multiple semantic feature maps (X 1 、X 2 、X 3 and X 4 ), the input of the semantic representation segmentation network SRRNet is the semantic feature map (X 1 、X 2 、X 3 and X 4 ), the output is the segmentation result.

[0136] It should be noted that the image segmentation method of the embodiment of the present invention adopts multi-scale feature information, captures the global semantic context through high-level features, and provides local text details through low-level features, and then refines the segmentation through cross-layer iteration. During segmentation, the semantic representation module SR is applied to all layers to further utilize the semantic context.

[0137] During training, deep supervision is performed through multi-scale loss to ensure that all 4 means Figure 3 The model shown has 4 layers and the final loss function is

[0138]

[0139] Among them, such as Figure 3 The GT shown is the ground truth, which represents the sample label, α s is the scale coefficient, which is used to balance the segmentation loss of different scales. For example, α is set based on experience. s , for example, we can set α with the scale s s =2 s-4 .

[0140] The inventors conducted experimental evaluations on the above method using the following two benchmark datasets: PASCAL VOC2012 and Cityscapes.

[0141] The PASCAL VOC 2012 dataset has 20 object classes and one background class, and contains 10,582 augmented images, 1,464 training images, 1,449 validation images, and 1,456 test images. The Cityscapes dataset has 19 semantic classes and provides 2,975, 500, and 1,525 finely annotated images for training, validation, and testing, respectively.

[0142] The encoder-decoder baseline model ASPP module is constructed using the pre-trained ResNet backbone as the feature extraction network. All models adopt the expansion strategy of the last two ResNet modules. The memory capacity of SRRNet is set to N=8. During the training phase, poly learning rate scheduling is used.

[0143] N iter is the current number of iterations, N total is the total number of iterations, γ0 is the base learning rate, and the stochastic gradient descent (SGD) optimizer with momentum 0.9 and weight decay coefficient 0.0001 is used.

[0144] During training, random horizontal flipping and random scaling strategies are used, with scale ratios ranging from 0.5 to 2.0. For the PASCAL VOC 2012 dataset, the training batch size is 16, the base learning rate is 0.001, the training epochs are 80, and the training size is 480 x 480. During testing, horizontal flipping and multi-scale input strategies are used to improve segmentation results, with scales set to {0.5, 0.75, 1.0, 1.25, 1.5, 1.75}.

[0145] For the Cityscapes dataset, the training batch size is 8, the base learning rate is 0.001, the training epochs are 240, and the training size is 768 x 768. During the test phase, horizontal flipping and multi-scale input strategies are used to improve the segmentation results. The scales are set to {0.75, 1.0, 1.25, 1.5, 1.75, 2.0}. In addition, a warm-up strategy with a training epoch of 5, the online hard example mining (OHEM) method, and a multi-grid method with the last ResNet module size of (4, 8, 16) are also used.

[0146] Through experiments, it was found that the semantic representation segmentation network SRRNet of the present invention achieved better performance (85.4%) than other methods on PASCAL VOC 2012. Among these comparison methods, EncNet uses an encoding layer to capture the encoding semantics of the global scene, while DFN uses a boundary network to capture semantic information and guides feature learning through semantic boundary supervision. In addition, as shown in Table 2, SRRNet also achieved an excellent performance of 82.3% on Cityscapes, which is much better than previous methods. In addition, SRRNet with a ResNet-50 backbone even outperforms some semantic context-based methods with ResNet-100, including DFN and CPNet, further verifying the effectiveness of the model of the present invention.

[0147] Table 1 Quantitative evaluation of PASCAL VOC 2012 dataset

[0148]

[0149] Table 2 Quantitative evaluation of Cityscapes PASCAL VOC 2012 dataset

[0150]

[0151] The image segmentation device provided by an embodiment of the present invention is described below. The image segmentation device described below and the image segmentation method described above can be referenced to each other.

[0152] like Figure 7 As shown, the image segmentation device according to the embodiment of the present invention includes: an extraction module 710 , a semantic segmentation module 720 and a determination module 730 .

[0153] Extraction module 710, used to extract features from the image to be segmented, obtain multiple semantic feature maps of different scales, sort the multiple semantic feature maps from high to low according to scale, and use the first semantic feature map as the first segmentation semantic;

[0154] Semantic segmentation module 720, configured to perform semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics, where the semantic feature map of the current scale includes all semantic feature maps except the first semantic feature map, and the target segmentation semantics is the segmentation semantics corresponding to the semantic feature map of the previous scale;

[0155] The determination module 730 is used to determine that the segmentation semantics corresponding to the last semantic feature map is the image segmentation result.

[0156] The image segmentation device provided by the embodiment of the present invention can fully utilize the semantic context by extracting semantic feature maps of multiple scales, and then combine semantic embedding and attention conversion to perform coarse-to-fine iterative segmentation across layers, which helps to improve the accuracy of the segmentation results.

[0157] In addition, through cross-layer iterative segmentation, the number of segmentation layers can be reduced while ensuring segmentation accuracy, thereby helping to streamline the network. In this way, this method can be directly used for independent operation of various terminals without relying on server calculations.

[0158] In some embodiments, the semantic segmentation module 720 is also used to perform semantic embedding on the semantic feature map of the current scale to obtain the embedded feature map of the current scale; perform attention conversion on the embedded feature map of the current scale through the target segmentation semantics to obtain the semantic representation features of the current scale; obtain the image features of the current scale based on the semantic representation features of the current scale; and fuse the image features of the current scale with the target segmentation semantics to obtain the current segmentation semantics.

[0159] In some embodiments, the semantic segmentation module 720 is further configured to embed the semantic feature map of the current scale into the memory tensor to obtain the embedded feature map of the current scale.

[0160] In some embodiments, the semantic segmentation module 720 is also used to determine, from the memory tensor, a target memory feature that has the greatest similarity to the target first feature in the semantic feature map of the current scale; embed the target first feature into the target memory feature to obtain the target embedded feature in the embedded feature map of the current scale.

[0161] In some embodiments, the semantic segmentation module 720 is further configured to perform a softmax operation on the target segmentation semantics to obtain a probability map; and perform an attention conversion on the embedded feature map of the current scale through the probability map to obtain the semantic representation feature of the current scale.

[0162] Through the inter-layer attention mechanism, the model can focus on feature learning during residual learning, thereby further improving the accuracy and reliability of feature extraction.

[0163] In some embodiments, the semantic segmentation module 720 is further configured to connect the semantic representation features of the current scale with the semantic feature map of the current scale, and perform feature extraction on the connected map to obtain image features of the current scale.

[0164] The image features retrieved in this way are combined with the original semantic feature map X and the semantic representation features obtained by semantic embedding and attention, which can focus on the connection between semantic contexts.

[0165] In some embodiments, the semantic segmentation module 720 is further configured to input the semantic feature map of the current scale into the current segmentation layer of the image segmentation model to obtain the segmentation semantics output by the current segmentation layer.

[0166] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the image segmentation method, which includes: extracting features from the image to be segmented to obtain multiple semantic feature maps of different scales, sorting the multiple semantic feature maps from high to low according to the scale, and taking the first semantic feature map as the first segmentation semantic; performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantic to obtain the current segmentation semantic, the semantic feature map of the current scale includes other semantic feature maps except the first semantic feature map, and the target segmentation semantic is the segmentation semantic corresponding to the semantic feature map of the previous scale; determining the segmentation semantic corresponding to the last semantic feature map as the image segmentation result.

[0167] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0168] Furthermore, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image segmentation method provided by the above-mentioned method embodiments, the method including: performing feature extraction on the image to be segmented to obtain multiple semantic feature maps of different scales, sorting the multiple semantic feature maps from high to low according to the scale, and taking the first semantic feature map as the first segmentation semantic; performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantic to obtain the current segmentation semantic, the semantic feature map of the current scale includes other semantic feature maps except the first semantic feature map, and the target segmentation semantic is the segmentation semantic corresponding to the semantic feature map of the previous scale; determining the segmentation semantic corresponding to the last semantic feature map as the image segmentation result.

[0169] On the other hand, an embodiment of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image segmentation method provided by the above-mentioned embodiments, the method comprising: performing feature extraction on the image to be segmented to obtain multiple semantic feature maps of different scales, sorting the multiple semantic feature maps from high to low according to scale, and taking the first semantic feature map as the first segmentation semantic; performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantic to obtain the current segmentation semantic, the semantic feature map of the current scale includes other semantic feature maps except the first semantic feature map, and the target segmentation semantic is the segmentation semantic corresponding to the semantic feature map of the previous scale; determining the segmentation semantic corresponding to the last semantic feature map as the image segmentation result.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0171] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0173] The above embodiments are intended to illustrate the present invention only and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that various combinations, modifications, or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and should be encompassed by the scope of the claims of the present invention.

Claims

1. An image segmentation method, characterized in that: include: Perform feature extraction on the image to be segmented to obtain multiple semantic feature maps of different scales, wherein the multiple semantic feature maps are sorted from high to low according to the scale, and the first semantic feature map is used as the first segmentation semantic; Performing semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics, wherein the semantic feature map of the current scale includes other semantic feature maps except the first semantic feature map, and the target segmentation semantics is the segmentation semantics corresponding to the semantic feature map of the previous scale; Determine the segmentation semantics corresponding to the last semantic feature map as the image segmentation result; The feature extraction is performed on the image to be segmented to obtain multiple semantic feature maps of different scales, including: Performing context semantic capture on the image to be segmented to obtain a plurality of first semantic features, wherein the dimensions of the first semantic features are the same; Based on the second semantic feature of the previous layer, the first semantic feature of the current layer is decoded to obtain the second semantic feature of the current layer; Feature extraction is performed on the second semantic feature of the current layer to obtain a semantic feature map of the current layer; wherein the second semantic feature of the highest layer is obtained by decoding based on the first semantic feature of the highest layer.

2. The image segmentation method according to claim 1, wherein: The semantic feature map of the current scale and the target segmentation semantics are semantically embedded and attention-converted to obtain the current segmentation semantics, including: Perform semantic embedding on the semantic feature map of the current scale to obtain the embedded feature map of the current scale; Performing attention conversion on the embedded feature map of the current scale through target segmentation semantics to obtain semantic representation features of the current scale; Obtaining image features at the current scale based on the semantic representation features at the current scale; The image features of the current scale are fused with the target segmentation semantics to obtain the current segmentation semantics.

3. The image segmentation method according to claim 2, characterized in that The semantic embedding of the semantic feature map of the current scale to obtain the embedded feature map of the current scale includes: Embed the semantic feature map of the current scale into the memory tensor to obtain the embedded feature map of the current scale.

4. The image segmentation method according to claim 3, characterized in that The embedding of the semantic feature map of the current scale into the memory tensor to obtain the embedded feature map of the current scale includes: Determine the target memory feature that has the greatest similarity to the first feature of the target in the semantic feature map of the current scale from the memory tensor; The target first feature is embedded into the target memory feature to obtain the target embedded feature in the embedded feature map of the current scale.

5. The image segmentation method according to claim 2, wherein: The step of performing attention conversion on the embedded feature map of the current scale by using target segmentation semantics to obtain semantic representation features of the current scale includes: Perform softmax operation on the target segmentation semantics to obtain a probability map; Attention conversion is performed on the embedded feature map of the current scale through the probability map to obtain the semantic representation feature of the current scale.

6. The image segmentation method according to claim 2, wherein: The obtaining of image features at the current scale based on the semantic representation features at the current scale includes: The semantic representation features of the current scale are connected with the semantic feature map of the current scale and features are extracted to obtain image features of the current scale.

7. The image segmentation method according to claim 1, wherein: The semantic feature map of the current scale and the target segmentation semantics are semantically embedded and attention-converted to obtain the current segmentation semantics, including: The semantic feature map of the current scale is input into the current segmentation layer of the image segmentation model to obtain the segmentation semantics output by the current segmentation layer.

8. The image segmentation method according to claim 7, characterized in that: The loss function of the image segmentation model is a cross entropy loss function, and the cross entropy loss function is used to characterize the segmentation loss of each segmentation layer.

9. An image segmentation device, characterized in that: include: An extraction module is used to extract features from the image to be segmented to obtain multiple semantic feature maps of different scales, wherein the multiple semantic feature maps are sorted from high to low according to the scale, and the first semantic feature map is used as the first segmentation semantic; A semantic segmentation module is configured to perform semantic embedding and attention conversion on the semantic feature map of the current scale and the target segmentation semantics to obtain the current segmentation semantics, wherein the semantic feature map of the current scale includes other semantic feature maps except the first semantic feature map, and the target segmentation semantics is the segmentation semantics corresponding to the semantic feature map of the previous scale; A determination module is used to determine the segmentation semantics corresponding to the last semantic feature map as the image segmentation result; The extraction module is further configured to perform context semantic capture on the image to be segmented to obtain a plurality of first semantic features, each of which has the same dimension; and decode the first semantic features of the current layer based on the second semantic features of the previous layer to obtain the second semantic features of the current layer; Feature extraction is performed on the second semantic feature of the current layer to obtain a semantic feature map of the current layer; wherein the second semantic feature of the highest layer is obtained by decoding based on the first semantic feature of the highest layer.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the image segmentation method according to any one of claims 1 to 8 are implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image segmentation method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image segmentation method according to any one of claims 1 to 8 are implemented.

Images