Icon semantic segmentation method, system and storage medium

Abstract

The present invention discloses an icon semantic segmentation method, system and storage medium, including: step S1, dividing an image data set into a training set and a test set, and performing data preprocessing on icon images and corresponding labels; step S2, constructing a first icon semantic segmentation model based on a multi-level feature aggregation mechanism; step S3, training the first icon semantic segmentation model with the training set icon images; step S4, constructing a second icon semantic segmentation model based on a nested encoding and decoding structure; step S5, training the second icon semantic segmentation model with the training set icon images; step S6, inputting non-transparent bottom icon test images into the trained model respectively, outputting the corresponding semantic segmentation mask map, calculating the similarity score between the semantic segmentation mask map and the text by comparing the learning model, and automatically selecting the semantic segmentation mask map with the highest score as the final semantic segmentation result. This case improves the performance and accuracy of the icon semantic segmentation algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision technology, and in particular relates to an icon semantic segmentation method based on dual-model prediction and automatic screening mechanism. Background Art

[0002] With the rapid development of digital media, icons, as an important carrier of visual information, play an indispensable role in user interface design, brand communication, advertising creativity and other fields. In recent years, the rise of large models of Wenshengtu has greatly enriched the possibilities of icon design, allowing designers to generate diverse and high-quality icons with unprecedented speed and flexibility. However, despite the great advantages of these large models in creativity and diversity, the generated icons often face a common problem: the background is opaque and the color is uncontrollable, and they may even come with redundant decorative patterns. In practical applications, such as poster design and advertising image production, icons often need to be embedded as independent elements into complex backgrounds to achieve visual harmony and unity. At this time, the uncontrollability and redundant elements of the icon background become bottlenecks that restrict its flexible application. Therefore, how to efficiently and accurately separate the icon foreground from the complex background has become a technical problem that needs to be solved urgently.

[0003] As an important technology in the field of computer vision, semantic segmentation technology is centered on assigning a category label to each pixel in the image, thereby achieving a detailed understanding of the image content. Through semantic segmentation, designers can easily obtain icons with transparent backgrounds, and then freely adjust their colors, sizes and layouts to meet different design requirements. However, when applied to icon semantic segmentation, existing semantic segmentation methods often face the problem of insufficient segmentation. Since icon design often pursues extreme details and creativity, the current mainstream semantic segmentation models are prone to mis-segmentation or missed segmentation when dealing with icons with complex structures and subtle features, especially when the icon edge is similar to the background color or there are small hollow patterns inside the icon. This problem is particularly prominent. Therefore, in response to the specific needs of icon semantic segmentation, it is of great practical significance and application value to develop a semantic segmentation method that can accurately identify and separate the foreground and background of icons, especially one that can effectively deal with complex structures and subtle features. This not only helps to improve the work efficiency of designers, but also promotes the innovation and enrichment of digital media content, bringing more possibilities to industries such as advertising and design. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to provide an icon semantic segmentation method based on a dual model prediction and automatic screening mechanism. The method constructs a dual pyramid pooling module, mines context information to obtain multi-scale features, designs a multi-scale feature fusion module, aggregates cross-scale feature information, constructs a residual encoding and decoding module, and adopts a nested encoding and decoding structure to extract multi-scale features within a single stage and fuse multi-scale features between multiple stages, thereby improving the network's ability to understand complex scenes and the recognition accuracy of subtle structures. At the same time, a contrast learning model is introduced for automatic screening, which effectively improves the performance of the icon semantic segmentation algorithm.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] The icon semantic segmentation method based on dual model prediction and automatic screening mechanism specifically includes the following steps:

[0007] Step S1: Divide the non-transparent background icon image dataset into a training set and a test set, and perform data preprocessing on the non-transparent background icon images and corresponding labels in the dataset;

[0008] Step S2: constructing a first icon semantic segmentation model based on a multi-level feature aggregation mechanism, the first icon semantic segmentation model comprising a dual pyramid pooling module and a multi-scale feature fusion module, mining context information through the dual pyramid pooling module to obtain multi-scale features, and aggregating cross-scale feature information through the multi-scale feature fusion module and predicting semantic segmentation results;

[0009] Step S3: constructing a training pipeline for the first icon semantic segmentation model, and using the training pipeline and training set icon images to train the first icon semantic segmentation model;

[0010] Step S4: constructing a second icon semantic segmentation model based on a nested codec structure, wherein the second icon semantic segmentation model includes a residual codec module, through which multi-scale features within a single stage are extracted and multi-scale features between multiple stages are fused;

[0011] Step S5: constructing a training pipeline for the second icon semantic segmentation, and using the training pipeline and the training set icon images to train the second icon semantic segmentation model;

[0012] Step S6: Input the non-transparent background icon test image into the trained first icon semantic segmentation model and the second icon semantic segmentation model respectively, output the corresponding semantic segmentation mask map, calculate the similarity score between the semantic segmentation mask map and the text description through the comparison learning model, automatically select the semantic segmentation mask map with the highest score as the final semantic segmentation result, and extract the icon with transparent background according to the selected semantic segmentation mask map.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] Step S11: dividing the non-transparent background icon image dataset into a training set and a test set according to a certain ratio;

[0015] Step S12: performing data enhancement processing on the icon images in the training set;

[0016] Step S13: preprocessing the icon image after data enhancement in step S12, converting it into the input of the icon segmentation network, randomly cropping the icon image to a uniform size, and then normalizing it so that the icon image data conforms to the standard normal distribution;

[0017] Step S14: Perform the same data enhancement and preprocessing operations on the labels as on the icon images.

[0018] Furthermore, in step S2, constructing a first icon semantic segmentation model based on a multi-level feature aggregation mechanism includes:

[0019] Using STDC as the backbone network of the first icon semantic segmentation model, the network can extract features from the non-transparent icon image after preprocessing in step S1, and obtain five feature maps with different scales and different numbers of channels. ;

[0020] Construct a dual-pyramid pooling module to connect to the end of the network to capture global dependencies;

[0021] A multi-scale feature fusion module is used as the decoder of the network to fuse feature maps of different dimensions for semantic segmentation; the input of the network is a non-transparent icon image And the corresponding truth mask , the output of the network is the icon semantic segmentation mask map .

[0022] Further, in step S2, the context information is mined by the dual pyramid pooling module to obtain multi-scale features and the cross-scale feature information is aggregated by the multi-scale feature fusion module to predict the semantic segmentation result, which specifically includes the following steps:

[0023] Step S21: Transform the non-transparent icon image Input into the encoder of the first icon semantic segmentation model network based on the multi-level feature aggregation mechanism to obtain the dimensions of Five-layer feature map ,in and Represent the height, width and number of channels of the feature map respectively;

[0024] Step S22: Get the data from step S21 As the input of the dual pyramid pooling module, the size of the feature map is first unified to , and concatenate along the channel dimension to obtain the feature , and then adaptive average pooling of different scales is used to optimize the features Downsampling is performed to obtain three layers of feature maps of different sizes , and then the Transformer encoder is used to capture global dependencies and semantic information to obtain an enhanced three-layer feature map , and then restore the size of the feature map to ,Will and Concatenate along the channel dimension, and finally obtain the dimension of Aggregate feature map of ;

[0025] Step S23: The five-layer feature maps obtained in steps S21 and S22 are and as the input of the decoder, the decoder is divided into 5 multi-scale feature fusion modules, which process features of different dimensions respectively and use the attention mechanism to perform weighted fusion of multi-scale features;

[0026] After multi-scale feature fusion processing, convolution kernels of different sizes are used to compare the input Perform depth-wise convolution to extract spatial features independently on each input channel, unify the size of the feature map by upsampling, and then perform point-wise convolution on the feature map to adjust the number of channels, and obtain , respectively perform average pooling and maximum pooling on the feature map, while mining global features and texture and edge information, and concatenate the two pooling results along the channel dimension to obtain the feature map and , the two layers of feature maps and It is spliced ​​again along the channel dimension, and then the attention map integrating multiple information is obtained through convolution and activation operations. , and then Respectively with attention map Perform element-wise multiplication to obtain the enhanced , and finally the enhanced Concatenate along the channel dimension to get the fused feature map ;

[0027] Step S24: fusion feature map obtained in step S23 Upsample to the same size as the input image Match and compare with the input image Concatenate in the channel dimension, then perform convolution, batch normalization and activation operations to obtain the semantic segmentation mask map .

[0028] Further, in step S3, the training of the first icon semantic segmentation model using the training pipeline and the training set icon images specifically includes the following steps:

[0029] Step S31: input a batch of non-transparent bottom icon images and corresponding real masks in the training set into the first icon semantic segmentation model network for training, and predict the icon semantic segmentation mask map;

[0030] Step S32: according to the loss function of the first icon semantic segmentation model network, the gradient of each parameter in the network is calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method;

[0031] Step S33: Repeat steps S31 to S32 in batches until the loss value calculated in step S32 converges and stabilizes, save the network parameters, and complete the training process of the first icon semantic segmentation model.

[0032] Furthermore, in step S4, constructing a second icon semantic segmentation model based on a nested codec structure includes:

[0033] The main semantic segmentation network of the second icon semantic segmentation model is UNet. The network replaces the convolutional layer in each stage with a residual codec structure, which can extract multi-scale local and global features from the non-transparent icon image after preprocessing in step S1. Through step-by-step upsampling, splicing and convolution operations, the multi-scale features are reintegrated into high-resolution feature maps, enhancing the network's ability to process complex image content and accurately capture subtle features. After convolution activation operations, the icon semantic segmentation mask map is finally output. .

[0034] Furthermore, in step S4, extracting multi-scale features in a single stage through the residual encoding and decoding module and fusing multi-scale features between multiple stages specifically includes the following steps:

[0035] Step S41: Transform the non-transparent icon image The dimensions of the four-stage encoder input into the second icon semantic segmentation model network are Four-layer feature map ,in and Represent the height, width and number of channels of the feature map respectively;

[0036] For input features Perform a convolution, batch normalization and activation operation to extract local features and obtain the initial feature map , then proceed Sub-sampling, convolution, batch normalization and activation operations are performed to continuously expand the receptive field to obtain richer global and local features. Layer feature map , then the last layer of feature map Perform a dilated convolution, batch normalization, and activation operation to capture a wider range of contextual information without losing spatial dimensions and obtain an enhanced feature map. , and then by The concatenation, convolution, batch normalization, activation and upsampling operations are performed to fuse multi-scale features and obtain Layer feature map , the last layer of feature maps containing the richest semantic information and Add together to get the composite feature map ;

[0037] Step S42: The feature map with the highest semantics obtained in step S41 As the input of the two-layer dilated residual codec module, it further extracts context information without losing spatial resolution.

[0038] First, the input features Perform a convolution, batch normalization and activation operation to extract local features to form an initial feature map , then proceed The dilated convolution, batch normalization and activation operations gradually expand the receptive field while maintaining the spatial resolution to obtain richer features. Layer feature map , and then the last two layers of feature maps and Splice along the channel dimension to obtain a composite feature map , and then perform a dilated convolution, batch normalization, and activation operation to capture more extensive context information and obtain the feature map , and then by Sub-residual connection, dilated convolution, batch normalization and activation operations are used to fuse multi-scale features to obtain Layer feature map , the last layer of feature maps containing the richest semantic information and Add to get the output feature map ;

[0039] Step S43: The feature map with the highest semantics obtained in step S42 As the input of the five-layer decoder, multi-scale feature fusion is performed while upsampling, and the final output is a six-layer enhanced feature map that takes into account both global semantic information and local detail information. , the dimensions are ;

[0040] Step S44: The six-layer feature map obtained in step S43 Convolution is performed separately, and the feature maps of the last five layers are Upsample to make it consistent with the feature map The six layers of feature maps are concatenated along the channel dimension, and convolution and activation operations are performed to finally obtain the semantic segmentation mask map. .

[0041] Further, in step S5, the training of the second icon semantic segmentation model using the training pipeline and the training set icon images specifically includes the following steps:

[0042] Step S51: inputting a batch of non-transparent bottom icon images and corresponding labels in the training set into the second icon semantic segmentation model network for training, and predicting an icon semantic segmentation mask map;

[0043] Step S52: according to the loss function of the second icon semantic segmentation model network, the gradient of each parameter in the network is calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method;

[0044] Step S53: Repeat steps S51 to S52 in batches until the loss value calculated in step S52 converges and stabilizes, save the network parameters, and complete the training process of the second icon semantic segmentation model.

[0045] Furthermore, step S6 specifically includes the following steps:

[0046] Step S61: Input the non-transparent icon images in the test set into the trained first icon semantic segmentation model and the second icon semantic segmentation model respectively, and output the corresponding icon semantic segmentation mask map and ;

[0047] Step S62: Using the data obtained from step S61 and From the input non-transparent bottom icon image Extract icons from and , and the corresponding text description The pre-trained CLIP model is input together to calculate the similarity score, and the one with the higher score is taken as the final semantic segmentation mask map to achieve automatic screening of the icon semantic segmentation results.

[0048] And, an icon semantic segmentation system based on dual model prediction and automatic screening mechanism, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, it can implement the icon semantic segmentation method based on dual model prediction and automatic screening mechanism as described above.

[0049] A computer-readable storage medium stores computer program instructions that can be executed by a processor. When the processor executes the computer program instructions, the icon semantic segmentation method based on dual-model prediction and automatic screening mechanism as described above can be implemented.

[0050] After adopting the above scheme, the present invention has the following beneficial effects:

[0051] The present invention and its preferred solution construct dual models. The first icon semantic segmentation model constructs a dual pyramid pooling module to mine context information to obtain multi-scale features. At the same time, a multi-scale feature fusion module is designed to aggregate cross-scale feature information. The second icon semantic segmentation model constructs a residual encoding and decoding module, adopts a nested encoding and decoding structure, extracts multi-scale features within a single stage, and fuses multi-scale features between multiple stages, thereby improving the network's ability to understand complex scenes and the recognition accuracy of subtle structures. At the same time, a contrast learning model is introduced for automatic screening, thereby effectively improving the performance of the icon semantic segmentation algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other deformation forms can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a flow chart of the icon semantic segmentation method based on dual model prediction and automatic screening mechanism of the present invention.

[0054] Figure 2 It is a structural diagram of a network model in an embodiment of the present invention.

[0055] Figure 3 It is a structural diagram of the dual pyramid pooling module and the multi-scale feature fusion module in the present invention.

[0056] Figure 4 It is a structural diagram of a residual coding and decoding module and a hole residual coding and decoding module in the present invention;

[0057] Figure 5 This is a comparison chart of semantic segmentation examples before and after using this method. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] like Figure 1 As shown, the icon semantic segmentation method based on dual model prediction and automatic screening mechanism disclosed in this embodiment specifically includes the following steps:

[0060] The icon semantic segmentation method based on dual model prediction and automatic screening mechanism specifically includes the following steps:

[0061] Step S1: Divide the non-transparent background icon image dataset into a training set and a test set, and perform data preprocessing on the non-transparent background icon images and corresponding labels in the dataset;

[0062] Step S2: constructing a first icon semantic segmentation model based on a multi-level feature aggregation mechanism, the first icon semantic segmentation model includes a dual pyramid pooling module and a multi-scale feature fusion module, mining context information through the dual pyramid pooling module to obtain multi-scale features, and aggregating cross-scale feature information and predicting semantic segmentation results through the multi-scale feature fusion module, thereby improving the network's recognition accuracy for subtle structures;

[0063] Step S3: constructing a training pipeline for the first icon semantic segmentation model, and using the training pipeline and training set icon images to train the first icon semantic segmentation model;

[0064] Step S4: constructing a second icon semantic segmentation model based on a nested codec structure, wherein the second icon semantic segmentation model includes a residual codec module. The overall model adopts a nested codec structure, through which multi-scale features within a single stage are extracted, and multi-scale features between multiple stages are integrated to enhance the network's ability to understand complex scenes;

[0065] Step S5: constructing a training pipeline for the second icon semantic segmentation, and using the training pipeline and the training set icon images to train the second icon semantic segmentation model;

[0066] Step S6: Input the non-transparent background icon test image into the trained first icon semantic segmentation model and the second icon semantic segmentation model respectively, output the corresponding semantic segmentation mask map, calculate the similarity score between the semantic segmentation mask map and the text description through the comparison learning model, and automatically select the semantic segmentation mask map with the highest score as the final semantic segmentation result, so as to extract the icon with transparent background according to the selected semantic segmentation mask map.

[0067] In this embodiment, step S1 specifically includes the following steps:

[0068] Step S11: dividing the non-transparent background icon image dataset into a training set and a test set according to a certain ratio;

[0069] Step S12: performing data enhancement processing on the icon images in the training set to increase the number of samples;

[0070] Step S13: preprocessing the icon image after data enhancement in step S12, converting it into the input of the icon segmentation network, randomly cropping the icon image to a uniform size, and then normalizing it so that the icon image data conforms to the standard normal distribution;

[0071] Step S14: Perform the same data enhancement and preprocessing operations on the label as on the icon image to ensure that the segmented area corresponds to the icon image.

[0072] Furthermore, in the above step S2, constructing the first icon semantic segmentation model based on the multi-level feature aggregation mechanism includes:

[0073] Using STDC as the backbone network of the first icon semantic segmentation model, the network can extract features from the non-transparent icon image after preprocessing in step S1, and obtain five feature maps with different scales and different numbers of channels. ;

[0074] Construct a dual-pyramid pooling module to connect to the end of the network to capture global dependencies;

[0075] A multi-scale feature fusion module is used as the decoder of the network to fuse feature maps of different dimensions for semantic segmentation; the input of the network is a non-transparent icon image And the corresponding truth mask , the output of the network is the icon semantic segmentation mask map .

[0076] The STDC described in this embodiment is a real-time semantic segmentation network. STDC has two main characteristics. First, as the network depth increases, the number of feature channels will gradually decrease, thereby reducing the amount of calculation; second, the output of the STDC module fuses the output feature maps of multiple blocks, thereby containing information at multiple scales.

[0077] In the above step S2, mining context information by the dual pyramid pooling module to obtain multi-scale features and aggregating cross-scale feature information by the multi-scale feature fusion module and predicting semantic segmentation results specifically include the following steps:

[0078] Step S21: Transform the non-transparent icon image Input into the encoder of the first icon semantic segmentation model network based on the multi-level feature aggregation mechanism to obtain the dimensions Five-layer feature map ,in and Represent the height, width and number of channels of the feature map respectively;

[0079] Step S22: Get the data from step S21 As the input of the dual pyramid pooling module, the size of the feature map is first unified to , and concatenate along the channel dimension to obtain the feature , and then adaptive average pooling of different scales is used to optimize the features Downsampling is performed to obtain three layers of feature maps of different sizes , and then the Transformer encoder is used to capture global dependencies and semantic information to obtain an enhanced three-layer feature map , and then restore the size of the feature map to ,Will and Concatenate along the channel dimension, and finally obtain the dimension of Aggregate feature map of , the specific expression is:

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] in, represents the average pooling operation, Indicates that features are concatenated in a new dimension. represents the adaptive average pooling operation, is the pooling size, surface It shows that the Transformer encoder is used to extract global features. represents the upsampling operation, Indicates convolution, batch normalization and ReLU activation operations on features;

[0088] Step S23: The five-layer feature maps obtained in steps S21 and S22 are and are used as the input of the decoder. The decoder is divided into 5 multi-scale feature fusion modules, which process features of different dimensions respectively. The attention mechanism is used to perform weighted fusion of multi-scale features to enhance semantic information. The specific expression is:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] in, Indicates that the scale of the feature is transformed by interpolation operation. Indicates multi-scale feature fusion. Represents the spatial dimension size of the input feature

[0095] After multi-scale feature fusion processing, in order to extract local detail features, we first use convolution kernels of different sizes to align the input Perform step-by-step depth convolution to extract spatial features independently on each input channel, unify the size of the feature map by upsampling, and then perform point-by-point convolution on the feature map to adjust the number of channels. , respectively perform average pooling and maximum pooling on the feature map, while mining global features and texture and edge information, and concatenate the two pooling results along the channel dimension to obtain the feature map and , the two layers of feature maps and It is spliced ​​again along the channel dimension, and then the attention map integrating multiple information is obtained through convolution and activation operations. , and then Respectively with attention map Perform element-wise multiplication to obtain the enhanced , and finally the enhanced Concatenate along the channel dimension to get the fused feature map , the specific expression is as follows:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] in, represents the depth-wise convolution operation, surface Indicates the upsampling operation, surface shows the point-by-point convolution operation, Indicates that features are concatenated in a new dimension. surface represents the average pooling operation, represents the maximum pooling operation, Indicates convolution and ReLU activation operations on features, ⨀ indicates Hadamard product;

[0105] Step S24: fusion feature map obtained in step S23 Upsample to the same size as the input image Match and compare with the input image Concatenate in the channel dimension, then perform convolution, batch normalization and activation operations to obtain the semantic segmentation mask map , the specific expression is as follows:

[0106]

[0107]

[0108]

[0109] in, represents the upsampling operation, Indicates that features are concatenated in a new dimension. Represents convolution, batch normalization, and ReLU activation operations on the features.

[0110] In the above step S3, the use of the training pipeline and the training set icon images to train the first icon semantic segmentation model specifically includes the following steps:

[0111] Step S31: input a batch of non-transparent bottom icon images and corresponding real masks in the training set into the first icon semantic segmentation model network for training, and predict the icon semantic segmentation mask map;

[0112] Step S32: According to the loss function of the first icon semantic segmentation model network, the gradient of each parameter in the network is calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method; the loss function of the network as follows:

[0113]

[0114]

[0115]

[0116]

[0117] in, is the loss function of the network, represents the mean relative squared error loss function; represents the gradient loss function; represents the predicted semantic segmentation mask map, represents the true semantic segmentation mask map, represents the number of pixels in the mask image, is a small positive number used to avoid division by zero. and Respectively x Axis and y The Sobel operator convolution kernel in the axial direction, Represents the convolution operation;

[0118] Step S33: Repeat steps S31 to S32 in batches until the loss value calculated in step S32 converges and stabilizes, save the network parameters, and complete the training process of the first icon semantic segmentation model.

[0119] In the above step S4, the construction of the second icon semantic segmentation model based on the nested codec structure includes:

[0120] The main semantic segmentation network of the second icon semantic segmentation model is UNet. The network replaces the convolutional layer in each stage with a residual codec structure, which can extract multi-scale local and global features from the non-transparent icon image after preprocessing in step S1. Through step-by-step upsampling, splicing and convolution operations, the multi-scale features are reintegrated into high-resolution feature maps, enhancing the network's ability to process complex image content and accurately capture subtle features. After convolution activation operations, the icon semantic segmentation mask map is finally output. The UNet described in this embodiment is a semantic segmentation model widely used in the field of computer vision, and is particularly suitable for medical image segmentation tasks. It extracts features from images and performs pixel-level classification through downsampling and upsampling processes, thereby achieving semantic segmentation of images.

[0121] In the above step S4, extracting multi-scale features in a single stage through the residual encoding and decoding module and fusing multi-scale features between multiple stages specifically include the following steps:

[0122] Step S41: Transform the non-transparent icon image The dimensions of the four-stage encoder input into the second icon semantic segmentation model network are Four-layer feature map ,in and Represent the height, width and number of channels of the feature map respectively. The specific expression is:

[0123]

[0124]

[0125] in, represents the maximum pooling operation, Indicates the context information of features extracted using the residual codec module. Indicates the number of encoding and decoding.

[0126] First, the input features Perform a convolution, batch normalization and activation operation to extract local features and obtain the initial feature map , then proceed Sub-sampling, convolution, batch normalization and activation operations are performed to continuously expand the receptive field to obtain richer global and local features. Layer feature map , then the last layer of feature map Perform a dilated convolution, batch normalization, and activation operation to capture a wider range of contextual information without losing spatial dimensions and obtain an enhanced feature map. , and then by The concatenation, convolution, batch normalization, activation and upsampling operations are performed to fuse multi-scale features and obtain Layer feature map , the last layer of feature maps containing the richest semantic information and Add together to get the composite feature map , the specific expression is:

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134] in, Indicates convolution, batch normalization and ReLU activation operations on the features. represents the maximum pooling operation, Indicates that the features are subjected to dilated convolution, batch normalization, and ReLU activation operations. represents the convolution expansion rate, represents the upsampling operation, Indicates that features are concatenated in a new dimension;

[0135] Step S42: The feature map with the highest semantics obtained in step S41 As the input of the two-layer hole residual codec module, the context information is further extracted without losing spatial resolution. The specific expression is:

[0136]

[0137] in ， represents the maximum pooling operation, represents the upsampling operation, Indicates the context information of features extracted using a residual codec module containing a dilated convolutional layer. Indicates the number of encoding and decoding.

[0138] First, the input features Perform a convolution, batch normalization and activation operation to extract local features to form an initial feature map , then proceed The dilated convolution, batch normalization and activation operations gradually expand the receptive field while maintaining the spatial resolution to obtain richer features. Layer feature map , and then the last two layers of feature maps and Splice along the channel dimension to get a composite feature map , and then perform a dilated convolution, batch normalization, and activation operation to capture more extensive context information and obtain the feature map , and then by Sub-residual connection, dilated convolution, batch normalization and activation operations are used to fuse multi-scale features to obtain Layer feature map , the last layer of feature maps containing the richest semantic information and Add to get the output feature map , the specific expression is:

[0139]

[0140]

[0141]

[0142]

[0143]

[0144]

[0145] in, Indicates convolution, batch normalization and ReLU activation operations on the features. Indicates that the features are subjected to dilated convolution, batch normalization, and ReLU activation operations. represents the convolution expansion rate, Indicates that features are concatenated in a new dimension;

[0146] Step S43: The feature map with the highest semantics obtained in step S42 As the input of the five-layer decoder, multi-scale feature fusion is performed while upsampling, and the final output is a six-layer enhanced feature map that takes into account both global semantic information and local detail information. , the dimensions are , the specific expression is:

[0147]

[0148]

[0149]

[0150] in, represents the upsampling operation, Indicates the context information of features extracted using a residual codec module containing a dilated convolutional layer. Indicates that features are concatenated in a new dimension. Indicates the context information of features extracted using the residual codec module. Indicates the number of encoding and decoding;

[0151] Step S44: The six-layer feature map obtained in step S43 Convolution is performed separately, and the feature maps of the last five layers are Upsample to make it consistent with the feature map The six layers of feature maps are concatenated along the channel dimension, and convolution and activation operations are performed to finally obtain the semantic segmentation mask map. , the specific expression is:

[0152]

[0153]

[0154]

[0155]

[0156] in, , It is indivual The weights and biases of the convolutional layers, represents the upsampling operation, , yes The weights and biases of the convolutional layers, Indicates that features are concatenated in a new dimension. Indicates Sigmoid activation operation.

[0157] Further, in step S5, the training of the second icon semantic segmentation model using the training pipeline and the training set icon images specifically includes the following steps:

[0158] Step S51: inputting a batch of non-transparent bottom icon images and corresponding labels in the training set into the second icon semantic segmentation model network for training, and predicting an icon semantic segmentation mask map;

[0159] Step S52: According to the loss function of the second icon semantic segmentation model network, the gradient of each parameter in the network is calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method; the loss function of the network as follows:

[0160]

[0161] in, represents the pixel value, represents the real mask, represents the prediction mask, represents pixel coordinates, and Represents the height and width of the input image;

[0162] Step S53: Repeat steps S51 to S52 in batches until the loss value calculated in step S52 converges and stabilizes, save the network parameters, and complete the training process of the second icon semantic segmentation model.

[0163] The above step S6 specifically includes the following steps:

[0164] Step S61: Input the non-transparent icon images in the test set into the trained first icon semantic segmentation model and the second icon semantic segmentation model respectively, and output the corresponding icon semantic segmentation mask map and ;

[0165] Step S62: Using the data obtained from step S61 and From the input non-transparent bottom icon image Extract icons from and , and the corresponding text description Input the pre-trained CLIP model together, calculate the similarity score, and take the one with the higher score as the final semantic segmentation mask map to realize automatic screening of icon semantic segmentation results. The calculation formula is as follows:

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] where ⨀ represents the Hadamard product, Represents the use of the CLIP model to calculate the similarity score between images and text.

[0172] The CLIP (Contrastive Language-Image Pre-Training) model described in this embodiment is a pre-trained neural network model for matching images and texts released by OpenAI. Its core idea is to establish the association between images and texts through contrastive learning and measure the similarity between them to achieve better mutual understanding between images and texts.

[0173] The comparison of semantic segmentation examples before and after using the icon semantic segmentation method of the present invention is shown in the figure below: Figure 5 As shown, it can be seen that the semantic segmentation method of the present invention can more accurately identify and segment the icon area, effectively reducing the situation of mis-segmentation and missed segmentation.

[0174] The icon semantic segmentation accuracy using the traditional semantic segmentation model and the icon semantic segmentation method of the present invention, as well as the similarity between the segmentation results and the text description are shown in Table 1 below. The data show that the icon semantic segmentation method of the present invention not only significantly improves the accuracy of icon semantic segmentation, but also enhances the matching degree between the segmentation results and the text description, further verifying the advancement of the icon semantic segmentation method of the present invention in icon semantic understanding and recognition.

[0175] Table 1: Comparison of the accuracy of the semantic segmentation algorithm of the present invention and the traditional semantic segmentation algorithm

[0176]

[0177] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] The present invention is described with reference to flowcharts of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process in the flowchart, as well as the combination of processes in the flowchart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.

[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple functions specified in a flowchart.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 The steps of a specified function in a process or multiple processes.

[0181] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0182] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of icon semantic segmentation methods based on dual model prediction and automatic screening mechanism under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A method for semantic segmentation of icons, characterized in that The following steps are involved: Step S1: Divide the non-transparent background icon image dataset into a training set and a test set, and perform data preprocessing on the non-transparent background icon images and corresponding labels in the dataset; Step S2: constructing a first icon semantic segmentation model based on a multi-level feature aggregation mechanism, the first icon semantic segmentation model comprising a dual pyramid pooling module and a multi-scale feature fusion module, mining context information through the dual pyramid pooling module to obtain multi-scale features, and aggregating cross-scale feature information through the multi-scale feature fusion module and predicting semantic segmentation results; Step S3: constructing a training pipeline for the first icon semantic segmentation model, and using the training pipeline and training set icon images to train the first icon semantic segmentation model; Step S4: constructing a second icon semantic segmentation model based on a nested codec structure, wherein the second icon semantic segmentation model includes a residual codec module, through which multi-scale features within a single stage are extracted and multi-scale features between multiple stages are fused; Step S5: constructing a training pipeline for the second icon semantic segmentation, and using the training pipeline and the training set icon images to train the second icon semantic segmentation model; Step S6: input the non-transparent bottom icon test image into the trained first icon semantic segmentation model and the second icon semantic segmentation model respectively, output the corresponding semantic segmentation mask map, calculate the similarity score between the semantic segmentation mask map and the text description by comparing the learning model, and automatically select the semantic segmentation mask map with the highest score as the final semantic segmentation result; In step S3, the use of the training pipeline and the training set icon images to train the first icon semantic segmentation model specifically includes the following steps: Step S31: input a batch of non-transparent bottom icon images and corresponding label real masks in the training set into the first icon semantic segmentation model network for training, and predict the icon semantic segmentation mask map; Step S32: according to the loss function of the first icon semantic segmentation model network, the gradient of each parameter in the network is calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method; Step S33: repeating steps S31 to S32 in batches until the loss value calculated in step S32 converges and becomes stable, saving the network parameters, and completing the training process of the first icon semantic segmentation model; In step S5, the use of the training pipeline and the training set icon images to train the second icon semantic segmentation model specifically includes the following steps: Step S51: inputting a batch of non-transparent bottom icon images and corresponding labels in the training set into the second icon semantic segmentation model network for training, and predicting an icon semantic segmentation mask map; Step S52: according to the loss function of the second icon semantic segmentation model network, the gradient of each parameter in the network is calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method; Step S53: Repeat steps S51 to S52 in batches until the loss value calculated in step S52 converges and stabilizes, save the network parameters, and complete the training process of the second icon semantic segmentation model.

2. The method for semantic segmentation of an icon as claimed in claim 1, characterized in that Step S1 specifically includes the following steps: Step S11: dividing the non-transparent background icon image dataset into a training set and a test set according to a certain ratio; Step S12: performing data enhancement processing on the icon images in the training set; Step S13: pre-processing the icon image after data enhancement in step S12, converting it into the input of the icon segmentation network, randomly cropping the icon image to a uniform size, and then normalizing it so that the icon image data conforms to the standard normal distribution; Step S14: Perform the same data enhancement and preprocessing operations on the labels as on the icon images.

3. The method for semantic segmentation of an icon as claimed in claim 1, characterized in that In step S2, constructing a first icon semantic segmentation model based on a multi-level feature aggregation mechanism includes: Using STDC as the backbone network of the first icon semantic segmentation model, the network can extract features from the non-transparent icon image after preprocessing in step S1, and obtain five feature maps with different scales and different numbers of channels. ; Construct a dual-pyramid pooling module to connect to the end of the network to capture global dependencies; A multi-scale feature fusion module is used as the decoder of the network to fuse feature maps of different dimensions for semantic segmentation; the input of the network is a non-transparent icon image And the corresponding truth mask , the output of the network is the icon semantic segmentation mask map .

4. The method for semantic segmentation of an icon as claimed in claim 3, characterized in that In step S2, the context information is mined by the dual pyramid pooling module to obtain multi-scale features and the cross-scale feature information is aggregated by the multi-scale feature fusion module to predict the semantic segmentation result, which specifically includes the following steps: Step S21: Transform the non-transparent icon image Input into the encoder of the first icon semantic segmentation model network based on the multi-level feature aggregation mechanism to obtain the dimensions Five-layer feature map ,in and Represent the height, width and number of channels of the feature map respectively; Step S22: Get the data from step S21 As the input of the dual pyramid pooling module, the size of the feature map is first unified to , and concatenate along the channel dimension to obtain the feature , and then adaptive average pooling of different scales is used to optimize the features Downsampling is performed to obtain three layers of feature maps of different sizes , and then the Transformer encoder is used to capture global dependencies and semantic information to obtain an enhanced three-layer feature map , and then restore the size of the feature map to ,Will and Concatenate along the channel dimension, and finally obtain the dimension of Aggregate feature map of ; Step S23: The five-layer feature maps obtained in steps S21 and S22 are and as the input of the decoder, the decoder is divided into 5 multi-scale feature fusion modules, which process features of different dimensions respectively and use the attention mechanism to perform weighted fusion of multi-scale features; After multi-scale feature fusion processing, convolution kernels of different sizes are used to compare the input Perform depth-wise convolution to extract spatial features independently on each input channel, unify the size of the feature map by upsampling, and then perform point-wise convolution on the feature map to adjust the number of channels, and obtain , respectively perform average pooling and maximum pooling on the feature map, while mining global features and texture and edge information, and concatenate the two pooling results along the channel dimension to obtain the feature map and , the two layers of feature maps and It is spliced ​​again along the channel dimension, and then the attention map integrating multiple information is obtained through convolution and activation operations. , and then Respectively with attention map Perform element-wise multiplication to obtain the enhanced , and finally the enhanced Concatenate along the channel dimension to get the fused feature map ; Step S24: fusion feature map obtained in step S23 Upsample to the same size as the input image Match and compare with the input image Concatenate in the channel dimension, then perform convolution, batch normalization and activation operations to obtain the semantic segmentation mask map .

5. The method for semantic segmentation of an icon as claimed in claim 1, characterized in that In step S4, constructing a second icon semantic segmentation model based on a nested codec structure includes: The main semantic segmentation network of the second icon semantic segmentation model is UNet. The network replaces the convolutional layer in each stage with a residual codec structure, which can extract multi-scale local and global features from the non-transparent icon image after preprocessing in step S1. Through step-by-step upsampling, splicing and convolution operations, the multi-scale features are reintegrated into high-resolution feature maps, enhancing the network's ability to process complex image content and accurately capture subtle features. After convolution activation operations, the icon semantic segmentation mask map is finally output. .

6. The method for semantic segmentation of an icon as claimed in claim 5, characterized in that In step S4, extracting multi-scale features in a single stage through the residual encoding and decoding module and fusing multi-scale features between multiple stages specifically includes the following steps: Step S41: Transform the non-transparent icon image The dimensions of the four-stage encoder input into the second icon semantic segmentation model network are Four-layer feature map ,in and Represent the height, width and number of channels of the feature map respectively; For input features Perform a convolution, batch normalization and activation operation to extract local features and obtain the initial feature map , then proceed Sub-sampling, convolution, batch normalization and activation operations are performed to continuously expand the receptive field to obtain richer global and local features. Layer feature map , then the last layer of feature map Perform a dilated convolution, batch normalization, and activation operation to capture a wider range of contextual information without losing spatial dimensions and obtain an enhanced feature map. , and then by The concatenation, convolution, batch normalization, activation and upsampling operations are performed to fuse multi-scale features and obtain Layer feature map , the last layer of feature maps containing the richest semantic information and Add together to get the composite feature map ; Step S42: The feature map with the highest semantics obtained in step S41 As the input of the two-layer dilated residual codec module, it further extracts context information without losing spatial resolution. For input features Perform a convolution, batch normalization and activation operation to extract local features to form an initial feature map , then proceed The dilated convolution, batch normalization and activation operations gradually expand the receptive field while maintaining the spatial resolution to obtain richer features. Layer feature map , and then the last two layers of feature maps and Splice along the channel dimension to get a composite feature map , and then perform a dilated convolution, batch normalization, and activation operation to capture more extensive context information and obtain the feature map , and then by Sub-residual connection, dilated convolution, batch normalization and activation operations are used to fuse multi-scale features to obtain Layer feature map , the last layer of feature maps containing the richest semantic information and Add to get the output feature map ; Step S43: The feature map with the highest semantics obtained in step S42 As the input of the five-layer decoder, multi-scale feature fusion is performed while upsampling, and the final output is a six-layer enhanced feature map that takes into account both global semantic information and local detail information. , the dimensions are ; Step S44: The six-layer feature map obtained in step S43 Convolution is performed separately, and the feature maps of the last five layers are Upsample to make it consistent with the feature map The six layers of feature maps are concatenated along the channel dimension, and convolution and activation operations are performed to finally obtain the semantic segmentation mask map. .

7. An icon semantic segmentation system, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, an icon semantic segmentation method as described in any one of claims 1 to 6 can be implemented.

8. A computer-readable storage medium storing computer program instructions that can be executed by a processor. When the processor executes the computer program instructions, an icon semantic segmentation method as described in any one of claims 1 to 6 can be implemented.

Images