National costume pattern semantic segmentation method and system based on deep learning
By constructing the ethnic clothing pattern data set and using deep learning technology, including convolutional neural network and attention mechanism module, the problem of inaccurate segmentation of complex ethnic clothing patterns in the existing technology is solved, and a high-precision and robust pattern segmentation effect is achieved.
Patent Information
- Application Number
- CN202411439564.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2025-02-25
AI Technical Summary
It is difficult for the prior art to accurately divide complex ethnic clothing patterns, especially when the edges of the patterns are complex and changeable.
Using a deep learning-based method, the combined use of ethnic clothing pattern data set, image preprocessing, convolutional neural network and attention mechanism modules is carried out to perform feature extraction and adaptive feature fusion to achieve semantic segmentation of ethnic clothing patterns.
It improves the segmentation accuracy and accuracy of ethnic clothing patterns, enhances the generalization ability and robustness of the model, and can effectively segment the patterns under different lighting conditions.
Smart Images
Figure CN119516184A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a method and system for semantic segmentation of ethnic costume patterns based on deep learning. Background Art
[0006] The existing image extraction generally uses learning models to extract features from images. Due to the numerous details of ethnic costume patterns, the edges of more complex ethnic costume patterns are often complex and changeable, making it difficult to accurately segment and fully present them. Therefore, there is an urgent need for a technical means to accurately segment pattern elements on clothing. Summary of the invention
[0007] In view of this, the present invention provides a method and system for semantic segmentation of ethnic costume patterns based on deep learning, so as to achieve accurate segmentation of pattern elements on costumes.
[0008] To this end, the present invention provides the following technical solutions:
[0009] The present invention discloses a method for semantic segmentation of ethnic costume patterns based on deep learning, comprising:
[0010] S1, build a national costume pattern dataset;
[0011] S2, preprocessing the ethnic costume images in the data set;
[0012] S3, using two convolutional neural networks (CNNs) to extract features from images and edge images respectively and fuse features;
[0013] S4, use the attention mechanism module to extract features from the image;
[0014] S5, using an attention gating mechanism to adaptively fuse the features extracted by the convolutional neural network and the features extracted by the attention mechanism module in the upsampling module of the network, and gradually restore the image to the input resolution size;
[0015] S6. Use the fused feature images to build and train the model, and use the trained model to perform semantic segmentation of ethnic costume images.
[0016] Furthermore, constructing a national costume pattern data set includes:
[0017] Use image acquisition equipment to take pictures of various ethnic costume patterns, collect pictures from the Internet, and form a dataset of ethnic costume patterns.
[0018] Annotating images of the ethnic costume pattern dataset;
[0019] Use semantic segmentation-related toolkits to generate segmentation maps corresponding to each image in the labeled ethnic costume pattern dataset for training the model.
[0020] Furthermore, the image preprocessing includes:
[0021] Perform random brightness and contrast adjustments on images;
[0022] Use Gaussian blur method to process the image;
[0023] Perform data augmentation on the input images using random horizontal flips, random vertical flips, and randomly applied translation, scaling, and rotation transformations;
[0024] The Canny edge detection algorithm is used to extract the image edges and obtain the edge map.
[0025] Furthermore, convolution and attention mechanism modules are used to extract features from the image, including: the model assigns different weights to each part of the input to extract key and important information.
[0026] Furthermore, the fused feature images are used to construct and train a model, including:
[0027] Use the image data in the training set to build a model, and preset the model loss value and iteration threshold;
[0028] Use the image data in the validation set to train the model and continuously adjust the overall parameters until the model's loss value and number of iterations reach the set threshold;
[0029] After the model training is completed, the accuracy of each category in the test set is calculated to obtain the average accuracy of the model.
[0030] Furthermore, an attention gating mechanism is used to adaptively fuse the features extracted by the convolutional neural network and the features extracted by the attention mechanism module in an upsampling module of the network, including:
[0031] The feature maps of different scales are connected through the concat operation.
[0032] Furthermore, the attention network used is spatial attention, and features of the image are extracted according to its local-global attention mechanism by using the shifted window transformer SwinTransformer module.
[0033] The present invention also provides a system for semantic segmentation of ethnic costume patterns based on deep learning, comprising:
[0034] Dataset construction module, constructs a dataset of ethnic costume patterns;
[0035] A preprocessing module, for preprocessing the ethnic costume images in the data set;
[0036] The feature extraction module uses two convolutional neural networks (CNNs) to extract features from images and edge images respectively and fuse the features;
[0037] Attention feature extraction module, which uses the attention mechanism module to extract features from images;
[0038] A feature fusion module, which uses an attention gating mechanism to adaptively fuse the features extracted by the convolutional neural network and the features extracted by the attention mechanism module in the network upsampling module, and gradually restores the image to the input resolution size;
[0039] The semantic segmentation module uses the fused feature images to build and train the model, and uses the trained model to perform semantic segmentation of ethnic costume images.
[0040] Advantages and positive effects of the present invention:
[0041] 1) During the image preprocessing process of the present invention, random brightness and contrast adjustment, Gaussian blur, color gamut change and other operations are performed on each image, which enriches the background of the image, increases the diversity of training scenes, removes noise, enhances image data, alleviates some interference caused by external factors such as lighting conditions and clothing wrinkles in actual applications, and enhances the generalization ability and robustness of the model.
[0042] 2) In the process of image preprocessing of the present invention, the Canny edge detection algorithm is further used to extract the image edge of the processed image to obtain an edge map, and the preprocessed image and the edge map are input into the network together. In view of the numerous details and complex edges of ethnic costume patterns, the integrity and accuracy of the edges are further improved by adding this processing.
[0043] 3) The present invention adopts feature extraction of attention network. The model assigns different weights to each part of the input, extracts key and important information, and enables the model to make more accurate judgments. At the same time, it will not bring greater overhead to the calculation and storage of the model, thereby improving the segmentation accuracy of the model and ensuring the segmentation speed.
[0044] 4) The present invention utilizes convolutional neural networks and attention modules to jointly perform feature extraction, image enhancement and other technologies, thereby enhancing detail features and the segmentation capability of the segmentation model for ethnic costume patterns, effectively alleviating the problems of easy loss of texture details and inaccurate segmentation of pattern shape contours in the semantic segmentation of ethnic costume patterns, and realizing pixel-level semantic segmentation of ethnic costume patterns. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0046] Figure 1 It is a technical framework for semantic segmentation of ethnic costume patterns in an embodiment of the present invention;
[0047] Figure 2 This is a flow chart of semantic segmentation of ethnic costume patterns in an embodiment of the present invention;
[0048] Figure 3 This is a flowchart of image preprocessing in an embodiment of the present invention;
[0049] Figure 4 The effect diagram of the image before and after preprocessing in the embodiment of the present invention;
[0050] Figure 5 Schematic diagram of the convolutional neural network and attention mechanism network of the present invention. DETAILED DESCRIPTION
[0051] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0053] The present invention provides a method for semantic segmentation of ethnic costume patterns based on deep learning, and its main concept is: firstly, the ethnic costume patterns are annotated, and various patterns in the image are annotated respectively, and then the corresponding segmentation map is generated according to the annotated image for training the model. After the image is preprocessed, the convolutional neural network (CNN) and the shift window transformer (Swin Transformer) attention module extract features from each image and the corresponding edge image, and the attention gate mechanism (Attention Gate) mechanism performs feature fusion. After training, a segmentation model with segmentation accuracy that meets the requirements is obtained; and the ethnic costume pattern is segmented using the segmentation model.
[0054] like Figure 1-2 The flowchart of the semantic segmentation of ethnic costume patterns of the present invention is shown, and the semantic segmentation method of ethnic costume patterns of the present invention is explained in a modular form:
[0055] The present invention consists of an image preprocessing module, a CNN convolutional neural network module, an attention network module, and an AttentionGate feature fusion mechanism. The image preprocessing performs image enhancement processing on the input image to generate an input image and an edge image, which are respectively used as inputs of two CNN convolutional neural network modules; the CNN convolutional neural network module first performs a convolution operation on each input image. The whole process divides the feature map obtained by the convolution into three sizes of feature maps, and the feature map output by the last layer is input into the attention module, which then performs global learning on the feature map; the two sets of features obtained from the input image and the edge image are added to obtain a fused feature map; the Attention Gate mechanism performs feature fusion on feature maps of different sizes, and outputs the fused results at the same time, and finally obtains a segmentation result map of the ethnic costume pattern.
[0056] The specific implementation steps are as follows:
[0057] S1. Collect and mark ethnic costume patterns;
[0058] S11 uses cameras and other devices to take pictures of various ethnic costume patterns, collects pictures from the Internet, and together forms an ethnic costume pattern dataset;
[0059] S12 circles various types of ethnic costume patterns appearing in the image on the obtained ethnic costume pattern data set, and adds corresponding type labels to the circled areas;
[0060] S13 uses the toolkit related to semantic segmentation to generate the segmentation map corresponding to each picture of the labeled ethnic costume pattern dataset, so as to be used for subsequent model training;
[0061] S2, preprocessing the collected images;
[0062] like Figure 3 The flowchart of the image preprocessing mechanism shown is implemented by the following steps:
[0063] S21 performs random brightness and contrast adjustments on the image;
[0064] The pixel value of the original image is I(x, y), where (x, y) is the position of the pixel. Brightness adjustment is achieved by weighted addition of the brightness value of each pixel. Specifically, the new brightness value I'(x, y) of each pixel can be calculated by the following formula:
[0065] I′(x,y)=I(x,y)+ΔB (1)
[0066] Where ΔB is a randomly generated brightness increment, which can be positive or negative. The size of this increment is limited by the range set by the user.
[0067] Contrast adjustment is achieved by scaling the brightness difference of each pixel. Specifically, the new brightness value I′(x, y) of each pixel can be calculated by the following formula:
[0068] I′(x,y)=α*(I(x,y)-μ)+μ (2)
[0069] Where α is a randomly generated contrast multiplier and μ is the average brightness of the image. The value of α is limited by the contrast range set by the user.
[0070] This method randomly generates brightness increment ΔB and contrast multiplier α, and then applies them to each pixel of the image to adjust the brightness and contrast. This can produce diverse images and help the model better generalize to input data under different brightness and contrast conditions.
[0071] S22 uses Gaussian blur method to process images; by applying Gaussian filter to reduce the details of the image, the image is smoothed and blurred. The following is the specific implementation process of this method:
[0072] The pixel value of the original image is I(x, y), where (x, y) is the position of the pixel. The kernel of the Gaussian filter is a two-dimensional Gaussian function, usually represented by a matrix. The size of the Gaussian kernel is determined by the size of the filter, usually an odd number, such as 3x3, 5x5, 7x7, etc. The center value of the kernel is the largest, and then gradually decreases to weight the values of the surrounding pixels when filtering. The one-dimensional form of the Gaussian kernel is as follows:
[0073]
[0074] Where G(x) is a Gaussian function, σ2 is the standard deviation and x is the offset from the center of the kernel.
[0075] The two-dimensional form of the Gaussian kernel is the superposition of the one-dimensional kernel in two directions, as shown below:
[0076] G(x, y) = G(x)*G(y) (4)
[0077] From the Gaussian convolution function G(x, y), we can see that the scale factor σ determines the range of the convolution kernel. When the value of σ is smaller, about 2% to 6% of the image size, the local details of the image are more prominent, but the color is easily distorted, and the grayscale dynamic range will be compressed more. When the value of σ is larger, about 30% to 60% of the image size, when calculating the pixel point, the surrounding pixel points have a larger influence on the calculation point, that is, the convolution kernel has a greater effect, the color can be maintained, the overall image processing effect is better, and the local details are not easy to stand out.
[0078] For each image pixel (x, y), the Gaussian kernel is convolved with the local area of the image to obtain a new pixel value I'(x, y). The convolution operation can be expressed by the following formula:
[0079] I′(x,y)=∑I(x+i,y+j)*G(i,j) (5)
[0080] Where ∑ represents the summation and (i, j) is the coordinate of the Gaussian kernel.
[0081] In the boundary regions of the image, since there are not enough pixels to perform a complete convolution, different boundary handling strategies are usually used, such as padding or truncation, to ensure that the output image has the same size as the input image.
[0082] The Gaussian blur method smoothes the image, reduces details, and blurs the image by applying a Gaussian filter kernel. The size and standard deviation of the filter kernel determine the degree of blurring. The larger the standard deviation, the more obvious the blurring. This operation is used to remove noise from an image or reduce image details for preprocessing for specific tasks.
[0083] S23 performs data augmentation on the input images using random horizontal flipping, random vertical flipping, random translation, scaling, and rotation transformations, and other methods.
[0084] S24 uses the Canny edge detection algorithm to extract edges and restore the results to the original size. The Canny edge detection algorithm is an edge detection method used in image processing, including grayscale, Gaussian filtering, gradient calculation, non-maximum suppression and double threshold processing.
[0085] The effect diagram before and after image preprocessing is as follows Figure 4 shown.
[0086] like Figure 5 The convolutional neural network and attention mechanism network diagram shown in FIG. 1 include steps S3, S4, and S5 as follows:
[0087] S3. Use two convolutional neural networks (CNNs) to extract features from the image and edge image respectively and fuse the features. The input layer of the network extracts features from the image and edge image respectively through the convolution kernel. The extracted features form a feature map. Add them together. As the depth of the network increases, the size of the feature map gradually decreases and the depth gradually deepens.
[0088] According to the order of data preprocessing, the image and edge image are input into the corresponding symmetric network respectively. The two networks have the same composition, both of which are composed of basic double convolution (DoubleConv) modules. A DoubleConv module consists of two convolution layers and a batch normalization layer. The downsampling module of the feature extraction network consists of 1, 1, and 1 DoubleConv modules, and their data specifications are Batchsize*224*224*48, Batchsize*112*112*96, and Batchsize*56*56*128 respectively. The convolution size is 3*3, and each convolution layer extracts image features according to formula (16).
[0089]
[0090] In the formula, l indicates the number of convolution layers where the current convolution is located. represents the convolution kernel connecting the i-th feature image of the l-1th layer and the j-th feature image of the lth layer, is the jth feature image of the lth layer, is the i-th feature image of the l-1th layer, represents the bias term of the lth layer, and f() represents the nonlinear activation function;
[0091] The nonlinear activation function uses the rectified linear unit function, and the stride of the convolution in the downsampling stage is set to 2, and the rest is 1. The learning rate is 0.01 and the number of iterations is 200.
[0092] S4 uses spatial attention network, which is composed of two identical networks like S3. It uses Swin Transformer module to extract features of images and edge images according to its local-global attention mechanism. The model assigns different weights to each part of the input, extracts key and important information, and enables the model to make more accurate judgments without causing greater overhead for the model's calculation and storage.
[0093] The calculation process is shown as follows:
[0094]
[0095]
[0096]
[0097]
[0098] S5, using the Attention Gate mechanism for adaptive feature fusion, the features obtained in the feature extraction stage are fused in the network upsampling module; the feature maps of different scales obtained in steps S3 and S4 are fused in this stage, and the image is gradually restored to the input resolution size; the feature maps of different scales are connected through the concat operation. If the dimensions of the two input features x and y are p and q, the dimension of the output feature z is p+q.
[0099] S6. Build and train the model using the fused feature image;
[0100] The model loss value and number of iterations are set when the model is established. When the model loss value is greater than the set loss threshold or does not reach the set iteration threshold, the parameters in the formula are adjusted to make the model converge.
[0101] Steps S3, S4, and S5 are a process of continuous iterative tuning. When the model is established, the loss value and the number of model iterations are set, the model is established using the image data in the training set, and the overall parameters are continuously adjusted using the image data in the validation set until the model loss value and the number of iterations reach the set threshold; and the accuracy is calculated. After the model training is completed, the accuracy of each category in the test set is calculated, and finally the final model accuracy is obtained by calculating the average accuracy.
[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for semantic segmentation of ethnic costume patterns based on deep learning, characterized in that: include: S1, build a national costume pattern dataset; S2, preprocessing the ethnic costume images in the data set; S3, using two convolutional neural networks (CNNs) to extract features from images and edge images respectively and fuse features; S4, use the attention mechanism module to extract features from the image; S5, using an attention gating mechanism to adaptively fuse the features extracted by the convolutional neural network and the features extracted by the attention mechanism module in the upsampling module of the network, and gradually restore the image to the input resolution size; S6. Use the fused feature images to build and train the model, and use the trained model to perform semantic segmentation of ethnic costume images.
2. The method for semantic segmentation of ethnic costume patterns based on deep learning according to claim 1, characterized in that: The construction of the national costume pattern data set includes: Use image acquisition equipment to take pictures of various ethnic costume patterns, collect pictures from the Internet, and form a dataset of ethnic costume patterns. Annotating images of the ethnic costume pattern dataset; Use semantic segmentation-related toolkits to generate segmentation maps corresponding to each image in the labeled ethnic costume pattern dataset for training the model.
3. The method for semantic segmentation of ethnic costume patterns based on deep learning according to claim 1, characterized in that: The image preprocessing comprises: Perform random brightness and contrast adjustments on images; Use Gaussian blur method to process the image; Perform data augmentation on the input images using random horizontal flips, random vertical flips, and randomly applied translation, scaling, and rotation transformations; The Canny edge detection algorithm is used to extract the image edges and obtain the edge map.
4. The method for semantic segmentation of ethnic costume patterns based on deep learning according to claim 1, characterized in that: The convolution and attention mechanism modules are used to extract features from the image, including: the model assigns different weights to each part of the input and extracts key and important information.
5. The method for semantic segmentation of ethnic costume patterns based on deep learning according to claim 1, characterized in that: Use fused feature images to build and train models, including: Use the image data in the training set to build a model, and preset the model loss value and iteration threshold; Use the image data in the validation set to train the model and continuously adjust the overall parameters until the model's loss value and number of iterations reach the set threshold; After the model training is completed, the accuracy of each category in the test set is calculated to obtain the average accuracy of the model.
6. The method for semantic segmentation of ethnic costume patterns based on deep learning according to claim 1, characterized in that: Adopting an attention gating mechanism to adaptively fuse the features extracted by the convolutional neural network and the features extracted by the attention mechanism module in the upsampling module of the network, including: The feature maps of different scales are connected through the concat operation.
7. The method for semantic segmentation of ethnic costume patterns based on deep learning according to claim 1, characterized in that: The attention network used is spatial attention, which uses the shift window transformer Swin Transformer module to extract features from the image based on its local-global attention mechanism.
8. A semantic segmentation system for ethnic costume patterns based on deep learning, characterized in that: include: Dataset construction module, constructs a dataset of ethnic costume patterns; A preprocessing module, for preprocessing the ethnic costume images in the data set; The feature extraction module uses two convolutional neural networks (CNNs) to extract features from images and edge images respectively and fuse the features; Attention feature extraction module, which uses the attention mechanism module to extract features from images; A feature fusion module, which uses an attention gating mechanism to adaptively fuse the features extracted by the convolutional neural network and the features extracted by the attention mechanism module in the network upsampling module, and gradually restores the image to the input resolution size; The semantic segmentation module uses the fused feature images to build and train the model, and uses the trained model to perform semantic segmentation of ethnic costume images.