A clothing image segmentation model establishment method and clothing image segmentation method

Abstract

The present invention provides a method for establishing a clothing image segmentation model and a clothing image segmentation method, which solves the problem of low segmentation accuracy in the clothing image segmentation algorithm in the prior art. The method for establishing a clothing image segmentation model includes making a clothing image data set; annotating the image at the pixel level, annotating different categories of clothing into different colors; using a data enhancement module to enhance the original image; building a deep neural network segmentation model: training the deep neural network segmentation model; using a post-processing module to post-process the output of the deep neural network segmentation model; the post-processing module includes a fully connected conditional random field. Using the encoding and decoding method, the extraction of edge and spatial information is relatively complete, and the segmentation result has smoother edges and obvious background distinction; adding a post-processing module improves the classification confusion of the segmentation result, reducing a large number of misclassified chaotic points.

Description

Technical Field

[0001] The invention belongs to the technical field of clothing image segmentation, and in particular relates to a method for establishing a clothing image segmentation model. Background Art

[0002] Image segmentation is a classic problem in computer vision and the foundation of computer vision. The segmentation of image content is an important part of image understanding. Traditional computer vision methods segment different entities or regions in an image through features such as grayscale, geometric shape, and spatial texture information. The features in the same region or entity remain consistent or similar, but show obvious differences in different instances. However, the image segmentation model of traditional methods is rough and the segmentation accuracy is generally low. It is not suitable for clothing images with multiple clothing styles or similar clothing colors and backgrounds.

[0003] Chinese patent document: CN201811082187.9, application date 2018.09.17, patent name: Fashion clothing image segmentation method based on deep learning, discloses a fashion clothing image segmentation method based on deep learning, including a deep neural network structure model composed of a feature extraction module, a clothing semantic information extraction module and a clothing segmentation prediction module; using the regression function of the key point position, the key visibility loss function, the weighted clothing prediction category cross entropy loss function and the clothing position regression loss function as the loss function; the weight parameter initialization method, the data preprocessing, the optimization algorithm and the training step as the optimization strategy to segment the clothing in the image.

[0004] Chinese patent document: CN202111034024.5, application date 2021.09.03, patent name: Model training method, clothing fine-grained segmentation method and related device, discloses a model training method, clothing fine-grained segmentation method and related device, including a feature extraction module composed of an architecture combining CBNet and FPN; the feature information corresponding to the training image is used to segment the training image through a prediction module, and the prediction information corresponding to the training image; the prediction information and annotation information are used to train through a deep neural network to obtain a clothing fine-grained segmentation model.

[0005] The fashion clothing image segmentation method based on deep learning in the above patent document CN201811082187.9 uses a single-line neural network extraction method, which extracts image information through a network structure composed of a feature extraction module, a clothing semantic information extraction module, and a clothing segmentation prediction module connected in series. The model training method and clothing segmentation method in the above patent document CN202111034024.5 combine two unconnected network structures into a new network structure, and train twice to achieve the purpose of fine-grained segmentation and improve precision and accuracy. However, the above segmentation algorithm still has the following defects:

[0006] 1. The image background is complex, and the edge information of the clothing and the complex background information are easily confused, resulting in problems such as edge blur.

[0007] 2. Different clothes have high similarity, and the segmentation algorithm classifies each pixel, resulting in a large number of false similarities during segmentation.

[0008] 3. The data set is difficult to produce. Clothes have a wide variety of patterns and textures, and the number of clothing types in the actual data set cannot meet the training requirements. Summary of the invention

[0009] The invention provides a clothing image segmentation model establishment method and a clothing image segmentation method, which solves the problem of low segmentation accuracy in the clothing image segmentation algorithm in the prior art.

[0010] The technical solution adopted by the present invention is:

[0011] In a first aspect, the present invention provides a method for establishing a clothing image segmentation model, comprising:

[0012] S1. Create clothing image dataset:

[0013] S11. crawling pictures containing clothing from social and e-commerce platforms, wherein the pictures contain a single piece of clothing or a plurality of different pieces of clothing; the categories of clothing include short-sleeved shirts, long-sleeved shirts, coats, shorts, short skirts, long pants, and long skirts;

[0014] S12, annotate the images at the pixel level, annotate different categories of clothing with different colors; for images of clothing worn on people, separate the human skin from the clothing and classify the background and skin into one category for annotation to obtain the original image; and make the original image into a clothing image dataset;

[0015] S2, using a data enhancement module to enhance the original image;

[0016] The data enhancement module includes a rotation data enhancement module, a flip data enhancement module, a splicing data enhancement module and a transformation data enhancement module;

[0017] S3. Build a deep neural network segmentation model:

[0018] A deep neural network segmentation model is constructed by using a feature encoding module and a feature decoding module, wherein the feature encoding module includes a feature extraction backbone network and a feature fusion module;

[0019] S4. Train the deep neural network segmentation model:

[0020] The deep neural network segmentation model is trained by freezing some weights first and then unfreezing all weights. During the training process, the loss value is calculated using the training results of the dice coefficient loss function and the weighted cross entropy loss function.

[0021] S5. Post-process the output of the deep neural network segmentation model using a post-processing module; the post-processing module includes a fully connected conditional random field.

[0022] Further improved solution: In step S2,

[0023] The rotation data enhancement module is used to rotate the original image and the annotation around the center point at the same angle, and the rotation angle takes a random value between 0 and 360 degrees;

[0024] The flip data enhancement module is used to flip the original image and annotation around the horizontal symmetry axis and the vertical symmetry axis of the image;

[0025] The splicing data enhancement module is used to randomly select four corresponding samples of the original image and the annotation, perform rotation, scaling and / or cropping operations, and then merge them into a new image;

[0026] The transformation data enhancement module is used to filter out redundant high-frequency information from the original image using wavelet transform, merge close brightness values ​​into the same brightness value, and directly exchange the corresponding sum when the brightness value and the randomly generated brightness value match. Finally, the image is reconstructed by wavelet transform and converted into RGB to obtain a new data enhancement result.

[0027] Further improved solution: In step S3,

[0028] The feature extraction backbone network includes step1-step 22 , step1-step 22 All are connected in series;

[0029] The step 1 is composed of a convolution module with a step size of 2 and a convolution module with a step size of 1 connected in series, and takes the original image as input;

[0030] The parameters of step 2-step 4 are the same except for the size of the convolution channel. Each step consists of two consecutive depth-wise separable convolution modules with a step size of 1 and a depth-wise separable convolution module with a step size of 2 connected in series, and a convolution module with a step size of 2 is residually connected to perform feature extraction on the output of the previous step.

[0031] Step 5-step 20 All parameters are the same; each step is composed of 3 consecutive depth-wise separable convolutional modules with a step size of 1, accompanied by an empty residual edge structure, which extracts features from the output of the previous step;

[0032] The step 21 Two consecutive depth-wise separable convolution modules with a step size of 1 and a depth-wise separable convolution module with a step size of 2 are connected in series, and a convolution module with a step size of 2 is residually connected to the previous step. 20 The output of is used for feature extraction;

[0033] The step 22 It is composed of three depthwise separable convolution modules with a step size of 1. The three depthwise separable convolution modules have the same parameters except for the size of the convolution channel. 21 The output is used for feature extraction.

[0034] Further improved solution: In step S3, the feature fusion module includes a parallel hole convolution module and a convolution module;

[0035] The parallel hole convolution module includes a convolution module step 23-1 , three hole convolution module steps 23-2 -step 23-4 And a global pooling module step 23-5 ;

[0036] The convolution module step 23-1 It consists of a normal convolution with a step size of 1. 22 The output of is used for feature extraction;

[0037] The three hole convolution modules step 23-2 -step 23-4 , each of which consists of a dilated convolution module. The three dilated convolution modules have the same parameters except for the dilation rate. 21 The output of is used for feature extraction;

[0038] The global pooling module step 23-5 is composed of a global pooling module and a normal convolution module in series. 21 The output of is used for feature extraction;

[0039] The convolution module, after weighting the features of the parallel dilated convolution module, performs a normal convolution step with a step size of 1. 24 , perform feature extraction;

[0040] Further improved solution: In step S3, the feature decoding module includes two inputs, and the two inputs are step3 and step 24 Output:

[0041] The feature decoding module takes the output of step 3 as input and undergoes a normal convolution step with a step size of 1. 25 , extract features;

[0042] The feature decoding module will step 24 The output is taken as input and goes through a 4-fold upsampling step 26 , decode and restore the features;

[0043] The feature decoding module will step 25 and step 26 The output features are fused and added, and then two ordinary convolution steps with a step size of 1 are connected in series. 27 , extract features;

[0044] The feature decoding module; step 27 The output is taken as input and goes through a 4-fold upsampling step 28 , decode the features and restore them to the original image size;

[0045] The feature decoding module will step 28 The output of is used to classify pixels, and the softmax module is used to output the probability of each pixel being classified into each of the following categories: short-sleeved shirt, long-sleeved shirt, jacket, shorts, short skirt, long pants, long skirt, and background.

[0046] Further improved solution: In step S4, the deep neural network segmentation model is trained:

[0047] Initializing the network parameters of the neural network segmentation model using the He initialization method;

[0048] During training, the Nesterov momentum optimization algorithm is used;

[0049] During training, a pre-trained model based on the VOC2012 dataset is used to freeze step 1-step in the feature extraction backbone network. 22 After training for 50 epochs, the weight parameters of step 1-step in the feature extraction backbone network are 22 Unfreeze the weight parameters and continue training for 50 epochs;

[0050] During training, the dice coefficient loss function is used to calculate the loss function for eight types of training results: short-sleeved shirts, long-sleeved shirts, jackets, shorts, short skirts, long pants, long skirts and backgrounds. At the same time, the weighted cross entropy loss function is used to weight the results of the dice coefficient loss function, and weights are added to the eight types of training results respectively. The two methods are performed alternately;

[0051] During training, a hierarchical learning rate decay strategy was used. The initial learning rate was set to 0.05. When the loss function value did not decrease for three epochs, the learning rate was reduced by half until the loss function value continued to not decrease, and then the training was stopped.

[0052] Further improved solution: The dice coefficient loss function uses the deep neural network segmentation model training evaluation index as the loss function, which represents the ratio of the intersection and union of the predicted result and the true value; the calculation formula of the dice coefficient loss function is as follows:

[0053]

[0054] Among them, x and y represent the pixel coordinates, P i Represents the predicted value of the pixel, G i Represents the true value of the pixel prediction value;

[0055] The weighted cross entropy loss function represents the distance between the actual probability and the expected probability. A weight parameter is added to each category according to the number of pixels of each category. The weighted cross entropy loss function is calculated as follows:

[0056]

[0057] Among them, w C Expressed as a weight parameter, the calculation formula is w C =(NN C ) / N, where N represents the total number of pixels, and N C It is represented by the number of pixels with the true value category c, p is represented by the actual probability, and y is represented by the expected probability.

[0058] Further improved solution: the feature extraction backbone network step1-step 22In the above figure, each layer of depth-wise separable convolutional module is followed by a batch normalization layer and a nonlinear activation function layer.

[0059] The size of the convolution kernel is 3x3, the number of convolution channels in step 1 is 32 and 64 respectively, the number of convolution channels in step 2 is 128, the number of convolution channels in step 3 is 256, and the number of convolution channels in step 4-step is 256. 20 The number of convolution channels is 728, and the step 21 The number of channels of the three consecutive depth-wise separable convolutions is 728, 1024, and 1024 respectively, and the number of channels of the convolution of the residual edge is 1024. 22 The number of channels of the three consecutive depth-wise separable convolutions are 1536, 1536, and 2048 respectively.

[0060] Further improved solution: In step S5,

[0061] A post-processing module is used to post-process the output of the deep neural network segmentation model; the post-processing module includes a fully connected conditional random field.

[0062] The fully connected conditional random field is inputted as the original image and the output of the feature decoding module, and each pixel point of the output image of the original feature decoding module has a category label; each pixel point in the corresponding original image has a corresponding observation value, and each pixel point is connected with all the pixels to form a connection edge, and the output result of the feature decoding module is processed in combination with the relationship between all the pixels in the original image;

[0063] The conditional probability of the fully connected conditional random field is defined as:

[0064]

[0065] Among them, f i 、f j Represented as the feature vector of pixel (i, j), It is represented as the Gaussian weight that measures the similarity between pixels i and j, m is represented as the logarithm of the pixel degree, μ(x i , x j ) is represented as a two-dimensional matrix of the consistency between different label pairs.

[0066] In a second aspect, the present invention provides a clothing image segmentation method, which inputs a clothing image into a clothing image segmentation model established by using any of the above-mentioned clothing image segmentation model establishment methods, and outputs the segmented clothing image.

[0067] The beneficial effects of the present invention are:

[0068] 1. Due to the use of encoding and decoding, the extraction of edge and spatial information is relatively complete, the edge of the segmentation result is smoother and clearly distinguished from the background;

[0069] 2. Due to the addition of the post-processing module, the classification confusion of the segmentation results has been improved, reducing a large number of misclassified points;

[0070] 3. Through the data enhancement module, the amount of training data can be increased to avoid underfitting of training caused by fewer types of clothing.

[0071] 4. Use the transformation data enhancement module (wavelet transform) to transform the texture of clothes in the original image. There are various textures of clothes, and it is impossible for the data set to collect clothes of all texture categories. Traditional data enhancement methods can only transform the size, angle, etc. of the picture, but cannot change the texture of the clothes. Therefore, a data enhancement method that can increase the types of clothing textures is needed. Using this data enhancement method, the generalization of the clothing segmentation system can be enhanced, that is, a certain effect can be achieved in segmenting clothes with different textures. Because the principle of deep learning is to use a large number of pictures for training, the pictures that can be recognized in the final test cannot be too different from the training pictures, so this data enhancement method has a certain improvement in the generalization ability of the network and can recognize clothes with different textures.

[0072] 4. In the present invention, the network model is mainly composed of convolution operations. Convolution can be generally understood as an operation of calculating the inner product of the grayscale values ​​of an image using an n*n matrix. In the encoding process, different clothes in the image will respectively obtain a value that can represent it after step-by-step convolution processing. This value can be called feature information, and at the same time, the original image will be reduced to a feature map 16 times the original size, and the feature information of each object will be marked on the feature map. In the decoding process, the values ​​representing different images will be processed by deconvolution (inverse operation of convolution). As the feature map is enlarged, the feature information is retained until it is enlarged to the size of the original image. At this time, different colors will be given to the areas where the original image is located according to the feature information of different objects to obtain a segmentation map.

[0073] 5. The post-processing module can reduce the confusion of classification points: the unary potential function describes the relationship between the category label xi of pixel point i and the corresponding observation value yi, that is, for each pixel i with category label x i There is also the corresponding observation value y i , so that each pixel is used as a node, and the relationship between pixels is used as an edge, which constitutes a conditional random field. We observe the variable y i To infer the category label x corresponding to pixel i iThe difference between the fully connected conditional random field and the binary potential function is that it describes the relationship between each pixel and all other pixels, so it is called "fully connected". The binary potential function encourages similar pixels to be assigned the same label, while pixels with large differences are assigned different labels, and the definition of this "distance" is related to the color value and the actual relative distance. Therefore, CRF can segment the image as close to the boundary as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a flow chart for implementing the technical solution of the present invention.

[0075] Figure 2 It is a schematic diagram of a data enhancement module based on wavelet transform of the present invention.

[0076] Figure 3 Schematic diagram of the feature extraction backbone network of the feature encoding module of the present invention.

[0077] Figure 4 It is a schematic diagram of a feature extraction module of the feature extraction backbone network of the present invention.

[0078] Figure 5 Schematic diagram of a feature fusion module of the feature encoding module of the present invention.

[0079] Figure 6 Schematic diagram of a feature decoding module of the present invention.

[0080] Figure 7 It is a schematic diagram of the post-processing module of the present invention.

[0081] Figure 8 It is a flow chart of the training steps of the present invention.

[0082] Fig. 9 This is the effect diagram of wavelet transform data enhancement; from left to right are the target texture image, the original image, and the image after texture transformation. DETAILED DESCRIPTION

[0083] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments.

[0084] See also Figure 1 ,In the present invention, a source image is input into the clothing image ,segmentation model, first enhanced by the data enhancement module, then encoded by the ,feature encoding module, decoded by the feature decoding module, ,processed by the post-processing module, and finally output the clothing prediction ,image.

[0085] Embodiment 1:

[0086] This embodiment provides a method for establishing a clothing image segmentation model, comprising:

[0087] S1. Create clothing image dataset:

[0088] S11. crawling pictures containing clothing from social and e-commerce platforms, wherein the pictures contain a single piece of clothing or a plurality of different pieces of clothing; the categories of clothing include short-sleeved shirts, long-sleeved shirts, coats, shorts, short skirts, long pants, and long skirts;

[0089] S12, annotate the images at the pixel level, annotate different categories of clothing with different colors; for images of clothing worn on people, separate the human skin from the clothing and classify the background and skin into one category for annotation to obtain the original image; and make the original image into a clothing image dataset;

[0090] S2, using a data enhancement module to enhance the original image;

[0091] The data enhancement module includes a rotation data enhancement module, a flip data enhancement module, a splicing data enhancement module and a transformation data enhancement module; specifically:

[0092] The rotation data enhancement module is used to rotate the original image and the annotation around the center point at the same angle, and the rotation angle takes a random value between 0 and 360 degrees;

[0093] The flip data enhancement module is used to flip the original image and annotation around the horizontal symmetry axis and the vertical symmetry axis of the image;

[0094] The splicing data enhancement module is used to randomly select four corresponding samples of the original image and the annotation, perform rotation, scaling and / or cropping operations, and then merge them into a new image;

[0095] See also Figure 2 The transformation data enhancement module is used to filter out redundant high-frequency information from the original image using wavelet transform, merge close brightness values ​​into the same brightness value, and directly exchange the corresponding sum when the brightness value matches the randomly generated brightness value. Finally, the image is reconstructed by wavelet transform and converted into RGB to obtain a new data enhancement result.

[0096] Wavelet transform converts the original RGB image into lαβ color space. The transformation is defined as follows:

[0097]

[0098] Wavelet transform, each time the 2x2 sub-region is subjected to wavelet transform, and the changing parameters are calculated as follows:

[0099]

[0100]

[0101]

[0102] Among them, A0 represents the parameters before conversion, (i, j) represents the pixel coordinates, a, b, c, d represent the sum of the grayscale values ​​of all pixels in the area enclosed by the upper left corner of the grayscale image and the current point, A1 represents the approximate component, H1 represents the horizontal detail component, V1 represents the vertical detail component and D1 represents the diagonal detail component.

[0103] See also Fig. 9 , from left to right are the target texture image, the original image, and the image after texture transformation. Wavelet transform is used to transform the texture of clothes in the original image. There are various textures of clothes. It is impossible for the data set to collect clothes of all texture categories. Traditional data enhancement methods can only transform the size, angle, etc. of the image, but they cannot change the texture of the clothes. Therefore, a data enhancement method that can increase the types of clothing textures is needed. Using this data enhancement method, the generalization of the clothing segmentation system can be enhanced, that is, a certain effect can be achieved in segmenting clothes with different textures. Because the principle of deep learning is to use a large number of pictures for training, the pictures that can be recognized in the final test cannot be too different from the training pictures, so this data enhancement method has a certain improvement in the generalization ability of the network and can recognize clothes with different textures.

[0104] S3. Build a deep neural network segmentation model:

[0105] A deep neural network segmentation model is constructed using a feature encoding module and a feature decoding module. The feature encoding module includes a feature extraction backbone network and a feature fusion module. Specifically:

[0106] See also Figure 3 , the feature extraction backbone network includes step1-step 22 , step1-step 22 All are connected in series;

[0107] See also Figure 4 , the step 1 is composed of a convolution module with a step size of 2 (ordinary convolution module) and a convolution module with a step size of 1 connected in series, and takes the original image as input;

[0108] The parameters of step 2-step 4 are the same except for the size of the convolution channel. Each step consists of two consecutive depth-wise separable convolution modules with a step size of 1 and a depth-wise separable convolution module with a step size of 2 connected in series, and a convolution module with a step size of 2 is residually connected to perform feature extraction on the output of the previous step.

[0109] Step 5-step 20All parameters are the same; each step is composed of 3 consecutive depth-wise separable convolutional modules with a step size of 1, accompanied by an empty residual edge structure, which extracts features from the output of the previous step;

[0110] The step 21 Two consecutive depth-wise separable convolution modules with a step size of 1 and a depth-wise separable convolution module with a step size of 2 are connected in series, and a convolution module with a step size of 2 is residually connected to the previous step. 20 The output of is used for feature extraction;

[0111] The step 22 It is composed of three depthwise separable convolution modules with a step size of 1 in series. The three depthwise separable convolution modules have the same parameters except for the size of the convolution channel. The output of the previous step 21 is used for feature extraction.

[0112] See also Figure 5 , the feature fusion module includes a parallel hole convolution module and a convolution module;

[0113] The parallel hole convolution module includes a convolution module step 23-1 , three hole convolution module steps 23-2 -step 23-4 And a global pooling module step 23-5 ;

[0114] The convolution module step 23-1 It consists of a normal convolution with a step size of 1. 22 The output of is used for feature extraction;

[0115] The three hole convolution modules step 23-2 -step 23-4 , each of which consists of a dilated convolution module. The three dilated convolution modules have the same parameters except for the dilation rate. 21 The output of is used for feature extraction;

[0116] The global pooling module step 23-5 It is composed of a global pooling module and a normal convolution module in series. 21 The output of is used for feature extraction;

[0117] The convolution module, after weighting the features of the parallel dilated convolution module, performs a normal convolution step with a step size of 1. 24 , perform feature extraction.

[0118] See also Figure 6The feature decoding module includes two inputs, and the two inputs are step3 and step 24 Output:

[0119] The feature decoding module takes the output of step 3 as input and performs a normal convolution step with a step size of 1. 25 , extract features;

[0120] The feature decoding module will step 24 The output is taken as input and goes through a 4-fold upsampling step 26 , decode and restore the features;

[0121] The feature decoding module will step 25 and step 26 The output features are fused and added, and then two ordinary convolution steps with a step size of 1 are connected in series. 27 , extract features;

[0122] The feature decoding module; step 27 The output is taken as input and goes through a 4-fold upsampling step 28 , decode the features and restore them to the original image size;

[0123] The feature decoding module will step 28 The output of is used to classify pixels, and the softmax module is used to output the probability of each pixel being classified into each of the following categories: short-sleeved shirt, long-sleeved shirt, jacket, shorts, short skirt, long pants, long skirt, and background.

[0124] The feature extraction backbone network step1-step 22 In the above figure, each layer of depth-wise separable convolutional module is followed by a batch normalization layer and a nonlinear activation function layer.

[0125] The size of the convolution kernel is 3x3, the number of convolution channels in step 1 is 32 and 64 respectively, the number of convolution channels in step 2 is 128, the number of convolution channels in step 3 is 256, and the number of convolution channels in step 4-step is 256. 20 The number of convolution channels is 728, and the step 21 The number of channels of the three consecutive depth-wise separable convolutions is 728, 1024, and 1024 respectively, and the number of channels of the convolution of the residual edge is 1024. 22 The number of channels of the three consecutive depth-wise separable convolutions are 1536, 1536, and 2048 respectively.

[0126] The network model in the present invention is mainly composed of convolution operations. Convolution can be generally understood as an operation of calculating the inner product of the grayscale value of an image using an n xn matrix. In the encoding process, different clothes in the image will be processed by convolution step by step, and a value representing it will be obtained respectively. This value can be called feature information, and at the same time, the original image will be reduced to a feature map 16 times the original, and the feature information of each object will be marked on the feature map; in the decoding process, the values ​​representing different images will be processed by deconvolution (inverse operation of convolution). As the feature map is enlarged, the feature information is retained until it is enlarged to the size of the original image. At this time, different colors will be given to the area in the original image according to the feature information of different objects to obtain a segmentation map.

[0127] S4. Training the deep neural network segmentation model: The deep neural network segmentation model is trained by first freezing a part of the weights and then unfreezing all the weights; during the training process, the loss value is calculated using the dice coefficient loss function and the weighted cross entropy loss function training results; refer to Figure 8 Specific:

[0128] Initializing the network parameters of the neural network segmentation model using the He initialization method;

[0129] During training, the Nesterov momentum optimization algorithm is used;

[0130] During training, a pre-trained model based on the VOC2012 dataset is used to freeze step 1-step in the feature extraction backbone network. 22 After training for 50 epochs, the weight parameters of step 1-step in the feature extraction backbone network are 22 Unfreeze the weight parameters and continue training for 50 epochs;

[0131] During training, the dice coefficient loss function is used to calculate the loss function for eight types of training results: short-sleeved shirts, long-sleeved shirts, jackets, shorts, short skirts, long pants, long skirts and backgrounds. At the same time, the weighted cross entropy loss function is used to weight the results of the dice coefficient loss function, and weights are added to the eight types of training results respectively. The two methods are performed alternately;

[0132] During training, a hierarchical learning rate decay strategy was used. The initial learning rate was set to 0.05. When the loss function value did not decrease for three epochs, the learning rate was reduced by half until the loss function value continued to not decrease, and then the training was stopped.

[0133] The dice coefficient loss function uses the deep neural network segmentation model training evaluation index as the loss function, which represents the ratio of the intersection and union of the predicted result and the true value; the calculation formula of the dice coefficient loss function is as follows:

[0134]

[0135] Among them, x and y represent the pixel coordinates, P i Represents the predicted value of the pixel, G i Represents the true value of the pixel prediction value;

[0136] The weighted cross entropy loss function represents the distance between the actual probability and the expected probability. A weight parameter is added to each category according to the number of pixels of each category. The weighted cross entropy loss function is calculated as follows:

[0137]

[0138] Among them, w C Expressed as a weight parameter, the calculation formula is w C =(NN C ) / N, where N represents the total number of pixels, and N C It is represented by the number of pixels with the true value category c, p is represented by the actual probability, and y is represented by the expected probability.

[0139] S5. Post-process the output of the deep neural network segmentation model using a post-processing module; the post-processing module includes a fully connected conditional random field. Specifically:

[0140] A post-processing module is used to post-process the output of the deep neural network segmentation model; the post-processing module includes a fully connected conditional random field.

[0141] The fully connected conditional random field is inputted as the original image and the output of the feature decoding module, and each pixel point of the output image of the original feature decoding module has a category label; each pixel point in the corresponding original image has a corresponding observation value, and each pixel point is connected with all the pixels to form a connection edge, and the output result of the feature decoding module is processed in combination with the relationship between all the pixels in the original image;

[0142] The conditional probability of the fully connected conditional random field is defined as:

[0143]

[0144] Among them, f i 、f j Represented as the feature vector of pixel (i, j), It is represented as the Gaussian weight that measures the similarity between pixels i and j, m is represented as the logarithm of the pixel degree, μ(x i , x j ) is represented as a two-dimensional matrix of the consistency between different label pairs.

[0145] See also Figure 7 , the unary potential function describes the category label x of pixel i i and the corresponding observation y i The relationship is that for each pixel i, it has a category label x i There is also the corresponding observation value y i , so that each pixel is used as a node, and the relationship between pixels is used as an edge, which constitutes a conditional random field. We observe the variable y i To infer the category label x corresponding to pixel i i The difference between the fully connected conditional random field and the binary potential function is that it describes the relationship between each pixel and all other pixels, so it is called "fully connected". The binary potential function encourages similar pixels to be assigned the same label, while pixels with large differences are assigned different labels, and the definition of this "distance" is related to the color value and the actual relative distance. Therefore, CRF can segment the image as close to the boundary as possible.

[0146] The present invention is further described below in conjunction with the image segmentation principle:

[0147] The original image is first input into step 1, and then passes through a convolution module with a step size of 2 and a convolution module with a step size of 1. The number of convolution channels is set to 32 and 64 respectively.

[0148] After the output of step1, features are extracted through step2-step4. Except for the different convolution kernel sizes, the other parameters of these steps are the same; step2-step4 is a series structure. Each step consists of two consecutive depth-wise separable convolutions with a step size of 1 and a depth-wise separable convolution with a step size of 2 in series, and a residual connection is made with a normal convolution with a step size of 2 to extract features from the output of the previous step; the number of convolution channels in step2 is set to 128, the number of convolution channels in step3 is set to 256, and the number of convolution channels in step4 is set to 728;

[0149] After step 4 output, go through step 5-step 20 Extract features. All parameters of these steps are the same. Each step is composed of 3 consecutive depth-separable convolution modules with a step size of 1, and is accompanied by an empty residual edge structure. The output of the previous step is used for feature extraction. Step 5-step 20 The number of convolution channels is set to 728;

[0150] step 20 After output, after step 21 Extract features, step 21 It is composed of two consecutive depth-wise separable convolutions with a step size of 1 and a depth-wise separable convolution with a step size of 2, and a residual connection with a normal convolution with a step size of 2. 20 The output of step 21 The number of channels of the three consecutive depth-wise separable convolutions is set to 728, 1024, and 1024 respectively, and the number of channels of the residual edge convolution is set to 1024;

[0151] step 21 After output, after step 22 Extract features, step 22 It is composed of three depth-wise separable convolutions with a step size of 1. The three convolution modules have the same parameters except for the size of the convolution kernel. 21 The output of step 22 The number of channels of the three consecutive depthwise separable convolutions is set to 1536, 1536, and 2048 respectively.

[0152] After the original image passes through the feature extraction backbone network, it passes through the feature fusion module:

[0153] The original image after step 22 After that, it will pass through a parallel dilated convolution module for feature extraction. The parallel dilated convolution module consists of a convolution module, three dilated convolution modules and a global pooling module.

[0154] The original image after step 22 After that, after the convolution module step 23-1 Feature extraction is performed, which consists of a normal convolution with a step size of 1. 22 The output of is used for feature extraction;

[0155] The original image after step 22 After that, it goes through three dilated convolution modules step 23-2 -step 23-4 For feature extraction, these three dilated convolutions are composed of a dilated convolution module respectively. Except for the different dilated rates, the other parameters are the same. 21 The output of is used for feature extraction, and the dilated convolution rates are set to 6, 12, and 18 respectively;

[0156] The image passes through step 22 After that, through the global pooling module step 23-5Feature extraction is performed by connecting a global pooling layer and a normal convolution module in series. 21 The output of is used for feature extraction;

[0157] The image passes through step 23 After that, five feature maps of different scales are obtained. After feature weighting, they are passed through a normal convolution step with a step size of 1. 24 , perform feature extraction;

[0158] After the image is extracted in step 3, it is used as the input of the feature decoding module and undergoes a normal convolution step with a step size of 1. 25 , extract features;

[0159] The image passes through step 24 After feature extraction, it is used as the input of the feature decoding module and undergoes a 4x upsampling step 26 , decode and restore the features;

[0160] The image passes through step 25 and step 26 Afterwards, their output features are fused and added, and then two concatenated ordinary convolution steps with a step size of 1 are performed. 27 , extract features;

[0161] The image passes through step 27 After feature extraction, a 4x upsampling step is performed 28 , decode the features and restore them to the original image size;

[0162] The image passes through step 28 After feature extraction, the softmax module is used to output the probability size, that is, the probability of each pixel being classified into one of the eight types: short-sleeved shirt, long-sleeved shirt, jacket, shorts, short skirt, long pants, long skirt and background.

[0163] After the feature encoding module predicts the image, the prediction result will go through a post-processing module, which consists of a fully connected conditional random field:

[0164] The predicted image and the original image are input into a fully connected conditional random field, each pixel of the predicted image has a category label, and each pixel in the corresponding original image has a corresponding observation value. At the same time, each pixel is connected to all pixels to form a connecting edge, and the output result of the feature decoding module is processed in combination with the relationship between all pixels in the original image.

[0165] Embodiment 2:

[0166] This embodiment provides a clothing image segmentation method, which inputs a clothing image into a clothing image segmentation model established by using any of the above-mentioned clothing image segmentation model establishment methods, and outputs a segmented clothing image.

[0167] The present invention is not limited to the above optional implementations, and anyone can derive other various forms of products under the enlightenment of the present invention. The above specific implementations should not be understood as limiting the scope of protection of the present invention. The scope of protection of the present invention should be based on the definition in the claims, and the description can be used to interpret the claims.

Claims

1. A method for establishing a clothing image segmentation model, characterized in that: include: S1. Create clothing image dataset: S11. crawling pictures containing clothing from social and e-commerce platforms, wherein the pictures contain a single piece of clothing or a plurality of different pieces of clothing; the categories of clothing include short-sleeved shirts, long-sleeved shirts, coats, shorts, short skirts, long pants, and long skirts; S12, annotate the images at the pixel level, annotate different categories of clothing with different colors; for images of clothing worn on people, separate the human skin from the clothing and classify the background and skin into one category for annotation to obtain the original image; and make the original image into a clothing image dataset; S2, using a data enhancement module to enhance the original image; The data enhancement module includes a rotation data enhancement module, a flip data enhancement module, a splicing data enhancement module and a transformation data enhancement module; S3. Build a deep neural network segmentation model: A deep neural network segmentation model is constructed by using a feature encoding module and a feature decoding module, wherein the feature encoding module includes a feature extraction backbone network and a feature fusion module; S4. Train the deep neural network segmentation model: The deep neural network segmentation model is trained by freezing some weights first and then unfreezing all weights. During the training process, the loss value is calculated using the training results of the dice coefficient loss function and the weighted cross entropy loss function. S5. Post-processing the output of the deep neural network segmentation model using a post-processing module; the post-processing module includes a fully connected conditional random field; In step S3, The feature extraction backbone network includes step1-step 22 , step1-step 22 All are connected in series; The step 1 is composed of a convolution module with a step size of 2 and a convolution module with a step size of 1 connected in series, and takes the original image as input; The parameters of step 2-step 4 are the same except for the size of the convolution channel. Each step consists of two consecutive depth-wise separable convolution modules with a step size of 1 and a depth-wise separable convolution module with a step size of 2 connected in series, and a convolution module with a step size of 2 is residually connected to perform feature extraction on the output of the previous step. Step 5-step 20 All parameters are the same; each step is composed of 3 consecutive depth-wise separable convolutional modules with a step size of 1, accompanied by an empty residual edge structure, which extracts features from the output of the previous step; The step 21 is composed of two consecutive depth-wise separable convolution modules with a step size of 1 and a depth-wise separable convolution module with a step size of 2, which are connected in series and residually connected to a convolution module with a step size of 2. 20 The output of is used for feature extraction; The step 22 is composed of three depth-wise separable convolution modules with a step size of 1 in series. The three depth-wise separable convolution modules have the same parameters except for the size of the convolution channel. 21 The output of is used for feature extraction; In step S3, the feature fusion module includes a parallel hole convolution module and a convolution module; The parallel hole convolution module includes a convolution module step 23-1 , three hole convolution module steps 23-2 -step 23-4 And a global pooling module step 23-5 ; The convolution module step 23-1 It consists of a normal convolution with a step size of 1. 22 The output of is used for feature extraction; The three hole convolution modules step 23-2 -step 23-4 , each of which consists of a dilated convolution module. The three dilated convolution modules have the same parameters except for the dilation rate. 21 The output of is used for feature extraction; The global pooling module step 23-5 It is composed of a global pooling module and a normal convolution module in series, and performs feature extraction on the output of the previous step 21; The convolution module, after weighting the features of the parallel dilated convolution module, performs a normal convolution step with a step size of 1. 24 , perform feature extraction; In step S3, the feature decoding module includes two inputs, and the two inputs are step3 and step 24 Output: The feature decoding module takes the output of step 3 as input and undergoes a normal convolution step with a step size of 1. 25 , extract features; The feature decoding module will step 24 The output is taken as input and goes through a 4-fold upsampling step 26 , decode and restore the features; The feature decoding module will step 25 and step 26 The output features are fused and added, and then two ordinary convolution steps with a step size of 1 are connected in series. 27 , extract features; The feature decoding module; step 27 The output is taken as input and goes through a 4-fold upsampling step 28 , decode the features and restore them to the original image size; The feature decoding module will step 28 The output of is used to classify pixels, and the softmax module is used to output the probability of each pixel being classified into each of the following categories: short-sleeved shirt, long-sleeved shirt, jacket, shorts, short skirt, long pants, long skirt, and background.

2. A method for establishing a clothing image segmentation model according to claim 1, characterized in that: In step S2, The rotation data enhancement module is used to rotate the original image and the annotation around the center point at the same angle, and the rotation angle takes a random value between 0 and 360 degrees; The flip data enhancement module is used to flip the original image and annotation around the horizontal symmetry axis and the vertical symmetry axis of the image; The splicing data enhancement module is used to randomly select four corresponding samples of the original image and the annotation, perform rotation, scaling and / or cropping operations, and then merge them into a new image; The transformation data enhancement module is used to filter out redundant high-frequency information from the original image using wavelet transform, merge close brightness values ​​into the same brightness value, and directly exchange the corresponding sum when the brightness value and the randomly generated brightness value match. Finally, the image is reconstructed by wavelet transform and converted into RGB to obtain a new data enhancement result.

3. A method for establishing a clothing image segmentation model according to claim 1, characterized in that: In step S4, the deep neural network segmentation model is trained: Initializing the network parameters of the neural network segmentation model using the He initialization method; During training, the Nesterov momentum optimization algorithm is used; During training, a pre-trained model based on the VOC2012 dataset was used to freeze the weight parameters of step 1--Step 22 in the feature extraction backbone network. After training for 50 epochs, the weight parameters of step 1--Step 22 in the feature extraction backbone network were frozen. 22 Unfreeze the weight parameters and continue training for 50 epochs; During training, the dice coefficient loss function is used to calculate the loss function for eight types of training results: short-sleeved shirts, long-sleeved shirts, jackets, shorts, short skirts, long pants, long skirts and backgrounds. At the same time, the weighted cross entropy loss function is used to weight the results of the dice coefficient loss function, and weights are added to the eight types of training results respectively. The two methods are performed alternately; During training, a hierarchical learning rate decay strategy was used. The initial learning rate was set to 0.

05. When the loss function value did not decrease for three epochs, the learning rate was reduced by half until the loss function value continued to not decrease, and then the training was stopped.

4. A clothing image segmentation model establishment method according to claim 3, characterized in that: The dice coefficient loss function uses the deep neural network segmentation model training evaluation index as the loss function, which represents the ratio of the intersection and union of the predicted result and the true value; the calculation formula of the dice coefficient loss function is as follows: Among them, x and y represent the pixel coordinates, P i Represents the predicted value of the pixel, G i Represents the true value of the pixel prediction value; The weighted cross entropy loss function represents the distance between the actual probability and the expected probability. A weight parameter is added to each category according to the number of pixels of each category. The weighted cross entropy loss function is calculated as follows: Among them, w C Expressed as a weight parameter, the calculation formula is w C =(NN c ) / N, where N represents the total number of pixels, and N c It is represented by the number of pixels with the true value category c, p is represented by the actual probability, and y is represented by the expected probability.

5. The method for establishing a clothing image segmentation model according to claim 1, characterized in that: The feature extraction backbone network step1-step 22 In the above figure, each layer of depth-wise separable convolutional module is followed by a batch normalization layer and a nonlinear activation function layer. The size of the convolution kernel is 3x3, the number of convolution channels in step 1 is 32 and 64 respectively, the number of convolution channels in step 2 is 128, the number of convolution channels in step 3 is 256, and the number of convolution channels in step 4-step is 256. 20 The number of convolution channels of step 21 is 728, the number of convolution channels of the three consecutive depth-wise separable convolutions of step 21 are 728, 1024, and 1024 respectively, the number of convolution channels of the residual edge is 1024, and the number of convolution channels of step 21 is 1024. 22 The number of channels of the three consecutive depth-wise separable convolutions are 1536, 1536, and 2048 respectively.

6. A clothing image segmentation model building method according to claim 1, characterized in that: In step S5, Post-processing the output of the deep neural network segmentation model using a post-processing module; the post-processing module includes a fully connected conditional random field; The fully connected conditional random field is inputted as the original image and the output of the feature decoding module, and each pixel point of the output image of the original feature decoding module has a category label; each pixel point in the corresponding original image has a corresponding observation value, and each pixel point is connected with all the pixels to form a connection edge, and the output result of the feature decoding module is processed in combination with the relationship between all the pixels in the original image; The conditional probability of the fully connected conditional random field is defined as: Among them, f i 、f j Represented as the feature vector of pixel (i, j), It is represented as the Gaussian weight that measures the similarity between pixels i and j, m is represented as the logarithm of the pixel degree, μ(x i , x j ) is represented as a two-dimensional matrix of the consistency between different label pairs.

7. A clothing image segmentation method, characterized in that: The clothing image is input into a clothing image segmentation model established by the clothing image segmentation model establishment method described in any one of claims 1 to 6, and the segmented clothing image is output.

Images