A data augmentation method and device for a clothing image segmentation model training data set

Abstract

This invention relates to the fields of artificial intelligence deep learning and fashion apparel technology, specifically providing a data augmentation method and apparatus for training datasets of clothing image segmentation models. It aims to solve the problem in existing technologies where the patterns in clothing images cannot be altered, thus failing to achieve satisfactory data augmentation results. The data augmentation method includes the following steps: receiving a target pattern image, an original clothing image, and pixel-level annotated real-world images; extracting the clothing portion from the original clothing image to obtain the original clothing image to be augmented; transforming the patterns in the original clothing image to be augmented into the patterns in the target pattern image, and then fitting it to a new background to obtain the data-augmented clothing image. The apparatus includes an annotation unit, a clothing extraction unit, a deep learning network unit, and a background transformation unit. By altering the pattern features in the original clothing image, the diversity of the dataset is increased, thereby achieving a good data augmentation effect.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence deep learning and fashion apparel technology, specifically relating to a data augmentation method and apparatus for training datasets of apparel image segmentation models. Background Technology

[0002] After modeling patterns in images and sounds, deep learning networks can recognize these patterns. Deep learning networks offer a method for computers to automatically learn pattern features and integrate feature learning into the modeling process, thereby reducing the incompleteness caused by manually designed features. Currently, some machine learning applications based on deep learning networks have achieved recognition or classification performance surpassing existing algorithms under specific conditions.

[0003] Image segmentation is an important problem in the field of deep learning networks. It is the foundation of computer vision and an important preprocessing step for various tasks such as image classification and style transfer. Due to the complexity and difficulty of segmentation tasks, a large number of images are needed to support the training of deep learning networks so that they can achieve better performance.

[0004] However, creating datasets for segmentation tasks is extremely difficult, requiring pixel-level image annotation and significant human and material resources. Furthermore, overfitting is prone to occur during the training of deep learning networks; without a large number of supporting images, the generalization ability of deep learning network models decreases, resulting in suboptimal recognition accuracy. Traditional data augmentation techniques, based on simple transformations in image processing such as cropping, scaling, rotation, or translation, cannot alter the pattern features of the object being detected. Pattern features include the fabric, color, and patterns on clothing. Since many clothing images share similar styles but differ in pattern features, traditional data augmentation techniques often fail to achieve satisfactory results. Summary of the Invention

[0005] This invention provides a data augmentation method and apparatus for training datasets of clothing image segmentation models, aiming to solve the problem that existing technologies cannot modify the pattern features in clothing images, thus failing to achieve good data augmentation results.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a data augmentation method for a training dataset of a clothing image segmentation model, comprising the following steps:

[0008] S100: Receive the target pattern image and label the pattern category; receive the original clothing image, and label the area of ​​the clothing in the original clothing image at the pixel level to obtain a real labeled image;

[0009] S200: Using the real-labeled image, the clothing portion in the original clothing image is extracted from the original clothing image by pixel-level segmentation to obtain the original clothing image to be enhanced;

[0010] S300: Convert the pattern in the original clothing image to be enhanced into the pattern in the target pattern image to obtain a virtual clothing image;

[0011] S400: The virtual clothing image is then overlaid with the new background to obtain a data-enhanced clothing image.

[0012] A further solution: In step S300, the pattern in the original clothing image to be enhanced is transformed into the pattern in the target pattern image using a deep learning network.

[0013] A further solution: The deep learning network includes a generator for generating the virtual clothing image and a discriminator for determining whether the virtual clothing image is real.

[0014] In a further embodiment, the generator includes:

[0015] Generator A is used to convert the pattern of the original clothing image to be enhanced into the pattern of the target pattern image, generate and output the virtual clothing image to be verified;

[0016] Generator B is used to restore the virtual clothing image output by generator A into the original clothing image to be enhanced and verified.

[0017] The discriminator includes:

[0018] Discriminator A is used to determine whether the virtual clothing image to be verified output by generator A is real;

[0019] Discriminator B is used to determine whether the original clothing image to be enhanced and verified output by generator B is genuine;

[0020] The model structure of generator A is the same as that of generator B, and generator A and generator B are updated synchronously.

[0021] In a further embodiment, the deep learning network is trained using a method that trains both the generator and the discriminator together.

[0022] Based on the above scheme, the deep learning network continuously optimizes the parameters inside the generator and the discriminator by training the generator and the discriminator together, and trains a deep learning network that can transfer the pattern in the target pattern image to the original clothing image to be enhanced.

[0023] A further approach: The specific steps for training the deep learning network include:

[0024] The original clothing image to be enhanced in the dataset is used as input to generator A, which generates and outputs the virtual clothing image to be verified.

[0025] Both the target pattern image and the original clothing image to be enhanced are input into the discriminator A. The discriminator A compares the target pattern image with the virtual clothing image to be verified output by the generator A. If the discriminator A determines that the virtual clothing image to be verified output by the generator A is false, the discriminator A sends feedback to the generator A, which regenerates and outputs the virtual clothing image to be verified. The discriminator A then continues to determine whether the virtual clothing image to be verified is genuine.

[0026] The virtual clothing image to be verified output by generator A is used as the input of generator B, and generator B outputs the original clothing image to be verified and enhanced.

[0027] Both the original clothing image to be enhanced and the original clothing image to be verified are input into the discriminator B. The discriminator B compares the original clothing image to be verified output by the generator B with the original clothing image to be enhanced. If the discriminator B determines that the original clothing image to be verified output by the generator B is false, the discriminator B feeds back to the generator B. The generator A regenerates and outputs the virtual clothing image to be verified, the generator B regenerates the original clothing image to be verified, and the discriminator B continues to determine whether the original clothing image to be verified is true.

[0028] The verification continues until the discriminator A determines that the virtual clothing image to be verified generated and output by the generator A is real, and the discriminator B determines that the original clothing image to be verified and enhanced output by the generator B is real;

[0029] The deep learning network is trained sequentially using each set of training data in the dataset to obtain the trained deep learning network.

[0030] Based on the above scheme, generator B is introduced. Generator B restores the virtual clothing image generated by generator A into the original clothing image to be enhanced. Then, discriminator B discriminates the restored original clothing image to be enhanced, and the discrimination result is fed back to generator B so that generator A and generator B are optimized synchronously, thereby realizing the model optimization of the deep learning network.

[0031] Discriminator A is used to discriminate the patterns in the virtual clothing image generated by generator A, and discriminator B is used to discriminate the clothing pattern in the original clothing image to be enhanced restored by generator B. This ensures that the virtual clothing image generated by the deep learning network presents a good effect, so that the virtual clothing image meets the usage requirements.

[0032] A further approach: Training the deep learning network also employs a loss function, the expression for which is:

[0033] L loss =L GAN1 +L GAN2 +L CLC

[0034] The L GAN1 Let L be the loss function of the generator A; GAN2 The loss function of the generator B; the L CLC This is a constraint on generator A and generator B.

[0035] Based on the above scheme, the loss function L of the deep learning network is used. loss The underlying model training logic of the generator and the discriminator are unified, making the obtained virtual clothing image more similar to the original clothing image to be enhanced. The virtual clothing image is closer to the original clothing image to be enhanced, making the pattern information more realistic, thereby meeting the usage requirements.

[0036] The L CLC This avoids the deep learning network from being underlearned, which would cause the pattern in the original clothing image to be enhanced to fail to be transformed into the pattern in the target pattern image in step S300; it also avoids the deep learning network from being overlearned, which would cause the pattern of the clothing in the original clothing image to be enhanced to be changed when the pattern in the original clothing image to be enhanced is transformed into the pattern in the target pattern image.

[0037] A further solution: The L GAN1 L GAN2 and L CLC The loss function expressions are as follows:

[0038] L GAN1 =E y～Y [lnD Y (y)]+E x～X [ln(1-D Y (G(x)))]

[0039] L GAN2 =E x～X [lnD X (x)]+E y～Y [ln(1-D X (F(yy)))]

[0040] L CLC =E x～X [||F(G(X))-x||1]+E y～Y [||G(F(y))-y||1]

[0041] X represents all the original clothing images, x represents a sample obtained from X, and x ~ X represent samples taken from X; Y represents all the virtual clothing images to be verified generated and output by generator A, y represents a sample obtained from Y, and y ~ Y represent samples taken from Y; E() represents the expected value, G() represents the process of generator A generating the virtual clothing images to be verified, and F() represents the process of generator B generating the original clothing images to be verified and enhanced; D X () represents the discrimination process of the discriminator A, D Y () The process of discrimination by the discriminator B.

[0042] A further solution: In step S400, the virtual clothing image is transformed and then matched with a new background; the real annotation is transformed to correspond to the virtual clothing image to obtain a virtual annotation image.

[0043] Secondly, the present invention provides a data augmentation apparatus for a training dataset of a clothing image segmentation model, comprising:

[0044] The annotation unit receives the target pattern image and annotates the pattern category; it also receives the original clothing image and annotates the area of ​​the clothing in the original clothing image at the pixel level to obtain a real annotated image.

[0045] The clothing extraction unit extracts the clothing portion from the original clothing image at the pixel level using the real labeled image obtained by the labeling unit, thereby obtaining the original clothing image to be enhanced.

[0046] The deep learning network unit transforms the patterns in the original clothing image to be enhanced obtained by the clothing extraction unit into the patterns in the target pattern image, thereby obtaining a virtual clothing image;

[0047] The background transformation unit combines the virtual clothing image obtained by the deep learning network unit with a new background to obtain a data-enhanced clothing image.

[0048] The beneficial effects of this invention are as follows:

[0049] The annotation unit annotates the received target pattern image with pattern categories and receives the original clothing image with pixel-level annotations on the clothing areas therein, obtaining the real annotated image. Then, the clothing extraction unit uses the real annotated image to perform pixel-level segmentation and extraction of the clothing portion from the original clothing image, obtaining the original clothing image to be enhanced. The deep learning network then transforms the patterns in the original clothing image to be enhanced into the patterns in the target pattern image, obtaining a virtual clothing image. Finally, in the background transformation unit, the virtual clothing image is pasted onto a new background to obtain the data-enhanced clothing image. This invention provides a data augmentation method and apparatus for training datasets of clothing image segmentation models. By using a deep learning network to transform patterns in the original clothing image into target patterns, the pattern features in the original clothing image are altered, further increasing the diversity of the dataset and achieving excellent data augmentation results. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a data augmentation method and apparatus for training datasets of clothing image segmentation models according to the present invention;

[0052] Figure 2 This is a pixel-level annotation diagram of the original clothing image, which is part of the data augmentation method and apparatus for training datasets of clothing image segmentation models according to the present invention.

[0053] Figure 3 This is a schematic diagram of a data augmentation method and apparatus for training a deep learning network for a clothing image segmentation model training dataset, according to the present invention.

[0054] Figure 4This is a schematic diagram illustrating the data augmentation method of the present invention, which is a data augmentation method and apparatus for training datasets of clothing image segmentation models.

[0055] Figure 5 This is a schematic diagram of the generator structure in a deep learning network of a data augmentation method and apparatus for training datasets of clothing image segmentation models according to the present invention;

[0056] Figure 6 This is a schematic diagram of the discriminator in a deep learning network, which is part of a data augmentation method and apparatus for training datasets of clothing image segmentation models according to the present invention.

[0057] Figure 7 This is a schematic diagram of a data augmentation method and apparatus for training datasets of clothing image segmentation models according to the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the protection scope of the present invention.

[0059] Example 1:

[0060] like Figure 1 As shown in this embodiment, a data augmentation method for a training dataset of a clothing image segmentation model includes the following steps:

[0061] S100: Receive the target pattern image and label the pattern category; receive the original clothing image, and label the area of ​​the clothing in the original clothing image at the pixel level to obtain a real labeled image;

[0062] The pattern category includes: fabric, pattern design, or pattern color. The implementation process of step S100 specifically includes the following steps:

[0063] S110. Download images containing clothing using a web crawler and manually select them as the original clothing images; use Labelme annotation software to manually annotate each original clothing image at the pixel level to obtain the real annotated image; similarly, download images of different patterns using a web crawler, manually annotate each target pattern image, and label the category of each pattern.

[0064] S120. Scale and crop each of the real labeled images and the original clothing images, and adjust them to a uniform size of 480x480.

[0065] S130. The real labeled images and the corresponding original clothing images are grouped together and divided into training set and validation set in a 9:1 ratio. In this example, we obtain a total of 200 sets of corresponding real labeled images and original clothing images, and 200 different target pattern images.

[0066] S200. Using the real-world labeled image, the clothing portion of the original clothing image is extracted pixel-level from the original clothing image to obtain the original clothing image to be enhanced, such as... Figure 2 As shown;

[0067] The process of segmenting and extracting the clothing portion from the original clothing image includes:

[0068] In Python, the `bitewise_and()` function from the OpenCV toolkit is used to obtain the original clothing image to be enhanced, taking the ground truth image and the original clothing image as input. The original clothing image to be enhanced is an image containing only the clothing area.

[0069] S300: Convert the pattern in the original clothing image to be enhanced into the pattern in the target pattern image to obtain a virtual clothing image;

[0070] S400: The virtual clothing image is then overlaid with the new background to obtain a data-enhanced clothing image.

[0071] One of the improved solutions is as follows:

[0072] In step S300, the pattern in the original clothing image to be enhanced is transformed into the pattern in the target pattern image using a deep learning network.

[0073] The deep learning network includes a generator for generating the virtual clothing image and a discriminator for determining whether the virtual clothing image is real.

[0074] The generator includes:

[0075] Generator A is used to convert the pattern of the original clothing image to be enhanced into the pattern of the target pattern image, generate and output the virtual clothing image to be verified;

[0076] Generator B is used to restore the virtual clothing image output by generator A into the original clothing image to be enhanced and verified.

[0077] The discriminator includes:

[0078] Discriminator A is used to determine whether the virtual clothing image to be verified output by generator A is real;

[0079] Discriminator B is used to determine whether the original clothing image to be enhanced and verified output by generator B is genuine;

[0080] The model structure of generator A is the same as that of generator B, and generator A and generator B are updated synchronously.

[0081] The deep learning network is trained using a method that trains both the generator and the discriminator together.

[0082] See Figure 3 The specific steps for training the deep learning network include:

[0083] The original clothing image to be enhanced in the dataset is used as input to generator A, which generates and outputs the virtual clothing image to be verified.

[0084] Both the target pattern image and the original clothing image to be enhanced are input into the discriminator A. The discriminator A compares the target pattern image with the virtual clothing image to be verified output by the generator A. If the discriminator A determines that the virtual clothing image to be verified output by the generator A is false, the discriminator A sends feedback to the generator A, which regenerates and outputs the virtual clothing image to be verified. The discriminator A then continues to determine whether the virtual clothing image to be verified is genuine.

[0085] The virtual clothing image to be verified output by generator A is used as the input of generator B, and generator B outputs the original clothing image to be verified and enhanced.

[0086] Both the original clothing image to be enhanced and the original clothing image to be verified are input into the discriminator B. The discriminator B compares the original clothing image to be verified output by the generator B with the original clothing image to be enhanced. If the discriminator B determines that the original clothing image to be verified output by the generator B is false, the discriminator B feeds back to the generator B. The generator A regenerates and outputs the virtual clothing image to be verified, the generator B regenerates the original clothing image to be verified, and the discriminator B continues to determine whether the original clothing image to be verified is true.

[0087] The verification continues until the discriminator A determines that the virtual clothing image to be verified generated and output by the generator A is real, and the discriminator B determines that the original clothing image to be verified and enhanced output by the generator B is real;

[0088] The deep learning network is trained sequentially using each set of training data in the dataset to obtain the trained deep learning network.

[0089] Specifically, both generator A and generator B are equipped with the same style extraction module; the style extraction module is used to extract feature information.

[0090] Specifically, such as Figure 5 As shown, the process by which generator A generates the virtual clothing image specifically includes the following steps:

[0091] S310. The original clothing image to be enhanced is used as the input of generator A and the original clothing image to be enhanced is converted into matrix information to obtain a feature map; a 3x3 convolutional layer is used to downsample the feature map so that the size of the feature map becomes 1 / 8 of the size of the original clothing image to be enhanced.

[0092] The original clothing image to be enhanced, with a size of 480x480x3, is used as the input to generator A. It is then subjected to convolution with a kernel size of 7 and a stride of 1 to expand the channel dimension, resulting in a feature map of 480x480x64.

[0093] Specifically, the feature map obtained above is subjected to three consecutive convolutions with a kernel size of 3 and a stride of 2. The feature map sizes obtained by the three convolutions are 240x240x128, 120x120x256, and 60x60x512, respectively. Reducing the feature map size facilitates subsequent feature extraction and increases the number of channels.

[0094] S320. Perform continuous convolution on the feature map after the size is reduced, and perform depth extraction on the feature information in the feature map through the style extraction module;

[0095] Specifically, the feature map with reduced size is subjected to feature information extraction by the same style extraction module nine times in a row, and the feature map size obtained each time is 60x60x512.

[0096] The structure of the style extraction module is as follows: Figure 5 As shown, a feature map of size 60x60x512 is used as input to the style extraction module. First, a convolution with a kernel size of 3 and a stride of 1 is performed, followed by normalization using an InstanceNorm layer. Then, it passes through a ReLU activation function to obtain a feature map with the same size. This feature map with the same size is then subjected to another convolution with a kernel size of 3 and a stride of 1, and normalized again using an InstanceNorm layer to obtain a feature map with a significantly changed size, which is then used as the output of the style extraction module.

[0097] S330. Upsample the above feature map by deconvolution to restore the feature map size to the size of the original clothing image to be enhanced, and obtain the virtual clothing image by activation function.

[0098] Specifically, the feature map is subjected to three consecutive deconvolutions with a kernel size of 3 and a stride of 2. The feature map sizes obtained by the three deconvolutions are 120x120x256, 240x240x128, and 480x480x64, respectively. The size of the feature map is gradually restored while the number of channels is reduced.

[0099] Specifically, the feature map obtained above is subjected to a final convolution with a kernel size of 7 and a stride of 1, restoring the number of channels to the initial number of 3, resulting in a feature map size of 480x480x3. After passing through the tanh activation function, the generated virtual clothing image is obtained.

[0100] Specifically, the model structure of generator B is the same as that of generator A; in generator B, the virtual clothing image generated by generator A is used as the input of generator B, and the same operation steps as generator A are performed to obtain the original clothing image to be enhanced and verified.

[0101] Specifically, such as Figure 6 As shown, the process by which the discriminator A determines whether the virtual clothing image generated by the generator A is real includes the following steps:

[0102] S340. The virtual clothing image and the target pattern image generated by the generator A are used as inputs to the discriminator A, and the virtual clothing image generated by the generator A is converted into matrix information to obtain the feature map of the virtual clothing image.

[0103] Specifically, the virtual clothing image with a size of 480x480x3 generated by generator A and the target pattern image with a size of 480x480x3 are used as inputs to discriminator A. The two images are stitched together by a Concat operation to obtain a feature map with a size of 480x480x6.

[0104] S350. Perform continuous 3x3 convolutions with a stride of 2 to continuously downsample and increase the number of channels in the feature map of the virtual clothing image, so that the feature map size becomes 1 / 16 of the size of the virtual clothing image.

[0105] Specifically, the feature map of the virtual clothing image is subjected to four consecutive convolutions with a kernel size of 3 and a stride of 2. The feature map sizes obtained from the four convolutions are 240x240x64, 120x120x128, 60x60x256, and 30x30x512, respectively. Reducing the feature map size facilitates feature extraction and increases the number of channels.

[0106] S360. Perform a 3x3 convolution with a stride of 1 to reduce the number of channels to 1, and determine whether the virtual clothing image generated by generator A is real.

[0107] Specifically, the feature map is subjected to a final convolution with a kernel size of 3 and a stride of 1, and channel reduction is performed to obtain a feature map with a size of 30x30x1; then, the Softmax function is used to determine whether the virtual clothing image generated by generator A is real.

[0108] Specifically, the model structure of the discriminator B is consistent with that of the discriminator A. In the discriminator B, the original clothing image to be enhanced and the original clothing image to be verified are used as inputs to the discriminator B. The same operation steps as those of the discriminator A are performed, and the original clothing image to be enhanced is compared with the original clothing image to be enhanced to determine whether the original clothing image to be enhanced generated by the generator B is genuine.

[0109] The deep learning network is trained sequentially using each set of training data in the dataset, and then validated using a validation set to obtain the trained deep learning network.

[0110] Training the deep learning network also employs a loss function, the expression of which is:

[0111] L loss ==L GAN1 +L GAN2 +L CLC

[0112] The L GAN1 The loss function is the adversarial loss function between the generator A and the discriminator B; the L GAN2 The loss function is the one used when the generator B and the discriminator A are in adversarial competition; the L CLC This is a constraint on generator A and generator B.

[0113] The L GAN1 L GAN2 and L CLC The loss function expressions are as follows:

[0114] L GAN1 =E y～Y [lnDY (y)]+E x～X [ln(1-D Y (G(x)))]

[0115] L GAN2 =E x～X [lnD X (x)]+E y～Y [ln(1-D X (F(yy)))]

[0116] L CLC =E x～X [||F(G(x))-x||1]+E y ~Y[||G(F(y))-y||1]

[0117] X represents all the original clothing images, x represents a sample obtained from X, and x ~ X represent samples taken from X; Y represents all the virtual clothing images to be verified generated and output by generator A, y represents a sample obtained from Y, and y ~ Y represent samples taken from Y; E() represents the expected value, G() represents the process of generator A generating the virtual clothing images to be verified, and F() represents the process of generator B generating the original clothing images to be verified and enhanced; D X () represents the discrimination process of the discriminator A, D Y () The process of discrimination by the discriminator B.

[0118] Specifically, for the L CLC The loss function is used because the training objective of the deep learning network is to make F(G(x)) as close to x as possible. CLC The loss function is used for constraint, that is, x→G(x)→F(G(x))≈x; similarly, L... CLC The loss function is used to constrain G(F(y)), that is, y→F(y)→G(F(y))≈y.

[0119] Specifically, in order to make the training process of the deep learning network more stable, the training actually uses a least squares loss method.

[0120] Specifically, with the L mentioned above GAN1 For example, the training of the deep learning network includes a generator A and a discriminator B. The optimization objective of the discriminator B becomes minimizing E. y～Y [D Y (y)]+E x～X [(1-D Y (G(x)))], the first part of the optimization objective of the discriminator B is to hope that D Y(y) should be as close to 1 as possible, meaning that when the virtual clothing image y to be verified is input, the discriminator B should consider y to be a sample taken from the Y space with the highest possible probability. The second part is to hope that D Y The discriminator B aims to minimize the probability that the input (G(x)) is in the Y space, meaning that when given a generated fake image G(x), the probability that the input is in the Y space is as low as possible. The optimization objective of the generator A then becomes minimizing E. x～X [ln(1-D Y (G(x)))], the optimization objective of the generator A is to hope that D Y (G(x)) should be as close to 1 as possible, while D Y The closer G(x) is to 1, the more the discriminator B believes that the image G(x) generated by the generator A is in the Y space. Therefore, the generator A and the discriminator B have different goals; the generator A hopes that L... GAN1 The smaller the value, the more discriminator B expects L to be. GAN1 The larger the loss, the greater the counter-loss.

[0121] In step S400, the virtual clothing image is transformed and then matched with a new background; the real annotations are transformed to correspond to the virtual clothing image to obtain a virtual annotation image of the data-enhanced clothing image.

[0122] Specifically, in Python, the getRotationMatrix2D() and getAffineTransform() functions from the OpenCV toolkit are used. The virtual clothing image generated by the deep learning network is taken as input, and after random rotation, translation or scaling by the functions, the transformed image is combined with multiple new background images at random positions by the bitewise_and() function to obtain multiple data-enhanced images.

[0123] Specifically, for the real labeled image, in Python, the getRotationMatrix2D() and getAffineTransform() functions from the OpenCV toolkit are used. Taking the real labeled image as input, the functions perform random rotation, translation, or scaling transformations to obtain a virtual labeled image corresponding to the data-augmented image.

[0124] Based on any of the above schemes, the extracted original clothing image to be enhanced and the trained deep learning network are used to perform data augmentation on the clothing image, generating a virtual clothing image with the target pattern. The real labeled image is then transformed by rotation, translation, or scaling to obtain a virtual labeled image corresponding to the real labeled image. The virtual clothing image and the virtual labeled image are used as the data augmentation structure for training subsequent image segmentation tasks.

[0125] When training the image segmentation task, a pre-training method is adopted. The pre-training method is as follows: the pre-training is divided into two stages. In the first stage, the virtual clothing image and the virtual labeled image are used for training to obtain the trained deep learning network model. In the second stage, the original clothing image and the real labeled image are used to continue training the deep learning network model trained in the first stage for fine-tuning.

[0126] The deep learning network model was trained on an A5000 graphics card with 24GB of VRAM, a batch size of 8, and SGD (Stochastic Steepest Descent) as the optimizer. The initial learning rate was set to 1e-2. All training images were set to 480x480 pixels. The learning rate was kept constant for the first 100 rounds. After 100 rounds, an early stopping strategy was implemented to automatically stop training when the deep learning network model reached overfitting. The dataset used consisted of 200 sets of original clothing images and target pattern images, divided into training and validation sets in a 9:1 ratio. 1e-2 is 10 to the power of negative 2, or 0.01.

[0127] The following section evaluates the effectiveness of the data augmentation method for training datasets of clothing image segmentation models proposed in this invention, using clothing image segmentation models as an example.

[0128] Two commonly used evaluation metrics for the clothing image segmentation task are mAP and mIoU. Higher metrics indicate better data augmentation results. Table 1 shows the results of comparing the real-labeled images and target pattern images of the 200 sets of original clothing images we collected with their virtual clothing images and virtual-labeled images.

[0129] Table 1 shows the comparison results of 200 datasets on the clothing image segmentation task:

[0130] Data volume mAP mIoU Original dataset 200 0.7689 0.6564 Original dataset + Data augmented dataset 2000 0.8976 0.7763

[0131] As shown in Table 1, the data augmentation method for training datasets of clothing image segmentation models proposed in this invention can increase the amount of data when the original dataset only has 200 images. Compared with using only the original dataset, the performance of clothing image segmentation is significantly improved after adding the data augmented dataset in the clothing image segmentation task. The mPA index and the mIoU index are both improved by nearly 12%, resulting in a clothing image segmentation model with better performance.

[0132] Example 2:

[0133] like Figure 7 As shown in this embodiment, a data augmentation device for training a training dataset for a clothing image segmentation model includes:

[0134] The annotation unit receives the target pattern image and annotates the pattern category; it also receives the original clothing image and annotates the area of ​​the clothing in the original clothing image at the pixel level to obtain a real annotated image.

[0135] The clothing extraction unit extracts the clothing portion from the original clothing image at the pixel level using the real labeled image obtained by the labeling unit, thereby obtaining the original clothing image to be enhanced.

[0136] The deep learning network unit transforms the patterns in the original clothing image to be enhanced obtained by the clothing extraction unit into the patterns in the target pattern image, thereby obtaining a virtual clothing image;

[0137] The background transformation unit combines the virtual clothing image obtained by the deep learning network unit with a new background to obtain a data-enhanced clothing image.

[0138] The invention will be further illustrated below with specific examples:

[0139] See Figure 4 A specific example is converting plain shirts into patterned shirts, which increases the diversity of shirt data and thus achieves a better data enhancement effect.

[0140] Step 1: The annotation unit receives the target pattern image and annotates the image with the pattern category. It also receives the original clothing image, i.e., the image of a shirt without a pattern, and annotates the image of the shirt without a pattern at the pixel level to obtain the real annotated image.

[0141] Step 2: The clothing extraction unit extracts the clothing portion from the plain shirt image using the real-labeled image, resulting in the original clothing image to be enhanced.

[0142] Step 3: The deep learning network unit transforms the plain shirt image into a patterned shirt image using a deep learning network, obtaining a virtual clothing image. The pattern in the patterned shirt image is the same as the pattern in the target patterned image.

[0143] Before the deep learning network transforms the image of a plain shirt into an image of a patterned shirt, the deep learning network is first trained.

[0144] The following section uses multiple sets of plain shirt images and target patterned images from the dataset as examples to illustrate the deep learning network trained to transform plain shirt images into patterned shirt images. Figure 5 and Figure 6 As shown.

[0145] Specifically, input a plain shirt image into generator A, generator A generates a patterned shirt image, input the patterned shirt image into generator B, generator B restores the plain shirt image.

[0146] The target patterned image and the patterned shirt image generated by generator A are input into discriminator A. Discriminator A compares the patterned shirt image with the target patterned image. If the patterned shirt image is false, it is fed back to generator A. Generator A regenerates the patterned shirt image. Discriminator A continues to determine whether the patterned shirt image is real.

[0147] Both the plain shirt image and the plain shirt image reconstructed by generator B are input into discriminator B. The discriminator compares the clothing patterns in the two images. If the plain shirt image reconstructed by generator B is false, it is fed back to generator B. Generator A regenerates a patterned shirt image, and generator B continues to reconstruct the plain shirt image. The discriminator B then continues to determine whether the plain shirt image reconstructed by generator B is genuine.

[0148] The process continues until the patterned shirt image generated by discriminator A and generator A is considered true, and the unpatterned shirt image restored by discriminator B and generator B is also considered true. The deep learning network is then trained sequentially using each set of training data from the dataset to obtain the trained deep learning network.

[0149] Step 4: The background transformation unit transforms the patterned image by translation, rotation, or scaling and then fits it with the new background to obtain different data-enhanced clothing images; the background transformation unit transforms the real annotation images in the annotation unit by translation, rotation, or scaling to obtain different virtual annotation images.

[0150] This invention is not limited to the above-described optional embodiments. Anyone can derive other various forms of products under the guidance of this invention. However, regardless of any changes made in their shape or structure, any technical solution that falls within the scope of the claims of this invention shall be protected by this invention.

Claims

1. A data augmentation method for training datasets of clothing image segmentation models, characterized in that: Includes the following steps: S100: Receive the target pattern image and label the pattern category; Receive the original clothing image, and annotate the area of ​​the clothing in the original clothing image at the pixel level to obtain a real annotated image; S200: Using the real-labeled image, the clothing portion in the original clothing image is extracted from the original clothing image by pixel-level segmentation to obtain the original clothing image to be enhanced; S300: The pattern in the original clothing image to be enhanced is transformed into the pattern in the target pattern image through a deep learning network to obtain a virtual clothing image; The deep learning network includes a generator for generating the virtual clothing image and a discriminator for determining whether the virtual clothing image is real; the generator includes: Generator A is used to convert the pattern of the original clothing image to be enhanced into the pattern of the target pattern image, generate and output the virtual clothing image to be verified; Generator B is used to restore the virtual clothing image to be verified output by generator A into the original clothing image to be enhanced. The discriminator includes: Discriminator A is used to determine whether the virtual clothing image to be verified output by generator A is real; Discriminator B is used to determine whether the original clothing image to be enhanced and verified output by generator B is genuine; The model structure of generator A is the same as that of generator B, and generator A and generator B are updated synchronously. S400: The virtual clothing image is then overlaid with the new background to obtain a data-enhanced clothing image.

2. The data augmentation method for a training dataset of a clothing image segmentation model according to claim 1, characterized in that: The deep learning network is trained using a method that trains both the generator and the discriminator together.

3. The data augmentation method for a training dataset of a clothing image segmentation model according to claim 2, characterized in that: The specific steps for training the deep learning network include: The original clothing image to be enhanced in the dataset is used as input to generator A, which generates and outputs the virtual clothing image to be verified. Both the target pattern image and the virtual clothing image to be verified are input into the discriminator A. The discriminator A compares the target pattern image with the virtual clothing image to be verified output by the generator A. If the discriminator A determines that the virtual clothing image to be verified output by the generator A is false, the discriminator A sends feedback to the generator A, which regenerates and outputs the virtual clothing image to be verified. The discriminator A then continues to determine whether the virtual clothing image to be verified is true. The virtual clothing image to be verified output by generator A is used as the input of generator B, and generator B outputs the original clothing image to be verified and enhanced. Both the original clothing image to be enhanced and the original clothing image to be verified are input into the discriminator B. The discriminator B compares the original clothing image to be verified output by the generator B with the original clothing image to be enhanced. If the discriminator B determines that the original clothing image to be verified output by the generator B is false, the discriminator B feeds back to the generator B. The generator A regenerates and outputs the virtual clothing image to be verified, the generator B regenerates the original clothing image to be verified, and the discriminator B continues to determine whether the original clothing image to be verified is true. The verification continues until the discriminator A determines that the virtual clothing image to be verified generated and output by the generator A is real, and the discriminator B determines that the original clothing image to be verified and enhanced output by the generator B is real; The deep learning network is trained sequentially using each set of training data in the dataset to obtain the trained deep learning network.

4. The data augmentation method for a training dataset of a clothing image segmentation model according to claim 2, characterized in that: Training the deep learning network also employs a loss function, the expression of which is: ； The The loss function of the generator A; The loss function of the generator B; This is a constraint on generator A and generator B.

5. The data augmentation method for a training dataset of a clothing image segmentation model according to claim 4, characterized in that: The , and The loss function expressions are as follows: ； X represents all the original clothing images, x represents a sample obtained from X, and x~X represents taking samples from X; Y represents all the virtual clothing images to be verified generated and output by generator A, y represents a sample obtained from Y, and y~Y represents taking samples from Y; E() represents the expected value, G() represents the process of generator A generating the virtual clothing images to be verified, and F() represents the process of generator B generating the original clothing images to be verified and enhanced. This describes the discrimination process of the discriminator A. This describes the discrimination process of the discriminator B.

6. The data augmentation method for a training dataset of a clothing image segmentation model according to claim 1, characterized in that: In step S400, the virtual clothing image is transformed and then matched with the new background; The real annotations are transformed to correspond to the virtual clothing image to obtain a virtual annotation image.

7. A data augmentation device for training datasets of clothing image segmentation models, characterized in that: include: The annotation unit receives the target pattern image and annotates the pattern category. Receive the original clothing image, and annotate the area of ​​the clothing in the original clothing image at the pixel level to obtain a real annotated image; The clothing extraction unit extracts the clothing portion from the original clothing image at the pixel level using the real labeled image obtained by the labeling unit, thereby obtaining the original clothing image to be enhanced. The deep learning network unit transforms the pattern in the original clothing image to be enhanced into the pattern in the target pattern image through a deep learning network, thereby obtaining a virtual clothing image. The deep learning network includes a generator for generating the virtual clothing image and a discriminator for determining whether the virtual clothing image is real; the generator includes: Generator A is used to convert the pattern of the original clothing image to be enhanced into the pattern of the target pattern image, generate and output the virtual clothing image to be verified; Generator B is used to restore the virtual clothing image to be verified output by generator A into the original clothing image to be enhanced. The discriminator includes: Discriminator A is used to determine whether the virtual clothing image to be verified output by generator A is real; Discriminator B is used to determine whether the original clothing image to be enhanced and verified output by generator B is genuine; The model structure of generator A is the same as that of generator B, and generator A and generator B are updated synchronously. The background transformation unit combines the virtual clothing image obtained by the deep learning network unit with a new background to obtain a data-enhanced clothing image.

Images