Clothing image segmentation method and device and medium

Abstract

The invention relates to a clothing image segmentation method and device, and a medium. The method comprises the following steps: obtaining a to-be-segmented clothing image; a first module is obtained by combining deformable space attention and PConv with an original C2F module in a YOLOv8 neck network, the original C2F module is replaced by the first module, a second module based on dynamic convolution and point sampling is constructed, an original up-sampling module of YOLOv8 is replaced by the second module, and an improved YOLOv8 network model is obtained; and inputting a garment image to be segmented into the improved YOLOv8 network model to obtain a segmentation result. Compared with the prior art, the method has the advantages of high segmentation speed, high accuracy and the like.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a clothing image segmentation method, device and medium. Background Art

[0002] With the rapid development of the e-commerce market, traditional industries represented by clothing have been transformed from offline to online. Online sales have become one of the important sales channels of the clothing industry, and related clothing scoring and recommendation systems have also developed rapidly. However, online clothing scoring and recommendation systems face the problem of lack of clothing item data sets. At the same time, existing clothing scoring and recommendation systems all need to divide the outfit into items before input. The effect of segmentation will greatly affect the effect of related scoring and recommendation systems.

[0003] Clothing images have characteristics such as texture, pattern, style, and pattern, which increase the difficulty of segmenting clothing images to a certain extent. Deep learning technology based on convolutional neural networks has achieved good results in image segmentation. Among many segmentation algorithms, YOLOv8 has attracted much attention for its advantages such as strong real-time performance and high precision. However, when it comes to clothing images, YOLOv8 has a long response time and poor segmentation effect when faced with a large number of clothing images, and cannot meet actual needs.

[0004] Therefore, a new clothing image segmentation method is needed to improve the image segmentation speed and segmentation effect. Summary of the invention

[0005] The purpose of the present invention is to provide a clothing image segmentation method, device and medium in order to overcome the defects of the above-mentioned prior art, such as long segmentation time and poor segmentation effect.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A clothing image segmentation method comprises the following steps:

[0008] S1: Obtain the clothing image to be segmented;

[0009] S2: Use deformable spatial attention and PConv to combine the original C2F module in the YOLOv8 neck network to obtain the first module, the first module replaces the original C2F module, and constructs the second module based on dynamic convolution and point sampling. The second module replaces the original upsampling module of YOLOv8 to obtain an improved YOLOv8 network model;

[0010] S3: Input the clothing image to be segmented into the improved YOLOv8 network model to obtain the segmentation result.

[0011] Furthermore, the first module includes three DSA modules and a connection module. The segmented features enter the first DSA module and the connection module respectively, the output of the first DSA module enters the second DSA module and the connection module respectively, the output of the second DSA module enters the first DSA module, the third DSA module and the connection module respectively, the output of the third DSA module enters the second DSA module, and the connection module splices all the inputs.

[0012] Furthermore, the PConv module replaces the 1x1 convolution layer and is combined with the DSA module. The PConv module divides the input feature map of the clothing image to be segmented into two parts, one part of the input feature map is convolved, and the other part of the input feature map remains unchanged.

[0013] Furthermore, the second module includes a first branch and a second branch, the input of the second module enters the first branch and the second branch respectively, the first branch includes an offset, and the output of the first branch and the second branch are multiplied and then point sampling is performed to obtain the output of the second module.

[0014] Furthermore, the detection head of the improved YOLOv8 network model generates specific feature maps according to the number of categories, and the specific feature maps include feature maps of predicted boxes, feature maps of predicted categories, feature maps of Mask coefficients, and feature maps of native segmentation.

[0015] Furthermore, the improved YOLOv8 network model includes a backbone network and a neck network, the backbone network includes a convolutional layer, multiple C2F modules and an SPPF module, each upsampling module of the neck network is replaced by a corresponding second module, each C2F module is replaced by a first module, and the output of each second module is merged with the output of the corresponding C2F module in the backbone network.

[0016] Furthermore, the improved YOLOv8 network model includes multiple detection heads, each detection head corresponds to a first module of a neck network, the input of the first module corresponding to the last detection head is the output post-processing result of the SPPF module, and the input of the first modules corresponding to the remaining detection heads is the result of merging the output of the corresponding second module and the output of the corresponding C2F module.

[0017] A second aspect of the present invention is a clothing image segmentation device, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements any one of the above clothing image segmentation methods when executing the program.

[0018] A third aspect of the present invention is a storage medium having a program stored thereon, which implements any of the above clothing image segmentation methods when the program is executed.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1) The present invention improves the YOLOv8 network model, reduces the model parameters and redundant information transmission through deformable spatial attention and partial convolution, adopts the second bypass dynamic convolution, performs upsampling from the perspective of point sampling, enhances the model's ability to capture key spatial information of clothing images and response speed, and improves the segmentation speed and segmentation accuracy of the model.

[0021] 2) The method of the present invention optimizes the network structure based on the YOLOv8 structure, which can effectively improve the speed and accuracy of the clothing image segmentation task, and has broad application prospects and important technical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The figure is a schematic diagram of the steps of the clothing image segmentation algorithm based on the improved YOLOv8 of the present invention.

[0023] Figure 2 It is a schematic diagram of the C2F_P_DSA structure in an example of the present invention.

[0024] Figure 3 This is a schematic diagram of the second module structure in an example of the present invention.

[0025] Figure 4 It is a schematic diagram of the structure of the improved YOLOv8 in the example of the present invention. DETAILED DESCRIPTION

[0026] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0027] Example 1

[0028] The present invention is a clothing image segmentation method. Figure 1 As shown, the following steps are included:

[0029] S1: Obtain the clothing image to be segmented;

[0030] A clothing image dataset to be segmented is formed. In this embodiment, the DeepFashion2 clothing dataset is used in the training process of the YOLOv8 model, and the dataset comes with annotations.

[0031] The backbone network of the YOLOv8 model is mainly used to extract features from input images. The feature extraction process is usually accompanied by a decrease in feature map resolution and an increase in the number of channels. The backbone network consists of a CBS module, a C2F module, and an SPPF module. C2F is a module designed for deep feature extraction. Its output feature map and output feature map remain the same size, but there are multiple branches inside it.

[0032] The detection head is designed to generate specific feature maps according to the number of categories. In addition to generating three feature maps (1, 64, 80, 80), (1, 64, 40, 40) and (1, 64, 20, 20) for predicting Boxes and three feature maps (1, nc, 80, 80), (1, nc, 40, 40) and (1, nc, 20, 20) for predicting CLs, the head layer also generates three feature maps with 32 channels each, namely mask_coefficients (1, 32, 80, 80), (1, 32, 40, 40) and (1, 32, 20, 20) for use as Mask coefficients. A Prototype Mask feature map of size (1, 32, 80, 80) is generated from the 80X80 feature map as the feature map for native segmentation.

[0033] S2: Using deformable spatial attention and PConv combined with the original C2F module in the YOLOv8 neck network to obtain the first module, the first module replaces the original C2F module, and constructs the second module based on dynamic convolution and point sampling. The second module replaces the original upsampling module of YOLOv8 to obtain an improved YOLOv8 network model;

[0034] The neck network of YOLOv8 also has a C2F module. The present invention first constructs Figure 2 The C2F_P_DSA module (first module) shown processes the segmented feature map layer by layer in multiple DSA modules to extract deeper features.

[0035] Inside the C2F_P_DSA module, the input feature map is first processed by PConv. PConv performs convolution operations on only 1 / 4 of the channels, and the remaining 3 / 4 channels remain unchanged. Through this design, unnecessary redundant information transmission is reduced while retaining most of the original feature information.

[0036] The input feature map processed by PConv is first processed by a convolution layer (usually 1x1 convolution), and the number of channels of the output feature map is doubled. The purpose of this step is to increase the feature expression ability of the model.

[0037] The input feature map is divided into two parts, one part enters the DSA module, and the other part directly participates in the subsequent splicing. In this embodiment, the C2F_P_DSA module includes three DSA modules and a connection module. The two parts of the input feature map after segmentation enter the first DSA module and the connection module respectively, and the output of the first DSA module enters the second DSA module and the connection module respectively. The output of the second DSA module enters the first DSA module, the third DSA module and the connection module respectively, and the output of the third DSA module enters the second DSA module.

[0038] The connection module concatenates all inputs to increase the diversity of features. DSA is 2.1 times faster than DCNv3, and efficiently completes the segmentation task with less computation. Finally, a convolutional layer is used to compress the number of channels of the concatenated feature map to the required number of output channels to meet the next processing requirements.

[0039] like Figure 3 As shown, the DySample module (second module) includes a first branch and a second branch. The input of the second module enters the first branch and the second branch respectively. The first branch includes an offset. The outputs of the first branch and the second branch are multiplied and then point sampling is performed to obtain the output of the second module.

[0040] Finally, the C2F of the neck in the original YOLOv8 model is replaced with the C2F_P_DSA module, and the DySample module replaces the original upsampling module of the YOLOv8 neck network. The improved YOLOv8 model is as follows Figure 4 As shown. The improved YOLOv8 network model includes a backbone network and a neck network. The backbone network includes a convolutional layer, multiple C2F modules and an SPPF module. Each upsampling module of the neck network is replaced by a corresponding second module, and each C2F module is replaced by a first module. The output of each second module is merged with the output of the corresponding C2F module in the backbone network. The improved YOLOv8 network model includes multiple detection heads, each detection head corresponds to a first module of the neck network, the input of the first module corresponding to the last detection head is the output post-processing result of the SPPF module, and the input of the first modules corresponding to the remaining detection heads is the result of merging the output of the corresponding second module with the output of the corresponding C2F module.

[0041] S3: Input the clothing image to be segmented into the improved YOLOv8 network model to obtain the segmentation result.

[0042] In the training phase, the public clothing dataset DeepFashion2 is divided into training set, validation set and test set in a ratio of 7:2:1. The test set is input into the clothing image segmentation network based on improved YOLOv8 obtained after training with the training set, and the segmentation performance of the clothing image segmentation network based on improved YOLOv8 is tested and evaluated.

[0043] Accuracy (Acc), precision (Precision, Pre) and recall (Recall, Re), average precision (Average Precision, AP) and multi-class average precision (Mean Average Precision, mAP) are used to evaluate the overall effect of the network.

[0044] Accuracy (Acc) is the ratio of correctly predicted classifications to the total sample size in all predictions. It is usually used to measure the overall accuracy. The calculation formula is as follows:

[0045]

[0046] Precision (Pre) is the ratio of all samples predicted to be positive that are actually positive based on the prediction results; and Recall (Re) is the ratio of the positive examples in the sample that are correctly predicted based on the sample. The formula is as follows:

[0047]

[0048] The recall rate and precision rate are used as the horizontal and vertical coordinates to draw the PR curve. The average precision (AP) is the average value of the accuracy at different recall rate points, which corresponds to the area under the PR curve on the PR curve. The AP calculation formula is as follows:

[0049]

[0050] The multi-class average precision (AP) reflects the accuracy of a single category. The multi-class average precision (mAP) is the average of the average precisions of all categories, reflecting the accuracy of the entire model. The mAP calculation formula is as follows:

[0051]

[0052] In the formula, TP represents the instance correctly predicted by the model as the target (correctly detected target), FP represents the instance incorrectly predicted by the model as the target (incorrectly detected non-target), FN represents the target not detected by the model (positive samples that actually exist but are not detected), K represents the number of categories, and i represents one of the specific categories.

[0053] The method of the present invention adds a DSA module to the C2F structure of the neck network to limit the deformation sampling to a single axis, further avoiding the quadratic increase of the computational load with the kernel size, improving the clothing image segmentation effect while reducing the amount of calculation.

[0054] The method of the present invention uses PConv to replace 1×1 convolution in the C2F module of the improved neck network, and enhances the ability of the model to process complex inputs through an adaptive convolution method, and enhances the accuracy of model feature extraction and the overall recognition speed; the method of the present invention replaces the original upsampling module of the YOLOv8 neck network with a DySample module, and DySample bypasses the dynamic convolution and formulates upsampling from the perspective of point sampling, which saves resources and expands the application scenarios; the method of the present invention optimizes the network structure based on the YOLOv8 structure, can effectively improve the speed and accuracy of performing segmentation tasks on clothing images, and has broad application prospects and important technical significance.

[0055] Example 2

[0056] On the basis of Example 1, the program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0057] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0058] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A clothing image segmentation method, characterized in that: The following steps are involved: S1: Obtain the clothing image to be segmented; S2: Using deformable spatial attention and PConv combined with the original C2F module in the YOLOv8 neck network to obtain the first module, the first module replaces the original C2F module, and constructs the second module based on dynamic convolution and point sampling. The second module replaces the original upsampling module of YOLOv8 to obtain an improved YOLOv8 network model; S3: Input the clothing image to be segmented into the improved YOLOv8 network model to obtain the segmentation result.

2. A clothing image segmentation method according to claim 1, characterized in that: The first module includes three DSA modules and a connection module. The segmented features enter the first DSA module and the connection module respectively. The output of the first DSA module enters the second DSA module and the connection module respectively. The output of the second DSA module enters the first DSA module, the third DSA module and the connection module respectively. The output of the third DSA module enters the second DSA module. The connection module splices all the inputs.

3. A clothing image segmentation method according to claim 2, characterized in that: The PConv module replaces the 1x1 convolution layer and is combined with the DSA module. The PConv module divides the input feature map of the clothing image to be segmented into two parts, one part of the input feature map is convolved, and the other part of the input feature map remains unchanged.

4. A clothing image segmentation method according to claim 1, characterized in that: The second module includes a first branch and a second branch. The input of the second module enters the first branch and the second branch respectively. The first branch includes an offset. The output of the first branch and the second branch is multiplied and then point sampling is performed to obtain the output of the second module.

5. A clothing image segmentation method according to claim 1, characterized in that: The detection head of the improved YOLOv8 network model generates specific feature maps according to the number of categories, and the specific feature maps include feature maps of prediction boxes, feature maps of prediction categories, feature maps of Mask coefficients, and feature maps of native segmentation.

6. A clothing image segmentation method according to claim 1, characterized in that: The improved YOLOv8 network model includes a backbone network and a neck network, the backbone network includes a convolutional layer, multiple C2F modules and an SPPF module, each upsampling module of the neck network is replaced by a corresponding second module, each C2F module is replaced by a first module, and the output of each second module is merged with the output of the corresponding C2F module in the backbone network.

7. A clothing image segmentation method according to claim 6, characterized in that: The improved YOLOv8 network model includes multiple detection heads, each detection head corresponds to a first module of a neck network, the input of the first module corresponding to the last detection head is the output post-processing result of the SPPF module, and the input of the first modules corresponding to the remaining detection heads is the result of merging the output of the corresponding second module and the output of the corresponding C2F module.

8. A clothing image segmentation method according to claim 1, characterized in that: Before obtaining the clothing image to be segmented, the method further includes: Collecting a data set, the data set includes initial clothing images; The initial clothing image is normalized and preprocessed to obtain the clothing image to be segmented.

9. A clothing image segmentation device, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, a clothing image segmentation method as described in any one of claims 1-8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, a clothing image segmentation method as described in any one of claims 1-8 is implemented.

Images