Laundry image processing method, storage medium, and electronic device

Abstract

The application discloses a clothes image processing method, a storage medium and an electronic device, relates to the technical field of image processing, and the clothes image processing method comprises the steps of acquiring an image containing clothes; performing feature extraction on the image containing clothes by using a feature extraction module in a segmentation model to obtain a feature map, wherein model parameters of the segmentation model are obtained by training sample segmentation results of training samples and superpixel value prediction results of the training samples, and the superpixel value prediction results are used for representing color values and predicted color values obtained based on the training samples; and performing segmentation on the feature map by using a segmentation prediction head module in the segmentation model to obtain a segmentation result. The application solves the technical problem of low efficiency when the clothes image is segmented.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a clothing image processing method, a storage medium and an electronic device. BACKGROUND

[0002] Clothing image segmentation plays an important role in the fashion field. The segmented clothing image can provide a good carrier for subsequent applications such as clothing fashion understanding and clothing management. In the existing clothing segmentation method, a segmentation model with multiple branches is usually used to integrate the segmentation results of clothing images, thereby introducing more parameters, resulting in slow speed and low efficiency when segmenting clothing images.

[0003] At present, there is no effective solution to the above technical problems. SUMMARY

[0004] At least one embodiment of the present application provides a clothing image processing method, a storage medium and an electronic device to at least solve the technical problem of low efficiency when segmenting clothing images.

[0005] According to one embodiment of the present application, a clothing image processing method is provided, comprising: obtaining an image containing clothing; using a feature extraction module in a segmentation model to extract features of the image containing clothing to obtain a feature map, wherein model parameters of the segmentation model are obtained by training sample segmentation results of training samples and superpixel value prediction results of the training samples, and the superpixel value prediction results are used to represent color values and predicted color values based on the training samples; using a segmentation prediction head module in the segmentation model to segment the feature map to obtain a segmentation result.

[0006] Optionally, the feature extraction module includes a backbone network and a feature pyramid network, and the feature extraction module is used to extract features of the image containing clothing to obtain the feature map, including: using the backbone network to process the image containing clothing to obtain a plurality of initial feature maps of different sizes; and using the feature pyramid network to fuse the plurality of initial feature maps of different sizes to obtain the feature map.

[0007] Optionally, the segmentation prediction head module includes a mask network and a label network, and the segmentation prediction head module in the segmentation model is used to predict the feature map to obtain the segmentation result, including: using the mask network to identify a target object in the feature map to obtain a predicted clothing segmentation mask; using the label network to predict a label of the target object in the feature map to obtain a predicted clothing category; and generating the segmentation result based on the predicted clothing segmentation mask and the predicted clothing category, wherein the predicted clothing segmentation mask and the predicted clothing category correspond one-to-one.

[0008] Optionally, the method further comprises: segmenting the training sample by using the initial segmentation model to obtain a sample segmentation result; predicting the superpixel value of the training sample by using the superpixel value prediction model to obtain a superpixel value prediction result; generating a target loss function based on the sample segmentation result and the superpixel value prediction result; and adjusting the parameters of the initial segmentation model based on the target loss function to obtain the segmentation model.

[0009] Optionally, the superpixel value prediction model comprises a superpixel segmentation module and a superpixel value prediction module, and the superpixel value prediction model is used to predict the superpixel value of the training sample to obtain a superpixel value prediction result, which comprises: segmenting the training sample based on the distance and color value of the training sample by using the superpixel segmentation module to obtain the color value of the superpixel block, wherein the distance is used to represent the distance between the pixel point in the training sample and the center point corresponding to the pixel point, and the center point is used to represent the clustering center of the plurality of pixel points in the training sample; and performing convolution on the first feature map by using the superpixel value prediction module to obtain the predicted color value of the first feature map; and confirming the color value of the superpixel block and the predicted color value as the superpixel value prediction result.

[0010] Optionally, the target loss function is generated based on the sample segmentation result and the superpixel value prediction result, which comprises: generating a first loss function based on the sample segmentation result, the real clothing category and the real clothing segmentation mask; generating a second loss function based on the superpixel value prediction result and the real color value; and generating the target loss function based on the first loss function and the second loss function.

[0011] Optionally, the parameters of the initial segmentation model are adjusted based on the target loss function to obtain the segmentation model, which comprises: adjusting the parameters of the initial segmentation model by using the gradient descent method in the case that the function value of the target loss function is greater than a preset threshold; and confirming the current segmentation model as the segmentation model in the case that the function value of the target loss function is equal to or less than the preset threshold.

[0012] Optionally, the sample segmentation result comprises a first predicted clothing category and a first clothing segmentation mask, and the first loss function is generated based on the sample segmentation result, the real clothing category and the real clothing segmentation mask, which comprises: generating a classification loss function based on the first predicted clothing category and the real clothing category; generating a mask loss function based on the first clothing segmentation mask and the real clothing segmentation mask; and generating the first loss function based on the classification loss function and the mask loss function.

[0013] Optionally, the sample segmentation result comprises a first predicted clothing category and a first clothing segmentation mask, and the first loss function is generated based on the sample segmentation result, the real clothing category and the real clothing segmentation mask, which comprises: generating a classification loss function based on the first predicted clothing category and the real clothing category; generating a mask loss function based on the first clothing segmentation mask and the real clothing segmentation mask; and generating the first loss function based on the classification loss function and the mask loss function.

[0014] According to an embodiment of the present application, a clothes image processing device is provided, comprising: an image containing clothes acquisition module, configured to acquire an image containing clothes; an extraction module, configured to extract features of the image containing clothes by using a feature extraction module in a segmentation model, to obtain a feature map, wherein model parameters of the segmentation model are obtained by training sample segmentation results of training samples and superpixel value prediction results of the training samples, and the superpixel value prediction results are used to represent color values and predicted color values obtained based on the training samples; and a prediction module, configured to segment the feature map by using a segmentation prediction head module in the segmentation model, to obtain a segmentation result.

[0015] Optionally, the feature extraction module comprises a backbone network and a feature pyramid network, and the extraction module comprises: a processing unit, configured to process the image containing clothes by using the backbone network, to obtain a plurality of initial feature maps of different sizes; and a fusion unit, configured to fuse the plurality of initial feature maps of different sizes by using the feature pyramid network, to obtain the feature map.

[0016] Optionally, the segmentation prediction head module comprises a mask network and a label network, and the prediction module comprises: an identification unit, configured to identify a target object in the feature map by using the mask network, to obtain a predicted clothes segmentation mask; a prediction unit, configured to predict a label of the target object in the feature map by using the label network, to obtain a predicted clothes category; and a segmentation result generation unit, configured to generate the segmentation result based on the predicted clothes segmentation mask and the predicted clothes category, wherein the predicted clothes segmentation mask and the predicted clothes category correspond to each other.

[0017] Optionally, the device further comprises: a segmentation module, configured to segment the training samples by using an initial segmentation model, to obtain sample segmentation results; a superpixel value prediction module, configured to predict superpixel values of the training samples by using a superpixel value prediction model, to obtain superpixel value prediction results; a loss function generation module, configured to generate a target loss function based on the sample segmentation results and the superpixel value prediction results; and an adjustment module, configured to adjust parameters of the initial segmentation model based on the target loss function, to obtain the segmentation model.

[0018] Optionally, the superpixel value prediction model comprises a superpixel segmentation module and a superpixel value prediction module, and the superpixel value prediction module comprises: a training unit, configured to segment the training samples based on distances and color values of the training samples by using the superpixel segmentation module, to obtain color values of superpixel blocks, wherein the distances are used to represent distances between pixel points in the training samples and center points corresponding to the pixel points, and the center points are used to represent clustering centers of the pixel points in the training samples; a convolution unit, configured to convolve the first feature map by using the superpixel value prediction module, to obtain predicted color values of the first feature map; and a result confirmation unit, configured to confirm the color values of the superpixel blocks and the predicted color values as the superpixel value prediction results.

[0019] Optionally, the loss function generation module comprises: a first loss function unit configured to generate a first loss function based on the sample segmentation result, the real clothing category, and the real clothing segmentation mask; a second loss function unit configured to generate a second loss function based on the superpixel value prediction result and the real color value; and a target loss function unit configured to generate a target loss function based on the first loss function and the second loss function.

[0020] Optionally, the adjustment module comprises: an adjustment unit configured to adjust the parameters of the initial segmentation model by using a gradient descent method in a case where the function value of the target loss function is greater than a preset threshold; and a model confirmation unit configured to confirm the current segmentation model as the segmentation model in a case where the function value of the target loss function is equal to or less than the preset threshold.

[0021] Optionally, the sample segmentation result comprises a first predicted clothing category and a first clothing segmentation mask, the first loss function unit is further configured to generate a classification loss function based on the first predicted clothing category and the real clothing category, generate a mask loss function based on the first clothing segmentation mask and the real clothing segmentation mask, and generate the first loss function based on the classification loss function and the mask loss function.

[0022] According to an embodiment of the present application, a computer readable storage medium is also provided, which comprises a stored program, wherein the program, when executed, performs any of the processing methods of the clothing image.

[0023] According to an embodiment of the present application, an electronic device is also provided, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform any of the processing methods of the clothing image.

[0024] In at least some embodiments of the present application, after obtaining an image containing clothing, a feature extraction module in the segmentation model is used to extract features of the image containing clothing to obtain a feature map, and then a segmentation prediction head module in the segmentation model is used to segment the feature map to obtain a segmentation result. It is easy to note that in the disclosure, in the process of image segmentation, the image containing clothing is not processed by using a multi-branch segmentation model, but after the feature map is extracted, the feature map is directly segmented by using the segmentation prediction head module to obtain the segmentation result, thereby achieving the technical effect of improving the efficiency of image segmentation and solving the technical problem of low efficiency in the prior art when segmenting clothing images. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, the other drawings can also be obtained based on these drawings without any creative work.

[0027] Figure 1 Fig. 1 is a hardware environment schematic diagram of an interaction method of an intelligent device according to an embodiment of the present application;

[0028] Figure 2 Fig. 2 is a flowchart of a clothes image processing method according to an embodiment of the present application;

[0029] Figure 3 Fig. 3 is a partial network structure schematic diagram of a segmentation model in a clothes image processing method according to an embodiment of the present application;

[0030] Figure 4 Fig. 4 is a structure schematic diagram of a segmentation model in a clothes image processing method according to an embodiment of the present application;

[0031] Figure 5 Fig. 5 is a flow schematic diagram of a clothes image processing method according to an embodiment of the present application;

[0032] Figure 6A Fig. 6 is a schematic diagram of a training sample in a clothes image processing method according to an optional embodiment of the present application;

[0033] Figure 6B Fig. 7 is a schematic diagram of a superpixel segmentation boundary in a clothes image processing method according to an optional embodiment of the present application;

[0034] Figure 6C Fig. 8 is a schematic diagram of Y* in a clothes image processing method according to an embodiment of the present application;

[0035] Figure 7 Fig. 9 is a mask sample distribution schematic diagram of a coefficient loss function in a clothes image processing method according to an optional embodiment of the present application;

[0036] Figure 8 Fig. 10 is a structure schematic diagram of a clothes image processing apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] According to one aspect of the embodiments of this application, a method for processing clothing images is provided. This method is widely applicable to clothing retrieval, recommendation, virtual try-on, clothing style recognition, and other application scenarios. Optionally, in this embodiment, the above-described clothing image processing method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0040] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to a PC, mobile phone, or tablet computer.

[0041] This embodiment provides a method for processing clothing images, applied to the aforementioned computer terminal. Figure 2is a flowchart of a clothes image processing method according to an embodiment of the present application, and the flow includes the following steps:

[0042] In step S202, an image containing clothes is acquired.

[0043] Specifically, the image containing clothes can be any image containing clothes, for example, an image of a mannequin wearing clothes, an image of a person wearing clothes, an image of a pet wearing clothes, or an image containing only clothes, and the image containing clothes can be an image or a video screenshot.

[0044] In step S204, a feature extraction module in the segmentation model is used to perform feature extraction on the image containing clothes to obtain a feature map, wherein model parameters of the segmentation model are trained by sample segmentation results of training samples and superpixel value prediction results of the training samples, and the superpixel value prediction results are used to represent color values and predicted color values obtained based on the training samples.

[0045] Specifically, the segmentation model can be a K-Net-based segmentation model, and the K-net is a segmentation prediction head based on deep learning, which mainly functions to perform image segmentation by segmenting an input image. The training sample can be an image containing clothes containing clothes. It should be noted that the superpixel value prediction result is only generated in the training process of the segmentation model, and is not generated in the application process. In the training stage, the superpixel value prediction result is introduced to establish the same association on local pixels, thereby reducing the holes of the clothes and improving the accuracy of the clothes segmentation boundary and the quality of the graphics. In the application stage, the superpixel prediction result is not introduced to ensure the processing efficiency of the model.

[0046] It should be noted that the color values mentioned in the present application are all RGB (Red-Green-Blue) values.

[0047] As an optional implementation manner, Figure 3 is a partial network structure diagram of a segmentation model in a clothes image processing method according to an embodiment of the present application, as shown in Figure 3 When the segmentation model performs feature extraction on the image containing clothes, it can extract features from high to low layer by layer, then combine the low-level features and the high-level features, and restore the feature map with an input size of one fourth to the original image size through the Conv-4x mode to obtain the feature map.

[0048] As an optional implementation manner, Figure 4 is a structure diagram of a segmentation model in a clothes image processing method according to an embodiment of the present application, as shown in Figure 4As shown, the input image x corresponds to the above-mentioned image containing clothes, and in the training stage, the segmentation model includes a Backbone model (main network) and an FPN (Feature Pyramid Networks), and after the input image x is input into the "Backbone model + FPN", a feature map of the input image x can be obtained, and then the feature map is input into a segmentation prediction head to obtain a segmentation result, wherein the predicted clothes segmentation mask can be used to generate a Mask (mask) loss function, and the predicted clothes category is used to generate a classification loss function. Secondly, the input image can also be input into a superpixel segmentation module to obtain a sample segmentation result, and the feature map can also be input into a superpixel segmentation prediction module to obtain a superpixel value prediction result. In the prediction stage, the model includes the "Backbone model + FPN" and the segmentation prediction head, and the superpixel segmentation module and the superpixel prediction module are only used in the model training stage. It should be noted that the introduction of the superpixel segmentation module and the superpixel prediction module in the training stage helps to retain the boundary information in image segmentation and establish the association between pixels, but since the processing speed of the superpixel segmentation module is relatively slow, the superpixel results corresponding to all training set data (including the input image x) can be generated offline in advance before training the segmentation model.

[0049] In step S206, the feature map is segmented by using the segmentation prediction head module in the segmentation model to obtain a segmentation result.

[0050] Specifically, the segmentation head prediction model can be composed of a dynamic Kernel (core) and a plurality of serially connected Kernels, and the output of the segmentation prediction head module includes the above-mentioned predicted clothes segmentation mask and the predicted clothes category.

[0051] The above-mentioned segmentation result can include the predicted clothes segmentation mask and the predicted clothes category, and the clothes segmentation mask and the clothes segmentation category correspond one-to-one. As an optional implementation manner, the size of the predicted clothes segmentation mask (Mask) in the output of the segmentation prediction head module is N*W*H, and the size of the predicted clothes category can be N*C.

[0052] As an optional implementation manner, Figure 5 is a flowchart of a clothes image processing method in an embodiment of the present application, as shown in Figure 5 After obtaining the image containing clothes, the image containing clothes is input into a segmentation model for feature extraction, and finally a segmentation result is obtained and output, as shown in Figure 5 The segmentation result includes a predicted clothes segmentation mask and a predicted clothes category, and in the case that the predicted clothes mask is an "image of a shirt", the "image of a shirt" corresponds to a predicted clothes category of "shirt", and in the case that the predicted clothes mask is an "image of trousers", the corresponding predicted clothes category is "trousers".

[0053] By the above method in the embodiments of the present application, after obtaining the image containing clothes, the feature extraction module in the segmentation model is used to extract features from the image containing clothes to obtain a feature map, and then the segmentation prediction head module in the segmentation model is used to segment the feature map to obtain a segmentation result. It is easy to note that in the disclosure, the image containing clothes is not processed by using a multi-branch segmentation model during image segmentation, but after the feature map is extracted, the segmentation prediction head module is directly used to segment the feature map to obtain the segmentation result, thereby achieving the technical effect of improving the efficiency of image segmentation and solving the technical problem of low efficiency in segmenting the image of clothes in the prior art.

[0054] Optionally, the feature extraction module includes a backbone network and a feature pyramid network, and the feature extraction module is used to extract features from the image containing clothes to obtain a feature map, including: using the backbone network to process the image containing clothes to obtain a plurality of initial feature maps of different sizes; and using the feature pyramid network to fuse the plurality of initial feature maps of different sizes to obtain the feature map.

[0055] Specifically, the above backbone network can be a Backbone model. During processing of the image containing clothes, the backbone network can extract features layer by layer from high to low to obtain initial feature maps of different sizes, i.e., high-level features and low-level features, and then the feature pyramid network fuses the feature maps of different sizes to obtain a first feature map, and then the feature map is processed by conv-4x to restore the size of the feature map of one-fourth to the size of the original image, where the size of the original image refers to the size of the image containing clothes.

[0056] Optionally, the segmentation prediction head module includes a mask network and a label network, and the segmentation prediction head module in the segmentation model is used to predict the feature map to obtain a segmentation result, including: using the mask network to identify a target object in the feature map to obtain a predicted clothing segmentation mask; using the label network to predict a label of the target object in the feature map to obtain a predicted clothing category; and generating the segmentation result based on the predicted clothing segmentation mask and the predicted clothing category, wherein the predicted clothing segmentation mask and the predicted clothing category correspond one-to-one.

[0057] Specifically, the segmentation prediction head can be a K-net segmentation prediction head, which updates the final output of the prediction head by using a dynamic kernel and a plurality of kernels connected in series to obtain the final segmentation result. The above target object can be a piece of clothing in the feature map. It should be noted that the mask network and the label network start processing the feature map at the same time.

[0058] Optionally, the method further comprises: segmenting the training sample by using the initial segmentation model to obtain a sample segmentation result; predicting the superpixel value of the training sample by using the superpixel value prediction model to obtain a superpixel value prediction result; generating a target loss function based on the sample segmentation result and the superpixel value prediction result; and adjusting the parameters of the initial segmentation model based on the target loss function to obtain the segmentation model.

[0059] As an optional implementation, as shown in Figure 4 The superpixel value prediction model is used to represent a model of a remaining training stage that does not include a segmentation prediction head, and the superpixel value prediction result can be an output of the superpixel prediction model. The target loss function includes a Mask loss function, a classification loss function, and a color regression error loss function.

[0060] Optionally, the superpixel value prediction model includes a superpixel segmentation module and a superpixel value prediction module, and the superpixel value prediction model is used to predict the superpixel value of the training sample to obtain the superpixel value prediction result, including: segmenting the training sample based on the distance and color value of the training sample by using the superpixel segmentation module to obtain the color value of the superpixel block, wherein the distance is used to represent the distance between the pixel point in the training sample and the center point corresponding to the pixel point, and the center point is used to represent the clustering center of the plurality of pixel points in the training sample; performing convolution on the first feature map by using the superpixel value prediction module to obtain the predicted color value of the first feature map; and confirming the color value and the predicted color value of the superpixel block as the superpixel value prediction result.

[0061] Specifically, the first feature map can be a feature map obtained by processing the training sample via a backbone network and a feature pyramid network in the superpixel prediction model.

[0062] As an optional implementation, the superpixel segmentation module uses the distance and the color value as features, and segments the pixels with similar color values and distances into the same class, which helps to retain the boundary information in image segmentation and establish the association between pixels. In actual implementation, the Quickshift algorithm is used to generate the superpixel block, and after the superpixel block is generated, the average color value of the region is taken as the color value of the region (i.e., the superpixel block), and a new result image with the same size as the original image is generated, i.e., Y*, for example, Figure 6A is a schematic diagram of a training sample in a processing method of a clothing image in an optional embodiment of the present application, Figure 6B is a schematic diagram of a superpixel segmentation boundary in a processing method of a clothing image in an optional embodiment of the present application, Figure 6C is a schematic diagram of Y* in a processing method of a clothing image in an embodiment of the present application, as shown in Figure 6B Figure 6B The lines in represent the superpixel segmentation boundary.​

[0063] In addition, the Quickshift algorithm has a kernel size of 7x7, and the distance feature and the color value feature have the same weight.

[0064] As an optional implementation, the superpixel value prediction module Xfuse feature (i.e., the first feature map described above) is taken as input, and the specific color value is taken as output. The specific structure is that Xfuse is input into two Conv1x1 layers, and the purpose is to further fuse the channel features. The first convolution outputs 128 channels, followed by a SiLU activation function to avoid excessive loss of negative direction information. The second convolution outputs 3 channels, and the image min(Relu, 1) function clips the output value range to (0, 1) to facilitate mapping the value to the value range space of (0, 255).

[0065] Optionally, a target loss function is generated based on the sample segmentation result and the superpixel value prediction result, including: generating a first loss function based on the sample segmentation result, the real clothing category, and the real clothing segmentation mask; generating a second loss function based on the superpixel value prediction result and the real color value; and generating the target loss function based on the first loss function and the second loss function.

[0066] Specifically, the first loss function described above includes a mask loss function and a classification function. The classification function can be generated according to the first predicted clothing category in the sample segmentation result and the real clothing category. The second loss function can be a color value error loss function.

[0067] As an optional implementation, the target loss function = the first loss function + 0.1*the second loss function, which adjusts the features in a small inhibition manner to avoid the technical problem that the function value of the target loss function is too large, causing the segmentation model to be unable to fit.

[0068] As an optional implementation, the second loss function can use a pixel-by-pixel MSE loss function, and its formula is as follows: wherein Y* represents the real color value, Y represents the superpixel value prediction result, i.e., the predicted color value, the dimension is (h, w, 3), and the value range is between 0 and 1, including 0 and 1.

[0069] Optionally, the parameters of the initial segmentation model are adjusted based on the target loss function to obtain the segmentation model, including: in the case that the function value of the target loss function is greater than a preset threshold, adjusting the parameters of the initial segmentation model using the gradient descent method; and in the case that the function value of the target loss function is equal to or less than the preset threshold, confirming the current segmentation model as the segmentation model.

[0070] Specifically, the preset threshold can be a parameter threshold set by the R&D personnel according to the robustness requirement of the segmentation model. During the training process of the initial segmentation model, if the function value of the target loss function is greater than the preset threshold, the gradient descent algorithm can be used to adjust the parameters, so that the function value of the target loss function is constantly close to the preset threshold, until it is less than or equal to the preset threshold, the training effect of the model reaches the expectation of the R&D personnel, and the current segmentation model is determined as the segmentation model.

[0071] Optionally, the sample segmentation result includes a first predicted clothing category and a first clothing segmentation mask, and the generating of the first loss function based on the sample segmentation result, the real clothing category and the real clothing segmentation mask includes: generating a classification loss function based on the first predicted clothing category and the real clothing category; generating a mask loss function based on the first clothing segmentation mask and the real clothing segmentation mask; and generating the first loss function based on the classification loss function and the mask loss function.

[0072] As an optional implementation, the mask loss function can be composed of a Dice coefficient loss (DiceLoss) and a cross-entropy loss. The DiceLoss is derived from a coefficient loss, which is used to evaluate the similarity between the predicted mask and the real labeled mask, and the value is between 0 and 1. The greater the value, the more similar the masks are, and the mask optimization is applied to solve the problem of imbalance between positive and negative samples in the mask. The value 0 in the true value represents the background, and the value 1 represents the foreground. Figure 7 is a mask sample distribution diagram of the coefficient loss in a clothing image processing method according to an optional embodiment of the present application, as shown in Figure 7 Due to the different sizes of the target, the area of the background region can be much larger than that of the foreground region.

[0073] The specific formula of the DiceLoss is as follows:

[0074] Wherein, ∈ represents a very small number, which is used to prevent the denominator from being predicted as 0, and I is the predicted value of the correctly predicted mask region, which can be calculated by using any one of the following formulas: Since the calculation of the DiceLoss is only related to the labeled mask and the predicted mask, the problem of too large difference between the foreground and the background caused by the small target will not have a great impact on the calculation of the DiceLoss.

[0075] The above cross-entropy loss function is mainly used to optimize the identification of the foreground and the background of the model, and the calculation formula is as follows:

[0076]

[0077] As an optional implementation, the classification loss function uses a Focal loss, which aims to solve the learning problem of difficult targets in the data set, and by applying a larger weight to the difficult targets and a smaller weight to the simple targets, the learning ability of the difficult targets is increased, and the Focal loss is also beneficial to solving the problem of data imbalance. The calculation formula of the classification loss function is as follows:

[0078]

[0079] It should be noted that in actual application, a t may be 0.25, and Y may be 2.

[0080] Figure 8 is a structural schematic diagram of a clothes image processing device according to an embodiment of the present application, which comprises:

[0081] The segmentation image acquisition module 82 is configured to acquire an image containing clothes.

[0082] The extraction module 84 is configured to perform feature extraction on the image containing clothes by using a feature extraction module in the segmentation model, to obtain a feature map, wherein model parameters of the segmentation model are obtained by training sample segmentation results of training samples and superpixel value prediction results of the training samples, and the superpixel value prediction results are used to represent color values and predicted color values obtained based on the training samples.

[0083] The prediction module 86 is configured to perform segmentation on the feature map by using a segmentation prediction head module in the segmentation model, to obtain a segmentation result.

[0084] Optionally, the feature extraction module comprises a backbone network and a feature pyramid network, and the extraction module comprises: a processing unit configured to process the image containing clothes by using the backbone network, to obtain a plurality of initial feature maps of different sizes; and a fusion unit configured to fuse the plurality of initial feature maps of different sizes by using the feature pyramid network, to obtain the feature map.

[0085] Optionally, the segmentation prediction head module comprises a mask network and a label network, and the prediction module comprises: an identification unit configured to identify a target object in the feature map by using the mask network, to obtain a predicted clothes segmentation mask; a prediction unit configured to predict a label of the target object in the feature map by using the label network, to obtain a predicted clothes category; and a segmentation result generation unit configured to generate the segmentation result based on the predicted clothes segmentation mask and the predicted clothes category, wherein the predicted clothes segmentation mask and the predicted clothes category correspond to each other.

[0086] Optionally, the apparatus further comprises a segmentation module configured to segment the training sample using an initial segmentation model to obtain a sample segmentation result; a superpixel value prediction module configured to predict a superpixel value of the training sample using a superpixel value prediction model to obtain a superpixel value prediction result; a loss function generation module configured to generate a target loss function based on the sample segmentation result and the superpixel value prediction result; and an adjustment module configured to adjust parameters of the initial segmentation model based on the target loss function to obtain the segmentation model.

[0087] Optionally, the superpixel value prediction model comprises a superpixel segmentation module and a superpixel value prediction module, and the superpixel value prediction module comprises: a training unit configured to segment the training sample based on distances and color values of the training sample using the superpixel segmentation module to obtain color values of superpixel blocks, wherein the distances are used to represent distances between pixel points in the training sample and center points corresponding to the pixel points, and the center points are used to represent clustering centers of the pixel points in the training sample; a convolution unit configured to convolve the first feature map using the superpixel value prediction module to obtain predicted color values of the first feature map; and a result confirmation unit configured to confirm the color values of the superpixel blocks and the predicted color values as the superpixel value prediction result.

[0088] Optionally, the loss function generation module comprises: a first loss function unit configured to generate a first loss function based on the sample segmentation result, a real clothing category and a real clothing segmentation mask; a second loss function unit configured to generate a second loss function based on the superpixel value prediction result and a real color value; and a target loss function unit configured to generate the target loss function based on the first loss function and the second loss function.

[0089] Optionally, the adjustment module comprises: an adjustment unit configured to adjust the parameters of the initial segmentation model using a gradient descent method in a case where a function value of the target loss function is greater than a preset threshold; and a model confirmation unit configured to confirm the current segmentation model as the segmentation model in a case where the function value of the target loss function is equal to or less than the preset threshold.

[0090] Optionally, the sample segmentation result comprises a first predicted clothing category and a first clothing segmentation mask, and the loss function generation module is further configured to generate a classification loss function based on the first predicted clothing category and the real clothing category; generate a mask loss function based on the first clothing segmentation mask and the real clothing segmentation mask; and generate the first loss function based on the classification loss function and the mask loss function.

[0091] According to one of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is configured to execute the processing method of the clothing image in any of the above embodiments when running.

[0092] Optionally, in the embodiment, the storage medium can be configured to store program code for performing the following steps:

[0093] S1, obtaining an image containing clothes;

[0094] S2, performing feature extraction on the image containing clothes by using a feature extraction module in the segmentation model to obtain a feature map, wherein model parameters of the segmentation model are trained by a sample segmentation result of a training sample and a superpixel value prediction result of the training sample, and the superpixel value prediction result is used to represent color values and predicted color values obtained based on the training sample;

[0095] S3, performing segmentation on the feature map by using a segmentation prediction head module in the segmentation model to obtain a segmentation result.

[0096] Optionally, in the embodiment, the storage medium can be configured to store program code for performing the following steps: the feature extraction module includes a backbone network and a feature map pyramid network, and the feature extraction module is used to perform feature extraction on the image containing clothes to obtain the feature map, including: the backbone network is used to process the image containing clothes to obtain a plurality of initial feature maps of different sizes; and the feature map pyramid network is used to fuse the plurality of initial feature maps of different sizes to obtain the feature map.

[0097] Optionally, in the embodiment, the storage medium can be configured to store program code for performing the following steps: the segmentation prediction head module includes a mask network and a label network, and the segmentation prediction head module in the segmentation model is used to predict the feature map to obtain the segmentation result, including: the mask network is used to identify a target object in the feature map to obtain a predicted clothes segmentation mask; the label network is used to predict a label of the target object in the feature map to obtain a predicted clothes category; and the segmentation result is generated based on the predicted clothes segmentation mask and the predicted clothes category, wherein the predicted clothes segmentation mask and the predicted clothes category correspond to each other one by one.

[0098] Optionally, in the embodiment, the storage medium can be configured to store program code for performing the following steps: the method further includes: performing segmentation on the training sample by using an initial segmentation model to obtain a sample segmentation result; performing superpixel value prediction on the training sample by using a superpixel value prediction model to obtain a superpixel value prediction result; generating a target loss function based on the sample segmentation result and the superpixel value prediction result; and adjusting parameters of the initial segmentation model based on the target loss function to obtain the segmentation model.

[0099] Optionally, in the embodiment, the storage medium can be configured to store program code for performing the following steps: the superpixel value prediction model comprises a superpixel segmentation module and a superpixel value prediction module, and the superpixel value prediction model is used to perform superpixel value prediction on the training sample to obtain a superpixel value prediction result, including: the superpixel segmentation module is used to segment the training sample based on distance and color value of the training sample to obtain color value of the superpixel block, wherein the distance is used to represent distance between a pixel point in the training sample and a center point corresponding to the pixel point, and the center point is used to represent a clustering center of a plurality of pixel points in the training sample; the superpixel value prediction module is used to perform convolution on the first feature map to obtain predicted color value of the first feature map; and the color value of the superpixel block and the predicted color value are confirmed as the superpixel value prediction result.

[0100] Optionally, in the embodiment, the storage medium can be configured to store program code for performing the following steps: the target loss function is generated based on the sample segmentation result and the superpixel value prediction result, including: the first loss function is generated based on the sample segmentation result, the real clothing category and the real clothing segmentation mask; the second loss function is generated based on the superpixel value prediction result and the real color value; and the target loss function is generated based on the first loss function and the second loss function.

[0101] Optionally, the parameters of the initial segmentation model are adjusted based on the target loss function to obtain the segmentation model, including: in the case that the function value of the target loss function is greater than a preset threshold, the parameters of the initial segmentation model are adjusted by using the gradient descent method; and in the case that the function value of the target loss function is equal to or less than the preset threshold, the current segmentation model is confirmed as the segmentation model.

[0102] Optionally, in the embodiment, the storage medium can be configured to store program code for performing the following steps: the sample segmentation result comprises a first predicted clothing category and a first clothing segmentation mask, and the first loss function is generated based on the sample segmentation result, the real clothing category and the real clothing segmentation mask, including: a classification loss function is generated based on the first predicted clothing category and the real clothing category; a mask loss function is generated based on the first clothing segmentation mask and the real clothing segmentation mask; and the first loss function is generated based on the classification loss function and the mask loss function.

[0103] According to one of the embodiments of the present application, an electronic device is also provided, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the processing method of the clothing image in any of the above embodiments.

[0104] Optionally, the electronic device can further comprise a transmission device and an input / output device, wherein the transmission device is connected with the processor, and the input / output device is connected with the processor.

[0105] Optionally, in the embodiment, the processor can be configured to execute the following steps by a computer program:

[0106] S1, acquiring an image containing clothes;

[0107] S2, performing feature extraction on the image containing clothes by using a feature extraction module in the segmentation model, to obtain a feature map, wherein model parameters of the segmentation model are trained by sample segmentation results of training samples and superpixel value prediction results of the training samples, and the superpixel value prediction results are used to represent color values and predicted color values obtained based on the training samples;

[0108] S3, performing segmentation on the feature map by using a segmentation prediction head module in the segmentation model, to obtain a segmentation result.

[0109] The above merely describes the preferred embodiments of the present application, and it should be noted that those of ordinary skill in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the scope of the present application.

Claims

1. A method of processing a clothing image, characterized by, The method comprises: obtaining an image containing clothes; performing feature extraction on the image containing clothes by using a feature extraction module in a segmentation model to obtain a feature map, wherein model parameters of the segmentation model are obtained by training sample segmentation results of training samples and superpixel value prediction results of the training samples, and the superpixel value prediction results are used to represent color values and predicted color values obtained based on the training samples; performing segmentation on the feature map by using a segmentation prediction head module in the segmentation model to obtain the segmentation results, wherein the segmentation prediction head module comprises a mask network and a label network; the segmentation prediction head module in the segmentation model is used to perform prediction on the feature map to obtain the segmentation results, comprising: identifying a target object in the feature map by using the mask network to obtain a predicted clothes segmentation mask; predicting a label of the target object in the feature map by using the label network to obtain a predicted clothes category; generating the segmentation results based on the predicted clothes segmentation mask and the predicted clothes category, wherein the predicted clothes segmentation mask and the predicted clothes category correspond to each other one by one; the method further comprises: segmenting the training samples by using an initial segmentation model to obtain the sample segmentation results; performing superpixel value prediction on the training samples by using a superpixel value prediction model to obtain the superpixel value prediction results, wherein the superpixel value prediction model comprises a superpixel segmentation module and a superpixel value prediction module; generating a target loss function based on the sample segmentation results and the superpixel value prediction results; adjusting parameters of the initial segmentation model based on the target loss function to obtain the segmentation model; wherein performing superpixel value prediction on the training samples by using the superpixel value prediction model to obtain the superpixel value prediction results comprises: segmenting the training samples based on distances and color values of the training samples by using the superpixel segmentation module to obtain color values of superpixel blocks, wherein the distances are used to represent distances between pixel points in the training samples and center points corresponding to the pixel points, and the center points are used to represent clustering centers of a plurality of pixel points in the training samples; performing convolution on a first feature map by using the superpixel value prediction module to obtain predicted color values of the first feature map; confirming that the color values of the superpixel blocks and the predicted color values are the superpixel value prediction results.

2. The method of claim 1, wherein, The feature extraction module comprises a backbone network and a feature pyramid network; performing feature extraction on the image containing clothes by using the feature extraction module to obtain a feature map, comprising: processing the image containing clothes by using the backbone network to obtain a plurality of initial feature maps with different sizes; fusing the plurality of initial feature maps with different sizes by using the feature pyramid network to obtain the feature map.

3. The method of claim 1, wherein, generating a target loss function based on the sample segmentation results and the superpixel value prediction results, comprising: generating a first loss function based on the sample segmentation results, real clothes categories, and real clothes segmentation masks; generate a second loss function based on the superpixel value prediction result and the real color value; generate the target loss function based on the first loss function and the second loss function.

4. The method of claim 1, wherein, adjust parameters of an initial segmentation model based on the target loss function to obtain the segmentation model, including: in a case where a function value of the target loss function is greater than a preset threshold, adjust the parameters of the initial segmentation model by using a gradient descent method; in a case where the function value of the target loss function is equal to or less than the preset threshold, confirm a current segmentation model as the segmentation model.

5. The method of claim 3, wherein, the sample segmentation result includes a first predicted clothing category and a first clothing segmentation mask; generate a first loss function based on the sample segmentation result, a real clothing category and a real clothing segmentation mask, including: generate a classification loss function based on the first predicted clothing category and the real clothing category; generate a mask loss function based on the first clothing segmentation mask and the real clothing segmentation mask; generate the first loss function based on the classification loss function and the mask loss function.

6. A computer readable storage medium, characterized in that, The computer readable storage medium includes a stored program, wherein the program executes the method of any one of claims 1-5 when running. 7.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory has stored therein a computer program, and the processor is configured to execute the method of any one of claims 1-5 by using the computer program.

Images