Model training method, clothing fine-grained segmentation method and related devices

Abstract

The present application provides a model training method, a clothing fine-grained segmentation method, and related devices. The model training method includes: obtaining a training image and its corresponding annotation information, inputting the training image into a feature extraction module of a preset deep neural network, extracting features from the training image through the feature extraction module, and obtaining feature information corresponding to the training image, wherein the feature extraction module adopts an architecture that combines CBNet and FPN, and CBNet contains two cascaded ResNeXT-101s; inputting the feature information corresponding to the training image into a prediction module of the preset deep neural network, segmenting the training image through the prediction module, and obtaining prediction information corresponding to the training image; and using the prediction information and annotation information corresponding to the training image to train the preset deep neural network to obtain a clothing fine-grained segmentation model. As a result, the clothing fine-grained segmentation model is relatively stable and meets the needs of practical applications.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a model training method, a clothing fine-grained segmentation method, and related devices. Background Art

[0002] Deep learning is a machine learning method based on learning to represent data. The benefit of deep learning is that it replaces manual feature extraction with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction. Deep learning is a new field in machine learning research, motivated by the goal of building neural networks that mimic the analytical learning of the human brain, mimicking the brain's mechanisms to interpret data such as images, sounds, and text.

[0003] With the advent of the era of big data and artificial intelligence, analyzing consumer clothing styles through deep learning technology and computer vision technology will help businesses capture consumption trends of various customer groups and develop targeted product portfolios, marketing plans and business decisions.

[0004] Therefore, it is urgent to design a fine-grained clothing segmentation algorithm to meet the needs of practical applications. Summary of the Invention

[0005] The purpose of this application is to provide a model training method, a fine-grained clothing segmentation method and related devices, which adopts a CBNet and FPN combined architecture as a feature extraction module, with stronger feature extraction capabilities. The resulting fine-grained clothing segmentation model is more stable and meets the needs of practical applications.

[0006] The purpose of this application is achieved by the following technical solutions:

[0007] In the first aspect, the present application provides a model training method, which includes: obtaining a training image and its corresponding annotation information, wherein the annotation information corresponding to the training image is used to indicate the labeled bounding box, labeled fine-grained category and labeled contour of the clothing in the training image; inputting the training image into a feature extraction module of a preset deep neural network, and performing feature extraction on the training image through the feature extraction module to obtain feature information corresponding to the training image, wherein the feature extraction module adopts an architecture combining CBNet and FPN, and CBNet contains two cascaded ResNeXT-101s; inputting the feature information corresponding to the training image into a prediction module of the preset deep neural network, and performing segmentation on the training image through the prediction module to obtain prediction information corresponding to the training image, and the prediction information corresponding to the training image is used to indicate the predicted bounding box, predicted fine-grained category and predicted contour of the clothing in the training image; using the prediction information and annotation information corresponding to the training image, the preset deep neural network is trained to obtain a clothing fine-grained segmentation model. The beneficial effect of this technical solution is that the training image is input into the feature extraction module of the preset deep neural network to obtain the feature information corresponding to the training image, and the preset deep neural network is trained using the prediction information and annotation information corresponding to the training image to obtain a clothing fine-grained segmentation model. This application adopts an architecture that combines CBNet and FPN as a feature extraction module. Compared with the traditional feature extraction network, CBNet has stronger feature extraction capabilities and higher instance segmentation accuracy. The resulting clothing fine-grained segmentation model is more stable and has higher accuracy, which meets the needs of practical applications.

[0008] In some optional embodiments, the step of inputting the training image into a feature extraction module of a preset deep neural network, performing feature extraction on the training image through the feature extraction module, and obtaining feature information corresponding to the training image includes: inputting the training image into Stage 1 of the feature extraction module, performing feature extraction on the training image through Stage 1, and obtaining a feature map F1 corresponding to the training image; inputting the feature map F1 corresponding to the training image into Stage 2 of the feature extraction module, and performing feature extraction on the training image through Stage 3 to obtain a feature map F1 corresponding to the training image; 1_1 , through Stage 1_1 Perform feature extraction on the feature map F1 corresponding to the training image to obtain the feature map F2 corresponding to the training image; input the feature map F1 corresponding to the training image into Stage 2 of the feature extraction module, perform feature extraction on the feature map F1 corresponding to the training image through Stage 2 to obtain the feature map F3 corresponding to the training image; add the feature map F3 corresponding to the training image and the feature map F2 and input the result into Stage 3 of the feature extraction module. 2_2 , through Stage 2_2Perform feature extraction on the sum of the feature map F3 and the feature map F2 corresponding to the training image to obtain the feature map F4 corresponding to the training image; input the feature map F3 corresponding to the training image into Stage 3 of the feature extraction module, perform feature extraction on the feature map F3 corresponding to the training image through Stage 3 to obtain the feature map F5 corresponding to the training image; add the feature map F5 corresponding to the training image and the feature map F4 and input them into Stage 2 of the feature extraction module. 3_3 , through Stage 3_3 Perform feature extraction on the sum of the feature map F5 and the feature map F4 corresponding to the training image to obtain the feature map F6 corresponding to the training image; input the feature map F5 corresponding to the training image into Stage 4 of the feature extraction module, perform feature extraction on the feature map F5 corresponding to the training image through Stage 4 to obtain the feature map F7 corresponding to the training image; add the feature map F7 corresponding to the training image and the feature map F6 and input them into Stage 5 of the feature extraction module. 4_4 , through Stage 4_4Perform feature extraction on the addition result of the feature map F7 and the feature map F6 corresponding to the training image to obtain the feature map F8 corresponding to the training image, and use the feature map F8 corresponding to the training image as the fused feature map M3 corresponding to the training image; input the feature map F8 corresponding to the training image into the third interpolation unit of the feature extraction module, interpolate the feature map F8 corresponding to the training image through the third interpolation unit, obtain the interpolation result of the feature map F8 corresponding to the training image, and add the interpolation result of the feature map F8 corresponding to the training image and the feature map F6 corresponding to the training image to obtain the fused feature map M2 corresponding to the training image; input the fused feature map M2 corresponding to the training image into the second interpolation unit of the feature extraction module, and interpolate the fused feature map M2 corresponding to the training image through the second interpolation unit. The feature map M2 is interpolated to obtain the interpolation result of the fused feature map M2 corresponding to the training image, and the interpolation result of the fused feature map M2 corresponding to the training image is added to the feature map F4 corresponding to the training image to obtain the fused feature map M1 corresponding to the training image; the fused feature map M1 corresponding to the training image is input into the first interpolation unit of the feature extraction module, and the fused feature map M1 corresponding to the training image is interpolated by the first interpolation unit to obtain the interpolation result of the fused feature map M1 corresponding to the training image, and the interpolation result of the fused feature map M1 corresponding to the training image is added to the feature map F2 corresponding to the training image to obtain the fused feature map M0 corresponding to the training image; the fused feature maps M3 to the fused feature map M0 corresponding to the training image are used as the feature information corresponding to the training image. The beneficial effect of this technical solution is that the training image is input into the feature extraction module, the multiple stages of the feature extraction module are used to obtain the feature map corresponding to the training image, and then the multiple interpolation units of the feature extraction module are used to perform interpolation to obtain the corresponding interpolation result, and the corresponding feature map is combined to obtain the corresponding fusion feature map, thereby obtaining the feature information corresponding to the training image. This method has a high level of intelligence.

[0009] In some optional embodiments, the Stage 1, Stage 2, Stage 3, Stage 4, Stage 1_1 、Stage 2_2 、Stage 3_3 、Stage 4_4Any one of the target stages is a target stage, and feature extraction is performed on the training image using the target stage, including: processing the image input to the target stage using multiple residual blocks of the target stage; adding the processing results of each residual block using a first addition layer of the target stage; and adding the sum of the processing results of each residual block to the image input to the target stage using a second addition layer of the target stage. The beneficial effect of this technical solution is that each residual block of the target stage can process the image input to the target stage, obtaining a highly accurate processing result.

[0010] In some optional embodiments, the image input to the target Stage is processed respectively by the multiple residual blocks of the target Stage, including: performing convolution processing on the image input to the target Stage respectively through the first convolution layer of each residual block of the target Stage to obtain the first convolution result corresponding to each residual block; performing convolution processing on the first convolution result corresponding to each residual block respectively through the deformable convolution layer of each residual block of the target Stage to obtain the deformable convolution result corresponding to each residual block; performing convolution processing on the deformable convolution result corresponding to each residual block respectively through the second convolution layer of each residual block of the target Stage. The beneficial effect of this technical solution is that, by utilizing the first convolution layer, the deformable convolution layer, and the second convolution layer of each residual block of the target Stage, features of the image input to the target Stage can be extracted more finely.

[0011] In some optional embodiments, the step of inputting the feature information corresponding to the training image into the prediction module of the preset deep neural network, segmenting the training image through the prediction module, and obtaining the prediction information corresponding to the training image includes: inputting the feature information corresponding to the training image into the RPN network of the preset deep neural network, performing candidate box generation and bounding box regression through the RPN network, and obtaining the reference bounding box information corresponding to the training image; inputting the feature information and reference bounding box information corresponding to the training image into the first head structure of the prediction module, performing classification, bounding box regression, fine-grained classification and segmentation through the first head structure, and obtaining the first category information, first bounding box information, first fine-grained category information and first contour information corresponding to the training image; inputting the feature information corresponding to the training image into the RPN network of the preset deep neural network, performing candidate box generation and bounding box regression through the RPN network, and obtaining the reference bounding box information corresponding to the training image; inputting the feature information and reference bounding box information corresponding to the training image into the first head structure of the prediction module, and performing classification, bounding box regression, fine-grained classification and segmentation through the first head structure, and obtaining the first category information, first bounding box information, first fine-grained category information and first contour information corresponding to the training image; The feature information and first bounding box information corresponding to the training image are input into the second head structure of the prediction module, and classification, bounding box regression, fine-grained classification and segmentation are performed through the second head structure to obtain the second category information, second bounding box information, second fine-grained category information and second contour information corresponding to the training image; the feature information and second bounding box information corresponding to the training image are input into the third head structure of the prediction module, and classification, bounding box regression, fine-grained classification and segmentation are performed through the third head structure to obtain the third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image; based on the third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image, the prediction information corresponding to the training image is obtained. The beneficial effect of this technical solution is that the feature information corresponding to the training image is input into the RPN network of the preset deep neural network to obtain the reference bounding box information corresponding to the training image, and the first head structure, second head structure and third head structure of the prediction module are used to obtain the corresponding category information, bounding box information, fine-grained category information and contour information, thereby obtaining the prediction information corresponding to the training image.

[0012] In some optional embodiments, any one of the first to third head structures is used as a target head structure, and classification, bounding box regression, fine-grained classification, and segmentation are performed using the target head structure, including: classification and bounding box regression using the Detection Head of the target head structure; fine-grained classification using the Attributes Head of the target head structure; and segmentation using the Mask Head of the target head structure. The beneficial effect of this technical solution is that the Detection Head of the target head structure can be used for classification and bounding box regression, the Attributes Head of the target head structure can be used for fine-grained classification, and the Mask Head of the target head structure can be used for segmentation.

[0013] In some optional embodiments, the classification and bounding box regression through the Detection Head of the target head structure include: performing ROIPooling processing through the ROI Pooling layer of the Detection Head of the target head structure; performing convolution processing through the convolution layer of the Detection Head of the target head structure; performing full connection processing through the first fully connected layer of the Detection Head of the target head structure; and performing classification and bounding box regression through the second fully connected layer of the Detection Head of the target head structure. The fine-grained classification through the Attributes Head of the target head structure includes: performing ROI Align processing through the ROIAlign layer of the Attributes Head of the target head structure; performing convolution processing through the convolution layer of the Attributes Head of the target head structure; performing full connection processing through the first fully connected layer of the Attributes Head of the target head structure; and performing fine-grained classification through the second fully connected layer of the Attributes Head of the target head structure. The segmentation through the Mask Head of the target head structure includes: performing ROIAlign processing through the ROI Align layer of the Mask Head of the target head structure to obtain mask features; and performing segmentation through the Mask Head of the target head structure. The semantic segmentation branch of the Mask Head performs semantic segmentation processing to obtain semantic segmentation features; the ROI Align layer of the Mask Head of the target head structure performs ROI Align processing on the semantic segmentation features and the bounding box information of the input target head structure to obtain a fusion feature; the addition layer of the Mask Head of the target head structure adds the fusion feature to the mask feature. The beneficial effect of this technical solution is that when segmentation is performed by the Mask Head of the target head structure, the semantic segmentation branch of the Mask Head of the target head structure can be used to perform semantic segmentation processing to obtain semantic segmentation features, thereby obtaining better spatial information, the ROI Align layer of the Mask Head of the target head structure performs ROI Align processing on the semantic segmentation features and the bounding box information of the input target head structure to obtain a fusion feature, and then the addition layer of the Mask Head of the target head structure adds the fusion feature to the mask feature, so that the clothing fine-grained segmentation model trained in this way has a stronger ability to distinguish targets.

[0014] In some optional embodiments, the classification and bounding box regression performed by the second fully connected layer of the Detection Head of the target head structure include: classification performed by the classification network FC-head of the second fully connected layer of the Detection Head of the target head structure; and bounding box regression performed by the regression network Conv-head of the second fully connected layer of the Detection Head of the target head structure. The beneficial effect of this technical solution is that the classification network FC-head of the second fully connected layer of the Detection Head of the target head structure can be used for classification, and the bounding box regression can be performed by the regression network Conv-head of the second fully connected layer of the Detection Head of the target head structure.

[0015] In some optional embodiments, the classification and bounding box regression performed by the Detection Head of the target head structure includes: using a detection algorithm based on spatial adaptive decoupling between tasks, performing classification and bounding box regression by the Detection Head of the target head structure, wherein adaptive proposals and feature extractors are learned for the classification task and the regression task, respectively. The beneficial effect of this technical solution is that the input and feature extractor are not shared for the classification task and the regression task, which can minimize the conflict caused by different optimization objectives, thereby significantly improving the performance of the fine-grained clothing segmentation model.

[0016] In some optional embodiments, the step of inputting the feature information and the first bounding box information corresponding to the training image into the second head structure of the prediction module, performing classification, bounding box regression, fine-grained classification and segmentation through the second head structure, and obtaining the second category information, second bounding box information, second fine-grained category information and second contour information corresponding to the training image, includes: inputting the feature information, the first bounding box information and the first contour information corresponding to the training image into the second head structure of the prediction module, performing classification, bounding box regression, fine-grained classification and segmentation through the second head structure, and obtaining the second category information, second bounding box information, second fine-grained category information and second contour information corresponding to the training image. Second contour information; the feature information and second bounding box information corresponding to the training image are input into the third head structure of the prediction module, and classification, bounding box regression, fine-grained classification and segmentation are performed through the third head structure to obtain third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image, including: inputting the feature information, second bounding box information and second contour information corresponding to the training image into the third head structure of the prediction module, and classification, bounding box regression, fine-grained classification and segmentation are performed through the third head structure to obtain third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image. The beneficial effect of this technical solution is that when using the first head structure, the second head structure and the third head structure to perform classification, bounding box regression, fine-grained classification and segmentation, the input information can include the contour information corresponding to the training image, and the accuracy of the resulting fine-grained clothing segmentation model is relatively high.

[0017] In some optional embodiments, the step of inputting the training images into a feature extraction module of a preset deep neural network, extracting features from the training images through the feature extraction module, and obtaining feature information corresponding to the training images comprises: determining a plurality of training images containing the training images; scaling the width and the long sides of the middle and high sides of each training image to a preset length value, and scaling the width and the short sides of the middle and high sides of each training image to any value within a preset length range; using the maximum value of the short sides of the plurality of training images as a reference value, and padding the short sides of the remaining training images to the reference value; inputting the plurality of training images into the feature extraction module in batches, extracting features from the plurality of training images through the feature extraction module, and obtaining feature information corresponding to the plurality of training images; and obtaining feature information corresponding to the training images based on the feature information corresponding to the plurality of training images. The beneficial effect of this technical solution is that, by performing corresponding scaling processing on the long and short sides of the training images, data augmentation can be achieved, the training images can be made as diverse as possible, and the trained fine-grained clothing segmentation model has a strong generalization ability.

[0018] In a second aspect, the present application provides a method for fine-grained clothing segmentation, comprising: obtaining an image to be segmented; inputting the image to be segmented into a fine-grained clothing segmentation model to obtain prediction information corresponding to the image to be segmented; wherein the fine-grained clothing segmentation model is trained using any of the above-mentioned model training methods. The beneficial effect of this technical solution is that the user inputs the image to be segmented into the fine-grained clothing segmentation model to obtain prediction information corresponding to the image to be segmented. The prediction information may include a predicted bounding box, a predicted fine-grained category, and a predicted outline of the image to be segmented. This method can be used to identify clothing categories and clothing attributes, thereby improving the user experience.

[0019] In a third aspect, the present application provides a model training device, which includes: a data acquisition module for acquiring a training image and its corresponding annotation information, wherein the annotation information corresponding to the training image is used to indicate the labeled bounding box, labeled fine-grained category and labeled contour of the clothing in the training image; a feature extraction module for inputting the training image into a feature extraction module of a preset deep neural network, and performing feature extraction on the training image through the feature extraction module to obtain feature information corresponding to the training image, wherein the feature extraction module adopts an architecture combining CBNet and FPN, and CBNet contains two cascaded ResNeXT-101s; an image prediction module for inputting the feature information corresponding to the training image into the prediction module of the preset deep neural network, and performing segmentation on the training image through the prediction module to obtain prediction information corresponding to the training image, and the prediction information corresponding to the training image is used to indicate the predicted bounding box, predicted fine-grained category and predicted contour of the clothing in the training image; a model training module for training the preset deep neural network using the prediction information and annotation information corresponding to the training image to obtain a clothing fine-grained segmentation model.

[0020] In some optional embodiments, the feature extraction module includes: a first feature map unit, which is used to input the training image into Stage 1 of the feature extraction module, perform feature extraction on the training image through Stage 1, and obtain a feature map F1 corresponding to the training image; a second feature map unit, which is used to input the feature map F1 corresponding to the training image into Stage 2 of the feature extraction module 1_1 , through Stage 1_1The feature map F1 corresponding to the training image is subjected to feature extraction to obtain the feature map F2 corresponding to the training image; the third feature map unit is used to input the feature map F1 corresponding to the training image into Stage 2 of the feature extraction module, and perform feature extraction on the feature map F1 corresponding to the training image through Stage 2 to obtain the feature map F3 corresponding to the training image; the fourth feature map unit is used to add the feature map F3 corresponding to the training image and the feature map F2 and then input the result into Stage 2 of the feature extraction module. 2_2 , through Stage 2_2 The feature map F3 and the feature map F2 corresponding to the training image are added to obtain the feature map F4 corresponding to the training image; the fifth feature map unit is used to input the feature map F3 corresponding to the training image into Stage 3 of the feature extraction module, and perform feature extraction on the feature map F3 corresponding to the training image through Stage 3 to obtain the feature map F5 corresponding to the training image; the sixth feature map unit is used to add the feature map F5 corresponding to the training image and the feature map F4 and then input the result into Stage 3 of the feature extraction module. 3_3 , through Stage 3_3 The feature map F5 and the feature map F4 corresponding to the training image are added to extract features to obtain the feature map F6 corresponding to the training image; the seventh feature map unit is used to input the feature map F5 corresponding to the training image into Stage 4 of the feature extraction module, and perform feature extraction on the feature map F5 corresponding to the training image through Stage 4 to obtain the feature map F7 corresponding to the training image; the third fusion feature unit is used to add the feature map F7 and the feature map F6 corresponding to the training image and input the result into Stage 4 of the feature extraction module. 4_4 , through Stage 4_4Perform feature extraction on the addition result of the feature map F7 and the feature map F6 corresponding to the training image to obtain the feature map F8 corresponding to the training image, and use the feature map F8 corresponding to the training image as the fused feature map M3 corresponding to the training image; a second fusion feature unit is used to input the feature map F8 corresponding to the training image into the third interpolation unit of the feature extraction module, interpolate the feature map F8 corresponding to the training image through the third interpolation unit to obtain the interpolation result of the feature map F8 corresponding to the training image, and add the interpolation result of the feature map F8 corresponding to the training image and the feature map F6 corresponding to the training image to obtain the fused feature map M2 corresponding to the training image; a first fusion feature unit is used to input the fused feature map M2 corresponding to the training image into the second interpolation unit of the feature extraction module, and interpolate the fused feature map M2 corresponding to the training image through the second interpolation unit The fused feature map M2 is interpolated to obtain the interpolation result of the fused feature map M2 corresponding to the training image, and the interpolation result of the fused feature map M2 corresponding to the training image is added to the feature map F4 corresponding to the training image to obtain the fused feature map M1 corresponding to the training image; the zeroth fusion feature unit is used to input the fused feature map M1 corresponding to the training image into the first interpolation unit of the feature extraction module, and the fused feature map M1 corresponding to the training image is interpolated by the first interpolation unit to obtain the interpolation result of the fused feature map M1 corresponding to the training image, and the interpolation result of the fused feature map M1 corresponding to the training image is added to the feature map F2 corresponding to the training image to obtain the fused feature map M0 corresponding to the training image; the feature information unit is used to use the fused feature map M3 corresponding to the training image to the fused feature map M0 as the feature information corresponding to the training image.

[0021] In some optional embodiments, the Stage 1, Stage 2, Stage 3, Stage 4, Stage 1_1 、Stage 2_2 、Stage 3_3 、Stage 4_4 Any one of them is a target Stage, and the feature extraction module includes: a residual block unit, which is used to process the image input to the target Stage respectively through multiple residual blocks of the target Stage; a processing result unit, which is used to add the processing results of each residual block through the first addition layer of the target Stage; and an image addition unit, which is used to add the addition result of the processing results of each residual block to the image input to the target Stage through the second addition layer of the target Stage.

[0022] In some optional embodiments, the residual block unit includes: a first convolution subunit, used to perform convolution processing on the image input to the target Stage through the first convolution layer of each residual block of the target Stage, to obtain a first convolution result corresponding to each residual block; a second convolution subunit, used to perform convolution processing on the first convolution result corresponding to each residual block through the deformable convolution layer of each residual block of the target Stage, to obtain a deformable convolution result corresponding to each residual block; a third convolution subunit, used to perform convolution processing on the deformable convolution result corresponding to each residual block through the second convolution layer of each residual block of the target Stage.

[0023] In some optional embodiments, the image prediction module includes: a reference bounding box unit, which is used to input the feature information corresponding to the training image into the RPN network of the preset deep neural network, and perform candidate box generation and bounding box regression through the RPN network to obtain the reference bounding box information corresponding to the training image; a first head structure unit, which is used to input the feature information and reference bounding box information corresponding to the training image into the first head structure of the prediction module, and perform classification, bounding box regression, fine-grained classification and segmentation through the first head structure to obtain the first category information, first bounding box information, first fine-grained category information and first contour information corresponding to the training image; a second head structure unit, which is used to input the feature information and first bounding box information corresponding to the training image into the prediction module a second head structure for performing classification, bounding box regression, fine-grained classification and segmentation through the second head structure to obtain second category information, second bounding box information, second fine-grained category information and second contour information corresponding to the training image; a third head structure unit for inputting the feature information and second bounding box information corresponding to the training image into the third head structure of the prediction module, performing classification, bounding box regression, fine-grained classification and segmentation through the third head structure to obtain third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image; a prediction information unit for obtaining prediction information corresponding to the training image based on the third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image.

[0024] In some optional embodiments, any one of the first head structure to the third head structure is used as the target head structure, and the image prediction module includes: a classification and regression unit, used to perform classification and bounding box regression through the DetectionHead of the target head structure; a fine-grained classification unit, used to perform fine-grained classification through the AttributesHead of the target head structure; and a segmentation unit, used to perform segmentation through the Mask Head of the target head structure.

[0025] In some optional embodiments, the classification and regression unit includes: a first processing subunit for performing ROI Pooling processing through the ROI Pooling layer of the Detection Head of the target head structure; a second processing subunit for performing convolution processing through the convolution layer of the Detection Head of the target head structure; a third processing subunit for performing full connection processing through the first fully connected layer of the Detection Head of the target head structure; a fourth processing subunit for performing classification and bounding box regression through the second fully connected layer of the Detection Head of the target head structure; the fine-grained classification unit includes: a fifth processing subunit for performing ROI Align processing through the ROI Align layer of the Attributes Head of the target head structure; a sixth processing subunit for performing convolution processing through the convolution layer of the Attributes Head of the target head structure; a seventh processing subunit for performing full connection processing through the first fully connected layer of the Attributes Head of the target head structure; an eighth processing subunit for performing fine-grained classification through the second fully connected layer of the Attributes Head of the target head structure; the segmentation unit includes: a mask subunit for performing ROI Align processing through the ROI Align layer of the Mask Head of the target head structure Align processing is performed to obtain mask features; a semantic segmentation subunit is used to perform semantic segmentation processing through the semantic segmentation branch of the Mask Head of the target head structure to obtain semantic segmentation features; a fusion feature subunit is used to perform ROI Align processing on the semantic segmentation features and the bounding box information input into the target head structure through the ROIAlign layer of the Mask Head of the target head structure to obtain a fusion feature; a feature addition subunit is used to add the first fusion feature and the mask feature through the addition layer of the Mask Head of the target head structure.

[0026] In some optional embodiments, the fourth processing subunit is used to: perform classification through the classification network FC-head of the second fully connected layer of the Detection Head of the target head structure; and perform bounding box regression through the regression network Conv-head of the second fully connected layer of the Detection Head of the target head structure.

[0027] In some optional embodiments, the classification and regression unit is used to: use a detection algorithm based on spatial adaptive decoupling between tasks to perform classification and bounding box regression through the Detection Head of the target head structure, wherein for the classification task and regression task, each adaptive proposal and feature extractor is learned respectively.

[0028] In some optional embodiments, the second head structure unit is used to: input the feature information, first bounding box information and first contour information corresponding to the training image into the second head structure of the prediction module, and perform classification, bounding box regression, fine-grained classification and segmentation through the second head structure to obtain the second category information, second bounding box information, second fine-grained category information and second contour information corresponding to the training image; the third head structure unit is used to: input the feature information, second bounding box information and second contour information corresponding to the training image into the third head structure of the prediction module, and perform classification, bounding box regression, fine-grained classification and segmentation through the third head structure to obtain the third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image.

[0029] In some optional embodiments, the feature extraction module includes: an image determination unit, used to determine multiple training images including the training image; an image scaling unit, used to scale the long sides of the width and height of each training image to a preset length value, and scale the short sides of the width and height of each training image to any value in a preset length range; an image filling unit, used to use the maximum value of the short sides in the multiple training images as a reference value, and fill the short sides of the remaining training images to the reference value; a first information unit, used to input the multiple training images into the feature extraction module in batches, and perform feature extraction on the multiple training images through the feature extraction module to obtain feature information corresponding to the multiple training images; a second information unit, used to obtain feature information corresponding to the training images based on the feature information corresponding to the multiple training images.

[0030] In a fourth aspect, the present application provides a fine-grained clothing segmentation device, which includes: an image to be segmented module, used to obtain the image to be segmented; an image segmentation module, used to input the image to be segmented into a clothing fine-grained segmentation model to obtain prediction information corresponding to the image to be segmented; wherein the clothing fine-grained segmentation model is trained using any of the above-mentioned model training methods.

[0031] In a fifth aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of any one of the above-mentioned model training methods or the above-mentioned steps of the fine-grained clothing segmentation method.

[0032] In a sixth aspect, the present application provides a computer-readable storage medium storing a computer program or a clothing fine-grained segmentation model; when the computer program is executed by a processor, the steps of any one of the above-mentioned model training methods or the above-mentioned clothing fine-grained segmentation method are implemented;

[0033] The clothing fine-grained segmentation model is trained using any of the above-mentioned model training methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present application is further described below with reference to the accompanying drawings and examples.

[0035] Figure 1 This is a flow chart of a model training method provided in an embodiment of the present application;

[0036] Figure 2 This is a schematic diagram of a process for obtaining feature information provided by an embodiment of the present application;

[0037] Figure 3 This is another flowchart for obtaining feature information provided by an embodiment of the present application;

[0038] Figure 4 This is a schematic diagram of a feature extraction process provided by an embodiment of the present application;

[0039] Figure 5 This is a schematic diagram of the structure of ResNext101 provided in an embodiment of the present application;

[0040] Figure 6 This is a schematic diagram of a process for processing an image provided by an embodiment of the present application;

[0041] Figure 7 This is a schematic diagram of a process for obtaining prediction information provided by an embodiment of the present application;

[0042] Figure 8 This is a flow chart of information processing using a target header structure provided by an embodiment of the present application;

[0043] Figure 9 This is a flow chart of a method for performing classification and bounding box regression using a Detection Head, as provided in an embodiment of the present application;

[0044] Figure 10 This is a flow chart of fine-grained classification using Attributes Head provided in an embodiment of the present application;

[0045] Figure 11This is a schematic diagram of a process of segmentation using Mask Head provided in an embodiment of the present application;

[0046] Figure 12 This is a schematic structural diagram of a target head structure provided by an embodiment of the present application;

[0047] Figure 13 This is a schematic diagram of another process for obtaining feature information provided by an embodiment of the present application;

[0048] Figure 14 This is a flow chart of another model training method provided in an embodiment of the present application;

[0049] Figure 15 This is a flow chart of a fine-grained clothing segmentation method provided in an embodiment of the present application;

[0050] Figure 16 This is a structural diagram of a model training device provided in an embodiment of the present application;

[0051] Figure 17 This is a schematic diagram of the structure of a feature extraction module provided in an embodiment of the present application;

[0052] Figure 18 This is a schematic diagram of the structure of another feature extraction module provided in an embodiment of the present application;

[0053] Figure 19 This is a structural diagram of a residual block unit provided in an embodiment of the present application;

[0054] Figure 20 This is a schematic diagram of the structure of an image prediction module provided in an embodiment of the present application;

[0055] Figure 21 is a structural diagram of another image prediction module provided in an embodiment of the present application;

[0056] Figure 22 This is a structural diagram of a classification and regression unit provided in an embodiment of the present application;

[0057] Figure 23 This is a schematic diagram of the structure of a fine-grained classification unit provided in an embodiment of the present application;

[0058] Figure 24 This is a structural diagram of a segmentation unit provided in an embodiment of the present application;

[0059] Figure 25 This is a schematic diagram of the structure of another feature extraction module provided in an embodiment of the present application;

[0060] Figure 26Schematic diagram of the structure of a fine-grained clothing segmentation device provided in an embodiment of the present application;

[0061] Figure 27 This is a structural block diagram of an electronic device provided in an embodiment of the present application;

[0062] Figure 28 This is a structural diagram of a program product for implementing a model training method or a fine-grained clothing segmentation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0064] See also Figure 1 , an embodiment of the present application provides a model training method, which includes steps S101 to S104.

[0065] Step S101: obtaining a training image and its corresponding annotation information, wherein the annotation information corresponding to the training image is used to indicate a labeled bounding box, a labeled fine-grained category, and a labeled outline of clothing in the training image.

[0066] The number of training images is, for example, 50,000, 80,000, or 100,000. The training images may be, for example, images of tops, trousers, shorts, scarves, and the like.

[0067] Step S102: Input the training image into a feature extraction module of a preset deep neural network. The feature extraction module extracts features from the training image to obtain feature information corresponding to the training image. The feature extraction module uses an architecture that combines CBNet and FPN, where CBNet includes two cascaded ResNeXT-101s. The preset deep neural network can use an HTC (Hybrid Task Cascade) network.

[0068] Step S103: Input the feature information corresponding to the training image into the prediction module of the preset deep neural network, segment the training image through the prediction module, and obtain the prediction information corresponding to the training image, which is used to indicate the predicted bounding box, predicted fine-grained category and predicted outline of the clothing in the training image.

[0069] Step S104: using the prediction information and annotation information corresponding to the training image, the preset deep neural network is trained to obtain a clothing fine-grained segmentation model.

[0070] Therefore, the training image is input into the feature extraction module of the preset deep neural network to obtain the feature information corresponding to the training image, and the preset deep neural network is trained using the prediction information and annotation information corresponding to the training image to obtain a fine-grained clothing segmentation model. This application adopts an architecture combining CBNet and FPN as a feature extraction module. Compared with the traditional feature extraction network, CBNet has stronger feature extraction capabilities and higher instance segmentation accuracy. The resulting fine-grained clothing segmentation model is more stable and has higher accuracy, meeting the needs of practical applications.

[0071] See also Figure 2 In some implementations, step S102 may include steps S1 to S12.

[0072] Step S1: input the training image into Stage 1 of the feature extraction module, perform feature extraction on the training image through Stage 1, and obtain a feature map F1 corresponding to the training image.

[0073] Step S2: Input the feature map F1 corresponding to the training image into the Stage of the feature extraction module 1_1 , through Stage 1_1 Feature extraction is performed on the feature map F1 corresponding to the training image to obtain a feature map F2 corresponding to the training image.

[0074] Step S3: inputting the feature map F1 corresponding to the training image into Stage 2 of the feature extraction module, performing feature extraction on the feature map F1 corresponding to the training image through Stage 2, and obtaining a feature map F3 corresponding to the training image.

[0075] Step S4: Add the feature map F3 and feature map F2 corresponding to the training image and input them into the Stage of the feature extraction module 2_2 , through Stage 2_2 Feature extraction is performed on the addition result of the feature map F3 and the feature map F2 corresponding to the training image to obtain the feature map F4 corresponding to the training image.

[0076] Step S5: inputting the feature map F3 corresponding to the training image into Stage 3 of the feature extraction module, performing feature extraction on the feature map F3 corresponding to the training image through Stage 3, and obtaining the feature map F5 corresponding to the training image.

[0077] Step S6: Add the feature map F5 and feature map F4 corresponding to the training image and input them into the Stage of the feature extraction module. 3_3 , through Stage 3_3Feature extraction is performed on the addition result of the feature map F5 and the feature map F4 corresponding to the training image to obtain a feature map F6 corresponding to the training image.

[0078] Step S7: inputting the feature map F5 corresponding to the training image into Stage 4 of the feature extraction module, performing feature extraction on the feature map F5 corresponding to the training image through Stage 4, and obtaining the feature map F7 corresponding to the training image.

[0079] Step S8: Add the feature map F7 and feature map F6 corresponding to the training image and input them into the Stage of the feature extraction module. 4_4 , through Stage 4_4 Feature extraction is performed on the addition result of the feature map F7 and the feature map F6 corresponding to the training image to obtain the feature map F8 corresponding to the training image, and the feature map F8 corresponding to the training image is used as the fused feature map M3 corresponding to the training image.

[0080] Step S9: Input the feature map F8 corresponding to the training image into the third interpolation unit of the feature extraction module, interpolate the feature map F8 corresponding to the training image through the third interpolation unit to obtain the interpolation result of the feature map F8 corresponding to the training image, and add the interpolation result of the feature map F8 corresponding to the training image and the feature map F6 corresponding to the training image to obtain the fused feature map M2 corresponding to the training image.

[0081] Step S10: Input the fused feature map M2 corresponding to the training image into the second interpolation unit of the feature extraction module, interpolate the fused feature map M2 corresponding to the training image through the second interpolation unit to obtain the interpolation result of the fused feature map M2 corresponding to the training image, and add the interpolation result of the fused feature map M2 corresponding to the training image and the feature map F4 corresponding to the training image to obtain the fused feature map M1 corresponding to the training image.

[0082] Step S11: Input the fused feature map M1 corresponding to the training image into the first interpolation unit of the feature extraction module, interpolate the fused feature map M1 corresponding to the training image through the first interpolation unit to obtain the interpolation result of the fused feature map M1 corresponding to the training image, and add the interpolation result of the fused feature map M1 corresponding to the training image and the feature map F2 corresponding to the training image to obtain the fused feature map M0 corresponding to the training image.

[0083] Step S12: taking the fused feature map M3 to the fused feature map M0 corresponding to the training image as feature information corresponding to the training image.

[0084] Therefore, the training image is input into the feature extraction module, and the feature map corresponding to the training image is obtained by using multiple stages of the feature extraction module. Then, the multiple interpolation units of the feature extraction module are used to perform interpolation to obtain the corresponding interpolation result. Combined with the corresponding feature map, the corresponding fusion feature map is obtained, thereby obtaining the feature information corresponding to the training image. This method has a high level of intelligence.

[0085] In a specific application, for each ResNeXT-101 in CBNet, the feature extraction process of the input image I is performed to obtain the feature information corresponding to the input image I. Figure 3 shown.

[0086] Before each stage enters the next stage, the output features of this stage are used as the input features of the stage horizontally parallel to it. For example, after the input image I passes through Stage1, the feature map F1 is generated. F1 is used as the input feature of the stage horizontally parallel to Stage1 (Stage 1_1 ) input features, F1 passes through Stage 1_1 After that, the feature map F2 is generated; after F1 passes through Stage2, the feature map F3 is generated. The sum of F3 and F2 is used as the Stage (Stage) of Stage2. 2_2 ) input features, after Stage 2_2 After that, the feature map F4 is generated; after F3 passes through Stage3, the feature map F5 is generated. The sum of F5 and F4 is used as the Stage (Stage 3) that is horizontally parallel to Stage 3. 3_3 ) input features, after Stage 3_3 After that, the feature map F6 is generated; after F5 passes through Stage4, the feature map F7 is generated. The sum of F7 and F6 is used as the Stage (Stage 4) horizontally side by side. 4_4 ) input features, after Stage 4_4 Then generate feature map F8.

[0087] Extract F2, F4, F6 and F8 generated by the above process. After interpolation, F8 forms a feature map of the same size and channel as F6. Add F8 and F6 to fuse the Stage 4_4 With Stage 3_3 The features of the stage are obtained to obtain M2; after interpolation, M2 forms a feature map of the same size and the same channel as F4, and M2 and F4 are added to fuse Stage 3_3 With Stage 2_2 The features of the stage are obtained to obtain M1; after interpolation, M1 forms a feature map of the same size and the same channel as F2, and M1 and F2 are added to fuse Stage 2_2 With Stage1_1 The characteristics of the stage are obtained, and M0 is obtained; F8 is directly output as M3.

[0088] See also Figure 4 In some embodiments, the Stage 1, Stage 2, Stage 3, Stage 4, Stage 1_1 、Stage 2_2 、Stage 3_3 、Stage 4_4 Any one of them is a target Stage, and the method for extracting features from the training image through the target Stage may include steps S201 to S203.

[0089] Step S201: Process the image input to the target stage through the multiple residual blocks of the target stage, wherein the residual block can be represented by Residual Block, and the target stage can be horizontally composed of multiple residual blocks.

[0090] Step S202: adding the processing results of the residual blocks through the first addition layer of the target Stage.

[0091] Step S203: adding the processing results of the residual blocks to the image input to the target stage through the second addition layer of the target stage.

[0092] Therefore, each residual block of the target stage can process the image input to the target stage to obtain a processing result with higher accuracy.

[0093] See also Figure 5 In some embodiments, each residual block in ResNext 101 can include three consecutive convolutional layers. The first layer has 256 input channels, 4 output channels, and a 1×1 convolution kernel size; the second layer has 4 input channels, 4 output channels, and a 3×3 convolution kernel size; and the third layer has 4 input channels, 256 output channels, and a 1×1 convolution kernel size. Each residual block consists of 32 groups of residual unit branch structures in parallel, and finally performs an additive operation on the channels.

[0094] See also Figure 6 In some implementations, step S201 may include steps S301 to S303.

[0095] Step S301: Convolution processing is performed on the input image of the target stage through the first convolution layer of each residual block of the target stage to obtain the first convolution result corresponding to each residual block. The size of the convolution kernel of the first convolution layer can be 1×1.

[0096] Step S302: Convolve the first convolution results corresponding to each residual block in the target stage through the deformable convolution layer of each residual block to obtain a deformable convolution result corresponding to each residual block. The deformable convolution layer is, for example, a DCN (Deformable Convolution Net) layer. The convolution kernel of the deformable convolution layer can offset the sampling of input features, thereby providing better robustness to deformed objects in the scene.

[0097] Step S303: Perform convolution processing on the deformable convolution results corresponding to each residual block through the second convolution layer of each residual block of the target stage, wherein the size of the convolution kernel of the second convolution layer can be 1×1.

[0098] Therefore, by utilizing the first convolutional layer, deformable convolutional layer, and second convolutional layer of each residual block of the target stage, features of the image input to the target stage can be extracted more finely.

[0099] See also Figure 7 In some implementations, step S103 may include steps S401 to S405.

[0100] Step S401: inputting feature information corresponding to the training image into the RPN network of the preset deep neural network, performing candidate box generation and bounding box regression through the RPN network, and obtaining reference bounding box information corresponding to the training image.

[0101] Step S402: Input the feature information and reference bounding box information corresponding to the training image into the first header structure of the prediction module. The first header structure performs classification, bounding box regression, fine-grained classification, and segmentation to obtain first category information, first bounding box information, first fine-grained category information, and first contour information corresponding to the training image. The first header structure can be represented by h1.

[0102] Step S403: Input the feature information and first bounding box information corresponding to the training image into the second header structure of the prediction module. The second header structure performs classification, bounding box regression, fine-grained classification, and segmentation to obtain second category information, second bounding box information, second fine-grained category information, and second contour information corresponding to the training image. The second header structure can be represented by h2.

[0103] Step S404: Input the feature information and second bounding box information corresponding to the training image into the third header structure of the prediction module. The third header structure performs classification, bounding box regression, fine-grained classification, and segmentation to obtain third category information, third bounding box information, third fine-grained category information, and third contour information corresponding to the training image. The third header structure can be represented by h3.

[0104] Step S405: Based on the third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image, obtain prediction information corresponding to the training image.

[0105] Therefore, the feature information corresponding to the training image is input into the RPN network of the preset deep neural network to obtain the reference bounding box information corresponding to the training image, and the first head structure, second head structure and third head structure of the prediction module are used to obtain the corresponding category information, bounding box information, fine-grained category information and contour information, thereby obtaining the prediction information corresponding to the training image.

[0106] See also Figure 8 In some embodiments, any one of the first to third head structures is used as a target head structure, and the method for performing classification, bounding box regression, fine-grained classification, and segmentation using the target head structure may include steps S501 to S503.

[0107] Step S501: performing classification and bounding box regression through the Detection Head of the target head structure.

[0108] Step S502: performing fine-grained classification through the Attributes Head of the target header structure.

[0109] Step S503: Segmentation is performed using the Mask Head of the target head structure.

[0110] Therefore, the Detection Head of the target head structure can be used for classification and bounding box regression, the Attributes Head of the target head structure can be used for fine-grained classification, and the Mask Head of the target head structure can be used for segmentation.

[0111] See also Figure 9-11 In some implementations, step S501 may include steps S601 to S604.

[0112] Step S601: performing ROIPooling processing through the ROI Pooling layer of the Detection Head of the target head structure.

[0113] Step S602: performing convolution processing through the convolution layer of the Detection Head of the target head structure.

[0114] Step S603: Perform full connection processing through the first fully connected layer of the Detection Head of the target head structure.

[0115] Step S604: Classification and bounding box regression are performed through the second fully connected layer of the Detection Head of the target head structure.

[0116] The step S502 may include steps S701 to S704.

[0117] Step S701: performing ROIAlign processing through the ROI Align layer of the Attributes Head of the target header structure.

[0118] Step S702: performing convolution processing through the convolution layer of the Attributes Head of the target header structure.

[0119] Step S703: Perform full connection processing through the first full connection layer of the Attributes Head of the target head structure.

[0120] Step S704: performing fine-grained classification through the second fully connected layer of the Attributes Head of the target head structure.

[0121] The step S503 may include steps S801 to S804.

[0122] Step S801: performing ROI Align processing through the ROI Align layer of the Mask Head of the target head structure to obtain mask features.

[0123] Step S802: performing semantic segmentation processing through the semantic segmentation branch of the Mask Head of the target head structure to obtain semantic segmentation features.

[0124] Step S803: performing ROI Align processing on the semantic segmentation features and the bounding box information input into the target head structure through the ROI Align layer of the Mask Head of the target head structure to obtain a fusion feature.

[0125] Step S804: adding the primary fusion feature and the mask feature through the addition layer of the Mask Head of the target head structure.

[0126] Therefore, when segmenting through the Mask Head of the target head structure, the semantic segmentation branch of the Mask Head of the target head structure can be used to perform semantic segmentation processing to obtain semantic segmentation features, thereby obtaining better spatial information. The semantic segmentation features and the bounding box information of the input target head structure are ROI Aligned through the ROI Align layer of the Mask Head of the target head structure to obtain a fusion feature, and then the fusion feature and the mask feature are added through the addition layer of the Mask Head of the target head structure. The fine-grained clothing segmentation model trained in this way has a strong ability to distinguish targets.

[0127] See also Figure 12 In a specific application, B i (i=1, 2, 3) represents the classification and regression branches of the Detection Head of the target head structure, H i (i=1, 2, 3) represents the template branch of the Mask Head of the target head structure, and S represents the semantic segmentation branch of the Mask Head of the target head structure.

[0128] In the adjacent head structure h i (i=1,2,3) template branch (H i ) to provide information flow for the template branch, so that H i+1 Know H i Characteristics of H i The features of are embedded through a 1x1 convolution and then input into H i+1 , so H i+1 It can obtain the features of the feature extraction module and H i characteristics.

[0129] In some embodiments, step S604 may include: performing classification through the classification network FC-head of the second fully connected layer of the DetectionHead of the target head structure; and performing bounding box regression through the regression network Conv-head of the second fully connected layer of the DetectionHead of the target head structure.

[0130] Therefore, the classification network FC-head of the second fully connected layer of the Detection Head of the target head structure can be used for classification, and the regression network Conv-head of the second fully connected layer of the Detection Head of the target head structure can be used for bounding box regression.

[0131] In some embodiments, step S501 may include: performing classification and bounding box regression using the detection head of the target head structure using a detection algorithm based on inter-task spatial adaptive decoupling, wherein adaptive proposals and feature extractors are learned for the classification task and the regression task, respectively. Task-aware spatial disentanglement can be represented by TSD (task-aware spatial disentanglement).

[0132] Therefore, on the one hand, the detection algorithm based on spatial adaptive decoupling between tasks can improve the detection accuracy of the fine-grained clothing segmentation model; on the other hand, for classification tasks and regression tasks, the input and feature extractor are not shared, which can minimize the conflicts caused by different optimization objectives, thereby significantly improving the performance of the fine-grained clothing segmentation model.

[0133] In a specific application, for classification tasks, the following formula can ensure that the classification task accuracy is higher than the traditional method, and the confidence of the constrained TSD is at least m higher than that of the sibling head c :

[0134]

[0135] Among them, cls represents the classification branch, M cls represents the progressive constraint of the classification branch of TSD, H1 represents the function that transforms the feature into a prediction function of a specific type and location object, H1 D Represents a new function that disentangles the classification and positioning tasks, F l represents the feature map output by the lth head (l is the letter “l”), τ c represents a predefined scalar for the adjustment amplitude, ΔC represents the point-wise deformation, H(y|·) represents the prediction confidence for the y-th class, and m c It is the predetermined margin.

[0136] For regression tasks, the following formula is used to ensure that the accuracy of the regression task is higher than that of the traditional method, and the IOU of the TSD prediction result is constrained to be at least m higher than that of the sibling head. r :

[0137]

[0138] Among them, M loc represents the asymptotic constraint of the positioning branch of TSD, m r Represents a preset constant, is the detection frame obtained by the original detector head, Is the detection box obtained by the TSD algorithm. If the current Proposal is a negative sample, the loss will be ignored.

[0139] During the entire training process, the optimization function of the overall detector is:

[0140]

[0141] During the inference phase, the original detector head is no longer used. rpn +L cls +L is the original loss function, is the TSD loss function.

[0142] In some embodiments, step S403 may include: inputting the feature information, first bounding box information, and first contour information corresponding to the training image into the second head structure of the prediction module, performing classification, bounding box regression, fine-grained classification, and segmentation through the second head structure to obtain second category information, second bounding box information, second fine-grained category information, and second contour information corresponding to the training image.

[0143] Step S404 may include: inputting the feature information, second bounding box information, and second contour information corresponding to the training image into the third head structure of the prediction module, and performing classification, bounding box regression, fine-grained classification, and segmentation through the third head structure to obtain third category information, third bounding box information, third fine-grained category information, and third contour information corresponding to the training image.

[0144] Therefore, when using the first head structure, the second head structure and the third head structure for classification, bounding box regression, fine-grained classification and segmentation, the input information can include contour information corresponding to the training image, and the resulting clothing fine-grained segmentation model has higher accuracy.

[0145] See also Figure 13 In some implementations, step S102 may include steps S901 to S905.

[0146] Step S901: determining a plurality of training images including the training image.

[0147] Step S902: scaling the width and the longer side of the middle and high sides of each training image to a preset length value, and scaling the width and the shorter side of the middle and high sides of each training image to any value within a preset length range.

[0148] Step S903: Taking the maximum value of the short sides of the plurality of training images as a reference value, the short sides of the remaining training images are filled to the reference value.

[0149] Step S904: inputting the plurality of training images into the feature extraction module in a batch form, performing feature extraction on the plurality of training images by the feature extraction module, and obtaining feature information corresponding to the plurality of training images.

[0150] Step S905: Based on the feature information corresponding to the plurality of training images, obtain the feature information corresponding to the training image.

[0151] Therefore, by performing corresponding scaling processing on the long and short sides of the training images, data augmentation can be achieved, making the training images as diverse as possible, and making the trained clothing fine-grained segmentation model have stronger generalization ability.

[0152] In a specific application, for each training image I i , compared to its own width I i_w and high I i_h , the width I i_w and high I i_h The longer side max(I i_w , I i_h ) is scaled to L, the shorter side min(I i_w , I i_h ) is scaled to S, where S is randomly selected from S1 to S2.

[0153] Multiple training images I i (i=1, 2, 3...n) are input to the feature extraction module in the form of batch processing I. The long side of all images in I is L, and the short side of the image is S in order to unify the size. i (i=1,2,3…n) the largest value max(S i ) is the base S_base, and the rest of S i Add padding to S_base.

[0154] The specific formula is as follows:

[0155] S_base=S i +padding

[0156] See also Figure 14 , the embodiment of the present application also provides a model training method, the model training method comprising the following steps:

[0157] Perform data augmentation on the original data to obtain augmented data;

[0158] The augmented data is input into the HTC (Hybrid Task Cascade) network, and the HTC network is used to output the corresponding feature information;

[0159] The corresponding feature information is input into the head structure, and the output result is obtained using the head structure. The head structure includes Detection Head, Attributes Head and Mask Head. A detection algorithm based on task-aware spatial disentanglement (TSD) is added to the Detection Head.

[0160] Visual analysis of clothing has been a topic of increasing interest in recent years. Identifying clothing products and related attributes from images can enhance the shopping experience for consumers and improve the work efficiency of fashion professionals.

[0161] Prior art CN111199248A discloses a clothing attribute recognition and detection method based on a deep learning target detection algorithm. This method labels and classifies original clothing images to obtain clothing attributes, such as sleeves and collars. The clothing image is then preprocessed by flipping and translating, and then the clothing attributes are recognized and detected using a deep learning target detection algorithm. The preprocessing of the clothing image involves labeling and classifying the locations of clothing attributes in the image. Traditional image algorithms are then used to perform preprocessing such as flipping and translating the image to achieve data augmentation. This method, based on a deep learning target detection algorithm, first uses a deep convolutional neural network to fully extract clothing attribute features. It then fuses multiple layers of features using a target detection algorithm feature pyramid. Finally, a fully convolutional neural network is used to recognize and detect clothing attributes. This method can achieve a recognition accuracy of 90% and a detection precision of 82%, with strong robustness and generalization capabilities, making it suitable for clothing attribute recognition and detection in the field of computer vision.

[0162] Although the above methods address the problem of relevant clothing attribute recognition, the needs of fashion insiders are not taken into consideration when designing these garments, which may be due to the research gap between clothing design and computer vision.

[0163] To address this issue, this paper uses deep learning computer vision to design a more stable and accurate fine-grained clothing segmentation method. This method can identify clothing items and their related attributes from consumer photos, thereby enhancing the consumer shopping experience. These attributes include sleeves, collars, and trouser legs.

[0164] Among them, fine-grained segmentation belongs to the technical field of fine-grained image classification. Fine-grained image classification is based on distinguishing basic categories and then making more detailed sub-category divisions, such as distinguishing types of birds, car styles, and clothing types.

[0165] See also Figure 15 , an embodiment of the present application further provides a method for fine-grained clothing segmentation, and the method for fine-grained clothing segmentation includes steps S21 to S22.

[0166] Step S21: Obtain the image to be segmented.

[0167] Step S22: inputting the image to be segmented into a clothing fine-grained segmentation model to obtain prediction information corresponding to the image to be segmented.

[0168] The clothing fine-grained segmentation model is trained using any of the above-mentioned model training methods.

[0169] Therefore, the user inputs the image to be segmented into the clothing fine-grained segmentation model to obtain the prediction information corresponding to the image to be segmented. The prediction information may include the predicted bounding box, predicted fine-grained category and predicted contour of the image to be segmented. This method can be used to identify clothing categories and clothing attributes, and improve the user experience.

[0170] See also Figure 16 The embodiment of the present application also provides a model training device, the specific implementation method of which is consistent with the implementation method and technical effects recorded in the embodiment of the above-mentioned model training method, and some contents will not be repeated here.

[0171] The model training device includes: a data acquisition module 101, used to acquire a training image and its corresponding annotation information, where the annotation information corresponding to the training image is used to indicate the labeled bounding box, labeled fine-grained category and labeled outline of the clothing in the training image; a feature extraction module 102, used to input the training image into a feature extraction module of a preset deep neural network, perform feature extraction on the training image through the feature extraction module, and obtain feature information corresponding to the training image, where the feature extraction module adopts an architecture combining CBNet and FPN, and CBNet includes two cascaded ResNeXT-101s; an image prediction module 103, used to input the feature information corresponding to the training image into the prediction module of the preset deep neural network, perform segmentation on the training image through the prediction module, and obtain prediction information corresponding to the training image, where the prediction information corresponding to the training image is used to indicate the predicted bounding box, predicted fine-grained category and predicted outline of the clothing in the training image; and a model training module 104, used to train the preset deep neural network using the prediction information and annotation information corresponding to the training image to obtain a clothing fine-grained segmentation model.

[0172] See also Figure 17In some embodiments, the feature extraction module 102 may include: a first feature map unit 1, which is used to input the training image into Stage 1 of the feature extraction module, perform feature extraction on the training image through Stage 1, and obtain a feature map F1 corresponding to the training image; a second feature map unit 2, which is used to input the feature map F1 corresponding to the training image into Stage 2 of the feature extraction module. 1_1 , through Stage 1_1 The feature map F1 corresponding to the training image is subjected to feature extraction to obtain the feature map F2 corresponding to the training image; the third feature map unit 3 is used to input the feature map F1 corresponding to the training image into Stage 2 of the feature extraction module, and perform feature extraction on the feature map F1 corresponding to the training image through Stage 2 to obtain the feature map F3 corresponding to the training image; the fourth feature map unit 4 is used to add the feature map F3 corresponding to the training image and the feature map F2 and then input the result into Stage 2 of the feature extraction module. 2_2 , through Stage 2_2 The feature map F3 and the feature map F2 corresponding to the training image are added to obtain the feature map F4 corresponding to the training image; the fifth feature map unit 5 is used to input the feature map F3 corresponding to the training image into Stage 3 of the feature extraction module, and perform feature extraction on the feature map F3 corresponding to the training image through Stage 3 to obtain the feature map F5 corresponding to the training image; the sixth feature map unit 6 is used to add the feature map F5 corresponding to the training image and the feature map F4 and then input the result into Stage 3 of the feature extraction module. 3_3 , through Stage 3_3 The feature map F5 and the feature map F4 corresponding to the training image are added together to obtain the feature map F6 corresponding to the training image; the seventh feature map unit 7 is used to input the feature map F5 corresponding to the training image into Stage 4 of the feature extraction module, and perform feature extraction on the feature map F5 corresponding to the training image through Stage 4 to obtain the feature map F7 corresponding to the training image; the third fusion feature unit 8 is used to add the feature map F7 and the feature map F6 corresponding to the training image and input the result into Stage 4 of the feature extraction module. 4_4 , through Stage 4_4Perform feature extraction on the addition result of the feature map F7 and the feature map F6 corresponding to the training image to obtain the feature map F8 corresponding to the training image, and use the feature map F8 corresponding to the training image as the fused feature map M3 corresponding to the training image; a second fusion feature unit 9 is used to input the feature map F8 corresponding to the training image into the third interpolation unit of the feature extraction module, interpolate the feature map F8 corresponding to the training image through the third interpolation unit, obtain the interpolation result of the feature map F8 corresponding to the training image, and add the interpolation result of the feature map F8 corresponding to the training image and the feature map F6 corresponding to the training image to obtain the fused feature map M2 corresponding to the training image; a first fusion feature unit 10 is used to input the fused feature map M2 corresponding to the training image into the second interpolation unit of the feature extraction module, interpolate the fused feature map M2 corresponding to the training image through the second interpolation unit The feature map M2 is interpolated to obtain the interpolation result of the fused feature map M2 corresponding to the training image, and the interpolation result of the fused feature map M2 corresponding to the training image is added to the feature map F4 corresponding to the training image to obtain the fused feature map M1 corresponding to the training image; the zeroth fusion feature unit 11 is used to input the fused feature map M1 corresponding to the training image into the first interpolation unit of the feature extraction module, and the fused feature map M1 corresponding to the training image is interpolated by the first interpolation unit to obtain the interpolation result of the fused feature map M1 corresponding to the training image, and the interpolation result of the fused feature map M1 corresponding to the training image is added to the feature map F2 corresponding to the training image to obtain the fused feature map M0 corresponding to the training image; the feature information unit 12 is used to use the fused feature map M3 corresponding to the training image to the fused feature map M0 as the feature information corresponding to the training image.

[0173] See also Figure 18 In some embodiments, the Stage 1, Stage 2, Stage 3, Stage 4, Stage 1_1 、Stage 2_2 、Stage 3_3 、Stage 4_4 Any one of them is a target Stage, and the feature extraction module 102 may include: a residual block unit 201, used to process the image input to the target Stage respectively through multiple residual blocks of the target Stage; a processing result unit 202, used to add the processing results of each residual block through the first addition layer of the target Stage; and an image addition unit 203, used to add the addition result of the processing results of each residual block to the image input to the target Stage through the second addition layer of the target Stage.

[0174] See also Figure 19 In some embodiments, the residual block unit 201 may include: a first convolution subunit 301, used to perform convolution processing on the image input to the target stage through the first convolution layer of each residual block of the target stage, to obtain a first convolution result corresponding to each residual block; a second convolution subunit 302, used to perform convolution processing on the first convolution result corresponding to each residual block through the deformable convolution layer of each residual block of the target stage, to obtain a deformable convolution result corresponding to each residual block; a third convolution subunit 303, used to perform convolution processing on the deformable convolution result corresponding to each residual block through the second convolution layer of each residual block of the target stage.

[0175] See also Figure 20 In some embodiments, the image prediction module 103 may include: a reference bounding box unit 401, which is used to input the feature information corresponding to the training image into the RPN network of the preset deep neural network, and perform candidate box generation and bounding box regression through the RPN network to obtain the reference bounding box information corresponding to the training image; a first head structure unit 402, which is used to input the feature information and reference bounding box information corresponding to the training image into the first head structure of the prediction module, and perform classification, bounding box regression, fine-grained classification and segmentation through the first head structure to obtain the first category information, first bounding box information, first fine-grained category information and first contour information corresponding to the training image; a second head structure unit 403, which is used to input the feature information and first bounding box information corresponding to the training image into the first head structure of the prediction module, and perform classification, bounding box regression, fine-grained classification and segmentation through the first head structure to obtain the first category information, first bounding box information, first fine-grained category information and first contour information corresponding to the training image. The second head structure of the prediction module performs classification, bounding box regression, fine-grained classification and segmentation through the second head structure to obtain second category information, second bounding box information, second fine-grained category information and second contour information corresponding to the training image; the third head structure unit 404 is used to input the feature information and second bounding box information corresponding to the training image into the third head structure of the prediction module, and performs classification, bounding box regression, fine-grained classification and segmentation through the third head structure to obtain third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image; the prediction information unit 405 is used to obtain prediction information corresponding to the training image based on the third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image.

[0176] See also Figure 21In some embodiments, with any one of the first to third head structures as the target head structure, the image prediction module 103 may include: a classification and regression unit 501, for performing classification and bounding box regression through the Detection Head of the target head structure; a fine-grained classification unit 502, for performing fine-grained classification through the Attributes Head of the target head structure; and a segmentation unit 503, for performing segmentation through the Mask Head of the target head structure.

[0177] See also Figure 22-24 In some embodiments, the classification and regression unit 501 may include: a first processing subunit 601, for performing ROI Pooling processing through the ROI Pooling layer of the Detection Head of the target head structure; a second processing subunit 602, for performing convolution processing through the convolution layer of the Detection Head of the target head structure; a third processing subunit 603, for performing full connection processing through the first fully connected layer of the Detection Head of the target head structure; a fourth processing subunit 604, for performing classification and bounding box regression through the second fully connected layer of the Detection Head of the target head structure; the fine-grained classification unit 502 may include: a fifth processing subunit 701, for performing ROI Align processing through the ROI Align layer of the Attributes Head of the target head structure; a sixth processing subunit 702, for performing convolution processing through the convolution layer of the Attributes Head of the target head structure; a seventh processing subunit 703, for performing full connection processing through the first fully connected layer of the Attributes Head of the target head structure; an eighth processing subunit 704, for performing full connection processing through the Attributes Head of the target head structure The second fully connected layer of the Head performs fine-grained classification; the segmentation unit 503 may include: a mask sub-unit 801, which is used to perform ROI Align processing through the ROI Align layer of the Mask Head of the target head structure to obtain mask features; a semantic segmentation sub-unit 802, which is used to perform semantic segmentation processing through the semantic segmentation branch of the Mask Head of the target head structure to obtain semantic segmentation features; a fusion feature sub-unit 803, which is used to perform ROI Align processing on the semantic segmentation features and the bounding box information of the target head structure input through the ROI Align layer of the Mask Head of the target head structure to obtain a fusion feature; a feature addition sub-unit 804, which is used to add the one fusion feature and the mask feature through the addition layer of the Mask Head of the target head structure.

[0178] In some embodiments, the fourth processing sub-unit 604 can be used to: perform classification through the classification network FC-head of the second fully connected layer of the Detection Head of the target head structure; and perform bounding box regression through the regression network Conv-head of the second fully connected layer of the Detection Head of the target head structure.

[0179] In some embodiments, the classification and regression unit 501 can be used to: use a detection algorithm based on spatial adaptive decoupling between tasks, perform classification and bounding box regression through the Detection Head of the target head structure, wherein for the classification task and regression task, each adaptive proposal and feature extractor is learned respectively.

[0180] In some embodiments, the second head structure unit 403 can be used to: input the feature information, first bounding box information and first contour information corresponding to the training image into the second head structure of the prediction module, and perform classification, bounding box regression, fine-grained classification and segmentation through the second head structure to obtain the second category information, second bounding box information, second fine-grained category information and second contour information corresponding to the training image; the third head structure unit 404 can be used to: input the feature information, second bounding box information and second contour information corresponding to the training image into the third head structure of the prediction module, and perform classification, bounding box regression, fine-grained classification and segmentation through the third head structure to obtain the third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image.

[0181] See also Figure 25 In some embodiments, the feature extraction module 102 may include: an image determination unit 901, used to determine multiple training images including the training image; an image scaling unit 902, used to scale the long side of the width and high and low sides of each training image to a preset length value, and scale the short side of the width and high and low sides of each training image to any value in a preset length range; an image filling unit 903, used to use the maximum value of the short sides in the multiple training images as a reference value, and fill the short sides of the remaining training images to the reference value; a first information unit 904, used to input the multiple training images into the feature extraction module in batches, and perform feature extraction on the multiple training images through the feature extraction module to obtain feature information corresponding to the multiple training images; a second information unit 905, used to obtain feature information corresponding to the training image based on the feature information corresponding to the multiple training images.

[0182] See also Figure 26The embodiment of the present application also provides a fine-grained clothing segmentation device, the specific implementation method of which is consistent with the implementation method and technical effects recorded in the embodiment of the above-mentioned fine-grained clothing segmentation method, and some contents will not be repeated here.

[0183] The clothing fine-grained segmentation device includes: an image to be segmented module 21, used to obtain an image to be segmented; an image segmentation module 22, used to input the image to be segmented into a clothing fine-grained segmentation model to obtain prediction information corresponding to the image to be segmented; wherein the clothing fine-grained segmentation model is trained using any of the above-mentioned model training methods.

[0184] See also Figure 27 , an embodiment of the present application further provides an electronic device 200, which includes at least one memory 210, at least one processor 220, and a bus 230 connecting different platform systems.

[0185] The memory 210 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 211 and / or a cache memory 212 , and may further include a read-only memory (ROM) 213 .

[0186] Among them, the memory 210 also stores a computer program, which can be executed by the processor 220, so that the processor 220 executes the steps of the model training method or the clothing fine-grained segmentation method in the embodiment of the present application. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiment of the above-mentioned model training method or the clothing fine-grained segmentation method, and some contents will not be repeated here.

[0187] Memory 210 may also include a utility 214 having at least one program module 215, such program modules 215 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0188] Accordingly, the processor 220 may execute the aforementioned computer program and the utility 214 .

[0189] The bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0190] The electronic device 200 may also communicate with one or more external devices 240, such as a keyboard, pointing device, Bluetooth device, etc., and may also communicate with one or more devices capable of interacting with the electronic device 200, and / or any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may be performed via an input / output interface 250. Furthermore, the electronic device 200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0191] The present application also provides a computer-readable storage medium for storing a computer program or a fine-grained clothing segmentation model. When executed, the computer program implements the steps of the model training method or fine-grained clothing segmentation method in the embodiments of the present application. The specific implementation method is consistent with the implementation method and technical effects described in the embodiments of the above-mentioned model training method or fine-grained clothing segmentation method, and some of the contents are not repeated here. The fine-grained clothing segmentation model is trained using the model training method in the embodiments of the present application.

[0192] Figure 28 The program product 300 provided in this embodiment for implementing the above-mentioned model training method is shown. It can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product 300 of the present invention is not limited to this. In this application, a readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, device or device. The program product 300 can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0193] A computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, or any suitable combination thereof. The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0194] This application is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and has complied with the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings of this application are only preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to those of this application, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A model training method, characterized in that: The model training method includes: Acquire a training image and its corresponding annotation information, wherein the annotation information corresponding to the training image is used to indicate a labeled bounding box, a labeled fine-grained category, and a labeled outline of clothing in the training image; Input the training image into a feature extraction module of a preset deep neural network, perform feature extraction on the training image through the feature extraction module to obtain feature information corresponding to the training image, wherein the feature extraction module adopts an architecture combining CBNet and FPN, and CBNet includes two cascaded ResNe XT-101s; Inputting feature information corresponding to the training image into a prediction module of the preset deep neural network, segmenting the training image by the prediction module to obtain prediction information corresponding to the training image, wherein the prediction information corresponding to the training image is used to indicate a predicted bounding box, a predicted fine-grained category, and a predicted outline of clothing in the training image; Using the prediction information and annotation information corresponding to the training image, the preset deep neural network is trained to obtain a fine-grained clothing segmentation model; The step of inputting the training image into a feature extraction module of a preset deep neural network, performing feature extraction on the training image by the feature extraction module, and obtaining feature information corresponding to the training image includes: The feature extraction module includes multiple stages and multi-level interpolation units; each stage performs feature extraction on the input training image or the feature map of the previous stage to generate feature maps of different levels; Through the interpolation and addition operations of the multi-level interpolation unit, the fused feature map generated in the previous step is added and interpolated with the corresponding feature map to gradually generate multiple fused feature maps; Use multiple fused feature maps as feature information of training images; The model training method further includes: Perform data augmentation on the original data to obtain augmented data; Input the augmented data into the HTC network and use the HTC network to output the corresponding feature information; The corresponding feature information is input into the head structure, and the output result is obtained using the head structure, wherein the head structure includes a Detection Head, an Attributes Head, and a Mask Head. A detection algorithm based on spatial adaptive decoupling between tasks is added to the Detection Head of the head structure.

2. The model training method according to claim 1, characterized in that Inputting the training image into a feature extraction module of a preset deep neural network, performing feature extraction on the training image by the feature extraction module to obtain feature information corresponding to the training image, includes: Input the training image into Stage 1 of the feature extraction module, perform feature extraction on the training image through Stage 1, and obtain a feature map F1 corresponding to the training image; Input the feature map F1 corresponding to the training image into the Stage of the feature extraction module 1_1 , through Stage 1_1 Perform feature extraction on the feature map F1 corresponding to the training image to obtain a feature map F2 corresponding to the training image; Input the feature map F1 corresponding to the training image into Stage 2 of the feature extraction module, perform feature extraction on the feature map F1 corresponding to the training image through Stage 2, and obtain a feature map F3 corresponding to the training image; The feature map F3 and feature map F2 corresponding to the training image are added and input into the Stage of the feature extraction module 2_2 , through Stage 2_2 Perform feature extraction on the sum of the feature map F3 and the feature map F2 corresponding to the training image to obtain a feature map F4 corresponding to the training image; Input the feature map F3 corresponding to the training image into Stage 3 of the feature extraction module, perform feature extraction on the feature map F3 corresponding to the training image through Stage 3, and obtain a feature map F5 corresponding to the training image; The feature map F5 and feature map F4 corresponding to the training image are added and input into the Stage of the feature extraction module 3_3 , through Stage 3_3 Perform feature extraction on the sum of the feature map F5 and the feature map F4 corresponding to the training image to obtain a feature map F6 corresponding to the training image; Input the feature map F5 corresponding to the training image into Stage 4 of the feature extraction module, perform feature extraction on the feature map F5 corresponding to the training image through Stage 4, and obtain a feature map F7 corresponding to the training image; The feature map F7 and feature map F6 corresponding to the training image are added and input into the Stage of the feature extraction module 4_4 , through Stage 4_4 Perform feature extraction on the sum of the feature map F7 and the feature map F6 corresponding to the training image to obtain a feature map F8 corresponding to the training image, and use the feature map F8 corresponding to the training image as the fused feature map M3 corresponding to the training image; Inputting the feature map F8 corresponding to the training image into the third interpolation unit of the feature extraction module, interpolating the feature map F8 corresponding to the training image through the third interpolation unit to obtain an interpolation result of the feature map F8 corresponding to the training image, and adding the interpolation result of the feature map F8 corresponding to the training image to the feature map F6 corresponding to the training image to obtain a fused feature map M2 corresponding to the training image; Inputting the fused feature map M2 corresponding to the training image into the second interpolation unit of the feature extraction module, interpolating the fused feature map M2 corresponding to the training image through the second interpolation unit to obtain an interpolation result of the fused feature map M2 corresponding to the training image, and adding the interpolation result of the fused feature map M2 corresponding to the training image to the feature map F4 corresponding to the training image to obtain a fused feature map M1 corresponding to the training image; Inputting the fused feature map M1 corresponding to the training image into the first interpolation unit of the feature extraction module, interpolating the fused feature map M1 corresponding to the training image through the first interpolation unit to obtain an interpolation result of the fused feature map M1 corresponding to the training image, and adding the interpolation result of the fused feature map M1 corresponding to the training image and the feature map F2 corresponding to the training image to obtain a fused feature map M0 corresponding to the training image; The fused feature maps M3 to M0 corresponding to the training image are used as feature information corresponding to the training image.

3. The model training method according to claim 2, characterized in that Based on the Stage 1, Stage 2, Stage 3, Stage 4, Stage 1_1 、Stage 2_2 、Stage 3_3 、Stage 4_4 Any one of them is a target stage, and feature extraction is performed on the training image through the target stage, including: Processing the images input to the target stage respectively through the multiple residual blocks of the target stage; Adding the processing results of each residual block through the first addition layer of the target stage; The addition result of the processing results of each residual block is added to the image input to the target stage through the second addition layer of the target stage.

4. The model training method according to claim 3, characterized in that The processing of the image input to the target stage by the multiple residual blocks of the target stage respectively includes: Performing convolution processing on the input image of the target Stage through the first convolution layer of each residual block of the target Stage, respectively, to obtain a first convolution result corresponding to each residual block; Performing convolution processing on the first convolution results corresponding to each residual block through the deformable convolution layer of each residual block of the target stage to obtain a deformable convolution result corresponding to each residual block; The deformable convolution results corresponding to each residual block are convolved through the second convolution layer of each residual block of the target stage.

5. The model training method according to claim 1, characterized in that Inputting the feature information corresponding to the training image into the prediction module of the preset deep neural network, segmenting the training image by the prediction module, and obtaining the prediction information corresponding to the training image includes: Inputting feature information corresponding to the training image into the RPN network of the preset deep neural network, performing candidate box generation and bounding box regression through the RPN network to obtain reference bounding box information corresponding to the training image; Inputting feature information and reference bounding box information corresponding to the training image into a first header structure of the prediction module, and performing classification, bounding box regression, fine-grained classification, and segmentation through the first header structure to obtain first category information, first bounding box information, first fine-grained category information, and first contour information corresponding to the training image; Inputting the feature information and the first bounding box information corresponding to the training image into a second head structure of the prediction module, and performing classification, bounding box regression, fine-grained classification, and segmentation through the second head structure to obtain second category information, second bounding box information, second fine-grained category information, and second contour information corresponding to the training image; Inputting the feature information and the second bounding box information corresponding to the training image into the third head structure of the prediction module, and performing classification, bounding box regression, fine-grained classification, and segmentation through the third head structure to obtain third category information, third bounding box information, third fine-grained category information, and third contour information corresponding to the training image; Based on the third category information, the third bounding box information, the third fine-grained category information, and the third contour information corresponding to the training image, prediction information corresponding to the training image is obtained.

6. The model training method according to claim 5, characterized in that Using any one of the first to third head structures as a target head structure, performing classification, bounding box regression, fine-grained classification, and segmentation using the target head structure, including: Classification and bounding box regression are performed through the Detection Head of the target head structure; Perform fine-grained classification through the Attributes Head of the target header structure; Segmentation is performed using the Mask Head of the target head structure.

7. The model training method according to claim 6, characterized in that The classification and bounding box regression are performed by the Detection Head of the target head structure, including: Perform ROIPooling processing through the ROIPooling layer of the Detection Head of the target head structure; Perform convolution processing through the convolution layer of the Detection Head of the target head structure; Performing full connection processing through the first fully connected layer of the Detection Head of the target head structure; Classification and bounding box regression are performed through the second fully connected layer of the Detection Head of the target head structure; The fine-grained classification is performed through the Attributes Head of the target header structure, including: Perform ROI Align processing through the ROI Align layer of the Attributes Head of the target head structure; Perform convolution processing through the convolution layer of the Attributes Head of the target head structure; Performing full connection processing through the first fully connected layer of the Attributes Head of the target head structure; Fine-grained classification is performed through the second fully connected layer of the Attributes Head of the target head structure; The segmentation using the Mask Head of the target head structure includes: Performing ROI Align processing through the ROI Align layer of the Mask Head of the target head structure to obtain mask features; Performing semantic segmentation processing through the semantic segmentation branch of the Mask Head of the target head structure to obtain semantic segmentation features; Performing ROI Align processing on the semantic segmentation features and the bounding box information of the input target head structure through the ROI Align layer of the Mask Head of the target head structure to obtain a fusion feature; The primary fusion feature and the mask feature are added through the addition layer of the Mask Head of the target head structure.

8. The model training method according to claim 7, characterized in that: The classification and bounding box regression are performed by the second fully connected layer of the Detection Head of the target head structure, including: Classification is performed through the classification network FC-head of the second fully connected layer of the Detection Head of the target head structure; Bounding box regression is performed through the regression network Conv-he ad of the second fully connected layer of the Detection Head of the target head structure.

9. The model training method according to claim 6, characterized in that: The classification and bounding box regression are performed by the Detection Head of the target head structure, including: A detection algorithm based on spatial adaptive decoupling between tasks is used to perform classification and bounding box regression through the detection head of the target head structure, wherein adaptive proposals and feature extractors are learned for the classification task and the regression task, respectively.

10. The model training method according to claim 5, characterized in that: Inputting the feature information and the first bounding box information corresponding to the training image into the second head structure of the prediction module, performing classification, bounding box regression, fine-grained classification, and segmentation through the second head structure to obtain second category information, second bounding box information, second fine-grained category information, and second contour information corresponding to the training image, includes: Inputting the feature information, first bounding box information, and first contour information corresponding to the training image into a second head structure of the prediction module, and performing classification, bounding box regression, fine-grained classification, and segmentation through the second head structure to obtain second category information, second bounding box information, second fine-grained category information, and second contour information corresponding to the training image; Inputting the feature information and the second bounding box information corresponding to the training image into the third head structure of the prediction module, performing classification, bounding box regression, fine-grained classification and segmentation through the third head structure, and obtaining third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image, including: The feature information, second bounding box information and second contour information corresponding to the training image are input into the third head structure of the prediction module, and classification, bounding box regression, fine-grained classification and segmentation are performed through the third head structure to obtain third category information, third bounding box information, third fine-grained category information and third contour information corresponding to the training image.

11. The model training method according to claim 1, characterized in that Inputting the training image into a feature extraction module of a preset deep neural network, performing feature extraction on the training image by the feature extraction module to obtain feature information corresponding to the training image, includes: determining a plurality of training images including the training image; Scale the width and the long side of the middle of each training image to a preset length value, and scale the width and the short side of the middle of each training image to any value within the preset length range; Taking the maximum value of the short sides of the plurality of training images as a reference value, padding the short sides of the remaining training images to the reference value; Inputting the plurality of training images into the feature extraction module in a batch form, performing feature extraction on the plurality of training images by the feature extraction module to obtain feature information corresponding to the plurality of training images; Based on the feature information corresponding to the plurality of training images, the feature information corresponding to the training images is acquired.

12. A fine-grained clothing segmentation method, characterized in that: The clothing fine-grained segmentation method includes: Obtain the image to be segmented; Inputting the image to be segmented into a clothing fine-grained segmentation model to obtain prediction information corresponding to the image to be segmented; The clothing fine-grained segmentation model is trained using the model training method described in any one of claims 1 to 11.

13. A model training device, characterized in that: The model training device comprises: a data acquisition module, configured to acquire a training image and its corresponding annotation information, wherein the annotation information corresponding to the training image is used to indicate a labeled bounding box, a labeled fine-grained category, and a labeled outline of the clothing in the training image; A feature extraction module is used to input the training image into a feature extraction module of a preset deep neural network, and extract features from the training image through the feature extraction module to obtain feature information corresponding to the training image. The feature extraction module adopts an architecture that combines CBNet and FPN, and CBNet includes two cascaded ResNeXT-101s; An image prediction module, configured to input feature information corresponding to the training image into a prediction module of the preset deep neural network, segment the training image through the prediction module, and obtain prediction information corresponding to the training image, wherein the prediction information corresponding to the training image is used to indicate a predicted bounding box, a predicted fine-grained category, and a predicted outline of clothing in the training image; A model training module, configured to train the preset deep neural network using the prediction information and annotation information corresponding to the training image to obtain a fine-grained clothing segmentation model; Wherein, the feature extraction module includes: The feature extraction module includes multiple stages and multi-level interpolation units; each stage performs feature extraction on the input training image or the feature map of the previous stage to generate feature maps of different levels; Through the interpolation and addition operations of the multi-level interpolation unit, the fused feature map generated in the previous step is added and interpolated with the corresponding feature map to gradually generate multiple fused feature maps; Use multiple fused feature maps as feature information of training images; Perform data augmentation on the original data to obtain augmented data; Input the augmented data into the HTC network and use the HTC network to output the corresponding feature information; The corresponding feature information is input into the head structure, and the output result is obtained using the head structure, wherein the head structure includes a Detection Head, an Attributes Head, and a Mask Head. A detection algorithm based on spatial adaptive decoupling between tasks is added to the Detection Head of the head structure.

14. A fine-grained clothing segmentation device, characterized in that: The clothing fine-grained segmentation device includes: The image to be segmented module is used to obtain the image to be segmented; An image segmentation module is used to input the image to be segmented into a clothing fine-grained segmentation model to obtain prediction information corresponding to the image to be segmented; The clothing fine-grained segmentation model is trained using the model training method described in any one of claims 1 to 10.

15. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the model training method according to any one of claims 1 to 11 or the steps of the clothing fine-grained segmentation method according to claim 12.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program or a fine-grained clothing segmentation model; When the computer program is executed by a processor, the steps of the model training method according to any one of claims 1 to 11 or the steps of the clothing fine-grained segmentation method according to claim 12 are implemented; The clothing fine-grained segmentation model is trained using the model training method described in any one of claims 1-11.

Images