Fashion field-based end-to-end image semantic description method and system

Abstract

The invention discloses a fashion field-based image language description generation method and system. The method comprises the following steps of preparing a data set; extracting features; carrying out downsampling operation on commodity features by using pooling layers of multiple scales, and sending the pooled features into a multi-layer perceptron to learn high-level visual features; designing a multi-scale transform network as a feature encoder for each visual high-level feature, and learning an interaction relationship between the interior of each feature; and calculating relative contribution values of the multi-scale features to the description by using a decoder with a gated attention mechanism, fusing the relative contribution values, and predicting the sentences. According to the method, commodity pictures of a real scene are described, so that the model has higher robustness and application value. By using multi-scale features of commodities and a gating attention mechanism designed for the fashion field, description expression generated by the model is more accurate and more humanized. Due to the end-to-end model framework, the landing threshold is lower.

Description

technical field [0001] The invention belongs to the cross field of computer vision and natural language processing, and in particular relates to a method and system for generating a semantic description of a picture. Background technique [0002] Image Semantic Description (Image Caption) is a comprehensive task that combines computer vision and natural language processing, and its goal is to translate a picture into a description. This task not only needs to use the model to understand the content of the picture, capture the semantic information of the image, but also need to use natural language to generate readable sentences. Automatically generating semantic descriptions for images can not only help visually impaired people understand image content by converting image content into text and then audio, but also assist in understanding complex images such as spectrograms and remote sensing images. In addition, tasks related to image understanding, such as image-text retri...

AI Technical Summary

Problems solved by technology

At the same time, since search engines and recommendation systems usually use keywords to search, such descriptions may obtain higher click-through rates

Therefore, the particularity of the fashion field leads to the fact that the image semantic description method directly transferred to the conventional field cannot achieve good results on this multi-attribute description. An image semantic description method specially designed for the fashion field is urgently needed

Images