Image captioning with weak supervision

Abstract

Techniques for image captioning with weak supervision are described herein. In implementations, weak supervision data regarding a target image is obtained and utilized to provide detail information that supplements global image concepts derived for image captioning. Weak supervision data refers to noisy data that is not closely curated and may include errors. Given a target image, weak supervision data for visually similar images may be collected from sources of weakly annotated images, such as online social networks. Generally, images posted online include ''weak'' annotations in the form of tags, titles, labels, and short descriptions added by users. Weak supervision data for the target image is generated by extracting keywords for visually similar images discovered in the different sources. The keywords included in the weak supervision data are then employed to modulate weights applied for probabilistic classifications during image captioning analysis.

Description

Background technique [0001] Automatically generated natural language descriptions of images continue to attract interest due to practical applications for image search, accessibility for the visually impaired, and management of image acquisition. Traditional techniques for image processing do not support high-precision natural language subtitles and image search due to the limitations of traditional image tagging and search algorithms. This is because conventional techniques only associate labels with images, but do not define the relationship between labels or between labels and images themselves. Alternatively, traditional techniques can include using a top-down approach, in which the entire "gist" of an image is first obtained and then refined into appropriate descriptive words or captions through language modeling and sentence generation. However, this top-down approach does not work well in capturing fine details of an image, such as local objects, attributes, and region...

AI Technical Summary

Problems solved by technology

However, this top-down approach does not work well in capturing fine details of an image, such as local objects, attributes, and regions that contribute to an accurate description of the image.

As such, it can be difficult to generate accurate and complex image captions using traditional methods, such as "person feeding a child holding a toy in a high chair"

Therefore, subtitles generated using conventional techniques may omit important image details, which makes it difficult for users to search for a specific image and fully understand the content of the image based on the associated subtitles

Images