Cross-language description-oriented adversarial data enhancement method, system and storage medium

Abstract

The invention discloses a cross-language description-oriented adversarial data enhancement method, a system and a storage medium. The method comprises the following steps of obtaining a clean image-text pair data set; using the sequence-to-sequence model to generate a text antagonism sample; training an image description generation model: if the current training stage is an adversarial training stage, generating an image adversarial sample, then expanding an image-text pair, then training the model by using an expanded image-text pair data set, and optimizing the model according to a joint loss function; if the current training stage is a non-adversarial training stage, training the model by using a clean image-text pair data set, and optimizing the model according to a loss function; and obtaining a trained image description generation model and an expanded image-text pair data set. Through an easy-to-operate data enhancement mode, the data set is expanded, and the robustness and performance of the image description generation model are improved.

Description

technical field [0001] The invention belongs to the technical field of data enhancement, and in particular relates to an adversarial data enhancement method, system and storage medium for cross-language description. Background technique [0002] Algorithms can directly benefit from the size of the data set, and the fit and robustness of models trained on large-scale data sets are often better than models obtained from small-scale data sets. For image description tasks in small languages, if you want to achieve the same performance as the English image description task, the first challenge you need to face is the acquisition of large-scale data sets. [0003] In order to ensure the quality, manually labeling the dataset is the best method, but this method is extremely time-consuming. In order to balance the model performance and the cost of manually labeling datasets, data augmentation methods are usually used to expand datasets. Data augmentation has been widely used in the...

AI Technical Summary

Problems solved by technology

Both geometric transformations and randomly cropped images can affect the accuracy of generated sentences

When it comes to orientation information, such as a person standing on the left side of the table, flipping or cropping the picture may cause the information obtained by the model to be inaccurate and comprehensive, resulting in problems with the generated text

In terms of text, it is also challenging to propose general language conversion rules, and general data enhancement techniques in Natural Language Processing (NLP) have not been fully explored

Images