Visual semantic embedding method and system based on data enhancement

Abstract

The invention discloses a visual semantic embedding method and system based on data enhancement, and belongs to the technical field of deep learning. Performing target identification on the image by using the first network model to select a plurality of image areas; extracting fine-grained features of the image region by using a second network model, and obtaining fine-grained representation of the image in the common embedding space through fine-tuning network mapping; first semantic graph reasoning is carried out, and unified pooling operation is carried out; using a first extraction model to extract word vector representation related to the text context; performing fine-tuning mapping to the common embedding space by using a second extraction model to obtain word vector representation of the text in the common embedding space; second semantic graph reasoning is carried out, and unified pooling operation is carried out; and performing semantic alignment on the first pooling result and the second pooling result, and enhancing data in model training. According to the method, the semantic association in the modal can be well constructed, and the unified representation can be generated in the common embedded subspace.

Description

technical field [0001] The invention belongs to the technical field of deep learning, and in particular relates to a visual semantic embedding method and system based on data enhancement. Background technique [0002] Image-text cross-modal entity discrimination aims to find image-text pairs with the same semantics. However, since images and texts are data belonging to different modalities, it poses a great challenge for the semantic alignment of images and texts [0003] Image-text cross-modal entity recognition methods can be divided into traditional methods and deep learning methods. Traditional methods generally use statistical analysis methods to learn the mapping matrix of cross-modal data by statistically analyzing the distribution of different modal information to achieve semantic alignment. Among them, the most representative method is Canonical Correlation Analysis (CCA). Researchers have proposed a variety of CCA-based methods, such as KCCA, Multi-view CCA and ...

AI Technical Summary

Problems solved by technology

[0007] Technical problem: Aiming at the problem that the existing visual semantic embedding methods are difficult to build semantic associations within the modal or generate a unified representation, the present invention provides a data-enhanced visual semantic embedding method and system, by reconstructing visual semantic embedding The whole process, and the generalization ability of the model is improved through data enhancement, so that the present invention can well build semantic associations within modals and generate unified representations

Images