News event searching method and system based on multistage image-text semantic alignment model

Abstract

The multi-level vision-text semantic alignment model MSAVT used for image-text matching is provided, the news event retrieval method based on the multi-level vision-text semantic alignment model MSAVT used for image-text matching is provided, news event cross-modal image-text search is achieved, and the current news retrieval requirement is met. The image-text alignment precision of the cross-modal retrieval model provided by the invention is higher, and when the cross-modal retrieval model is applied to news event cross-modal image-text retrieval, indexes such as recall rates of multiple levels, average accuracy and the like are remarkably improved. And meanwhile, a pre-trained BERT model is introduced to extract text features, so that the generalization performance of the algorithm is improved. The model adopts a public space feature learning method, vector representations of images and texts can be independently obtained, namely, vector representations of retrieval results can be stored in advance, retrieval time is short, and the method can be applied to actual scenes.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a news event search method based on a multi-level image-text semantic alignment model. Background technique [0002] Cross-modal retrieval [0003] Mode refers to the existence form of data, such as text, picture, video, etc. Cross-modal retrieval aims to use data from one modality as a query to retrieve data from another modality. The most common is image-text retrieval (image-text retrieval). Given a piece of text, retrieve related images, or conversely given an image, retrieve related text. The main difficulty of cross-modal retrieval lies in the "heterogeneous gap". The heterogeneous gap refers to the inconsistency between the query input and the retrieval result, and the two data are in different distribution spaces. Although the high-level semantics are related, the similarity cannot be directly measured. Therefore, the focus of research is how to represent ...

AI Technical Summary

Problems solved by technology

[0067] Although the combination of graphic and text features can provide more cross-feature information for the hidden layer of the model, it is impossible to use the top-level embedding vector to independently represent the input signals of images and text

Compared with common space feature learning methods, the search process of cross-modal similarity measurement methods is time-consuming

Specifically, when the user enters a text query q, the system needs to calculate the feature combination of all images and q online to obtain the similarity score between q and each image. The calculation performance is a huge bottleneck, making it impossible to apply in practice.

Images