A cross-modal image-text association anomaly detection method

Abstract

The invention belongs to the technical field of computer multimedia, and specifically relates to a cross-modal image-text association anomaly detection method. The present invention judges whether the image-text pair association is abnormal through the following steps: 1) In the image multi-label classification stage, the image is input into a CNN-RNN-based codec to accurately extract the label information of the image; 2) Text multi-label In the classification stage, the text is input into the BiLSTM-based network to obtain the label information of the text; 3) in the association anomaly detection stage, the classification results of the image and text are fused to determine whether the image-text pair is abnormal. The method proposed by the invention can accurately realize the abnormal detection of image-text pair association, and the model has strong robustness.

Description

technical field [0001] The invention relates to a cross-modal image-text correlation abnormal detection method, which belongs to the technical field of computer multimedia. Background technique [0002] With the application of related technologies such as mobile Internet, Internet of Things, and social media networks, the amount of data that can be collected and analyzed is growing rapidly, and the carrier of information is also developing from traditional text records to richer multimedia records. Unlike written records, which contain a large number of abstract concepts, the information content of multimedia is mostly figurative sensory information description, and how to let artificial intelligence learn to understand multimedia content can associate abstract text semantic information with intuitive multimedia content. has become a subject of increasing attention in recent years. Image-text association anomaly detection is an important research content in this topic. In t...

AI Technical Summary

Problems solved by technology

[0003] However, it is very difficult to model the relationship between cross-modal data. There are great differences in data between different modalities. For example, the representation of images is continuous, while the representation of text is usually discrete, so it is difficult to It is difficult to establish links between data of different modalities at this level

Some traditional image-text association anomaly detection algorithms based on machine learning can map heterogeneous data features into a common latent space by introducing typical correlation analysis methods. In the public latent space, different modal data The correlation coefficient can be directly calculated between different modal data. Due to the heterogeneity of different modal data, it is difficult for these 'shallow' models to fully learn the correlation between different modal data.

Images