Multi-modal media asset intelligent auditing system and method based on unified semantic space

Abstract

The invention discloses a multi-modal media asset intelligent auditing system and method based on a unified semantic space. The method belongs to the technical field of media asset auditing. The method is characterized in that text / image / audio modal weights are dynamically distributed through an eight-head self-attention architecture, and the problem of misjudgment caused by traditional static fusion is solved by combining a GRU gating network to suppress modal contradictions (cosine similarity variance gt, weight drop 30% + when 0.3); a 768-dimensional unified semantic space is constructed, BERT text features, ResNet-50 image features and Wav2Vec2.0 audio features are aligned through a cross-modal projection layer, and 92% cross-modal semantic alignment accuracy is achieved; and starting incremental training by adopting a double-threshold triggering mechanism (the misjudgment rate is gt, 15% and the sample size is greater than or equal to 50 cases), and realizing 30-sample / second real-time reasoning on edge equipment. The MSCOCO2017 public test set violence-related misjudgment rate is reduced from 18% to 7.2%, the sensitive tag recall rate reaches 98%, the auditing efficiency is improved by 5 times, and a social media high-concurrency auditing scene is effectively supported.