Joint multi-modal entity relationship extraction method and system based on information representation and semantic alignment

By employing multi-angle visual representation and fine-grained filtering of irrelevant features, combined with progressive modal alignment, the problems of unutilized visual information and irrelevant interference in images are solved, thereby improving the accuracy of multimodal entity relationship extraction.

CN121859903AActive Publication Date: 2026-04-14YANBIAN UNIV
2 Cites 4 Cited by

Patent Information

Application Number
CN202512055231.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-31
Publication Date
2026-04-14
Estimated Expiration
2045-12-31

AI Technical Summary

Technical Problem

Existing joint multimodal entity relationship extraction methods ignore the rich visual information in images, leading to a decrease in the accuracy of task decision-making. Furthermore, image-text alignment methods lack explicit relevance and irrelevant visual information interference, affecting model decision-making.

Method used

By employing multi-angle visual representation, fine-grained relevance filtering, and progressive modal alignment, the model accuracy is improved. Multiple encoders are used to acquire visual and textual features. Combined with a fine-grained guided multi-layer relevance mapping mechanism and a routing weighting function, irrelevant features are filtered out to perform interactive modeling of images and text.

Benefits of technology

It improves the accuracy of joint multimodal entity relationship extraction, effectively utilizes image information, reduces irrelevant interference, and enhances the accuracy of task decision-making.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a joint multi-modal entity relationship extraction method and system based on information representation and semantic alignment, and the method comprises the steps: obtaining sample data composed of an original image and a text, and obtaining the feature representation of visual information and text information based on the sample data; aligning the feature representation of the visual information and the feature representation of the text information by using a progressive modal semantic alignment strategy; by introducing a multi-layer correlation mapping mechanism guided by fine granularity, the correlation coefficient represented by the features of the aligned visual information and text information is judged, and features irrelevant to a task core are filtered; performing visual representation and text interaction by using a multi-modal interaction module to obtain multi-modal semantic features; carrying out weighted mapping on the multi-modal semantic features by utilizing a routing weighting function, and finally obtaining multi-modal feature representation; and sending the multi-modal feature representation into a word pair relation label extractor, and extracting an entity, an entity relation and an entity attribute quintuple.
Need to check novelty before this filing date? Find Prior Art