Layering-based graph and text semantic consistency intelligent evaluation method

A consistent and semantic technology, applied in semantic analysis, natural language data processing, instruments, etc., can solve problems such as inability to evaluate the semantic consistency between images and text, and cannot completely cover image information without considering reference sentences, so as to achieve applicable scenarios wide effect

Active Publication Date: 2022-02-08
HANGZHOU INNOVATION RES INST OF BEIJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Most of the existing studies on the semantic consistency between pictures and texts are local semantic consistency calculations for pictures and single sentence texts, which treat the importance of each word equally, and cannot well explain the semantic elements (ontology, attribute, etc.) , relationship) on the semantic consistency between the image and the text; the rule-based evaluation methods all assume that the reference sentence can express the image content well, and do not consider the situation that the reference sentence cannot completely cover the image information, and cannot evaluate the graph and text from a global perspective. Semantic Consistency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Layering-based graph and text semantic consistency intelligent evaluation method
  • Layering-based graph and text semantic consistency intelligent evaluation method
  • Layering-based graph and text semantic consistency intelligent evaluation method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] In order to understand the purpose, technical solution and advantages of the present invention more clearly, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The specific embodiments described here are only used to further explain the present invention, but not to limit the present invention.

[0049] Such as figure 1 As shown, the steps of the intelligent evaluation method for semantic consistency between images and their titles based on layers in the present invention are as follows: first, feature extraction is performed on images and texts, and an attention mechanism including bottom-up, channel and space is used for image feature extraction . Next, the semantic classification of ontology, attributes, and relationships is carried out on image features and text features, and then the attention cross-mapping of "image to text" and "text to image" is carried out by the SCAN method to obtain ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a layering-based graph and text semantic consistency intelligent evaluation method. The method comprises the following steps of: representing image and text features by adopting deep learning; understanding an image scene by adopting a bottom-up, space and channel mixed attention mechanism, and taking information such as illumination, a visual angle, a scale and shielding into consideration to extract remarkable image region features; considering semantic consistency from semantics, wherein evaluation index models of four levels of ontology, attribute, relationship and global are constructed for hierarchical evaluation; finally, giving the final semantic consistency degree of the image and the title content of the image by weighting, wherein the evaluation result considers the information of the image and the semantic element level, and the different degrees of the influence of each level on the semantic consistency of the image and the image title can be reflected more objectively and clearly.

Description

technical field [0001] The invention relates to the fields of computer simulation and natural language processing, in particular to a hierarchically-based intelligent evaluation method for the semantic consistency of images and texts, which can be used for the semantic consistency evaluation of images and image titles in multimedia content such as news, comments, and microblogs . Background technique [0002] Existing multimedia content includes modal information such as images, texts, audio, and video, among which image and text content is the most common multi-modal expression form of multimedia content. False content deceives and induces the audience by "adding and subtracting" facts for some improper purposes, and will have a negative impact on the audience and society after dissemination. Image and text semantic consistency evaluation is to determine whether the image and text in the multimedia content are consistent from the semantic content. If they are not consisten...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F40/279G06F40/30G06F40/216
CPCG06F40/279G06F40/30G06F40/216
Inventor 郭雷李玉红王岩乔建忠
Owner HANGZHOU INNOVATION RES INST OF BEIJING UNIV OF AERONAUTICS & ASTRONAUTICS
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap