Coherence story generation system and method based on vision and theme collaborative attention

A generation system and attention technology, applied in the direction of neural learning methods, biological neural network models, resources, etc., can solve the problems of theme coherence and expression diversity to be further improved, so as to maintain theme coherence and overcome content theme inconsistency Coherent, optimized build-quality effects

Pending Publication Date: 2021-12-10
TONGJI UNIV
View PDF7 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] However, the stories generated by the above two types of methods still need to be further improved in terms of theme coherence and expression diversity.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Coherence story generation system and method based on vision and theme collaborative attention
  • Coherence story generation system and method based on vision and theme collaborative attention
  • Coherence story generation system and method based on vision and theme collaborative attention

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0053] The invention will be described in detail below with reference to the accompanying drawings and specific examples. The present embodiment is implemented in terms of the technical solution of the present invention, and a detailed embodiment and a specific operation process are given, but the scope of the present invention is not limited to the following examples.

[0054] The present invention provides a method of generating a coincided with the coincidence of the synergistic attention of visual and the subject, and can be applied to early education, guide, human-machine interaction, by bridging the semantic gap between the two modal data between the computer vision and the natural language. Security monitoring, automatic driving, traffic monitoring and robotic visual field, such as figure 1 and 2 As shown, including the following steps:

[0055] 1) Image album feature encoding module: Sequentially use the image album feature encoding module of the image album in each album ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a coherence story generation system and method based on vision and theme collaborative attention. The method comprises the following steps: 1) extracting photo album feature vectors and time dynamic information; 2) obtaining topic probability distribution of each description statement and predicting topic distribution information in each image in the photo album; 3) generating an image description statement with theme coherence based on vision and theme collaborative attention; and 4) carrying out phrase beam search on the image description statement through a phrase beam search algorithm considering n-gram diversity, so that the accuracy and diversity of visual story description expression are improved. Compared with the prior art, the invention has the advantages that the theme coherence of the description statement is enhanced, the expression diversity of the story text is improved, and the generation quality of the visual story is optimized.

Description

Technical field [0001] The present invention relates to the field of computer visual story description, in particular, involving a coherent story generation system and method based on visual and topical synergies. Background technique [0002] At present, although the visual description method based on deep learning has made a series of progress, the image album story generating task has put forward higher requirements for the expression diversity of the topic coherence and description statements of the description. [0003] At this stage, the research method based on deep learning image album generating method can be divided into the following two categories: [0004] (1) Image story generation model based on strengthening learning: Introducing strengthening learning in the training stage of the model to improve the evaluation index value of the generated story; [0005] (2) Image Album Story Generation Model Based on Visual Characteristics: The expression diversity of the gener...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F40/166G06F40/216G06F16/35G06K9/62G06N3/04G06N3/08G06Q10/06
CPCG06F40/166G06F40/216G06F16/35G06N3/08G06Q10/06393G06N3/044G06N3/045G06F18/2132
Inventor 王瀚漓谷金晶
Owner TONGJI UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap