Image description generating method based on neural network and image attention focuses

A neural network and image description technology, applied in the field of computer vision, can solve problems such as high computational complexity, lack of attention between image objects or attributes, etc.

Active Publication Date: 2017-05-31
SYSU CMU SHUNDE INT JOINT RES INST +1
View PDF3 Cites 35 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, RNNs rely on good semantic expression input. For the field of image understanding, the degree of attention between image objects or ...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Image description generating method based on neural network and image attention focuses
  • Image description generating method based on neural network and image attention focuses
  • Image description generating method based on neural network and image attention focuses

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] Such as figure 1 As shown, an image description generation method based on neural network and image concerns, including the following steps:

[0044]S1: Construct a multimodal model of the image of frame t at each moment:

[0045] 1) The text description information of the marked images in the training set is divided into a single word set, and the corresponding word is represented by a one-hot vector, which is used as the input of the text module of the model, and is projected into a dense word expression space through two embedding layers to become a semantic The word expression vector W t ;

[0046] 2) The word expression vector is used for the input of the circular convolutional neural network RNN ​​at a certain time frame t to perform the calculation of the circular convolutional neural network RNN, and the circular layer of the frame t at this time activates R t is the word expression vector of the current time frame and the recurrent layer R of the previous ti...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an image description generating method based on a neural network and image focuses. The method is characterized in that an original one-layer word embedding structure is replaced by a two-layer word embedding structure, and accordingly word expression can be learned effectively; image feature expression is directly used as the input of an m-RNN model, and accordingly the capacity of a circulating layer can be fully utilized, and the small-dimension circulating layer can be used; by a decision soft attention mechanism, the attention of an image salient region is reflected and used as one input of a multi-modal layer. By the method, the focus relation between targets or scenes is utilized effectively, and the semantic features of the image is described in a targeted manner.

Description

technical field [0001] The present invention relates to the field of computer vision, and more specifically, to an image description generation method based on a neural network and image attention points. Background technique [0002] Obtaining text-level image descriptions has become an important research topic in the current field of computer vision, and in real life, it has many application scenarios. Examples include early childhood education, image retrieval, and navigation for the blind. With the rapid development of computer vision and natural language processing technology, a large number of effective works on this topic appear, many of which regard it as a retrieval problem. The researchers project the features of text sentences and images into the same semantic space by learning a node embedding layer. These methods generate image descriptions by retrieving similar descriptions from text sentence datasets, but they lack image descriptions that can effectively com...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30G06N3/08
CPCG06F16/51G06F16/5866G06N3/084
Inventor 胡海峰杨梁王腾张俊轩王伟轩
Owner SYSU CMU SHUNDE INT JOINT RES INST
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap