Image-sentence description generation system and method based on hierarchical neural network

A neural network and generation system technology, applied in the field of image-sentence description generation system based on hierarchical neural network, can solve problems such as inconsistency in description habits and achieve performance improvement

Active Publication Date: 2021-08-31
FUDAN UNIV
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The problem solved by the present invention is that the sentences generated by the existing CNN+RNN model in the image-sentence description generation technology are inconsistent with human description habits; in order to solve the problem, the present invention provides an image-sentence description generation system and method based on hierarchical neural networks

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Image-sentence description generation system and method based on hierarchical neural network
  • Image-sentence description generation system and method based on hierarchical neural network
  • Image-sentence description generation system and method based on hierarchical neural network

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0025] It can be seen from the background technology that the sentences generated by the existing image-sentence description generation method based on the hierarchical network do not match the language habits of human beings; the applicant conducted research on the problem and believed that the reason is that the existing methods did not consider human beings in the image. General knowledge information in the description process. For example, when humans describe images, they usually have a cascading segmentation process, that is, they first look at the objects in the image, the attributes of the objects, and the relationship between objects, and then organize appropriate language based on the recognized objects. Describe the image. On the other hand, humans usually use an attention mechanism when describing images, which means that human attention will change dynamically during the expression process, and may focus on different areas of the image or different scene.

[...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention provides an image-sentence description generation system and method based on a hierarchical neural network. The image-sentence description generation system based on a hierarchical neural network provided by the present invention includes: a deep convolutional network, a first deep loop neural network, and a second deep loop Neural network; the deep convolution network acquires images, and calculates the embedding vector of the image feature vector in the embedding space; and outputs to the first deep recurrent neural network and the second deep recurrent neural network; the first deep recurrent neural network recognizes the image The object contained in is output to the second deep recurrent neural network; the second deep recurrent neural network outputs a sentence describing the image according to the embedding vector and the object sequence.

Description

technical field [0001] The invention belongs to the technical field of cross-media generation, and in particular relates to an image-sentence description generation system and method based on a hierarchical neural network. Background technique [0002] Among various multimodal information processing tasks, automatic image description generation (Image Captioning) is a very important task. Its task is given an image, the computer automatically analyzes the visual content of the image, and generates a sentence or text that can describe the main content of the image. Automatic image description generation is a very challenging task because it involves two very important research fields of artificial intelligence, computer vision and natural language processing. From the perspective of computer vision, image description can in principle involve any visual information in the image, such as the objects and attributes contained in the image, or describe the type of scene in the im...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/58G06N3/08
CPCG06N3/082G06F16/58
Inventor 张玥杰程勇周练张涛
Owner FUDAN UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap