Image description method based on intra-layer and inter-layer joint global representation

An image description and global technology, applied in neural learning methods, instruments, biological neural network models, etc., can solve problems such as bias, no explicit modeling of global features, missing objects, and relationships

Pending Publication Date: 2021-05-18
XIAMEN UNIV
View PDF3 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to provide a more comprehensive and instructive global feature by modeling a more comprehensive and instructive global feat...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Image description method based on intra-layer and inter-layer joint global representation
  • Image description method based on intra-layer and inter-layer joint global representation
  • Image description method based on intra-layer and inter-layer joint global representation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0058] The following embodiments will describe the technical solutions and beneficial effects of the present invention in detail in conjunction with the accompanying drawings.

[0059] The purpose of the present invention is to solve the problem that the traditional transformer-based image description method does not explicitly model global features, resulting in missing objects and biased relationships. It proposes to connect different local features by modeling a more comprehensive and instructive global feature. information, thereby improving the accuracy of generated descriptions, and providing an image description method based on a joint global representation between layers within layers. The specific method flow is as figure 1 shown.

[0060] Embodiments of the present invention include the following steps:

[0061] 1) For the images in the image library, first use the convolutional neural network to extract the corresponding image features;

[0062] 2) Sen...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an image description method based on intra-layer and inter-layer joint global representation, and relates to artificial intelligence. The method comprises: the step 1, extracting a plurality of candidate areas of a to-be-described image and features corresponding to the candidate areas by adopting a target detector; and the step 2, inputting the features extracted in the step 1 into the trained neural network so as to output a description result of the to-be-described image. By utilizing the characteristics of a Transformer structure and explicitly modeling intra-layer and inter-layer joint global features, the global features of pictures are effectively utilized, object missing and relation bias existing between image description tasks are reduced, and the accuracy and comprehensiveness of generated sentences are improved; the method has very strong mobility and can be suitable for any image description model based on a Transformer structure, and the model performance is improved; and the problems of target missing and relation offset of image description are solved, complex multi-modal reasoning is expanded, description is automatically generated, and the method can be applied to the fields of image retrieval, blind navigation, automatic generation of medical reports and early education.

Description

technical field [0001] The invention relates to automatic image description in the field of artificial intelligence, in particular to an image description method based on intra-layer and inter-layer joint global representation for describing the objective content of the image in natural language based on the image. Background technique [0002] Automatic image description (Image Captioning) is an ultimate machine intelligence task proposed by the artificial intelligence community in recent years. Its task is to describe the objective content of the image in natural language for a given image. With the development of computer vision technology, tasks such as target detection, recognition, and segmentation can no longer meet people's production needs, and there is an urgent need for how to automatically and objectively describe image content automatically. Different from tasks such as target detection and semantic segmentation, automatic image description requires an overall a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06K9/46G06K9/62G06N3/04G06N3/08
CPCG06N3/049G06N3/082G06V10/40G06V10/44G06V2201/07G06N3/047G06N3/045G06F18/22
Inventor 孙晓帅纪荣嵘纪家沂
Owner XIAMEN UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap