Supercharge Your Innovation With Domain-Expert AI Agents!

Bilingual image subtitle generation method and system, storage medium and computer equipment

An image and subtitle technology, applied in the fields of computer vision and natural language processing, can solve problems such as inability to effectively use hidden semantics, inability to generate two types of subtitles at the same time, and ignoring the characteristics of inter-translation

Active Publication Date: 2021-03-26
GUANGDONG UNIV OF TECH
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this solution and other existing technologies are only used to generate subtitles in a single language. For example, the above-mentioned patents can generate Chinese subtitles or English subtitles separately by using different training sets and test sets or pictures to be processed. but cannot generate both subtitles at the same time
Although it is possible to use a monolingual image subtitle generation model when generating subtitles for each language, or directly translate the output of subtitles generated in one language into another language, the above methods often ignore the existence of subtitles in two languages. The characteristics of inter-translation cannot effectively use the deep hidden semantics

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Bilingual image subtitle generation method and system, storage medium and computer equipment
  • Bilingual image subtitle generation method and system, storage medium and computer equipment
  • Bilingual image subtitle generation method and system, storage medium and computer equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0051] Please refer to figure 1 , a method for generating bilingual image subtitles, comprising the following steps:

[0052] S01. Obtain a bilingual image subtitle data set, the bilingual image subtitle data set includes an image set and a bilingual subtitle data set; construct a bilingual dictionary according to the bilingual subtitle data set;

[0053] S02, performing feature extraction training on the residual network model according to the image set, to obtain image features of the encoder and the image set;

[0054] S03, according to the image features of the image set and the bilingual dictionary, perform word embedding-based bilingual image subtitle alternate generation training on the two recurrent neural network models to obtain a first language decoder and a second language decoder;

[0055] S04, based on the encoding-decoding model framework, constructing a bilingual image subtitle joint generation model according to the encoder, the first language decoder and the...

Embodiment 2

[0126] A bilingual image subtitle generation system, see Figure 7 , including bilingual image subtitle data set acquisition processing module 1, feature extraction training module 2, bilingual image subtitle alternate generation training module 3, bilingual image subtitle joint generation model building module 4 and image acquisition processing module 5 to be processed; the feature extraction training Module 2 is connected to the bilingual image subtitle data set acquisition processing module 1; the bilingual image subtitle data set acquisition processing module 3 is connected to the bilingual image subtitle data set acquisition processing module 1 and the feature extraction training module 2; the bilingual image subtitle joint Generate model construction module 4 to connect described feature extraction training module 2 and bilingual image subtitles to generate training module 3 alternately; Described image acquisition processing module 5 to be processed connects described...

Embodiment 3

[0133] A storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for generating bilingual image subtitles in Embodiment 1 are realized.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a bilingual image subtitle generation method and system, a storage medium and computer equipment, aiming at solving the technical problem that bilingual image subtitle generation cannot be realized in the prior art, and constructing a bilingual image subtitle joint generation model by training a residual network model and a recurrent neural network model. Bilingual image subtitles are generated in an alternating mode, the inter-translation characteristic between subtitles of two languages is fully utilized in the subtitle generation process, and therefore when a next word of a certain language is predicted, historical information of the subtitles of the language can be utilized, and historical information of subtitles of another language can also be utilized. The hidden information of the image is fully mined to obtain an accurate bilingual caption output result.

Description

technical field [0001] The present invention relates to the technical fields of computer vision and natural language processing, in particular to the application of deep neural networks in image subtitle tasks, and more specifically, to a bilingual image subtitle generation method, system, storage medium and computer equipment. Background technique [0002] The task of image captioning (Image Caption) is for a given image, let the machine automatically generate a fluent subtitle that conforms to the content of the image or an annotation that describes the content of the image. It is essentially a visual-language (Visual- to-language) task. [0003] Publication time is 2018-04-13, and the Chinese patent application with the publication number CN107909115A: a method for generating Chinese subtitles for images, discloses a method that attempts to link the semantic information of each word with the local features of the image and utilize the attention A scheme for modeling the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F40/242G06F40/289G06K9/62G06N3/04H04N5/278
CPCG06F40/242G06F40/289H04N5/278G06N3/044G06N3/045G06F18/214Y02D10/00
Inventor 王耀葛原玲张壮裕庞贵杰文瑞森
Owner GUANGDONG UNIV OF TECH
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com