Image description generation method of multi-modal feature fusion network

Abstract

The invention relates to the field of image data processing, and discloses an image description generation method of a multi-modal feature fusion network, which comprises the following steps: constructing the multi-modal feature fusion network; designing a decoding end on the infrastructure of the Up-Down model; integrating the multi-modal feature fusion network into a decoding end to form an image description generation model based on the multi-modal feature fusion network; training an image description generation model based on the multi-modal feature fusion network; and inputting a test image, and verifying the performance of the image description generation model based on the multi-modal feature fusion network. According to the invention, a hierarchical structure is constructed by using a recurrent neural network to fuse coding features, and input information is weighted by using an attention mechanism, so that single image features extracted by an encoder are associated with each other, feature interaction is enhanced, the association between hidden layer vectors and object features is better mined, and higher image description generation performance is realized.

Description

technical field [0001] The invention relates to the field of image data processing, in particular to an image description generation method of a multimodal feature fusion network. Background technique [0002] Understanding an image largely depends on the obtained image features, and the techniques used to obtain features include traditional machine learning techniques and deep machine learning techniques. Traditional machine learning techniques extract artificial features such as LBPs, SIFT, and HOG, and send them or their combination into classifiers such as SVM to determine object categories. This method has two shortcomings. One is that these artificial features are Task-oriented, it is not feasible to extract such features from large and diverse datasets; second, real-world data is complex and has different semantic interpretations. On the contrary, deep machine learning technology can automatically learn features from the training set, and is suitable for processing l...

AI Technical Summary

Problems solved by technology

[0003] Existing representative models such as LSTM-A, Plstm-a-2, VS-LSTM, DAA, RFNet, Up-Down, and VSV-VRV-POS have insufficient extraction and utilization of image features, and semantic gaps still exist

Images