Image description method based on convolutional neural network, computer readable storage medium and electronic equipment

Abstract

The invention provides an image description method based on a convolutional neural network, a computer readable storage medium and electronic equipment, and solves the problems that an existing recurrent neural network method cannot process sequence signals in parallel and is time-consuming in operation. The method comprises the following steps: 1) pre-training the convolutional neural network; 2) extracting global features and local features of the image, and projecting the global features and the local features of the image into a multi-modal mapping space; 3) performing convolutional coding on the image expression in the multi-modal mapping space in the step 2); 4) carrying out word feature expression; 5) carrying out convolution coding on the description statement in the step 4), 6) calculating the attention, and obtaining the probability of word generation corresponding to the input image; 7) constructing a target loss function between input and output, and performing neural network training by using the loss function to obtain position parameters of the neural network, and 8) inputting the test image into the trained neural network system to obtain a descriptive natural statement corresponding to the test image.

Description

technical field [0001] The invention relates to image and text multimodal fusion technology, in particular to an image description method based on convolutional neural network, computer readable storage medium, and electronic equipment, which can be used for early childhood education, human-computer interaction, and visual aid for visually impaired people. Wait. Background technique [0002] With the development of science and technology, artificial intelligence has gradually become a decisive force to push mankind into the intelligent age. Artificial intelligence studies how to let the machine simulate the human thinking process and intelligent behavior, let the computer automatically generate a descriptive text from the natural image, and describe the content of the image in one sentence. In recent years, deep learning has made great breakthroughs in the fields of computer vision, natural language processing, and speech information processing, and has also received widesp...

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to solve the problem that the existing recursive neural network method cannot process sequence signals in parallel an

Images