Multimodal model training and image recognition method, device, and electronic equipment

Abstract

The invention discloses a multimodal model training and image recognition method, device and electronic equipment, and relates to the field of image recognition. The method includes: acquiring a sample image and a character feature vector corresponding to the sample image; inputting the sample image into a feature extraction network of an initial multimodal model to generate an image feature vector corresponding to the sample image. The feature extraction network is used to encode the sample image, and generate image feature vectors according to the relationship between the features to be generated and the generated features; input the text feature vectors and image feature vectors to the transformer structure of the initial multimodal model In , the candidate text corresponding to the sample image is output; according to the target text and the candidate text corresponding to the text feature vector, the parameters of the initial multimodal model are updated to determine the target multimodal model. Using this method can ensure the accuracy of the generated image feature vector, so that the local receptive field will not be damaged during the image generation process.

Description

technical field [0001] The invention relates to the field of image recognition, in particular to a multimodal model training and image recognition method, device, and electronic equipment. Background technique [0002] Since the transformer was born, it has achieved great success in both images and text. However, real artificial intelligence can understand both images and text, not just images or text. Therefore, in recent years, there has been a lot of related research on multimodal (text, image) understanding problems. [0003] Existing multimodal autoregressive models still use a method similar to that of natural language processing autoregressive models, focusing on how to convert images into features similar to text. as attached figure 1 As shown, the mainstream method is to use the feature extraction part of the variational autoencoder to perform operations such as convolution and pooling on the image, and finally obtain a vector matrix of V_size*N_h*N_w size, where...

AI Technical Summary

Problems solved by technology

[0006] In view of this, the embodiment of the present invention provides a multi-modal model training and image recognition method, device, and electronic equipment, aiming to solve the problem that the local receptive field is destroyed during the image generation process in the prior art

Images