Neural framework search method for general multi-modal learning

Abstract

The invention discloses a neural framework search method for general multi-modal learning. The method comprises the following steps of 1, performing data preprocessing on image and text data, and extracting features; 2, initializing an encoder decoder structure redundant network and corresponding architecture parameters; 3, sampling a substructure from the architecture parameter distribution, andcalculating a result. 4, ensuring the stability of a search structure through model search, hot start and alternate update. and 5, model training: retraining the searched optimal sub-network to obtainan optimal network model. The invention provides a neural frame search method for image text multi-modal modeling. A better sub-network is particularly searched for different tasks, so that the parameter amount calculation amount of the network is reduced, the deep features of each modal are fully utilized, the expression capability of modal expansion features is improved, and a leading effect isobtained in the three multi-modal tasks.

Description

technical field [0001] The invention proposes a general multimodal learning-oriented neural frame search method MMNasNet. Background technique [0002] Visual Question Answering is an emerging task in the multimodal domain, which aims to answer a given question about an image based on the provided image. Specifically, input an image and a question, and answer the corresponding answer to the question after passing through the model. For example, the image content is a street with houses of various colors, different types of cars parked, and a lot of pedestrians walking on the street. When given a specific question, such as "What color is the car on the left of the pedestrian in black walking on the zebra crossing?", the model needs to recognize the object in the image and the different properties of the object and then infer to get the answer. The Visual Grounding task aims to find the corresponding object in the image based on the provided object description. For example,...

AI Technical Summary

Problems solved by technology

At the same time, the image carrier in natural scenes has various themes, complex and varied content, and there may be high similarity and redundancy between object frames, which makes the required architecture and also makes the architecture search method face a huge challenge

Images