Self-attention mechanism representation network search method, device and equipment

Abstract

The embodiment of the invention relates to the technical field of data processing, in particular to a self-attention mechanism representation network searching method, device and equipment, and aims to accelerate the design process of a self-attention mechanism representation network and improve the design efficiency and effect of a deep model. The method comprises the following steps: constructing a self-attention mechanism representation network search space according to self-attention mechanism representation networks of various architectures, wherein the self-attention mechanism representation network search space comprises a plurality of sub-networks; constructing a super network through the plurality of sub-networks; and inputting to-be-analyzed data into the super network, and searching a corresponding self-attention mechanism representation network from the self-attention mechanism representation network search space through the super network.

Description

technical field [0001] The embodiments of the present application relate to the technical field of data processing, and in particular, relate to a self-attention mechanism characterization network search method, device and equipment. Background technique [0002] The self-attention mechanism is currently widely used in the tasks of natural language processing and graph representation learning. The deep neural network constructed by the self-attention mechanism can greatly improve the training speed and representation ability of the neural network. Processing and graphing is learning for each task and dataset in the task domain by manually designing a suitable self-attention mechanism to represent the network. Neural network architecture search is to improve the efficiency and effect of deep network model design by automating the design of deep network. This type of technology generally includes two parts: search space and search algorithm. The search space contains deep neur...

AI Technical Summary

Problems solved by technology

[0003] In the existing technology, the method of self-attention mechanism design mainly relies on expert experience and manual debugging, which has low efficiency, high manpower and technical costs, and is easy to introduce design deviation and suboptimal design

The existing neural network search architecture technology is not designed for the representation network of the self-attention mechanism, and the search space and search algorithm cannot be directly applied to the representation network of the self-attention mechanism, and cannot solve the high flexibility of the representation network of the self-attention mechanism Sexuality, heterogeneity and other issues

Images