Network representation generation method and device and network representation encoding method and device in neural network

Abstract

The invention discloses a network representation generation method and device and a network representation encoding method and device in a neural network, a coder and machine equipment. The method comprises the following steps: acquiring spatial representations corresponding to elements in an input sequence; generating a key value pair vector sequence and a request vector sequence mapped to the key vector sequence in the key value pair vector sequence by encoding the spatial representation; carrying out key vector sequence extraction on local elements relative to the elements to obtain a key vector sequence set taking the elements as centers; obtaining a weight coefficient of a value vector sequence set corresponding to the key vector sequence set by calculating the correlation between therequest vector sequence and the key vector sequence set; generating network representation for element output through a weight coefficient and a value vector sequence set corresponding to the key vector sequence set; the weight coefficient is not distributed in a scattered manner any more, and the weight coefficient is obtained based on the relevance of the element request vector sequence because, so that the local information is enhanced, and the local context capture capability of the neural network is correspondingly improved.

Description

technical field [0001] The invention relates to the field of computer application technology, in particular to a network representation generation and encoding method and device, an encoder, and a machine device in a neural network. Background technique [0002] With the application and development of neural networks in various fields, more and more attention mechanisms are introduced into neural networks as a basic module, so as to dynamically select the relevant representation of the input sequence in the neural network on demand, and then relatively Recurrent Neural Network (RNN for short) obtains better output quality. [0003] In the neural network, each upper-layer network representation will establish a direct connection with all lower-layer network representations, that is, the output of each layer is used as the input of the next layer, and it is repeated many times until the encoding completes the network representation. [0004] Through the attention mechanism in...

AI Technical Summary

Problems solved by technology

[0004] Through the attention mechanism introduced in the neural network, all network representations in each layer will be fully considered, and a weighted sum operation will be performed on each network representation, which will disperse the distribution of the obtained attention weights to a certain extent , for the elements in the input sequence, the information of its adjacent elements will also be weakened. In other words, the network representation generated by the neural network for element encoding has the limitation that local information is weakened

Images