Compressed representation learning method based on tensor decomposition

Abstract

The invention discloses a compressed representation learning method based on tensor decomposition. The method comprises the following steps that firstly, a representation learned through micro neuralnetwork preprocessing is converted into a tensor to be decomposed; tensor decomposition is carried out on the basis of an optimization algorithm, a subspace of the tensor decomposition is solved, low-rank reconstruction is carried out, and finally, low-rank representation extracted through processing tensor decomposition of another micro neural network is fused into representation learned by a backbone network to play a role of regularization; and a truncated single-step gradient optimization method is combined to improve an optimization algorithm with a multi-step time axis iterative model. According to the method, regularization and supplementation are successfully provided for large-scale pre-training and representation learning in a calculation-friendly and parameter-saving mode, the effectiveness of the method is verified by a large number of tasks and applications of computer vision, and a remarkable effect is achieved in image recognition, semantic segmentation and target detection; and the attention mechanism commonly used by computer vision is destroyed by lighter calculation and parameter quantity.

Description

technical field [0001] The invention belongs to the technical field of representation learning and deep neural network structure design, and in particular relates to a compressed representation learning method based on tensor decomposition. Background technique [0002] In recent years, representation learning has achieved great success in the field of machine learning. Representation learning extracts distributed representations from data and applies regularization to the representations to achieve disentanglement. Distributed representations can represent exponentially large-scale information with polynomial-level complexity. A disentangled vector representation, on the other hand, can separate the varying independent information factors from the data. Representation learning is general and proven to be beneficial for different downstream tasks. [0003] The concept of representation learning is quite broad. It is generally believed that transformations based on multi-...

AI Technical Summary

Problems solved by technology

[0006] However, both of the above approaches have their limitations

The representation learned by the first idea lacks a priori constraints. Even through the pre-training of massive data, it still shows redundancy or insufficiency for specific downstream tasks.

The second approach to encode prior constraints into representations based on generative models, reconstruction, and time series prediction is limited by the model size and training methods, and often stays in the reasoning tasks of small data sets, which is difficult to migrate to specific supervisory tasks

Images