Word vector matrix compression method and device and word vector acquisition method and device

Abstract

The invention discloses a word vector matrix compression method and device and a word vector acquisition method and device, which relate to the technical field of data processing, and are beneficial to saving the storage space of client equipment. The word vector matrix compression method comprises the following steps of generating a word vector matrix for representing a to-be-processed vocabularybased on a word vector model, and taking the generated word vector matrix as a to-be-compressed word vector matrix, wherein one row or one column of the to-be-compressed word vector matrix is a wordvector, and one word vector in the to-be-compressed word vector matrix is used for representing one word in the to-be-processed vocabulary; classifying the word vectors included in the to-be-compressed word vector matrix according to the semantic information of the to-be-processed vocabulary to obtain at least two categories; and compressing at least one of the at least two categories, constructing a compressed word vector matrix according to the word vector obtained by compressing the at least one category, and storing the compressed word vector matrix.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a word vector matrix compression method and device, and a method and device for obtaining word vectors. Background technique [0002] With the development of deep learning technology, the performance of natural language processing (NLP) tasks has been greatly improved. Among them, NLP tasks can include: word segmentation, part-of-speech tagging, named-entity recognition (NER ), sentence classification, dialogue system, etc. Based on deep learning technology, the core of NLP tasks is to represent vocabulary through language models. The language model not only contains the meaning of the words themselves, but also reflects the relationship between different words, such as synonyms, antonyms, and contextual relationships. [0003] The word vector matrix is ​​a specific representation of the language model. A word vector matrix is ​​a matrix composed of one or...

AI Technical Summary

Problems solved by technology

[0004] Because the language has the characteristics of large vocabulary and complex syntax and grammar, a large number of features are required to describe it. Therefore, the generated word vector matrix is ​​large in size and occupies a relatively large storage space. Usually, it cannot be directly applied to the client device.

Images