Text semantic representation method based on aggregation weighting matrix compression algorithm

Abstract

The invention belongs to the technical field of data mining, and specifically provides a text semantic representation method based on an aggregation weighting matrix compression algorithm. According to the method, through constructing a global smooth context matrix, influences of pragmatic habits on word semantics are weakened; through utilizing a word vector aggregation algorithm to weight global smooth contexts, a word vector weight measurement method is corrected; and finally, texts are represented through utilizing an infinite norm compression algorithm to convert global smooth contexts into vectors. Compared with the existing method, the method has the advantages that the highest possible value of each dimension of the vectors is reserved and the text information is represented more fully.

Description

technical field [0001] The invention belongs to the technical field of data mining, and specifically proposes a text representation method based on an aggregation weighted matrix compression algorithm. Background technique [0002] As a widespread data form (different types of web pages, digitized books, log files, etc.), plain text has always been the focus of research in the fields of natural language processing, machine learning, and artificial intelligence. Under the condition of continuous expansion of network data, the analysis and processing technology for text data is becoming more and more important. A common denominator of traditional text processing techniques is the need to transform text data presented in natural paragraphs into a computable data form. Among them, converting text into a fixed-size vector form as input is a prerequisite for many text processing techniques. Taking text clustering analysis as an example, the core idea of ​​many clustering algorit...

AI Technical Summary

Problems solved by technology

However, such word-vector-based document vector generation methods usually focus on how word vectors are generated, but ignore how word vectors are combined

The weighted accumulation method tends to use some word vectors with larger weights to represent the entire document, which makes the generated document vector information missing, which further causes misestimation of the similarity between documents

In addition, the existing methods do not consider the semantic similarity of the word vector itself when estimating the weight of the word vector, resulting in biased weight estimation.

Images