A machine learning-based similarity computation method for multi-feature text data

Abstract

The invention discloses a method for calculating similarity of multi-feature text data based on machine learning. Each feature of each data is converted into a vector array by using a text vectorization algorithm; the vector data generated by multiple features of each data is spliced and regularized, and all the corresponding vector arrays of the data are formed into a matrix; alternatively, a PCAalgorithm is used to reduce the dimension of the matrix; a series of similar data pairs are marked out by the business experts in the above data, and each data pair is composed of two similar data pairs; a vector distance mapping matrix is calculated based on the similar data pairs, and a vector distance calculation formula is obtained based on the vector distance mapping matrix; a low-precisionaggregation algorithm is used. The method uses machine learning algorithm to realize the distance calculation of multi-feature text data, and uses low-precision clustering method to reduce the computational load and improves the performance of the algorithm.

Description

technical field [0001] The invention belongs to the technical field of computer software, in particular to a method for calculating similarity of multi-feature text data based on machine learning. Background technique [0002] In the field of big data processing, one requirement is to calculate the similarity of multi-featured text data. The so-called multi-feature text data refers to such data: the data contains multiple features (also called domains), and each feature is a piece of text composed of multiple characters, or other data types that can be converted into text. [0003] Such as Figure 1 As shown, this is a set of e-commerce product data, which contains three features of "product number", "brand" and "product model", each feature is a piece of text data (or can be simply converted into text other datatypes of the data). [0004] For such a set of multi-featured text data, many data applications involve calculating the similarity between the data. For example: ...

AI Technical Summary

Problems solved by technology

[0007] 2. Text is a kind of non-quantitative information, and an effective quantitative algorithm is required to calculate the similarity between data; for a data group with N data sets, if you want to obtain the data with the highest similarity with one of the data, Then in theory, N×(N-1) calculations are required. When N is large, the efficiency of the algorithm is very low.

Images