Method and device for realizing attribute normalization

Abstract

The invention discloses a method and a device for realizing attribute normalization, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of obtaining first data by taking a word of to-be-normalized data as a KEY and taking the to-be-normalized data as VALUE; aggregating the first data with the same KEY into a first data group, aggregating the VALUE of each piece of first data in the first data group into second data, and selecting one word from all words of the first data group as a normalized word of the second data; taking each original word in the second data as a KEY, and taking the second data as VALUE to obtain third data; aggregating the third data with the same KEY into a second data group, aggregating the VALUE of each piece ofthird data in the second data group into fourth data, selecting one of all the normalization words of the second data group as a normalization word of the fourth data, and adding the normalization word into an expansion word set to form result data; and determining the normalized attribute value according to the result data. According to the embodiment, attribute normalization can be achieved without manual maintenance, the accuracy is high, the real-time performance is good, and manpower is saved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a method and device for realizing attribute normalization. Background technique [0002] The basis of various data mining algorithms is the characteristics of data. However, in today's Internet, in order to increase the exposure rate in various search engines, users often reflect the various ways of writing the same attribute in the text as much as possible. That is, a certain attribute value of a piece of data is often represented by splicing multiple redundant words that can represent the same meaning. Using different writing methods to write the attribute value of the same attribute will bring a lot of trouble to data processing. Taking the e-commerce field as an example, if a certain brand A has four writing styles: A1, A2, A3, and A4, these four writing styles all represent brand A. The number, sequence, etc. are not necessarily the same. If "A1 A2 A3" is the o...

AI Technical Summary

Problems solved by technology

[0005] 1) Manpower consumption: Maintaining the corresponding table requires continuous manpower, because whenever the original words of the same attribute appear in different ways, they must be manually maintained and added to this corresponding table

[0006] 2) Low accuracy and real-time performance: In massive data, it is difficult to manually accurately, comprehensively, and quickly find the original words of a certain attribute that appear in different ways. Classification errors caused by untimely and inaccurate updates, or situations that do not correspond at all

Images