A multi-dimensional information combination method, device, equipment and storage medium

Abstract

A multi-dimensional information merging method, device, device, and storage medium, the method comprising: extracting feature keywords of a text; performing hash calculations on the feature keywords respectively, so as to obtain a set containing multiple hash values; Take the sum of all hash values ​​as the main fingerprint; combine each hash value in the set and take the sum as the secondary fingerprint set; merge the secondary fingerprint set and the primary fingerprint into a union, use the union as the Key, and the primary fingerprint as the Value constitutes a key-value pair as the fingerprint feature sub-base; match the fingerprint feature sub-base to the fingerprint feature general library, if the match is successful, the fingerprint feature sub-base will be discarded, if the match fails, the hash value of the feature subject word will be Combine the sum as the Value of the fingerprint feature sub-library, and add the text fingerprint feature sub-library to the fingerprint feature total library. The invention solves the problems of slow calculation speed and exponential growth of simhash. It can provide high-quality deduplication data and reduce the phenomenon of re-reporting and omission of push data.

Description

technical field [0001] The invention relates to the similarity merging of massive texts in the field of text analysis. The network public opinion information is similarly merged to achieve the effect of deduplication. Specifically, it relates to a method, device, equipment and storage medium for combining multi-dimensional information. Background technique [0002] With the popularization of the Internet, the Internet has gradually become the main carrier for people to publish, obtain and transmit information. With the advent of the self-media era, every individual can become a reporter and a messenger. Public opinion data is crucial to timely understanding of public opinion, reflecting social information, controlling and guiding the correct development of public opinion, social stability and national development. The analysis of public opinion is helpful to understand the development trend of the event, avoid the vicious spread of the event, and provide important decision...

AI Technical Summary

Problems solved by technology

However, Internet public opinion information has the characteristics of massive data and high repetition, and the amount of information is expanding at an unprecedented speed. Therefore, the development of public opinion will explode and disappear in a short period of time. Timely viewing of users causes pressure and wastes manpower

[0003] At present, the commonly used text deduplication method simhash is to convert all the texts into binary and then perform pairwise comparison. When there is a large amount of data, the calculation increment will be exponential, and it will be unusable in the massive data.

Images