Data association processing method and system and electronic equipment

Abstract

The invention relates to a data association processing method and system and electronic equipment. The data association processing method comprises the steps that a first information table is generated according to basic log data, wherein each record in the first information table is uniquely identified with a first key field and a second key field; in a preset time period, the first key fields and the corresponding second key fields in the first information table are self-associated, and a second information table is generated; the quantity of the first key fields corresponding to at least two second key fields is subjected to statistical analysis after self-association in the second information table, and a third information table is generated; and a fourth information table is generatedaccording to the first information table, the second information table and the third information table, wherein the fourth information table comprises indexes representing association among at leasttwo second key fields.

Description

technical field

[0001] The present invention relates to data association processing technology, in particular to a data association processing method, system and electronic equipment. Background technique

[0002] In the big data environment, many companies will choose Hadoop's big data association processing architecture, and hive is more commonly used as a data query language on Hadoop. But for the exploration of large amounts of data, sometimes simple query logic cannot meet the needs of deep data exploration in actual scenarios, such as some complex relational operations in data mining, such as association rules. In many cases, users need to first query the basic data through Hadoop, then export or download (down) to the local, and then import these basic data into related data mining software for further processing.

[0003] In this processing method, data and processing software are separated, manual processing is too much, data security is low, data export consumes a...

Examples

Embodiment Construction

[0033]Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and fully convey the concept of example embodiments to those skilled in the art. The drawings are merely schematic illustrations of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted.

[0034] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the invention. However, those skilled in the art will appreciate ...

technical field

[0001] The present invention relates to data association processing technology, in particular to a data association processing method, system and electronic equipment. Background technique

[0002] In the big data environment, many companies will choose Hadoop's big data association processing architecture, and hive is more commonly used as a data query language on Hadoop. But for the exploration of large amounts of data, sometimes simple query logic cannot meet the needs of deep data exploration in actual scenarios, such as some complex relational operations in data mining, such as association rules. In many cases, users need to first query the basic data through Hadoop, then export or download (down) to the local, and then import these basic data into related data mining software for further processing.

[0003] In this processing method, data and processing software are separated, manual processing is too much, data security is low, data export consumes a...

Examples

Embodiment Construction

[0033]Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and fully convey the concept of example embodiments to those skilled in the art. The drawings are merely schematic illustrations of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted.

[0034] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the invention. However, those skilled in the art will appreciate ...

Images