A method and device for realizing visualization processing of big data statistics

Abstract

The application provides a kind of visualization processing method and device for realizing big data statistics, method includes consuming the original vehicle data in the topic of first Kafka on NIFI, processing the statistical field of original vehicle data, the result stream after processing is sent to the topic of second Kafka;Create a data source table on KYLIN, the data source of data source table selects the topic of second Kafka;According to the definition of Model and Cube according to statistical demand, build Cube to obtain pre-computed result data;Obtain the pre-computed result data obtained by KYLIN on NIFI, define a process to process the pre-computed result data obtained, and store the final data after processing into intermediate table.The application has the advantages that: it can provide intuitive web interface operation and management for users, without complicated code development, can effectively reduce development cycle and development cost, is very convenient for later maintenance, and can improve query efficiency and performance.

Description

[Technical Field]

[0001] This invention relates to the field of big data processing technology, and in particular to a visualization processing method and apparatus for realizing big data statistics. [Background Technology]

[0002] With the advent of the big data era, there is a need for statistical analysis of massive amounts of data. Current technologies typically store source data in Elasticsearch, then use task code to periodically analyze the source data stored in Elasticsearch, finally generating a statistical intermediate table. Elasticsearch is a search server based on Lucene, providing a distributed, multi-user, full-text search engine based on a RESTful web interface.

[0003] However, existing technologies have the following drawbacks: 1. Storing source data in Elasticsearch consumes a large amount of storage space, and coupled with complex statistical conditions, limits Elasticsearch's query performance; 2. Developing code for scheduled statistical tasks is cumbersome, has a long development cycle, is inconvenient to maintain, and has high overall implementation costs. In view of the above-mentioned problems, the inventors of this case conducted in-depth research on this issue, resulting in this case. [Summary of the Invention]

[0004] The technical problem to be solved by the present invention is to provide a visualization processing method and device for big data statistics, which solves the problems of limited query performance, cumbersome task code development, long development cycle, inconvenient maintenance, and high overall implementation cost in the existing technology.

[0005] This invention is implemented as follows:

[0006] Firstly, a visualization processing method for big data statistics is provided, the method comprising the following steps:

[0007] Step S1: Consume the raw vehicle passage data from the first Kafka topic on NIFI, process the statistical fields of the raw vehicle passage data, and send the processed result stream to the second Kafka topic.

[0008] Step S2: Create a data source table on Kylin, selecting the second Kafka topic as the data source for the data source table; define the Model and Cube according to the statistical requirements, and build the Cube to obtain the pre-calculated result data;

[0009] Step S3: Obtain the pre-calculation result data from KYLIN on NIFI, define a process to process the obtained pre-calculation result data, and store the processed final data into an intermediate table.

[0010] Furthermore, step S1 specifically includes:

[0011] Step S11: Obtain the raw vehicle passage data from the first Kafka topic through the Kafka consumer component on NIFI. The raw vehicle passage data is a stream file in JSON format.

[0012] Step S12: Set the component property Destination of the attribute extraction component to flowfile-attribute, and extract the data fields to be used from the flow file and put them into the custom flow property;

[0013] Step S13: The attribute update component processes the statistical fields of the stream file using custom stream attributes or rules;

[0014] Step S14: Set the component property JoltTransformationDSL of the JSON format conversion component to Modify-Overwrite, and fill in the statistical fields in the component property JoltSpecification of the JSON format conversion component, so as to merge the statistical fields into the stream file to form the result stream;

[0015] Step S15: The Kafka production component sends the processed result stream to the second Kafka topic.

[0016] Furthermore, in step S2, the specific steps of defining the Model and Cube according to statistical requirements and constructing the Cube to obtain the pre-calculated result data are as follows:

[0017] Create a Model, select the data source table, dimension fields, and measure fields for the Model according to statistical requirements, and set partitions for the Model;

[0018] Create the Cube corresponding to the Model, select the Cube's dimension fields, measure fields, and expressions, and configure the aggregation group fields, row keys, and column families in the advanced settings;

[0019] Select the successfully created Cube and monitor the task through the Monitor to obtain pre-calculated result data.

[0020] Furthermore, in step S2, the field names of the data source table include device ID, data entry with an 8-second delay, data entry with a 1-minute delay, inverted, creation time, and vehicle passage time.

[0021] Furthermore, step S3 specifically includes:

[0022] Step S31: The data table query component executes SQL statements to obtain the pre-calculated result data of the data source table;

[0023] Step S32: The Avro to JSON component converts the pre-calculated result data into an operable JSON format stream file;

[0024] Step S33: The JSON splitting component splits the converted stream file into multiple JSON format stream files;

[0025] Step S34: Set the component property Destination of the attribute extraction component to flowfile-attribute, and extract the data fields to be used from the split flow file and put them into the custom flow property;

[0026] Step S35: The attribute update component updates the attribute id;

[0027] Step S36: The attribute hash value encryption component uses the MD5 algorithm to calculate the hash value of the attribute id as the data primary key;

[0028] Step S37: Obtain device, location, and region information from the cache through the mapping cache acquisition component;

[0029] Step S38: Set the EvaluationMode property of the text replacement component to Line-by-Line, and replace the content of the stream file with device, location and area information, so that the attribute extraction component can obtain the attributes of device, location and area information;

[0030] Step S39: Add the SQL statement for the attribute through the attribute update component. The SQL execution component executes the SQL statement and stores the processed final data into the intermediate table.

[0031] Secondly, a visualization processing device for realizing big data statistics, the processing device includes a statistical field processing module, a data table creation module and a result data processing module;

[0032] The statistical field processing module is used to consume the raw vehicle passage data in the first Kafka topic on NIFI, process the statistical fields of the raw vehicle passage data, and send the processed result stream to the second Kafka topic.

[0033] The data table creation module is used to create a data source table on Kylin, with the data source of the data source table selected as a second Kafka topic; the Model and Cube are defined according to statistical requirements, and the Cube is constructed to obtain pre-calculated result data;

[0034] The result data processing module is used to obtain the pre-calculated result data obtained from KYLIN on NIFI, define a process to process the obtained pre-calculated result data, and store the processed final data into an intermediate table.

[0035] Furthermore, the statistical field processing module specifically includes a vehicle data acquisition unit, a first field extraction unit, a statistical field processing unit, a statistical field merging unit, and a result stream sending unit;

[0036] The vehicle passage data acquisition unit is used to obtain raw vehicle passage data from the first Kafka topic through the Kafka consumer component on NIFI. The raw vehicle passage data is a stream file in JSON format.

[0037] The first field extraction unit is used to set the component attribute Destination of the attribute extraction component to flowfile-attribute, and extract the data fields to be used from the flow file and put them into a custom flow attribute;

[0038] The statistical field processing unit is used by the attribute update component to process the statistical fields of the stream file using custom stream attributes or rules;

[0039] The statistical field merging unit is used to set the component property JoltTransformationDSL of the JSON format conversion component to Modify-Overwrite, and fill in the statistical fields in the component property JoltSpecification of the JSON format conversion component, thereby merging the statistical fields into the stream file to form a result stream;

[0040] The result stream sending unit is used by the Kafka production component to send the processed result stream to the topic of the second Kafka.

[0041] Furthermore, in the data table creation module, the specific steps of defining the Model and Cube according to statistical requirements and constructing the Cube to obtain the pre-calculated result data are as follows:

[0042] Create a Model, select the data source table, dimension fields, and measure fields for the Model according to statistical requirements, and set partitions for the Model;

[0043] Create the Cube corresponding to the Model, select the Cube's dimension fields, measure fields, and expressions, and configure the aggregation group fields, row keys, and column families in the advanced settings;

[0044] Select the successfully created Cube and monitor the task through the Monitor to obtain pre-calculated result data.

[0045] Furthermore, in the data table creation module, the field names of the data source table include device ID, data entry with an 8-second delay, data entry with a 1-minute delay, inverted data entry, creation time, and vehicle passage time.

[0046] Furthermore, the result data processing module specifically includes a result data acquisition unit, a format conversion unit, a file splitting unit, a second field extraction unit, an attribute ID update unit, an encryption unit, a cache acquisition unit, a file replacement unit, and a storage execution unit;

[0047] The result data acquisition unit is used by the data table query component to execute SQL statements and obtain the pre-calculated result data of the data source table;

[0048] The format conversion unit is used by the Avro to JSON component to convert the pre-calculated result data into an operable JSON format stream file;

[0049] The file splitting unit is used by the JSON splitting component to split the converted stream file into multiple JSON format stream files;

[0050] The second field extraction unit is used to set the component attribute Destination of the attribute extraction component to flowfile-attribute, and extract the data fields to be used from the split flow file and put them into a custom flow attribute;

[0051] The attribute ID update unit is used by the attribute update component to update the attribute ID;

[0052] The encryption unit is used by the attribute hash value encryption component to calculate the hash value of the attribute id as the data primary key using the MD5 algorithm;

[0053] The cache acquisition unit is used to acquire device, location, and area information in the cache through the mapping cache acquisition component;

[0054] The file replacement unit is used to set the component attribute EvaluationMode of the text replacement component to Line-by-Line, and replace the content of the stream file with device, location and area information, so that the attribute extraction component can obtain the attributes of device, location and area information;

[0055] The storage execution unit is used to add SQL statements for attributes through the attribute update component, and the SQL execution component executes the SQL statements to store the processed final data into an intermediate table.

[0056] By adopting the technical solution of this invention, at least the following beneficial effects are achieved: NIFI provides a stream-based programming experience, enabling data extraction and storage through visual operation without cumbersome code development; simultaneously, KYLIN pre-computation, through a space-for-time data processing model, improves query efficiency and performance, supports multi-dimensional data modeling for more flexible data analysis, and facilitates statistical calculations; by combining NIFI and KYLIN and applying them to big data statistics, users are provided with an intuitive web interface for operation and management, enabling automated statistical analysis without cumbersome code development, effectively reducing development cycle and costs, and facilitating later maintenance. It also improves query efficiency and performance and is suitable for various usage scenarios. [Attached Image Description]

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] Figure 1 This is an execution flowchart of a visualization processing method for big data statistics according to the present invention;

[0059] Figure 2 This is a flowchart of the execution process of each component when processing the statistical fields of raw vehicle data on NIFI according to the present invention;

[0060] Figure 3 This is a flowchart of the data source table and Model in Kylin according to the present invention;

[0061] Figure 4 This is a flowchart of the execution process of each component in the process of retrieving the pre-calculated result data of KYLIN from NIFI to the intermediate table in this invention.

[0062] Figure 5 This is a structural principle block diagram of a visualization processing device for realizing big data statistics according to the present invention.

Detailed Implementation Methods

[0063] To better understand the technical solution of the present invention, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] Example 1

[0065] Please see Figures 1 to 4As shown in the figure, a preferred embodiment of the present invention for a visualization processing method for big data statistics includes the following steps:

[0066] Step S1: Consume the raw vehicle passage data in the topic of the first Kafka (i.e., Kafka1) on NIFI, process the statistical fields of the raw vehicle passage data, and send the processed result stream to the topic of the second Kafka (i.e., Kafka2).

[0067] Step S2: Create a data source table CARPASS on KYLIN. The data source for the CARPASS data source table is selected from the second Kafka topic. Define the Model and Cube according to the statistical requirements, and build the Cube to obtain the pre-calculated result data.

[0068] Step S3: Obtain the pre-calculation result data from KYLIN on NIFI, define a process to process the obtained pre-calculation result data, and store the processed final data into an intermediate table.

[0069] This invention utilizes NIFI to provide a stream-based programming experience, enabling data extraction and storage through visual operations without cumbersome code development. Simultaneously, it leverages KYLIN for pre-computation, employing a space-for-time data processing model to improve query efficiency and performance. It also supports multi-dimensional data modeling, making data analysis more flexible and facilitating statistical calculations. By combining NIFI and KYLIN and applying them to big data statistics, this invention provides users with an intuitive web interface for operation and management, enabling automated statistical analysis without the need for complex code development. This effectively reduces development cycles and costs, facilitates maintenance, improves query efficiency and performance, and is applicable to various usage scenarios.

[0070] In a preferred embodiment of the present invention, please refer to the following: Figure 2 As shown, step S1 specifically includes:

[0071] Step S11: Obtain the raw vehicle passage data from the first Kafka topic through the Kafka consumer component (ConsumeKafka) on NIFI. The raw vehicle passage data is a stream file in JSON format. The Kafka consumer component (ConsumeKafka) is used to consume messages from Kafka, that is, to obtain a stream file in JSON format.

[0072] Step S12: Set the component attribute Destination of the attribute extraction component (EvaluateJsonPath) to flowfile-attribute, and extract the required data fields from the stream file and place them in a custom stream attribute. The required data fields include PASS_CREATE_TIME and PASS_DATETIME. In a specific implementation of this invention, when the Destination of the attribute extraction component (EvaluateJsonPath) is set to flowfile-attribute, certain required data can be extracted from the stream file and placed in a user-defined stream attribute.

[0073] Step S13: The attribute update component (UpdateAttribute) processes the statistical fields of the stream file using custom stream attributes or rules. These statistical fields include PASS_CREATE_TIME, PASS_EIGHT_SEC, PASS_ONE_MINUTE, and PASS_REVERSIAL. Specifically, PASS_CREATE_TIME needs to be converted to timestamp format PASS_CREATE_TIME_STAMP for use with partitioned time periods on Kylin. In practical application, the attribute update component (UpdateAttribute) can use user-defined stream attributes or rules to update or add stream attributes to the stream file.

[0074] Step S14: Set the component property JoltTransformationDSL of the JSON format conversion component (JoltTransformJSON) to Modify-Overwrite, and fill in the statistical fields in the component property JoltSpecification of the JSON format conversion component, thereby merging the statistical fields into the stream file to form a result stream; In specific implementation of this invention, when the JoltTransformationDSL of the JSON format conversion component (JoltTransformJSON) is set to Modify-Overwrite, the JSON content to be merged can be filled in the JoltSpecification, and the filled JSON content can be merged into the stream file;

[0075] Step S15: The Kafka production component (PublishKafka) sends the processed result stream to the topic of the second Kafka; the Kafka production component (PublishKafka) is mainly used to send the stream file to the specified topic of Kafka.

[0076] It should be noted that ConsumeKafka, EvaluateJsonPath, UpdateAttribute, JoltTransformJSON, and PublishKafka are all common components on NIFI, and their specific functions are well known to those skilled in the art. Therefore, we will not go into detail about ConsumeKafka, EvaluateJsonPath, UpdateAttribute, JoltTransformJSON, PublishKafka, etc. here.

[0077] In a preferred embodiment of the present invention, please refer to the following: Figure 3 As shown, in step S2, the specific steps of defining the Model and Cube according to statistical requirements and constructing the Cube to obtain the pre-calculated result data are as follows:

[0078] Create a Model, select the data source table CARPASS, dimension fields (including PASS_DEVICE_UNID and DAY_START), and measure fields (including PASS_EIGHT_SEC, PASS_ONE_MINUTE, PASS_REVERSIAL, and PASS_UNRECOGNIZED) according to the statistical requirements, and set the partition (DAY_START) for the Model.

[0079] Create the Cube corresponding to the Model, select the Cube's dimension fields (including PASS_DEVICE_UNID and DAY_START), measure fields (including PASS_EIGHT_SEC, PASS_ONE_MINUTE and PASS_REVERSIAL), and expression (select SUM for the expression). At the same time, configure the aggregation group fields (including PASS_DEVICE_UNID and DAY_START), row key (PASS_DEVICE_UNID and DAY_START), and column family (including PASS_EIGHT_SEC, PASS_ONE_MINUTE and PASS_REVERSIAL) in the advanced settings.

[0080] Select the successfully created Cube and monitor the task through the Monitor to obtain pre-calculated result data.

[0081] In a preferred embodiment of the present invention, in step S2, the field names of the data source table CARPASS include device id, 8-second delay for data entry, 1-minute delay for data entry, inverted status, creation time, and vehicle passage time. The specific fields of the data source table CARPASS are shown in Table 1 below:

[0082] Table 1. Detailed field descriptions of the data source table.

[0083] field name Field Name Field type PASS_DEVICE_UNID Device ID VARCHAR2(32) PASS_EIGHT_SEC 8-second delay for data entry INT PASS_ONE_MINUTE Delayed entry into the warehouse by 1 minute INT PASS_REVERSIAL upside down INT PASS_CREATE_TIME Creation time DATE PASS_DATETIME Passing time DATE

[0084] In a preferred embodiment of the present invention, please refer to the following: Figure 4 As shown, step S3 specifically includes:

[0085] Step S31: The data table query component (QueryDatabaseTable) executes SQL statements to obtain pre-calculated result data from the data source table; the data table query component (QueryDatabaseTable) is used to execute SQL statements to query the database table and obtain the record file in the database;

[0086] Step S32: The Avro to JSON component (ConvertAvroToJSON) converts the pre-calculated result data into a stream file of operable JSON format; the Avro to JSON component (ConvertAvroToJSON) is used to convert the acquired record file into an operable JSON format;

[0087] Step S33: The JSON splitting component (SplitJson) splits the converted stream file into multiple JSON format stream files; the JSON splitting component (SplitJson) is used to split each record file into multiple separate stream files;

[0088] Step S34: Set the component property Destination of the attribute extraction component (EvaluateJsonPath) to flowfile-attribute, and extract the data fields to be used from the split flow file and put them into the custom flow attribute; when the Destination of the attribute extraction component (EvaluateJsonPath) is set to flowfile-attribute, certain data to be needed can be extracted from the flow file and put into the user-defined flow attribute.

[0089] Step S35: The attribute update component (UpdateAttribute) updates the attribute id; the attribute update component (UpdateAttribute) can use user-defined stream attributes or rules to update or add stream attributes of the stream file;

[0090] Step S36: The CryptographicHashAttribute component uses the MD5 algorithm to calculate the hash value of the attribute id as the data primary key; the CryptographicHashAttribute component is used to calculate the hash value of each specified attribute using the given algorithm and write it to the output attribute.

[0091] Step S37: Obtain device, location, and region information from the cache through the FetchDistributedMapCache component; the FetchDistributedMapCache component is used to calculate the cache key from the attributes of each incoming stream file and obtain the value from the distributed map cache associated with each cache key for each incoming stream file.

[0092] Step S38: Set the EvaluationMode component property of the ReplaceText component to Line-by-Line, and replace the content of the stream file with device, location, and region information, so that the attribute extraction component can obtain the attributes of device, location, and region information; the ReplaceText component is used to replace the content of the stream file line by line or entirely with regular expressions;

[0093] Step S39: Add the SQL statement for the attribute through the attribute update component (UpdateAttribute). The SQL execution component (ExecuteSQL) executes the SQL statement and stores the processed final data into the intermediate table. The SQL execution component (ExecuteSQL) is used to execute the provided SQL statement.

[0094] It should be noted that QueryDatabaseTable, ConvertAvroToJSON, SplitJson, EvaluateJsonPath, UpdateAttribute, CryptographicHashAttribute, FetchDistributedMapCache, ReplaceText, and ExecuteSQL are all common components on NIFI, and their specific functions are well known to those skilled in the art. Therefore, these components will not be described in detail here.

[0095] In the technical solution of this invention:

[0096] NIFI is an easy-to-use, powerful, and reliable data processing and distribution system that can move data within a system and provide tools for processing that data. NIFI is primarily used to connect files of different types, formats, and from different systems, and distribute them to the required locations after appropriate processing.

[0097] KYLIN is an open-source distributed analytics engine that provides a SQL query interface on top of Hadoop and multidimensional analysis (OLAP) capabilities to support ultra-large-scale data, job management and monitoring, and a user-friendly web interface for easy monitoring and use.

[0098] Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action streams of data from consumers on a website. Messages stored in Kafka come from any process called a "producer," and data can be allocated to different "partitions" and different "topics." Topics are used to categorize messages, and each message entering Kafka is placed under a topic. Within a partition, these messages are indexed and stored together with timestamps, and other processes called "consumers" can query messages from the partition.

[0099] A Model is the relationship between certain tables in a Hive database, i.e., a data model; a Cube is used to perform maximum pre-computation based on the Model and store the pre-computation results in HBase. Each Model corresponds to one Cube.

[0100] Example 2

[0101] Please see Figures 2 to 5 As shown, a preferred embodiment of the visualization processing device for big data statistics of the present invention is provided. The processing device includes a statistical field processing module, a data table creation module, and a result data processing module.

[0102] The statistical field processing module is used to consume the raw vehicle passage data in the topic of the first Kafka (i.e., Kafka1) on NIFI, process the statistical fields of the raw vehicle passage data, and send the processed result stream to the topic of the second Kafka (i.e., Kafka2).

[0103] The data table creation module is used to create a data source table CARPASS on KYLIN, and the data source of the data source table CARPASS is selected from the second Kafka topic; the Model and Cube are defined according to statistical requirements, and the Cube is constructed to obtain the pre-calculated result data;

[0104] The result data processing module is used to obtain the pre-calculated result data obtained from KYLIN on NIFI, define a process to process the obtained pre-calculated result data, and store the processed final data into an intermediate table.

[0105] This invention utilizes NIFI to provide a stream-based programming experience, enabling data extraction and storage through visual operations without cumbersome code development. Simultaneously, it leverages KYLIN for pre-computation, employing a space-for-time data processing model to improve query efficiency and performance. It also supports multi-dimensional data modeling, making data analysis more flexible and facilitating statistical calculations. By combining NIFI and KYLIN and applying them to big data statistics, this invention provides users with an intuitive web interface for operation and management, enabling automated statistical analysis without the need for complex code development. This effectively reduces development cycles and costs, facilitates maintenance, improves query efficiency and performance, and is applicable to various usage scenarios.

[0106] In a preferred embodiment of the present invention, please refer to the following: Figure 2 As shown, the statistical field processing module includes a vehicle data acquisition unit, a first field extraction unit, a statistical field processing unit, a statistical field merging unit, and a result stream sending unit;

[0107] The vehicle passage data acquisition unit is used to obtain raw vehicle passage data from the first Kafka topic through the Kafka consumer component (ConsumeKafka) on NIFI. The raw vehicle passage data is a stream file in JSON format. The Kafka consumer component (ConsumeKafka) is used to consume messages from Kafka, that is, to obtain a stream file in JSON format.

[0108] The first field extraction unit is used to set the component attribute Destination of the attribute extraction component (EvaluateJsonPath) to flowfile-attribute, and extract the data fields to be used from the stream file and place them in a custom stream attribute. The data fields to be used include PASS_CREATE_TIME and PASS_DATETIME. In a specific implementation of the present invention, when the Destination of the attribute extraction component (EvaluateJsonPath) is set to flowfile-attribute, certain required data can be extracted from the stream file and placed in a user-defined stream attribute.

[0109] The statistical field processing unit is used by the attribute update component (UpdateAttribute) to process the statistical fields of the stream file using custom stream attributes or rules. These statistical fields include PASS_CREATE_TIME, PASS_EIGHT_SEC, PASS_ONE_MINUTE, and PASS_REVERSIAL. Specifically, PASS_CREATE_TIME needs to be converted to timestamp format PASS_CREATE_TIME_STAMP for use with partitioned time periods on Kylin. In practical application, the attribute update component (UpdateAttribute) can use user-defined stream attributes or rules to update or add stream attributes to the stream file.

[0110] The statistical field merging unit is used to set the component property JoltTransformationDSL of the JSON format conversion component (JoltTransformJSON) to Modify-Overwrite, and fill in the statistical fields in the component property JoltSpecification of the JSON format conversion component, thereby merging the statistical fields into the stream file to form a result stream; in a specific implementation of the present invention, when the JoltTransformationDSL of the JSON format conversion component (JoltTransformJSON) is set to Modify-Overwrite, the JSON content to be merged can be filled in the JoltSpecification, and the filled JSON content can be merged into the stream file;

[0111] The result stream sending unit is used by the Kafka production component (PublishKafka) to send the processed result stream to a second Kafka topic; the Kafka production component (PublishKafka) is mainly used to send the stream file to the specified Kafka topic.

[0112] It should be noted that ConsumeKafka, EvaluateJsonPath, UpdateAttribute, JoltTransformJSON, and PublishKafka are all common components on NIFI, and their specific functions are well known to those skilled in the art. Therefore, we will not go into detail about ConsumeKafka, EvaluateJsonPath, UpdateAttribute, JoltTransformJSON, PublishKafka, etc. here.

[0113] In a preferred embodiment of the present invention, please refer to the following: Figure 3 As shown, in the data table creation module, the specific steps of defining the Model and Cube according to statistical requirements and constructing the Cube to obtain the pre-calculated result data are as follows:

[0114] Create a Model, select the data source table CARPASS, dimension fields (including PASS_DEVICE_UNID and DAY_START), and measure fields (including PASS_EIGHT_SEC, PASS_ONE_MINUTE, PASS_REVERSIAL, and PASS_UNRECOGNIZED) according to the statistical requirements, and set the partition (DAY_START) for the Model.

[0115] Create the Cube corresponding to the Model, select the Cube's dimension fields (including PASS_DEVICE_UNID and DAY_START), measure fields (including PASS_EIGHT_SEC, PASS_ONE_MINUTE and PASS_REVERSIAL), and expression (select SUM for the expression). At the same time, configure the aggregation group fields (including PASS_DEVICE_UNID and DAY_START), row key (PASS_DEVICE_UNID and DAY_START), and column family (including PASS_EIGHT_SEC, PASS_ONE_MINUTE and PASS_REVERSIAL) in the advanced settings.

[0116] Select the successfully created Cube and monitor the task through the Monitor to obtain pre-calculated result data.

[0117] In a preferred embodiment of the present invention, in the data table creation module, the field names of the data source table CARPASS include device id, 8-second delay for data entry, 1-minute delay for data entry, inverted, creation time, and vehicle passage time. The specific fields of the data source table CARPASS are shown in Table 1 below:

[0118] Table 1. Detailed field descriptions of the data source table.

[0119] field name Field Name Field type PASS_DEVICE_UNID Device ID VARCHAR2(32) PASS_EIGHT_SEC 8-second delay for data entry INT PASS_ONE_MINUTE Delayed entry into the warehouse by 1 minute INT PASS_REVERSIAL upside down INT PASS_CREATE_TIME Creation time DATE PASS_DATETIME Passing time DATE

[0120] In a preferred embodiment of the present invention, please refer to the following: Figure 4 As shown, the result data processing module specifically includes a result data acquisition unit, a format conversion unit, a file splitting unit, a second field extraction unit, an attribute ID update unit, an encryption unit, a cache acquisition unit, a file replacement unit, and a storage execution unit;

[0121] The result data acquisition unit is used by the data table query component (QueryDatabaseTable) to execute SQL statements and obtain pre-calculated result data from the data source table; the data table query component (QueryDatabaseTable) is used to execute SQL statements to query the database table and obtain the record file in the database;

[0122] The format conversion unit is used by the Avro to JSON component (ConvertAvroToJSON) to convert the pre-calculated result data into an operable JSON format stream file; the Avro to JSON component (ConvertAvroToJSON) is used to convert the acquired record file into an operable JSON format;

[0123] The file splitting unit is used by the JSON splitting component (SplitJson) to split the converted stream file into multiple JSON format stream files; the JSON splitting component (SplitJson) is used to split each record file into multiple separate stream files;

[0124] The second field extraction unit is used to set the component property Destination of the attribute extraction component (EvaluateJsonPath) to flowfile-attribute, and extract the data fields to be used from the split flow file and put them into a custom flow attribute; when the Destination of the attribute extraction component (EvaluateJsonPath) is set to flowfile-attribute, certain data to be needed can be extracted from the flow file and put into a user-defined flow attribute.

[0125] The attribute ID update unit is used by the attribute update component (UpdateAttribute) to update the attribute ID; the attribute update component (UpdateAttribute) can use user-defined stream attributes or rules to update or add stream attributes of the stream file;

[0126] The encryption unit is used by the attribute hash value encryption component (CryptographicHashAttribute) to calculate the hash value of the attribute id as the data primary key using the MD5 algorithm; the attribute hash value encryption component (CryptographicHashAttribute) is used to calculate the hash value of each specified attribute using a given algorithm and write it to the output attribute.

[0127] The cache acquisition unit is used to acquire device, location, and region information in the cache through the FetchDistributedMapCache component; the FetchDistributedMapCache component is used to calculate the cache key from the attributes of each incoming stream file and acquire the value from the distributed map cache associated with each cache key for each incoming stream file.

[0128] The file replacement unit is used to set the EvaluationMode component property of the text replacement component (ReplaceText) to Line-by-Line, and replace the content of the stream file with device, location, and region information, so that the attribute extraction component can obtain the attributes of device, location, and region information; the text replacement component (ReplaceText) is used to replace the content of the stream file line by line or entirely with regular expressions;

[0129] The storage execution unit is used to add SQL statements for attributes through the attribute update component (UpdateAttribute), and the SQL execution component (ExecuteSQL) executes the SQL statements to store the processed final data into an intermediate table. The SQL execution component (ExecuteSQL) is used to execute the provided SQL statements.

[0130] It should be noted that QueryDatabaseTable, ConvertAvroToJSON, SplitJson, EvaluateJsonPath, UpdateAttribute, CryptographicHashAttribute, FetchDistributedMapCache, ReplaceText, and ExecuteSQL are all common components on NIFI, and their specific functions are well known to those skilled in the art. Therefore, these components will not be described in detail here.

[0131] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for visualizing processing of big data statistics, characterized in that: The processing method comprises the following steps: Step S1, consuming original vehicle data in a topic of a first Kafka on NIFI, processing statistical fields of the original vehicle data, and sending the processed result stream to a topic of a second Kafka; Step S2, creating a data source table on KYLIN, selecting the topic of the second Kafka as the data source of the data source table, defining a Model and a Cube according to statistical requirements, and constructing the Cube to obtain pre-computed result data; Step S3, obtaining the pre-computed result data obtained by KYLIN on NIFI, defining a process to process the obtained pre-computed result data, and storing the processed final data into an intermediate table, specifically comprising: Step S31, a data table query component executes an SQL statement to obtain pre-computed result data of the data source table; Step S32, an Avro to JSON component converts the pre-computed result data into an operable JSON format stream file; Step S33, a JSON splitting component splits the converted stream file into multiple JSON format stream files; Step S34, setting a component attribute Destination of an attribute extraction component to flowfile-attribute, and extracting data fields required to be used from the split stream file into a custom stream attribute; Step S35, an attribute updating component updates an attribute id; Step S36, an attribute hash value encryption component calculates a hash value of the attribute id as a data primary key by using an MD5 algorithm; Step S37, a mapping cache obtaining component obtains device, point and area information in the cache; Step S38, setting a component attribute EvaluationMode of a text replacement component to Line-by-Line, and replacing content of the stream file by using the device, point and area information, so that the attribute extraction component can obtain attributes of the device, point and area information; Step S39, adding an attribute by using an attribute updating component, and executing an SQL statement by using an SQL execution component to store the processed final data into the intermediate table.

2. The method of claim 1, wherein: The step S1 specifically comprises: Step S11, obtaining original vehicle data from a topic of a first Kafka by using a Kafka consumption component on NIFI, wherein the original vehicle data is a stream file in a JSON format; Step S12, setting a component attribute Destination of an attribute extraction component to flowfile-attribute, and extracting data fields required to be used from the stream file into a custom stream attribute; Step S13, using the custom stream attribute or a rule to process statistical fields of the stream file by using an attribute updating component; Step S14, set the component attribute JoltTransformationDSL of the JSON format conversion component to Modify-Overwrite, and fill in the statistical field in the component attribute JoltSpecification of the JSON format conversion component, so as to realize merging the statistical field into the stream file to form a result stream; Step S15, the Kafka production component sends the processed result stream to the topic of the second Kafka.

3. The method of claim 1, wherein: In the step S2, the Model and the Cube are defined according to the statistical requirement, and the Cube is constructed to obtain the pre-computed result data, which is specifically: A Model is created, the data source table, dimension field and measure field of the Model are selected according to the statistical requirement, and the partition of the Model is set; The Cube corresponding to the Model is created, the dimension field, measure field and expression of the Cube are selected, and the aggregation group field, row key and column family in the advanced setting are configured; The created Cube is selected, the task is monitored through the Monitor, and thus the pre-computed result data is obtained.

4. The method of claim 1, wherein: In the step S2, the field names of the data source table include device id, delay 8 seconds into the database, delay 1 minute into the database, inverted, creation time and passing time.

5. A device for visualizing processing of big data statistics, characterized in that it comprises: The processing device includes a statistical field processing module, a data table creation module and a result data processing module; The statistical field processing module is configured to consume the original passing data in the topic of the first Kafka on the NIFI, process the statistical field of the original passing data, and send the processed result stream to the topic of the second Kafka; The data table creation module is configured to create a data source table on the KYLIN, and the data source table is selected from the topic of the second Kafka; the Model and the Cube are defined according to the statistical requirement, and the Cube is constructed to obtain the pre-computed result data; The result data processing module is configured to obtain the pre-computed result data of the KYLIN on the NIFI, define a process to process the obtained pre-computed result data, and store the processed final data into an intermediate table, and the result data processing module specifically includes a result data obtaining unit, a format conversion unit, a file splitting unit, a second field extracting unit, an attribute id updating unit, an encryption unit, a cache obtaining unit, a file replacing unit and a storage executing unit; The result data obtaining unit is configured to execute an SQL statement by the data table query component, and obtain the pre-computed result data of the data source table; The format conversion unit is configured to convert the pre-computed result data into an operable stream file in JSON format by the Avro to JSON component; The file splitting unit is configured to split the converted stream file into multiple stream files in JSON format by the JSON splitting component; The second field extraction unit is configured to set a component attribute Destination of the attribute extraction component to flowfile-attribute, and extract data fields required to be used from the split flow file and place the data fields in a custom flow attribute; The attribute id updating unit is configured to update the attribute id by using the attribute updating component; The encryption unit is configured to use an MD5 algorithm to calculate a hash value of the attribute id as a data primary key by using the attribute hash value encryption component; The cache obtaining unit is configured to obtain device, point and area information in the cache by using the mapping cache obtaining component; The file replacing unit is configured to set a component attribute EvaluationMode of the text replacing component to Line-by-Line, and replace content of the flow file by using the device, point and area information, so that the attribute extraction component can obtain attributes of the device, point and area information; The storage executing unit is configured to store the final data processed by using the attribute updating component to add the attribute in the SQL statement into the intermediate table by using the SQL execution component to execute the SQL statement.

6. The device for visualizing big data statistics according to claim 5, wherein: The statistical field processing module specifically comprises a passing vehicle data obtaining unit, a first field extraction unit, a statistical field processing unit, a statistical field merging unit and a result flow sending unit. The passing vehicle data obtaining unit is configured to obtain original passing vehicle data from a topic of a first Kafka by using a Kafka consumption component on the NIFI, and the original passing vehicle data is a flow file in a JSON format. The first field extraction unit is configured to set a component attribute Destination of the attribute extraction component to flowfile-attribute, and extract data fields required to be used from the flow file and place the data fields in a custom flow attribute. The statistical field processing unit is configured to process statistical fields of the flow file by using a custom flow attribute or a rule by using the attribute updating component. The statistical field merging unit is configured to set a component attribute JoltTransformationDSL of the JSON format conversion component to Modify-Overwrite, and fill in the statistical fields in a component attribute JoltSpecification of the JSON format conversion component, so as to merge the statistical fields into the flow file to form a result flow. The result flow sending unit is configured to send the result flow processed to a topic of a second Kafka by using a Kafka production component.

7. The device for visualizing big data statistics according to claim 5, wherein: In the data table creating module, the Model and the Cube are defined according to the statistical requirement, and the pre-computed result data obtained by constructing the Cube is specifically as follows: A Model is created, data source tables, dimension fields and measurement fields of the Model are selected according to the statistical requirement, and the Model is set to be partitioned; A Cube corresponding to the Model is created, dimension fields, measurement fields and expressions of the Cube are selected, and an aggregation group field, a row key and a column family in advanced settings are configured; Select the created Cube successfully, monitor the task through Monitor, and obtain the pre-computed result data.

8. The device for visualizing big data statistics of claim 5, wherein: In the data table creation module, the field names of the data source table include device id, delay 8 seconds into the database, delay 1 minute into the database, upside down, creation time and passing time.

Images