Intelligent park data analysis compensation system

Abstract

The application discloses a kind of intelligent park data analysis compensation systems, comprising: rule configuration module, for configuring data model;Data collector, with rule configuration module communication;Data extraction distributor, with rule configuration module communication, also with data collector communication, for extracting and distributing the data that data collector collects after being configured by rule configuration module;Fink computing task module, with data extraction distributor communication;Result data distributor, with Fink computing task module communication, also with external data user communication connection.The intelligent park data analysis compensation system of the application, in the field of park, can start data computing task by using simple function configuration using this scheme, cooperate with low code component or front-end engineering, real-time data cleaning and display can be simply realized;The data between different systems can be combined, and unified management and analysis is carried out;As a general tool, it can reduce repeated data development work.

Description

TECHNICAL FIELD

[0001] The present application relates to a data analysis compensation system, more particularly to a smart park data analysis compensation system. BACKGROUND

[0002] The big data analysis under the Internet of Things environment is to collect device data and synchronize business data of the business system module, to summarize and process the collected data according to the analysis purpose, to analyze the processing results by selecting different methods and ideas, and to output the indicators.

[0003] However, the past data analysis needs to use a large amount of resources and time, and needs to use the entire big data environment, which is not conducive to data analysis in a low-cost small environment. SUMMARY

[0004] In view of the deficiencies in the prior art, the purpose of the present application is to provide a smart park data analysis compensation system which internally defines many general computing functions and can adapt to most park data computing scenarios.

[0005] To achieve the above purpose, the present application provides the following technical scheme: a smart park data analysis compensation system, comprising:

[0006] A rule configuration module is configured to configure a data model.

[0007] A data collector is in communication with the rule configuration module and is configured to communicate with an external data provider after being configured by the rule configuration module to collect data content by subscribing to data or data pulling.

[0008] A data extraction distributor is in communication with the rule configuration module and the data collector and is configured to extract and distribute the data collected by the data collector after being configured by the rule configuration module.

[0009] A Fink computing task module is in communication with the data extraction distributor to receive the data distributed by the data extraction distributor and to calculate the data to output the result data.

[0010] A result data distributor is in communication with the Fink computing task module and an external data user and is configured to receive the result data output by the Fink computing task module and to transmit the result data to the data user.

[0011] As a further improvement of the present application, the information of the data model configured by the rule configuration module is data collection configuration, calculation configuration and release configuration, wherein the data collection configuration is configuration of request address, parameters and scheduling period.

[0012] As a further improvement of the present application, the specific steps of the data collector collecting data content are as follows: step 1, identifying the type of data source of the external data provider, which is divided into API, Mysql and Kafka; step 2, when the type is identified as API and Mysq1, a unified timer is built in, and the timer is used to uniformly manage and schedule the pulling task;

[0013] Step 3, when the type is identified as Kafka, a client with message queue is built in, and the client is used to listen to data pushing;

[0014] Among them, the way of the timer uniformly managing and scheduling the pulling task includes time cron expression analysis, retry after pulling failure and guarantee of incremental data, wherein the guarantee of incremental data is to guarantee the reduction of data repeated consumption processing in the case of service exception by applying redis.

[0015] As a further improvement of the present application, the specific way of the Fink calculation task module for data calculation is to first calculate the event queue FIFO, then perform window calculation and real-time output calculation according to the obtained event queue FIFO, and then pack the window calculation result and the real-time output calculation result into result data and output.

[0016] As a further improvement of the present application, the specific steps of calculating the event queue FIFO are to receive the data output by API, Mysql and Kafka through the Fink stream data receiver, and then match and push the events to the window calculation and real-time output calculation according to the configuration rules of the rule configuration module.

[0017] As a further improvement of the present application, the specific way of the result data distributor for distributing result data is to call JAVA service to notify the result data, and then store and distribute the result data by the JAVA service.

[0018] As a further improvement of the present application, the Fink calculation task module performs data sedimentation after outputting the calculation result, then performs asynchronous processing on the result data, and finally outputs the asynchronous processed result data.

[0019] The beneficial effects of this invention are that data rules can be effectively configured through the rule configuration module, data content can be effectively collected through the data collector, and the data extraction and distribution device distributes the data according to the rules configured by the rule configuration module. Then, the Fink calculation task module calculates the data to obtain the corresponding result data. Finally, the result data distributor transmits the result data to the data user. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the smart park data analysis and compensation system applying the present invention;

[0021] Figure 2 for Figure 1 A flowchart of the calculation process in the Fink computing task module;

[0022] Figure 3 This is a schematic diagram of the data accumulation process;

[0023] Figure 4 This is a sample page for the smart park data analysis and compensation system of the present invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0025] Reference Figures 1 to 4 As shown, a smart park data analysis and compensation system according to this embodiment includes:

[0026] The rules configuration module is used to configure the data model;

[0027] The data collector communicates with the rule configuration module and is used to connect with external data providers after being configured by the rule configuration module, so as to collect data content by subscribing to data or pulling data.

[0028] The data extractor and distributor communicates with the rule configuration module and the data collector. It is used to extract and distribute the data collected by the data collector after being configured by the rule configuration module.

[0029] The Fink computation task module communicates with the data extraction and distribution unit to receive data distributed by the data extraction and distribution unit, perform calculations on the data, and output the calculated results.

[0030] The result data distributor is in communication with the Fink computing task module and also connected with an external data user to receive the result data output by the Fink computing task module and transmit to the data user, and in the process of using the system of the embodiment, only the system needs to be carried between the data provider and the data user, so that the data transmission and calculation can be effectively realized, and the embodiment learns from the flink streaming computing framework in big data, and is matched with the self-developed data analysis system to realize the scheme of the application, and the ability of data processing and analysis and index output can be provided in a small-cost and general way in the scenario of the smart park.

[0031] As an improved specific embodiment, the rule configuration module configures information of the data model as data collection configuration, calculation configuration and release configuration, wherein the data collection configuration is configuration of a request address, parameters and a scheduling period, and as a simple data analysis tool, the ability of defining data sources and calculation methods needs to be provided before providing the calculation ability; for example, data is acquired by calling an API of a third party or event data is subscribed and consumed through a message middleware, and then functions such as accumulation and difference are defined to clean the data; for the calculation method, JEXL (Java Expression Language Engine) is used to support the dynamic running of some basic expressions and scripts, and a batch of functions such as SUM (sum), AVG (average), SQL_COL (SQL column) and the like are built in by using the feature of dynamically calling Java methods; for the data source, the present application has realized the configuration definition and access of data sources such as Mysql, Kafka, Mqtt and API; taking the API as an example, only the request address, parameters and scheduling period need to be configured, and the collection module can dynamically pull data according to the configuration and add the data to the calculation queue, so that the configuration of the data model can be effectively realized.

[0032] As an improved specific embodiment, the specific steps of the data collector collecting data content are as follows: step 1, identifying the type of the data source of the external data provider, which is divided into API, Mysql and Kafka; step 2, when the type is identified as API and Mysq1, a unified timer is built in to uniformly manage and schedule the pulling task through the timer;

[0033] Step 3, when the type is identified as Kafka, a client with a message queue is built in to listen to data pushing through the client;

[0034] The timer unified management and scheduling of the pull task mode includes time cron expression analysis, retry after pull failure, and guarantee of incremental data, wherein the guarantee of incremental data is to reduce data repeated consumption processing in the case of service exception by applying redis, after the data model is configured, the collection module acquires data according to different types of data sources by using different strategies, such as API, Mysql and other active pull data acquisition modes, an overall timer is internally implemented, which is used for unified management and scheduling of pull tasks, including not only time cron expression analysis, retry after pull failure, guarantee of incremental data and the like, wherein the incremental data will also use redis to guarantee the case of reducing data repeated consumption processing in the case of service exception, such as Kafka and other pushed data, the product internally implements a message queue client to listen to data push, which is also a common real-time stream computing mode structure. As an improved specific implementation, the specific way of the Fink computing task module for data calculation is to first calculate the event queue FIFO, then perform window calculation and real-time output calculation according to the obtained event queue FIFO, and then package the window calculation result and the real-time output calculation result into result data and output, the specific steps of calculating the event queue FIFO are receiving the data output by API, Mysql and Kafka through the Fink stream data receiver, and then matching and pushing events to window calculation and real-time output calculation according to the configuration rules of the rule configuration module, the Flink stream computing framework is introduced in the application, and the main computing process is built by using the characteristics of stateful computing and event-driven stream computing; Event-driven stream: processing in units of events, it extracts data from one or more event streams, and triggers calculation, state update or other external actions according to incoming events; this speeds up the calculation frequency and can also obtain real-time data results faster; Stateful computing: state management is to save the intermediate results of the operator in the stream computing process in the memory or file system, so that the current event value can be integrated and calculated with the historical value after the next event enters the operator; the embodiment relies on the characteristics of Flink to realize T+1 time window calculation, for example, to calculate the hourly power consumption, after the data event after 13:30 is reported to the computing service, the data aggregation calculation between 12:00 and 12:59:59 will be ended, and the data result of this hour will be output; at the same time, we also use the memory storage technology to do the stateful calculation of non-event window, associate and integrate the last calculation structure with the input data, and complete real-time calculation and output.

[0035] As an embodiment of the improvement, the specific way of the result data distributor to distribute the result data is to call the JAVA service to notify the result data, and then the JAVA service stores and distributes the result data, so that the JAVA service can be used to distribute the result data.

[0036] As an embodiment of the improvement, the Fink computing task module performs data sinking after outputting the computing result, then performs asynchronous processing on the result data, and finally outputs the asynchronous processing result data. In the aspect of outputting the data result, a unified data format and storage calling and adaptation (time + data value) are used in the general data storage scene. Therefore, the data sinking is used to realize the storage and calling of the data, and the asynchronous processing result is used to ensure the efficiency of the processing flow.

[0037] In the embodiment, a unified data storage method is also used.

[0038] In the storage format, the main information stored includes event aggregation time and computing value in the time, which is suitable for the scene of querying the aggregated data in the time range.

[0039] In the storage technology selection, different databases can be selected according to different project configurations and data conditions. Currently, the schemes of mysql, ClickHousr and InfluxDB are provided for selection.

[0040] In the embodiment, the data publishing method provided to the outside is mainly the general API, and the general query interface is provided. The data format query of the commonly used data chart is supported, such as the column chart, pie chart and top data query. In cooperation with the front end, the page example in the Figure 4 In addition to the general API, the latest result data is pushed to the business user in the form of message push. For example, the page can subscribe to the specified webSocket to listen to the real-time result data, and configure the dynamic change of the page display value.

[0041] As described above, the embodiment defines many general computing functions, which can adapt to most park data computing scenes. If the scene involves complex or high-order computing methods, further function development process is needed to enrich the computing ability through function extension, which is similar to the form of plug-in expansion, so as to adapt to complex scenes.

[0042] The above merely describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-described embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A smart park data analysis and compensation system, characterized in that: The application relates to a data processing system, which comprises the following parts: a rule configuration module for configuring a data model; a data collector in communication with the rule configuration module, which is connected with an external data provider after being configured by the rule configuration module to collect data content by subscribing data or data pulling, and the specific steps are as follows: Step 1: identifying the type of data source of the external data provider, which is divided into API, Mysql and Kafka; Step 2: when the type is identified as API and Mysql, a unified timer is built to uniformly manage and schedule the pulling task through the timer; Step 3: when the type is identified as Kafka, a client with a message queue is built to listen to data pushing through the client; wherein the way of uniformly managing and scheduling the pulling task by the timer comprises the analysis of time cron expression, the retry after pulling failure and the guarantee of incremental data, wherein the guarantee of incremental data is to reduce the repeated consumption of data in the case of service exception by applying redis; a data extraction distributor in communication with the rule configuration module and also in communication with the data collector, which is used to extract and distribute the data collected by the data collector after being configured by the rule configuration module; a Fink computing task module in communication with the data extraction distributor to receive the data distributed by the data extraction distributor, and output the result data after computing the data; the specific way of computing data by the Fink computing task module is as follows: firstly, computing the event queue FIFO: the data output by API, Mysql and Kafka is received through the Fink stream data receiver; then, window computing and real-time output computing: the event queue FIFO obtained by matching and pushing computing according to the configuration rule of the rule configuration module is matched to the window for real-time computing and output, and then the window computing result and the real-time output computing result are packaged into result data and output; a result data distributor in communication with the Fink computing task module and also connected with an external data user to receive the result data output by the Fink computing task module and transmit the result data to the data user. 2.The smart park data analysis compensation system of claim 1, wherein: The information of the rule configuration module for configuring the data model is data collection configuration, computing configuration and publishing configuration, wherein the data collection configuration is the configuration of request address, parameter and scheduling period. 3.The smart park data analysis compensation system of claim 1, wherein: The specific way of the result data distributor for distributing the result data is to call JAVA service to notify the result data, and then the JAVA service stores and distributes the result data. 4.The smart park data analysis compensation system of claim 1, wherein: The Fink computing task module performs data sedimentation after outputting the computing result, then performs asynchronous processing on the result data, and finally outputs the result data after asynchronous processing.

Images