Government affair data statistical method, system and equipment based on real-time calculation and medium

Abstract

The invention discloses a government affair data statistical method, system and device based on real-time calculation and a medium, and belongs to the technical field of government affair services. In order to solve the technical problems of how to uniformly converge and analyze government affair data with various types, wide data and uneven quality and how to realize real-time statistics of massive government affair data, the technical scheme is as follows: real-time data acquisition: deploying flink middleware by adopting a cluster deployment mode, establishing an flink task, and establishing flink middleware; obtaining business data of a source from different data sources as original data, and realizing communication between the different data sources through flink middleware; data preprocessing: carrying out preprocessing operations of cleaning, duplicate removal and conversion on the original data to obtain preprocessed data; the method comprises the following steps: deploying an Elasticsearch cluster; analyzing service requirements, and creating statistical indexes; according to the analyzed service requirements, a query condition of an Elasticsearch cluster is configured; creating a timer; and designing a page prototype for data display.

Description

Technical Field

[0001] The present invention relates to the field of government service technology, and in particular to a government data statistics method, system, device and medium based on real-time calculation. Background Art

[0002] With the in-depth development of the information age, the processing and utilization of data in government services is particularly important. The field of government services involves the processing and statistical analysis of a large amount of real-time data, and traditional data processing methods often cannot meet actual needs in terms of processing speed and accuracy.

[0003] At present, most of the data statistics methods in the field of government services are based on batch processing mode, which cannot achieve real-time statistics and analysis of data. In addition, with the development of big data technology, the amount of government service data has exploded, which has put forward higher requirements for the speed and accuracy of data processing. On the one hand, government services involve a wide range of business types, a wide range of data sources, and uneven data quality, which brings great difficulty to data statistics. On the other hand, government services have high requirements for the real-time nature of data and need to respond quickly and accurately to various government service needs. However, traditional data processing methods often cannot meet these requirements in terms of processing speed and accuracy.

[0004] In summary, the existing data processing methods in the field of government services have the following shortcomings:

[0005] ① Difficulty in statistics: There are many types of government service businesses, data sources are wide-ranging, and data quality is uneven, which makes it impossible to complete comprehensive statistical analysis using traditional statistical methods;

[0006] ② Timeliness cannot be guaranteed: With the continuous growth of government data, massive amounts of data cannot be counted quickly and effectively based on traditional statistical methods, resulting in the displayed statistical results not being timely.

[0007] Therefore, how to uniformly aggregate and analyze government data of various types, extensive data and uneven quality and how to achieve real-time statistics of massive government data are technical problems that need to be solved urgently. Summary of the invention

[0008] The technical task of the present invention is to provide a government data statistics method, system, equipment and medium based on real-time computing to solve the problem of how to uniformly aggregate and analyze government data of various types, extensive data and uneven quality, and how to realize real-time statistics of massive government data.

[0009] The technical task of the present invention is achieved in the following way: a government affairs data statistics method based on real-time calculation, the method is as follows:

[0010] Real-time data collection: Flink middleware is deployed in clusters. By creating Flink tasks, business data from different data sources is obtained as raw data. The Flink middleware is used to connect different data sources, thereby achieving efficient data reading.

[0011] Data preprocessing: Perform preprocessing operations such as cleaning, deduplication and conversion on the original data to obtain the preprocessed data;

[0012] Deploy an Elasticsearch cluster: extract the preprocessed data into the Elasticsearch cluster by creating an index;

[0013] Analyze business needs and create statistical indicators;

[0014] Configure the query conditions of the Elasticsearch cluster according to the business needs of the analysis;

[0015] Create a timer: control the timing within the set time interval, call the statistical conditions of the Elasticsearch cluster, design the library table structure, and store the statistical data in the relational database;

[0016] Design a page prototype for data display: display data through the vue+springboot front end, and provide data query on the back end to realize page display to meet the user's data result display requirements.

[0017] Preferably, the data sources include databases, log files and message queues (kafka).

[0018] Preferably, when the preprocessed data is extracted into the Elasticsearch cluster by creating an index, the flink streaming method is adopted to extract data from different data sources through different connectors provided by the flink middleware.

[0019] Preferably, the database uses MySQL, reads binlog logs, and then performs structured data extraction. The read data will be stored in the source layer of the flink middleware. After the source layer obtains the data, the sink implementation in flink is customized and developed. The data is then pre-processed by cleaning, deduplication, and conversion, and the data is stored in the Elasticsearch cluster.

[0020] As a preferred option, analyze business needs and create statistical indicators as follows:

[0021] When counting the number of cases handled each week in each district, analyze the data needs based on the statistical indicators;

[0022] Grouping by district;

[0023] Filter the data according to the time requirements (data screening needs to be limited to this week) and obtain the corresponding screening results;

[0024] Sum the screening results.

[0025] As a preferred option, the query conditions of the Elasticsearch cluster are configured as follows according to the business requirements of the analysis:

[0026] Time filter: add time interval conditions, the range of the conditions is the set time interval;

[0027] Sum statistics: Based on the full amount of zoning data, sum statistics are performed on each zone.

[0028] A government data statistics system based on real-time computing, which uses real-time computing technology to collect and process government service data in real time, and realizes rapid and accurate statistics and analysis of government service data by building an efficient data statistics model; the data statistics model includes:

[0029] The access layer is used to obtain data from different data sources, including databases, log files, and message queues (kafka);

[0030] The computing layer is used to deploy the Flink middleware in a cluster deployment mode. By creating Flink tasks, the source business data is obtained from different data sources as raw data, and data is extracted through the Flink middleware to achieve connectivity between different data sources. It is also used to perform pre-processing operations such as cleaning, deduplication, and conversion on the raw data to obtain pre-processed data.

[0031] The storage layer is used to store the preprocessed data in the Elasticsearch cluster by creating indexes;

[0032] The service layer is used to use timers to calculate results in real time through time intervals and aggregate the results into a relational database. At the same time, it adopts an agile development model to respond quickly and provide users with a visual interface.

[0033] The application layer is used to display government service data in the form of regional statistics, time statistics and user portraits.

[0034] As a preferred embodiment, the working process of the system is as follows:

[0035] (1) Analyze the data source structure and sort out the existing data structure types, including databases, log files, and message queues;

[0036] (2) Deploy the flink middleware, develop a package for extracting data, execute different flink tasks according to different data types, and start the flink task to extract data into the Elasticsearch cluster;

[0037] (3) Design a timer and write a scheduled task service for the timer to perform real-time statistics on the data in the Elasticsearch cluster and store the results in the database;

[0038] (4) Deploy front-end applications as service layer support and provide interface display.

[0039] An electronic device comprising: a memory and at least one processor;

[0040] Wherein, the memory stores a computer program;

[0041] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the government data statistics method based on real-time calculation as described above.

[0042] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the government data statistics method based on real-time calculation as mentioned above.

[0043] The government data statistics method, system, device and medium based on real-time calculation of the present invention have the following advantages:

[0044] (i) The present invention is superior to traditional data statistics methods in terms of processing speed and accuracy, and can significantly improve the efficiency and accuracy of government services; at the same time, the stability and reliability of the system are tested, and the results show that the system has good stability and reliability;

[0045] (ii) The present invention reduces the difficulty of data statistics, uniformly aggregates and analyzes data of various types, extensive data and uneven quality, and performs data modeling based on statistical indicators; it guarantees the timeliness of data, and achieves real-time statistics, real-time analysis and real-time display of statistical results for massive data;

[0046] (III) The present invention mainly uses two open source middleware, flink and Elasticsearch, and uses a relational database, such as mysql, for data storage; the timer uses the timer provided by the spring framework; the data display adopts a customized development model, and agile development is performed based on VUE+springboot, which realizes the processing of a large amount of real-time data in a short time, and performs database processing according to corresponding rules, reflecting the high efficiency of data processing and the accuracy of data results. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention is further described below in conjunction with the accompanying drawings.

[0048] Attached Figure 1 It is a structural diagram of the data statistical model;

[0049] Attached Figure 2 This is a schematic diagram of the flink middleware and Elasticsearch cluster. DETAILED DESCRIPTION

[0050] The government data statistics method, system, device and medium based on real-time calculation of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.

[0051] Embodiment 1:

[0052] The present invention provides a method for government data statistics based on real-time calculation, which is as follows:

[0053] S1. Real-time data collection: Flink middleware is deployed in cluster mode. By creating Flink tasks, business data from different data sources is obtained as raw data. The Flink middleware is used to connect different data sources, thereby achieving efficient data reading.

[0054] S2. Data preprocessing: Perform preprocessing operations such as cleaning, deduplication and conversion on the original data to obtain preprocessed data;

[0055] S3, deploying the Elasticsearch cluster: extracting the preprocessed data into the Elasticsearch cluster by creating an index;

[0056] S4. Analyze business needs and create statistical indicators;

[0057] S5. Configure the query conditions of the Elasticsearch cluster according to the business requirements of the analysis;

[0058] S6. Create a timer: control the timing within the set time interval, call the statistical conditions of the Elasticsearch cluster, design the library table structure, and store the statistical data in the relational database;

[0059] S7. Design a page prototype for data display: display data through the vue+springboot front end, and provide data query on the back end to realize page display to meet the user's data result display requirements.

[0060] The data sources in step S1 of this embodiment include a database, a log file, and a message queue (kafka).

[0061] In step S3 of this embodiment, when extracting the preprocessed data into the Elasticsearch cluster by creating an index, the flink streaming method is adopted to extract data from different data sources through different connectors provided by the flink middleware.

[0062] The database in this embodiment uses MySQL. By reading the binlog log, the data is structured and extracted. The read data will be stored in the flink middleware source layer, as shown in the attached figure. Figure 2 As shown in the figure, after the source layer obtains the data, the sink implementation in Flink is customized and developed, and then the data is preprocessed by cleaning, deduplication, and conversion, and the data is stored in the Elasticsearch cluster.

[0063] The specific details of analyzing business needs and creating statistical indicators in step S4 of this embodiment are as follows:

[0064] S401. When counting the number of cases handled in each zone each week, analyze the data requirements based on the statistical indicators;

[0065] S402, grouping according to the districts;

[0066] S403, filter the data according to the time requirement (data filtering needs to be limited to this week) to obtain the corresponding filtering results;

[0067] S404, summing up the screening results.

[0068] In step S5 of this embodiment, the query conditions of the Elasticsearch cluster are configured according to the analyzed business requirements as follows:

[0069] S501, time screening: add a time interval condition, the range of the condition is the set time interval;

[0070] S502, summing and statistics: performing summing and statistics on each zone based on the full amount of zone data.

[0071] Embodiment 2:

[0072] As attached Figure 1 As shown, this embodiment provides a government data statistics system based on real-time computing. The system uses real-time computing technology to collect and process government service data in real time, and builds an efficient data statistics model to achieve rapid and accurate statistics and analysis of government service data; the data statistics model includes:

[0073] The access layer is used to obtain data from different data sources, including databases, log files, and message queues (kafka);

[0074] The computing layer is used to deploy the Flink middleware in a cluster deployment mode. By creating Flink tasks, the source business data is obtained from different data sources as raw data, and data is extracted through the Flink middleware to achieve connectivity between different data sources. It is also used to perform pre-processing operations such as cleaning, deduplication, and conversion on the raw data to obtain pre-processed data.

[0075] The storage layer is used to store the preprocessed data in the Elasticsearch cluster by creating indexes;

[0076] The service layer is used to use timers to calculate results in real time through time intervals and aggregate the results into a relational database. At the same time, it adopts an agile development model to respond quickly and provide users with a visual interface.

[0077] The application layer is used to display government service data in the form of regional statistics, time statistics and user portraits.

[0078] The working process of the system is as follows:

[0079] (1) Analyze the data source structure and sort out the existing data structure types, including databases, log files, and message queues;

[0080] (2) Deploy the flink middleware, develop a package for extracting data, execute different flink tasks according to different data types, and start the flink task to extract data into the Elasticsearch cluster;

[0081] (3) Design a timer and write a scheduled task service for the timer to perform real-time statistics on the data in the Elasticsearch cluster and store the results in the database;

[0082] (4) Deploy front-end applications as service layer support and provide interface display.

[0083] Embodiment 3:

[0084] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0085] Wherein, the memory stores computer-executable instructions;

[0086] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the government data statistics method based on real-time calculation in any embodiment of the present invention.

[0087] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0088] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0089] Embodiment 4:

[0090] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the government data statistics method based on real-time calculation in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0091] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0092] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0093] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0094] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A government data statistics method based on real-time calculation, characterized in that: The method is as follows: Real-time data collection: Flink middleware is deployed in cluster mode. By creating Flink tasks, business data from different data sources is obtained as raw data, and the connectivity between different data sources is achieved through Flink middleware. Data preprocessing: Perform preprocessing operations such as cleaning, deduplication and conversion on the original data to obtain the preprocessed data; Deploy an Elasticsearch cluster: extract the preprocessed data into the Elasticsearch cluster by creating an index; Analyze business needs and create statistical indicators; Configure the query conditions of the Elasticsearch cluster according to the business needs of the analysis; Create a timer: control the timing within the set time interval, call the statistical conditions of the Elasticsearch cluster, design the library table structure, and store the statistical data in the relational database; Design a page prototype for data display: display data through the vue+springboot front end, and provide data query on the back end to realize page display to meet the user's data result display requirements.

2. The government affairs data statistics method based on real-time calculation according to claim 1 is characterized in that: Data sources include databases, log files, and message queues.

3. The government data statistics method based on real-time calculation according to claim 1 is characterized in that: When the preprocessed data is extracted into the Elasticsearch cluster by creating an index, the flink streaming method is adopted to extract data from different data sources through different connectors provided by the flink middleware.

4. The government affairs data statistics method based on real-time calculation according to claim 2 is characterized in that: The database uses MySQL. By reading binlog logs and then performing structured data extraction, the read data will be stored in the source layer of the flink middleware. After the source layer obtains the data, the sink implementation in flink is customized and developed. The data is then pre-processed by cleaning, deduplication and conversion, and the data is stored in the Elasticsearch cluster.

5. The government data statistics method based on real-time calculation according to claim 1 is characterized in that: Analyze business needs and create statistical indicators as follows: When counting the number of cases handled each week in each district, analyze the data needs based on the statistical indicators; Grouping by district; Filter the data according to the time requirements and obtain the corresponding filtering results; Sum the screening results.

6. The government affairs data statistics method based on real-time calculation according to claim 1 is characterized in that: The query conditions for configuring the Elasticsearch cluster according to the business requirements of the analysis are as follows: Time filter: add time interval conditions, the range of the conditions is the set time interval; Sum statistics: Based on the full amount of zoning data, sum statistics are performed on each zone.

7. A government data statistics system based on real-time computing, characterized in that: The system uses real-time computing technology to collect and process government service data in real time, and builds a data statistical model to achieve rapid and accurate statistics and analysis of government service data; The statistical models include: The access layer is used to obtain data from different data sources, including databases, log files, and message queues; The computing layer is used to deploy the Flink middleware in a cluster deployment mode. By creating Flink tasks, the source business data is obtained from different data sources as raw data, and data is extracted through the Flink middleware to achieve connectivity between different data sources. It is also used to perform pre-processing operations such as cleaning, deduplication, and conversion on the raw data to obtain pre-processed data. The storage layer is used to store the preprocessed data in the Elasticsearch cluster by creating indexes; The service layer is used to use timers to calculate results in real time through time intervals and aggregate the results into a relational database. At the same time, it adopts an agile development model to respond quickly and provide users with a visual interface. The application layer is used to display government service data in the form of regional statistics, time statistics and user portraits.

8. The government affairs data statistics system based on real-time calculation according to claim 7 is characterized in that: The working process of the system is as follows: (1) Analyze the data source structure and sort out the existing data structure types, including databases, log files, and message queues; (2) Deploy the flink middleware, develop a package for extracting data, execute different flink tasks according to different data types, and start the flink task to extract data into the Elasticsearch cluster; (3) Design a timer and write a scheduled task service for the timer to perform real-time statistics on the data in the Elasticsearch cluster and store the results in the database; (4) Deploy front-end applications as service layer support and provide interface display.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the government data statistics method based on real-time calculation as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the government data statistics method based on real-time calculation as described in any one of claims 1 to 6.

Images