Basic data statistical analysis method and system

Abstract

The invention relates to the field of data governance technologies and management, and particularly provides a basic data statistical analysis method and system.The basic data statistical analysis method is based on a database log system and comprises the steps that firstly, configuration services are developed and include analyzing storage objects and conditions, and whether logs are stored or not is judged according to configuration conditions after log analysis; then, a log analysis service is developed, and database logs are analyzed regularly according to SQL grammar through log analysis; and finally, developing a data analysis service, and performing simple analysis and statistics on the basic data by the data analysis service according to the data table operation record obtained by the log analysis service. Compared with the prior art, the method has the advantages that the data table operation can be acquired and stored in a manner of regularly analyzing and extracting the log data of the database, and the statistical analysis of the basic data is realized under the condition of not changing a service system.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology and management, and specifically provides a basic data statistical analysis method and system. Background Art

[0002] Logging is a mechanism in the database that records all modification operations and events on the database, including operations such as insert, update, and delete, as well as information such as the operation status of the database system and abnormal events.

[0003] In the process of grassroots data governance, basic data collection and statistical analysis are important parts. Basic data has the characteristics of wide variety, many data items, simple business, and large changes in data requirements. When doing simple statistical analysis on this part of data, we usually focus on statistical analysis of additions, deletions, and modifications.

[0004] If user operations are recorded in the business system, a large amount of repeated coding will often occur, and data requirements change greatly, requiring frequent changes to the business system. Summary of the invention

[0005] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a basic data statistical analysis method with strong practicability.

[0006] A further technical task of the present invention is to provide a basic data statistical analysis system that is reasonably designed, safe and applicable.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A basic data statistical analysis method, based on the database log system, first, develops a configuration service, including parsing storage objects and conditions, and determines whether to save the logs according to the configuration conditions after log parsing;

[0009] Then, we developed a log parsing service that regularly parses database logs based on SQL syntax.

[0010] Finally, develop a data analysis service that performs simple analysis and statistics on the basic data based on the data table operation records obtained by the log parsing service.

[0011] Furthermore, in the configuration service, the configuration items include the name of the data to be parsed, the table name corresponding to the data, the operation type, and the extraction cycle.

[0012] Furthermore, in the log parsing service, the parsing service is executed periodically. After parsing each log data, the log information that meets the conditions will be stored according to the configured filtering conditions. The stored content includes the table name, operation type and time.

[0013] Further, in the data analysis service, the analysis service provides two parts: real-time query and periodic statistical analysis. The real-time query queries the number of various operations of the current data and the total amount of data;

[0014] The periodic statistical analysis realizes the periodic statistical storage of data. The statistical period and statistical content are configurable, and the data analyzed at regular intervals are used for historical analysis of the data.

[0015] A basic data statistical analysis system, based on the database log system, firstly, develops configuration services, including parsing storage objects and conditions, and determines whether to save the logs according to the configuration conditions after log parsing;

[0016] Then, we developed a log parsing service that regularly parses database logs based on SQL syntax.

[0017] Finally, develop a data analysis service that performs simple analysis and statistics on the basic data based on the data table operation records obtained by the log parsing service.

[0018] Furthermore, in the configuration service, the configuration items include the name of the data to be parsed, the table name corresponding to the data, the operation type, and the extraction cycle.

[0019] Furthermore, in the log parsing service, the parsing service is executed periodically. After parsing each log data, the log information that meets the conditions will be stored according to the configured filtering conditions. The stored content includes the table name, operation type and time.

[0020] Further, in the data analysis service, the analysis service provides two parts: real-time query and periodic statistical analysis. The real-time query queries the number of various operations of the current data and the total amount of data;

[0021] The periodic statistical analysis realizes the periodic statistical storage of data. The statistical period and statistical content are configurable, and the data analyzed at regular intervals are used for historical analysis of the data.

[0022] Compared with the prior art, the basic data statistical analysis method and system of the present invention have the following outstanding beneficial effects:

[0023] The present invention obtains and saves data table operations by regularly parsing and extracting database log data, thereby achieving statistical analysis of basic data without changing the business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Attached Figure 1 It is a flowchart of a basic data statistical analysis method. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] A best embodiment is given below:

[0028] like Figure 1 As shown, a basic data statistical analysis method in this embodiment is based on a database log system. First, a configuration service is developed, including parsing storage objects and conditions. After the log is parsed, it is determined whether to save it according to the configuration conditions. The configuration items should include the name of the data to be parsed, the table name corresponding to the data, the operation type (add, delete, modify), and the extraction cycle.

[0029] Then, a log parsing service is developed. Log parsing regularly parses database logs according to SQL syntax. The parsing service is executed periodically. After parsing each log data, it will store the log information that meets the conditions according to the configured filtering conditions. The stored content includes table name, operation type, and time.

[0030] Finally, develop a data analysis service that performs simple analysis and statistics on the basic data based on the data table operation records obtained by the log parsing service.

[0031] In the data analysis service, the analysis service provides two parts: real-time query and regular statistical analysis. The real-time query queries the number of various operations and the total amount of data of the current data;

[0032] Regular statistical analysis realizes periodic statistical storage of data. The statistical period and statistical content are configurable, and the data analyzed regularly is used for historical analysis of data.

[0033] Statistical analysis of basic data in the grassroots data governance process can be achieved by regularly parsing database logs, storing operation records of corresponding data tables, and analyzing the stored operation records.

[0034] The log parsing service reads the configuration content of the configuration service and executes periodically according to the configuration. During the parsing process, the data is filtered by table name and operation type according to the configuration read from the configuration service, and the filtered operation data is stored. The storage should include table name, operation type, and operation time for use by the data analysis service.

[0035] Based on the above method, a basic data statistical analysis system in this embodiment is based on a database log system. First, a configuration service is developed, including parsing storage objects and conditions. After the log is parsed, it is determined whether to save according to the configuration conditions;

[0036] Then, we developed a log parsing service that regularly parses database logs based on SQL syntax.

[0037] Finally, develop a data analysis service that performs simple analysis and statistics on the basic data based on the data table operation records obtained by the log parsing service.

[0038] Among them, in the configuration service, the configuration items include the name of the data to be parsed, the table name corresponding to the data, the operation type and the extraction cycle.

[0039] In the log parsing service, the parsing service is executed periodically. After parsing each log data, it will store the log information that meets the conditions according to the configured filtering conditions. The stored content includes the table name, operation type and time.

[0040] In the data analysis service, the analysis service provides two parts: real-time query and regular statistical analysis. The real-time query queries the number of various operations and the total amount of data of the current data;

[0041] Regular statistical analysis realizes periodic statistical storage of data. The statistical period and statistical content are configurable, and the data analyzed regularly is used for historical analysis of data.

[0042] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0043] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A basic data statistical analysis method, characterized in that: Based on the database log system, first, develop the configuration service, including parsing the storage objects and conditions, and determine whether to save the logs according to the configuration conditions after log parsing; Then, we developed a log parsing service that regularly parses database logs based on SQL syntax. Finally, develop a data analysis service that performs simple analysis and statistics on the basic data based on the data table operation records obtained by the log parsing service.

2. A basic data statistical analysis method according to claim 1, characterized in that: In the configuration service, the configuration items include the name of the data to be parsed, the table name corresponding to the data, the operation type, and the extraction cycle.

3. A basic data statistical analysis method according to claim 2, characterized in that: In the log parsing service, the parsing service is executed periodically. After parsing each log data, it will store the log information that meets the conditions according to the configured filtering conditions. The stored content includes the table name, operation type and time.

4. A basic data statistical analysis method according to claim 3, characterized in that: In the data analysis service, the analysis service provides two parts: real-time query and regular statistical analysis. The real-time query queries the number of various operations and the total amount of data of the current data; The periodic statistical analysis realizes the periodic statistical storage of data. The statistical period and statistical content are configurable, and the data analyzed at regular intervals are used for historical analysis of the data.

5. A basic data statistical analysis system, characterized in that: Based on the database log system, first, develop the configuration service, including parsing the storage objects and conditions, and determine whether to save the logs according to the configuration conditions after log parsing; Then, we developed a log parsing service that regularly parses database logs based on SQL syntax. Finally, develop a data analysis service that performs simple analysis and statistics on the basic data based on the data table operation records obtained by the log parsing service.

6. A basic data statistical analysis system according to claim 5, characterized in that: In the configuration service, the configuration items include the name of the data to be parsed, the table name corresponding to the data, the operation type, and the extraction cycle.

7. A basic data statistical analysis system according to claim 6, characterized in that: In the log parsing service, the parsing service is executed periodically. After parsing each log data, it will store the log information that meets the conditions according to the configured filtering conditions. The stored content includes the table name, operation type and time.

8. A basic data statistical analysis system according to claim 7, characterized in that: In the data analysis service, the analysis service provides two parts: real-time query and regular statistical analysis. The real-time query queries the number of various operations and the total amount of data of the current data; The periodic statistical analysis realizes the periodic statistical storage of data. The statistical period and statistical content are configurable, and the data analyzed at regular intervals are used for historical analysis of the data.

Images