Data aggregation method, system, device and medium

Abstract

The invention provides a data aggregation method, system and device and a medium, and belongs to the technical field of data aggregation. The data aggregation method comprises the following steps: firstly, creating user permission information capable of reading related data handled by each business process in an original database; setting configuration information, and determining the range of a data table needing to be monitored in the original database according to the configuration information; the changed data table in the original database is collected based on the user permission information and the determined range of the data table needing to be monitored in the original database, and the collected changed data table is received; the received changed data table is analyzed and stored; and finally, converging the stored data tables. The problems that the real-time performance of data aggregation is poor, and the production data of a power plant cannot be monitored in real time are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data aggregation, and in particular relates to a data aggregation method, system, device and medium. Background Art

[0002] In order to meet the needs of artificial intelligence and unified monitoring, power plants need to use data centers to aggregate data from various business systems. In the process of power plant data aggregation, data aggregation from various business databases to data centers is the most common scenario.

[0003] Currently, the methods used by power plants for data aggregation are: 1) incremental aggregation through the time stamp timing of the business table. The method is simple, has good stability, and is not easy to lose data. However, this method not only must the target table have fields indicating the update time and deletion marks, but also has poor real-time performance of data aggregation, so it is impossible to monitor the production data of the power plant; 2) Real-time synchronization of the change data is achieved by capturing the database change log, such as Flink and Canal tools. However, during the use of Flink tools, each table must be configured separately, which is quite cumbersome. If you want to add new tables to the business library, you need to manually add new synchronization tasks, and automatic aggregation of data tables cannot be achieved. Although the Canal tool can monitor the entire database, it can only be adapted to MySQL database and has a small scope of application. Summary of the invention

[0004] The object of the present invention is to provide a data aggregation method, system, device and medium, which are used to solve the problem in the prior art that the real-time performance of data aggregation is poor and the production data of a power plant cannot be monitored in real time.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a data aggregation method, comprising the following steps: Create user permission information in the original database that can read relevant data for each business process; Set configuration information, and determine the range of data tables that need to be monitored in the original database according to the configuration information; Based on the user authority information and the determined range of data tables that need to be monitored in the original database, the changed data tables in the original database are collected, and the collected changed data tables are received; Parse and store the received changed data tables; Aggregate the stored data tables.

[0006] A further improvement of the present invention is that the change refers to the addition, modification or deletion of a data table in the original database, as well as the modification of the data table structure or the modification of the data table name.

[0007] A further improvement of the present invention is that the configuration information includes original database login information, a data table whitelist and a data table blacklist.

[0008] In a second aspect, the present invention provides a data aggregation system, including a source data module, a data acquisition module, a data consumption module and a target data module; The source data module is used to create user authority information in the original database that can read relevant data for various business process handling; The data acquisition module is used to set configuration information, and determine the range of data tables that need to be monitored in the original database according to the configuration information; the data acquisition module is also used to collect changed data tables in the original database based on user authority information and the determined range of data tables that need to be monitored in the original database, and send the collected changed data tables; The data consumption module is used to receive the collected changed data table, parse and store the received changed data table; The target data module is used to aggregate the stored data tables.

[0009] A further improvement of the present invention is that the data collection module sends the collected changed data table in JSON format. A further improvement of the present invention is that the data acquisition module sends the collected changed data table in a manner including interface call, file transfer and TCP transmission, wherein the interface call refers to the data acquisition module directly calling the relevant interface of the data consumption module, the file transfer refers to the data acquisition module writing the changed data table into a file and putting it into a designated folder, and the data consumption module reads it periodically, and the TCP transmission refers to the data acquisition module connecting to the TCP server unit of the data consumption module and sending the changed data table directly to the TCP server unit of the data consumption module. A further improvement of the present invention is that the process of the data consumption module parsing the received changed data table is: A) Parse the changed data table name; B) Query the mapping relationship configured in the target data module. If the changed data table has a corresponding mapping table name, the data table change operation is performed in the target data module according to the mapping table name; if the changed data table does not have a corresponding mapping table name, the data table change operation is performed in the target data module according to the unchanged data table name.

[0010] In a third aspect, the present invention provides a data aggregation system, including a user authority information creation module, a data table range determination module, a changed data table receiving module, a changed data table parsing and storage module, and a data table aggregation module; The user authority information creation module is used to create user authority information in the original database that can read relevant data for various business process handling; The data table range determination module is used to set configuration information and determine the range of the data table to be monitored in the original database according to the configuration information; The changed data table receiving module is used to collect the changed data tables in the original database based on the user authority information and the determined range of the data tables that need to be monitored in the original database, and receive the collected changed data tables; The changed data table parsing and storage module is used to parse and store the received changed data table; The data table aggregation module is used to aggregate the stored data tables.

[0011] In a fourth aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the data aggregation method introduced above when executing the computer program.

[0012] In a fifth aspect, the present invention provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data aggregation method introduced above.

[0013] Compared with the prior art, the present invention has the following beneficial effects: The present invention belongs to an improved invention. Compared with the existing data aggregation method, the data aggregation method proposed by the present invention first creates user authority information in the original database that can read the relevant data of each business process, and then sets the configuration information. According to the configuration information, the range of data tables that need to be monitored in the original database is determined; based on the user authority information and the determined range of data tables that need to be monitored in the original database, the changed data tables in the original database are collected, the collected changed data tables are received, the received changed data tables are parsed and stored, and finally the stored data tables are aggregated. It can be seen that when the present invention successfully receives the changed data tables, all data tables in the original database can be monitored in real time. Even if a data table is newly created or deleted in the original database, it will be automatically included in the monitoring range without manual configuration. Compared with the prior art, the present invention effectively solves the problem of poor real-time data aggregation and inability to monitor the production data of power plants in real time in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the data aggregation method of the present invention; Figure 2 It is a working principle diagram between the modules of the data aggregation system of the present invention; Figure 3 is a schematic diagram of a data aggregation system of the present invention; Figure 4 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0015] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0016] The data aggregation method proposed in the present invention first creates user authority information in the original database that can read the relevant data of each business process, then sets configuration information, and determines the range of data tables that need to be monitored in the original database based on the configuration information, and then collects the changed data tables in the original database based on the user authority information and the determined range of data tables that need to be monitored in the original database, receives the collected changed data tables, and then parses and stores the received changed data tables, and finally aggregates the stored data tables. Compared with the prior art, the present invention effectively solves the problem of poor real-time data aggregation and inability to monitor the production data of power plants in real time in the prior art.

[0017] Embodiment 1: The flow chart of the data aggregation method of the present invention is as follows Figure 1 As shown, the data aggregation method of the present invention comprises the following steps: S1. Create user permission information in the original database that can read relevant data for each business process.

[0018] S2. Set configuration information, and determine the range of data tables in the original database that need to be monitored based on the configuration information.

[0019] S3. Based on the user authority information and the determined range of data tables in the original database that need to be monitored, the changed data tables in the original database are collected, and the collected changed data tables are received.

[0020] S4. Parse and store the received changed data table.

[0021] S5. Aggregate the stored data tables.

[0022] Embodiment 2: The data aggregation system of this embodiment includes a source data module, a data acquisition module, a data consumption module and a target data module.

[0023] The source data module is used to create user permission information in the original database that can read relevant data for each business process.

[0024] The data collection module is used to set configuration information and determine the range of data tables in the original database that need to be monitored based on the configuration information. The data collection module is also used to collect the changed data tables in the original database based on user authority information and the determined range of data tables in the original database that need to be monitored, and send the collected changed data tables.

[0025] The data consumption module is used to receive the collected changed data table, and parse and store the received changed data table.

[0026] The target data module is used to aggregate the stored data tables.

[0027] Changes refer to the addition, modification or deletion of data tables in the original database, as well as the modification of data table structure or data table name. Configuration information includes original database login information, data table whitelist, data table blacklist, data transmission method between data acquisition module and data consumption module, and address name and port name of data consumption module (for example, http: / / 10.41.41.55:8080 / receiver, where receiver is a relative path, 8080 is a port name, and http: / / 10.41.41.55:8080 is an address name). The data table whitelist means that multiple data tables that need to be synchronized can be written, separated by semicolons. If this item is empty, all data tables of the source database are synchronized by default. The data table blacklist means that multiple data tables that do not need to be synchronized can be written, separated by semicolons. If this item is empty, it is considered that there are no data tables that do not need to be synchronized; when both the data table whitelist and the data table blacklist have content, only the data table whitelist is effective. The transmission format of data table changes is shown in Table 1.

[0028] Table 1 Transmission format of data table changes

[0029] The KEY field is in JSON format, and its structure is: {"primary key field name":"actual value"}.

[0030] CDATA and ADATA are in JSON format, and their structure is: {"field name 1":"field value 1","field name 2":"field value 2","field name 3":"field value 3","field name 4":"field value 4"}.

[0031] The data collection module of this embodiment sends the collected changed data table in JSON format. In this embodiment, the data acquisition module sends the collected changed data table in a manner including interface call, file transfer and TCP transfer. Interface call refers to the data acquisition module directly calling the relevant interface of the data consumption module. File transfer refers to the data acquisition module writing the changed data table into a file and putting it into a specified folder, and the data consumption module reads it regularly. TCP transmission refers to the TCP server unit (also called TCP server module) of the data acquisition module and the data consumption module, which directly sends the changed data table to the TCP server unit (also called TCP server module) of the data consumption module. The process of parsing the received changed data table by the data consumption module in this embodiment is as follows: A) Parse the changed data table name; B) Query the mapping relationship configured in the target data module. If the changed data table has a corresponding mapping table name, the data table change operation is performed in the target data module according to the mapping table name; if the changed data table does not have a corresponding mapping table name, the data table change operation is performed in the target data module according to the unchanged data table name.

[0032] The working principle between the source data module (also called source database), the data acquisition module (also called data collector), the data consumption module (also called data consumer) and the target data module (also called target database) in this embodiment is as follows: Figure 2 shown.

[0033] The data acquisition module of this embodiment includes Debezium connector (log capture component), change data parsing module, interface call module, file writing module and TCP client module. Specifically, in this embodiment, Debezium connector is used to read the log of the source database and capture the change data. The change data parsing module is used to parse and convert the captured change data table into a specified transmission format. The interface call module, file writing module and TCP client module are used to send the changed data table to the data consumption module according to the configured transmission mode.

[0034] The data consumption module of this embodiment includes an interface publishing module, a file reading module, a TCP server module, a change data parsing module and a data writing module. Specifically, the interface publishing module, the file reading module and the TCP server module in this embodiment are used to receive the changed data table sent by the data collector. The change data parsing module is used to parse the received changed data table and read the mapping relationship to perform data conversion. The data writing module is used to store the changed data table in the target database.

[0035] The databases supported by the source data module and the target data module in this embodiment are MySQL database, PostgreSQL database, Oracle database, Renmin University of China Jincang database and Vastbase database.

[0036] Embodiment 3: The schematic diagram of the data aggregation system of the present invention is as follows Figure 3 As shown, the data aggregation system of the present invention includes a user authority information creation module, a data table range determination module, a changed data table receiving module, a changed data table parsing and storage module and a data table aggregation module.

[0037] The user authority information creation module is used to create user authority information in the original database that can read relevant data for various business process handling.

[0038] The data table range determination module is used to set configuration information and determine the range of the data table to be monitored in the original database according to the configuration information; The changed data table receiving module is used to collect the changed data tables in the original database based on the user authority information and the determined range of the data tables that need to be monitored in the original database, and receive the collected changed data tables; The changed data table parsing and storage module is used to parse and store the received changed data table; The data table aggregation module is used to aggregate the stored data tables.

[0039] Embodiment 4: See also Figure 4 As shown, the present invention also provides an electronic device 100 for a data aggregation method; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0040] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the data aggregation method described in Example 2 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0041] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0042] The memory 101 in the electronic device 100 stores a plurality of instructions to implement the data aggregation method, and the processor 102 can execute the plurality of instructions to implement: Create user permission information in the original database that can read relevant data for each business process; Set configuration information, and determine the range of data tables that need to be monitored in the original database according to the configuration information; Based on the user authority information and the determined range of data tables that need to be monitored in the original database, the changed data tables in the original database are collected, and the collected changed data tables are received; Parse and store the received changed data tables; Aggregate the stored data tables.

[0043] Embodiment 5: If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0044] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0045] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0046] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0047] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A data aggregation method, characterized in that: The following steps are involved: Create user permission information in the original database that can read relevant data for each business process; Set configuration information, and determine the range of data tables that need to be monitored in the original database according to the configuration information; Based on the user authority information and the determined range of data tables that need to be monitored in the original database, the changed data tables in the original database are collected, and the collected changed data tables are received; Parse and store the received changed data tables; Aggregate the stored data tables.

2. The data aggregation method according to claim 1, characterized in that: The change refers to the addition, modification or deletion of a data table in the original database, as well as the modification of the data table structure or the modification of the data table name.

3. The data aggregation method according to claim 1, characterized in that: The configuration information includes original database login information, a data table whitelist and a data table blacklist.

4. A data aggregation system, characterized in that: It includes source data module, data acquisition module, data consumption module and target data module; The source data module is used to create user authority information in the original database that can read relevant data for various business process handling; The data acquisition module is used to set configuration information, and determine the range of data tables that need to be monitored in the original database according to the configuration information; the data acquisition module is also used to collect changed data tables in the original database based on user authority information and the determined range of data tables that need to be monitored in the original database, and send the collected changed data tables; The data consumption module is used to receive the collected changed data table, parse and store the received changed data table; The target data module is used to aggregate the stored data tables.

5. The data aggregation system according to claim 4, characterized in that: The data collection module sends the collected changed data table in JSON format.

6. The data aggregation system according to claim 4, characterized in that: The data acquisition module sends the collected changed data table in a manner including interface call, file transfer and TCP transfer. The interface call refers to the data acquisition module directly calling the relevant interface of the data consumption module. The file transfer refers to the data acquisition module writing the changed data table into a file and putting it into a designated folder, and the data consumption module reads it periodically. The TCP transmission refers to the data acquisition module connecting to the TCP server unit of the data consumption module and sending the changed data table directly to the TCP server unit of the data consumption module.

7. The data aggregation system according to claim 4, characterized in that: The process of the data consumption module parsing the received changed data table is as follows: A) Parse the changed data table name; B) Query the mapping relationship configured in the target data module. If the changed data table has a corresponding mapping table name, perform the data table change operation in the target data module according to the mapping table name; If the changed data table does not have a corresponding mapping table name, the data table change operation is performed in the target data module according to the unchanged data table name.

8. A data aggregation system, characterized in that: It includes a user authority information creation module, a data table range determination module, a changed data table receiving module, a changed data table parsing and storage module, and a data table aggregation module; The user authority information creation module is used to create user authority information in the original database that can read relevant data for various business process handling; The data table range determination module is used to set configuration information and determine the range of the data table to be monitored in the original database according to the configuration information; The changed data table receiving module is used to collect the changed data tables in the original database based on the user authority information and the determined range of the data tables that need to be monitored in the original database, and receive the collected changed data tables; The changed data table parsing and storage module is used to parse and store the received changed data table; The data table aggregation module is used to aggregate the stored data tables.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the data aggregation method according to any one of claims 1 to 3 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data aggregation method according to any one of claims 1 to 3 are implemented.

Images