Data aggregation method and apparatus

Abstract

The application provides a data collection method and device, which can be used in the financial field or other technical fields. The method comprises the following steps: when a time trigger condition is met, searching for to-be-collected data in a database according to preset query parameters; and sending the to-be-collected data to a subscription party of the to-be-collected data, so that the subscription party sends the to-be-collected data to a data collector for data collection. The device is used for executing the above method. The data collection method and device provided by the embodiment of the application can automatically collect to-be-collected data through an aggregation query method, and actively push the to-be-collected data, thereby getting rid of the dependence on batch submission at fixed time intervals, and improving the effectiveness of data collection.

Description

Technical Field

[0001] This invention relates to the financial field or other technical fields, specifically to a data collection method and apparatus. Background Technology

[0002] In large financial enterprises, to ensure uninterrupted 24 / 7 service for core business operations, multiple physical data centers are typically created to guarantee data security and high system availability. These data centers are usually physically deployed in different cities or different locations within the same city, ensuring mutual disaster recovery. If an emergency occurs in one location or city, services can be switched to another location to continue providing services and ensuring the normal operation of core business operations.

[0003] During core business operations, these operations are typically evenly distributed across multiple data centers for multi-active processing. Key transaction data is often distributed across multiple data centers, and is aggregated daily through asynchronous batch processing to support subsequent data analysis and mining. While asynchronous batch processing allows for scheduled data aggregation, it also presents several challenges: First, it cannot adapt to real-time data aggregation, or requires increasing the number of batch runs to shorten the data aggregation interval. This method necessitates underlying modifications to existing core business systems and may create performance bottlenecks for transaction systems experiencing rapid data growth. Second, periodic data reporting may also lead to data omissions or delays due to occasional system malfunctions or human error processing, requiring subsequent manual queries and reprocessing to resolve.

[0004] In summary, there is currently no good method to achieve real-time data collection in the process of data aggregation in multiple locations and centers. Summary of the Invention

[0005] To address the problems in the prior art, embodiments of the present invention provide a data collection method and apparatus that can at least partially solve the problems existing in the prior art.

[0006] On the one hand, the present invention provides a data aggregation method, comprising:

[0007] When the time-triggered conditions are met, the database is searched for the data to be collected according to the preset query parameters;

[0008] The data to be aggregated is sent to the subscribers of the data to be aggregated, so that the subscribers send the data to the data aggregator for data aggregation.

[0009] Optionally, the query parameters include at least one of the following: query start time, query end time, data table name, data cleanup cycle, and / or field name.

[0010] Optionally, the step of searching for the data to be collected in the database according to preset query parameters when the time trigger condition is met includes:

[0011] When the time-triggered conditions are met, determine whether to perform an aggregate query based on the preset query start time, query end time, and data cleanup cycle;

[0012] If an aggregation query is performed, the data to be aggregated will be searched in the latest data table and the historical data table corresponding to the preset data table name.

[0013] Optionally, the method further includes: if it is determined that no aggregation query will be performed, then searching for the data to be aggregated in the latest data table or historical data table corresponding to the preset data table name.

[0014] Optionally, sending the data to be aggregated to the subscribers of the data to be aggregated includes:

[0015] The data to be collected, the search time of the data to be collected, and the data source information of the data to be collected are packaged together to generate a data package of the data to be collected.

[0016] The data packet is sent to the subscriber of the data to be aggregated.

[0017] Optionally, sending the data to be aggregated to the subscriber of the data to be aggregated, so that the subscriber sends the data to the data aggregator for data aggregation, includes:

[0018] Based on the data type of the data to be aggregated, the data to be aggregated is sent to subscribers who subscribe to the data to be aggregated of that data type, so that the subscribers send the data to be aggregated to the data aggregator for data aggregation.

[0019] Optionally, the subscriber and the data aggregator are nodes on the target blockchain network.

[0020] On the other hand, the present invention provides a data collection device, comprising:

[0021] The search module is used to search for data to be collected in the database according to preset query parameters when the time trigger condition is met.

[0022] The sending module is used to send the data to be collected to the subscribers of the data to be collected, so that the subscribers send the data to the data aggregator for data collection.

[0023] Optionally, the query parameters include at least one of the following: query start time, query end time, data table name, data cleanup cycle, and / or field name.

[0024] Optionally, the search module is specifically used for:

[0025] When the time-triggered conditions are met, determine whether to perform an aggregate query based on the preset query start time, query end time, and data cleanup cycle;

[0026] If an aggregation query is performed, the data to be aggregated will be searched in the latest data table and the historical data table corresponding to the preset data table name.

[0027] Optionally, the search module is further configured to: if it is determined that no aggregation query will be performed, search for the data to be aggregated in the latest data table or historical data table corresponding to the preset data table name.

[0028] Optionally, the sending module is specifically used for:

[0029] The data to be collected, the search time of the data to be collected, and the data source information of the data to be collected are packaged together to generate a data package of the data to be collected.

[0030] The data packet is sent to the subscriber of the data to be aggregated.

[0031] Optionally, the sending module is specifically used to: send the data to be collected to a subscriber that subscribes to the data to be collected of that data type, so that the subscriber sends the data to be collected to the data aggregator for data collection.

[0032] Optionally, the subscriber and the data aggregator are nodes on the target blockchain network.

[0033] In another aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the data collection method described in any of the above embodiments.

[0034] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data collection method described in any of the above embodiments.

[0035] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the data aggregation method described in any of the above embodiments.

[0036] The data collection method and apparatus provided in this invention, when a time-triggered condition is met, searches for data to be collected in a database according to preset query parameters; the data to be collected is then sent to the subscribers of the data to be collected, so that the subscribers send the data to the data collector for data collection. In this way, data to be collected can be automatically collected and proactively pushed through an aggregation query method, eliminating the need for batch reporting on a regular schedule and improving the effectiveness of data collection. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0038] Figure 1 This is a flowchart illustrating a data aggregation method provided in an embodiment of the present invention.

[0039] Figure 2 This is a partial flowchart of a data collection method provided in an embodiment of the present invention.

[0040] Figure 3 This is a partial flowchart of a data collection method provided in an embodiment of the present invention.

[0041] Figure 4 This is a schematic diagram of the structure of a target blockchain network built using blockchain technology according to an embodiment of the present invention.

[0042] Figure 5 This is a schematic diagram of the structure of a data collection device according to an embodiment of the present invention.

[0043] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0045] The terms "first," "second," etc., used in this document do not specifically refer to any order or sequence, nor are they intended to limit the invention; they are merely used to distinguish elements or operations described using the same technical terms.

[0046] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0047] The term "and / or" as used herein includes any or all of the things mentioned.

[0048] The data collection method provided in this embodiment of the invention can be executed by a computer, among other things.

[0049] Figure 1 This is a flowchart illustrating a data collection method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the data collection method provided in this embodiment of the invention includes:

[0050] S101. When the time triggering condition is met, search for the data to be collected in the database according to the preset query parameters;

[0051] In step S101, the probe program can be triggered to collect data to be aggregated in a rolling manner at pre-set time interval parameters, such as 10 minutes. In one embodiment of the present invention, the database includes detailed transaction data generated by various data centers of financial institutions. The probe program filters the transaction data generated by financial institutions in the database through aggregation query methods, and reserves a personalized probe window to support updating data by timestamp. It also supports setting specific query parameters in SQL form. In this way, it adapts to different types of databases of various financial institutions and retains key transaction data (i.e., data to be aggregated).

[0052] S102. Send the data to be collected to the subscriber of the data to be collected, so that the subscriber sends the data to the data collector for data collection.

[0053] In step S102, Kafka can be used for data transmission, establishing a transmission channel for data providers and data consumers (i.e., subscribers of the data to be aggregated). Kafka is an open-source message engine system with significant performance advantages. It supports real-time and accurate message publishing, and through the AMQP protocol, it is compatible with cross-platform transmission, making it adaptable to various information systems of major financial institutions.

[0054] Specifically, a program unit is created to publish messages to the Kafka cluster. This unit can publish messages (data to be aggregated) to any object in a topic. Message publishing typically has two modes: queuing and publish-subscribe. In queuing mode, consumers can read messages from the server simultaneously, but each message is read by only one consumer. In publish-subscribe mode, messages are broadcast to all consumers. To ensure that the reported data is accurately retrieved by consumers, in one embodiment of this invention, the unit uses queuing mode to implement message publishing.

[0055] Optionally, sending the data to be aggregated to the subscribers of the data to be aggregated, so that the subscribers send the data to the data aggregator for data aggregation, may include: sending the data to be aggregated to the subscribers of the data to be aggregated according to the data type of the data to be aggregated, so that the subscribers send the data to the data aggregator for data aggregation.

[0056] In this embodiment, since the data is usually composed of multiple data packets, in order to better adapt to multiple data patterns, the Kafka cluster can allocate two topics, IMPMESSAGE_GROUP and RETMESSAGE_GROUP, to push regular aggregated data and special aggregated data respectively. If financial institutions or regulatory agencies have other data needs, other topics can be added to place more data information, and then consumers can obtain the data. The consumers can subscribe to one or more topics.

[0057] This invention provides a data aggregation method. When a time-triggered condition is met, data to be aggregated is searched in a database according to preset query parameters. The data to be aggregated is then sent to its subscribers, who in turn forward the data to a data aggregator for aggregation. This method automatically collects and proactively pushes data to be aggregated through aggregated query methods, eliminating the need for periodic batch reporting and improving the effectiveness of information aggregation.

[0058] Optionally, the query parameters include at least one of the following: query start time, query end time, data table name, data cleanup cycle, and / or field name.

[0059] In one embodiment of this invention, the query parameters for data collection may include the query start date (uqrybegindate), the query end date (uqryenddate), the table name (tablename), the cleanup period (cleantime), and the conditional SQL parameters (field names) (whereqry). `uqrybegindate` and `uqryenddate` can be set via the page, defaulting to the current server system date; the table name is the data table for the specified data collection business entity; for tables with excessively high transaction volume, the business system may have a periodic cleanup mechanism, storing historical data in a history table. The cleanup period parameter `cleantime` represents the cleanup period for the business system to periodically clean the data table. For example, assuming the table name is table A, as business volume increases, the data volume in table A will increase. Therefore, table A will periodically move historical data to the `A_history` table to clean up the historical data in table A. Assuming `cleantime` is 24 hours, if data collection requires the collection of data from the current 48 hours, it is necessary to query the 24-hour data from table A and the 24-hour data from table A_history, merge them, and then summarize.

[0060] In addition, fields in the data table often contain information that is not sensitive to data collection, such as spare fields and transaction notes, which do not need to be collected in real time. Therefore, a condition setting window can be set to support entering query parameters via SQL statements. Multiple SQL statements are supported, and each SQL statement has a WHERE clause parameter. If no parameter is set, the entire table will be collected and exported by default. It is also possible to set an independent time window (qrytime) for each collection query. The query results of each SQL statement will generate an independent data package. Each SQL statement needs to have a time interval parameter set to trigger the probe mode, which collects the data to be aggregated in a rolling manner at time intervals. The default value is 10 minutes.

[0061] like Figure 2 As shown, optionally, the step of searching for the data to be collected in the database according to preset query parameters when the time trigger condition is met includes:

[0062] S1011. When the time triggering condition is met, determine whether to perform an aggregate query based on the preset query start time, query end time, and data cleanup cycle.

[0063] In step S1011, before data collection, the most important thing for the system is to determine whether the data to be collected includes data that has already been processed by the business system. Therefore, a data type parameter isunionqry can be set to distinguish whether historical data is needed. Typically, each database of the business entity stores the transaction data currently being processed and the historical data that has already been processed. The data collector may only need the data currently being processed, or it may need both the data currently being processed and the historical data that has already been processed.

[0064] For example, if the table to be queried is named `tablename`, the query start time plus the data cleanup period is calculated. If the calculated time is greater than the current time, it indicates that `tablename` is the query table name, and the aggregation query flag is 0 (no), requiring no further judgment. If it is null, the query end time plus the data cleanup period is calculated. If the calculated time is less than the current time, it indicates that `tablename_h` (the historical table of `tablename`) is used as the query table name, and the aggregation query flag is 0 (no), requiring no further judgment. Otherwise, the aggregation query flag is 1 (yes). After determining the aggregation query flag, the data to be aggregated can be filtered according to the preset query parameters. The parameter `isunionqry` is used to determine whether two tables need to be queried together.

[0065] S1012. If it is determined to perform an aggregation query, the data to be aggregated will be searched in the latest data table and the historical data table corresponding to the preset data table name.

[0066] In step S1012, if it is determined that an aggregation query will be performed, the data to be aggregated will be searched in the latest data table and the historical data table corresponding to the preset data table name; if it is determined that no aggregation query will be performed, the data to be aggregated will be searched in the latest data table or the historical data table corresponding to the preset data table name.

[0067] For example, if the default data table is named Table A, there are three scenarios for querying data:

[0068] (1) If the time between the end time and the start time does not exceed the data registration time in table A, then only table A is searched;

[0069] (2) If the data between the end time and the start time is not in the data recorded in table A and all the dates are historical dates, then only table A_h needs to be searched;

[0070] (3) If the end time to start time includes both the data registration time in table A and the data registration time in table A_h, then both table A and table A_h need to be queried at the same time.

[0071] For example, table A's data is cleared to the A_h historical table every 24 hours. Today is April 15th. The three scenarios are as follows:

[0072] (1) To find the data from 00:00 to 23:59 on April 15, look up table A;

[0073] (2) To retrieve data from 00:00 to 23:59 on April 14, consult the A_h table;

[0074] (3) To retrieve data from 12:00 on April 14 to 12:00 on April 15, you need to retrieve data from 12:00 to 23:59 in table A_h and from 00:00 to 12:00 in table A.

[0075] After finding the data to be collected, the method may further include: regularizing the output of the data to be collected, transforming it into a data source that conforms to the format of the information transmission device.

[0076] like Figure 3 As shown, optionally, sending the data to be aggregated to the subscriber of the data to be aggregated includes:

[0077] S1021. Package the data to be collected, the search time of the data to be collected, and the data source information of the data to be collected to generate a data package of the data to be collected.

[0078] In step S1021, a timestamp is created for each filtered data packet. The timestamp format can be "xxxx-xx-xx xx:xx:xx.xxxxxxxx", containing the year, date, time, and an 8-digit random number, creating a unique index. Furthermore, to distinguish the source of the data packets, a reserved field can be provided for each data packet to record the data center node value from which the data packet originated, with the key value denoted as DZONE. Assuming an institution has data centers located in Shanghai and Shenzhen (two in Shanghai and one in Shenzhen), the DZONE values ​​can be set as sh-01, sh-02, and sz-03 respectively. These values ​​can be customized by each financial institution.

[0079] S1022, Send the data packet to the subscriber of the data to be aggregated.

[0080] In step S1022, a data packet including the data to be collected, the search time of the data to be collected, and the data source information of the data to be collected is sent to the subscriber of the data to be collected.

[0081] Optionally, the subscriber and the data aggregator are nodes on the target blockchain network.

[0082] In this embodiment, as Figure 4 As shown, a target blockchain network is built using blockchain technology. A set of blockchain nodes is created for each data center. The consensus algorithm ensures the trustworthiness and immutability of the data. An independent set of nodes is also created for the corresponding data aggregator nodes to obtain and store the data from each node, ensuring the immutability and trustworthiness of the data.

[0083] In one embodiment of the present invention, a consumer is created for the node, a producer is created for the data source, and a Kafka cluster is created to receive messages sent by the producer and route these messages to queues in the server, so that Kafka can send the messages sent by the producer to the message subscriber.

[0084] Consumer nodes receive the data to be collected, first call the consensus module to complete the consensus of this data packet, and then execute the smart contract to register the data packet information in each node, ensuring that the main information of the data packet on all nodes is consistent. They also calculate the hash value (let's say H) of the splicing result of multiple key elements of the data packet (which can be selected according to actual needs) as a feature value and store it in the blockchain for subsequent comparison.

[0085] After receiving information registration requests from multiple entities, the blockchain network is responsible for receiving the data packet information attached to the request, including the data center number and the batch number of the aggregated data packet. It verifies the signature to confirm identity, and uses the Practical Byzantine Fault Tolerance (PBFT) algorithm to sort the data packets, generate blocks, and broadcast them to all nodes to achieve consensus and upload them to the chain. At this point, the data aggregator node has received the data packet locally.

[0086] The data aggregator is responsible for receiving query verification requests from the identity verification system, querying based on the data center number and the batch number of the aggregated data package, calculating the hash value (assuming it is H'), checking the data in the blockchain, and if a record with the characteristic value H = H' exists, querying the on-chain address based on the data center number and the batch number of the aggregated data package, querying the aggregated data submitted by the data center corresponding to the number in the blockchain network based on the queried on-chain address, and returning the result to the query system. Figure 4 This is a diagram of the entire blockchain reporting network structure. Assume an organization has four data centers. The four block nodes represent the corresponding nodes of the four data centers, which are interconnected. The fifth node represents the data collection node (data collector). This node can be a physically independent node or a logically virtual node. It is mainly used for future visualization of the collected data analysis and for information retrieval.

[0087] The data aggregation method provided in this embodiment of the invention has at least the following advantages:

[0088] (1) The probe program can automatically collect real-time transaction data within a certain period of time, get rid of the dependence on the underlying database master-slave synchronization, identify data packets by timestamp and data center label, and collect key transaction data to be collected in real time by time interval parameter.

[0089] (2) Realize near real-time push of transaction data through Kafka technology, and get rid of the time dependency of needing to collect data in batches at regular intervals.

[0090] (3) The blockchain-based network structure can ensure the immutability and accuracy of data after it is uploaded to the chain, guarantee the trustworthiness of the collected data, and quickly collect real and accurate transaction information during the data collection process, thereby improving the effectiveness and accuracy of information collection.

[0091] Figure 5 This is a schematic diagram of the structure of a data collection device according to an embodiment of the present invention, as shown below. Figure 5 As shown in the figure, a data collection device proposed in an embodiment of the present invention includes:

[0092] The search module 21 is used to search for the data to be collected in the database according to the preset query parameters when the time triggering condition is met.

[0093] The sending module 22 is used to send the data to be collected to the subscriber of the data to be collected, so that the subscriber sends the data to be collected to the data aggregator for data collection.

[0094] This invention provides a data collection device that, when a time-triggered condition is met, searches for data to be collected in a database according to preset query parameters; the data to be collected is then sent to its subscribers, who in turn forward the data to a data collector for collection. This allows for the automatic collection of data through aggregated query methods and proactive push of the data, eliminating the need for periodic batch reporting and improving the effectiveness of information collection.

[0095] Optionally, the query parameters include at least one of the following: query start time, query end time, data table name, data cleanup cycle, and / or field name.

[0096] Optionally, the search module is specifically used for:

[0097] When the time-triggered conditions are met, determine whether to perform an aggregate query based on the preset query start time, query end time, and data cleanup cycle;

[0098] If an aggregation query is performed, the data to be aggregated will be searched in the latest data table and the historical data table corresponding to the preset data table name.

[0099] Optionally, the search module is further configured to: if it is determined that no aggregation query will be performed, search for the data to be aggregated in the latest data table or historical data table corresponding to the preset data table name.

[0100] Optionally, the sending module is specifically used for:

[0101] The data to be collected, the search time of the data to be collected, and the data source information of the data to be collected are packaged together to generate a data package of the data to be collected.

[0102] The data packet is sent to the subscriber of the data to be aggregated.

[0103] Optionally, the sending module is specifically used to: send the data to be collected to a subscriber that subscribes to the data to be collected of that data type, so that the subscriber sends the data to be collected to the data aggregator for data collection.

[0104] Optionally, the subscriber and the data aggregator are nodes on the target blockchain network.

[0105] The embodiments of the device provided in this invention can be used to execute the processing flow of the above-described method embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the above-described method embodiments.

[0106] It should be noted that the data collection method and apparatus provided in the embodiments of the present invention can be used in the financial field, or in any technical field other than the financial field. The embodiments of the present invention do not limit the application field of the data collection method and apparatus.

[0107] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the electronic device may include a processor 301, a communications interface 302, a memory 303, and a communication bus 304. The processor 301, communications interface 302, and memory 303 communicate with each other via the communication bus 304. The processor 301 can call logical instructions stored in the memory 303 to execute the methods described in any of the above embodiments, such as: when a time-triggered condition is met, searching for data to be collected in a database according to preset query parameters; and sending the data to be collected to the subscribers of the data to be collected, so that the subscribers send the data to the data collector for data collection.

[0108] Furthermore, the logical instructions in the aforementioned memory 303 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by the computer, the computer can execute the methods provided in the above-described method embodiments, such as: when a time trigger condition is met, searching for data to be collected in a database according to preset query parameters; sending the data to be collected to the subscriber of the data to be collected, so that the subscriber sends the data to the data collector for data collection.

[0110] This embodiment provides a computer-readable storage medium storing a computer program that causes the computer to execute the methods provided in the above-described method embodiments. For example, the methods include: when a time-triggered condition is met, searching for data to be collected in a database according to preset query parameters; and sending the data to be collected to the subscriber of the data to be collected, so that the subscriber sends the data to the data aggregator for data collection.

[0111] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0116] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data aggregation method, characterized in that, include: When the time-triggered conditions are met, the system searches for the data to be collected in the database according to the preset query parameters. The time interval parameter is preset to trigger the probe program to collect the data to be collected in a rolling manner at time intervals. The data to be collected is sent to the subscriber of the data to be collected, so that the subscriber sends the data to the data aggregator for data aggregation. The query parameters include at least one of the following: query start time, query end time, data table name, data cleanup cycle and / or field name; The step of searching for the data to be collected in the database according to preset query parameters when the time trigger condition is met includes: When the time-triggered conditions are met, determine whether to perform an aggregate query based on the preset query start time, query end time, and data cleanup cycle; If an aggregation query is performed, the data to be aggregated will be searched in the latest data table and the historical data table corresponding to the preset data table name. If it is determined that no aggregation query will be performed, the data to be aggregated will be searched in the latest data table or historical data table corresponding to the preset data table name; The subscriber and the data aggregator are nodes on the target blockchain network. The target blockchain network is built using blockchain technology, a set of blockchain nodes is created for each data center, and the credibility and immutability of the data are guaranteed by a consensus algorithm. An independent set of nodes is also created for the corresponding data aggregator nodes to acquire and store the data from each node.

2. The method according to claim 1, characterized in that, Sending the data to be aggregated to the subscribers of the data to be aggregated includes: The data to be collected, the search time of the data to be collected, and the data source information of the data to be collected are packaged together to generate a data package of the data to be collected. The data packet is sent to the subscriber of the data to be aggregated.

3. The method according to claim 1, characterized in that, The step of sending the data to be aggregated to the subscriber of the data to be aggregated, so that the subscriber sends the data to the data aggregator for data aggregation, includes: Based on the data type of the data to be aggregated, the data to be aggregated is sent to subscribers who subscribe to the data to be aggregated of that data type, so that the subscribers send the data to be aggregated to the data aggregator for data aggregation.

4. A data collection device, characterized in that, include: The search module is used to search for data to be collected in the database according to preset query parameters when the time trigger condition is met; wherein, the time interval parameter is preset to trigger the probe program to collect the data to be collected in a rolling manner at time intervals; The sending module is used to send the data to be collected to the subscriber of the data to be collected, so that the subscriber sends the data to be collected to the data collector for data collection. The query parameters include at least one of the following: query start time, query end time, data table name, data cleanup cycle and / or field name; The search module is specifically used for: When the time-triggered conditions are met, determine whether to perform an aggregate query based on the preset query start time, query end time, and data cleanup cycle; If an aggregation query is performed, the data to be aggregated will be searched in the latest data table and the historical data table corresponding to the preset data table name. If it is determined that no aggregation query will be performed, the data to be aggregated will be searched in the latest data table or historical data table corresponding to the preset data table name; The subscriber and the data aggregator are nodes on the target blockchain network. The target blockchain network is built using blockchain technology, a set of blockchain nodes is created for each data center, and the credibility and immutability of the data are guaranteed by a consensus algorithm. An independent set of nodes is also created for the corresponding data aggregator nodes to acquire and store the data from each node.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 3.

Images