Data collection method, device, system and storage medium

Abstract

The present invention provides a data collection method, device, system and storage medium. The method is applied in a system including a terminal collection node, an intermediate collection node and a terminal collection node. The method comprises: collecting initial data through the terminal collection node and uploading the initial data to the intermediate collection node; filtering the initial data according to business requirements through the intermediate collection node to obtain intermediate data, and uploading the intermediate data to the terminal collection node; filtering the intermediate data through the terminal collection node to obtain target data. Thus, data can be processed in layers, the amount of data processing at a high level can be reduced, the data can be traceable, data management is convenient, and the data has strong versatility.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device, system and storage medium. Background Art

[0002] With the development of industrialization, the amount of data generated in the industrial control process is increasing. In order to effectively control the industrial production process, a large amount of data needs to be collected and processed.

[0003] Current data collection systems are generally designed for specific technical scenarios, and managers then manually filter and process the collected data.

[0004] However, the system designed in this way has strong coupling and poor versatility. When collecting data, due to the non-standard business process, the collected data is mixed, which greatly increases the amount of calculation for later data processing. Summary of the invention

[0005] The embodiments of the present invention provide a data collection method, device, system and storage medium, which can perform layered processing on data, reduce the amount of high-level data processing, make the data traceable, facilitate data management, and have strong versatility.

[0006] In a first aspect, an embodiment of the present invention provides a data aggregation method, which is applied in a system including an end aggregation node, an intermediate aggregation node, and a terminal aggregation node, and the method includes:

[0007] Collecting initial data through the terminal aggregation node and uploading the initial data to the intermediate aggregation node;

[0008] The intermediate aggregation node filters the initial data according to business requirements to obtain intermediate data, and uploads the intermediate data to the terminal aggregation node;

[0009] The intermediate data is filtered through the terminal aggregation node to obtain target data.

[0010] In a possible design, collecting initial data through the terminal aggregation node and uploading the initial data to the intermediate aggregation node includes:

[0011] Collecting the initial data according to a preset collection frequency;

[0012] Sending a timing request to the intermediate aggregation node;

[0013] Receiving time information sent by the intermediate aggregation node;

[0014] Perform time calibration using the time information so that the time of the terminal converging node is consistent with the time of the intermediate converging node;

[0015] Packing the initial data to obtain a first data packet;

[0016] After serializing the first data packets according to the collection time sequence, the first data packets are uploaded to the intermediate aggregation node in sequence.

[0017] In a possible design, filtering the initial data according to business requirements by the intermediate aggregation node to obtain intermediate data, and uploading the intermediate data to the terminal aggregation node includes:

[0018] Obtain all first data packets uploaded at the same time in chronological order;

[0019] Filtering the first data packet to remove data irrelevant to the business requirement to obtain the intermediate data;

[0020] Packing the intermediate data to obtain a second data packet;

[0021] After serializing the second data packets according to the time sequence, the second data packets are uploaded to the terminal aggregation node in sequence.

[0022] In a possible design, after filtering the first data packet to remove data irrelevant to the service requirement and obtaining the intermediate data, the method further includes:

[0023] After the intermediate data is aggregated, it is stored in a relational database;

[0024] A data report corresponding to the intermediate data is generated according to the first query request input by the user.

[0025] In a possible design, filtering the intermediate data by the terminal aggregation node to obtain target data includes:

[0026] Obtain all the second data packets uploaded at the same time in chronological order;

[0027] The second data packet is unpacked and filtered to obtain the target data.

[0028] In one possible design, it also includes:

[0029] After the target data is aggregated, the target data is stored in a relational database;

[0030] According to the second query request input by the user, a data report corresponding to the target data is generated.

[0031] In a possible design, when the number of the terminal aggregation nodes is multiple and the number of the intermediate aggregation nodes is greater than 1, the method further includes:

[0032] Any of the intermediate aggregation nodes transmits the intermediate data to another intermediate aggregation node, so that the other intermediate aggregation node filters the intermediate data and then uploads it to the terminal aggregation node.

[0033] In a second aspect, an embodiment of the present invention provides a data collection device, including:

[0034] The terminal collection node is used to collect initial data and upload the initial data to the intermediate collection node;

[0035] The intermediate aggregation node is used to filter the initial data according to business requirements to obtain intermediate data, and upload the intermediate data to the terminal aggregation node;

[0036] The terminal aggregation node is used to filter the intermediate data to obtain target data.

[0037] In a possible design, the terminal collection node is specifically used for:

[0038] Collecting the initial data according to a preset collection frequency;

[0039] Sending a timing request to the intermediate aggregation node;

[0040] Receiving time information sent by the intermediate aggregation node;

[0041] Perform time calibration using the time information so that the time of the terminal converging node is consistent with the time of the intermediate converging node;

[0042] Packing the initial data to obtain a first data packet;

[0043] After serializing the first data packets according to the collection time sequence, the first data packets are uploaded to the intermediate aggregation node in sequence.

[0044] In a possible design, the intermediate aggregation node is specifically used to:

[0045] Obtain all first data packets uploaded at the same time in chronological order;

[0046] Filtering the first data packet to remove data irrelevant to the business requirement to obtain the intermediate data;

[0047] Packing the intermediate data to obtain a second data packet;

[0048] After serializing the second data packets according to the time sequence, the second data packets are uploaded to the terminal aggregation node in sequence.

[0049] In a possible design, the intermediate aggregation node is specifically used to:

[0050] After the intermediate data is aggregated, it is stored in a relational database;

[0051] A data report corresponding to the intermediate data is generated according to the first query request input by the user.

[0052] In a possible design, the terminal aggregation node is specifically used to:

[0053] Obtain all the second data packets uploaded at the same time in chronological order;

[0054] The second data packet is unpacked and filtered to obtain the target data.

[0055] In a possible design, the terminal aggregation node is specifically used to:

[0056] After the target data is aggregated, the target data is stored in a relational database;

[0057] According to the second query request input by the user, a data report corresponding to the target data is generated.

[0058] In a possible design, when the number of the terminal aggregation nodes is multiple and the number of the intermediate aggregation nodes is greater than 1, any intermediate aggregation node transmits the intermediate data to another intermediate aggregation node, so that the other intermediate aggregation node filters the intermediate data and then uploads it to the terminal aggregation node.

[0059] In a third aspect, an embodiment of the present invention provides a data collection system, comprising: a memory and a processor, wherein the memory stores executable instructions of the processor; wherein the processor is configured to execute the data collection method described in any one of the first aspects by executing the executable instructions.

[0060] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data collection method described in any one of the first aspects.

[0061] In a fifth aspect, an embodiment of the present invention provides a program product, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of a server can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the server executes any data collection method described in the first aspect.

[0062] The embodiment of the present invention provides a data collection method, device, system and storage medium, which are applied in a system including a terminal collection node, an intermediate collection node and a terminal collection node, wherein the terminal collection node collects initial data and uploads the initial data to the intermediate collection node; the intermediate collection node filters the initial data according to business requirements to obtain intermediate data, and uploads the intermediate data to the terminal collection node; the terminal collection node filters the intermediate data to obtain target data. Thus, data can be processed in layers, the amount of data processing at a high level can be reduced, the data can be traceable, data management is convenient, and the data has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0064] Figure 1 A schematic diagram of a data collection topology structure for an application scenario of the present invention;

[0065] Figure 2 A schematic diagram of a data collection topology structure for another application scenario of the present invention;

[0066] Figure 3 A schematic diagram of a data collection topology structure for another application scenario of the present invention;

[0067] Figure 4 A flow chart of a data collection method provided in Embodiment 1 of the present invention;

[0068] Figure 5 A schematic diagram of a service process of a terminal aggregation node provided by an embodiment of the present invention;

[0069] Figure 6 A schematic diagram of a service process of an intermediate aggregation node provided by an embodiment of the present invention;

[0070] Figure 7 A logical schematic diagram of data collection provided by an embodiment of the present invention;

[0071] Figure 8 A schematic diagram of a service process of a terminal aggregation node provided by an embodiment of the present invention;

[0072] Fig. 9 A schematic diagram of the structure of a data collection device provided in Embodiment 2 of the present invention;

[0073] Fig.10 This is a structural diagram of a data collection system provided in Embodiment 3 of the present invention.

[0074] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0077] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0078] With the development of industrialization, the amount of data generated in the industrial control process is also increasing. In order to effectively control the industrial production process, a large amount of data needs to be collected and processed. The current data collection system is generally designed for specific technical scenarios, and then the management personnel manually filter and process the collected data. However, the system designed in this way has strong coupling and poor versatility. When collecting data, due to the non-standard business process, the collected data is mixed, which greatly increases the amount of calculation for later data processing.

[0079] In view of the above technical problems, the present invention provides a data collection method, device, system and storage medium, which are applied in a system including a terminal collection node, an intermediate collection node and a terminal collection node, and can process data in layers, reduce the amount of data processing at a high level, make the data traceable, facilitate data management, and have strong versatility. The method provided by this implementation can also be applied to a distributed data acquisition system, which is integrated by a flexible and easily expandable real-time data analysis module, and has a clear topological structure at each level, and can realize data collection efficiently and in real time, ensuring the reusability and ease of use of business implementation, and is easy to implement. Figure 1 Schematic diagram of a data collection topology structure of an application scenario of the present invention, such as Figure 1 As shown in the figure, this is a standard data collection topology, which includes three types of data collection nodes, namely, terminal collection nodes, intermediate collection nodes and terminal collection nodes. After one or more terminal collection nodes collect data, they upload it to the intermediate collection node. In addition to the data collection function, the intermediate collection node also includes the function of filtering the collection results, and can perform real-time data analysis, and write the analysis results back to the second packet as ordinary data items, and upload them to the upper node together with the data collection results. The terminal collection node can collect data uploaded by multiple intermediate collection nodes. The data collection method in the terminal collection node is slightly different from that of the other two types of nodes, which is mainly determined by the environment in which the terminal node is located. The terminal collection node is located on the data collector computer with the data collection process, and is the closest place to the data in the data collection system. The data collection process here also serves as the data time synchronization and is the origin of the data time. The functions included in the terminal collection node and the intermediate collection node are generally the same, the only difference is that the data upload function is blocked in the terminal collection node. Among the three data aggregation nodes, only the terminal aggregation node does not have the authority to store data, while the other two nodes have the authority to store data. The terminal and intermediate aggregation nodes also include the function of filtering the aggregation results, which determines whether to upload the upper data to the upper-level aggregation node through filtering.

[0080] Figure 2 FIG. 1 is a schematic diagram of a data collection topology structure for another application scenario of the present invention. Figure 2 As shown, this is a data aggregation topology structure of a multi-level hybrid intermediate aggregation node. The data aggregation topology structure includes three types of data aggregation nodes, namely, terminal aggregation nodes, intermediate aggregation nodes, and terminal aggregation nodes. The function of each aggregation node is as described in the standard data aggregation topology structure, and will not be repeated here. The difference from the standard data aggregation topology structure is that in addition to receiving data from the terminal aggregation node, the intermediate aggregation node can also receive data from other intermediate aggregation nodes to realize the aggregation of data from the terminal aggregation node and the intermediate aggregation node. This type of data aggregation topology structure is suitable for the field of industrial control. For example, there is a data association between the parts produced by production line 1 and the quality inspection intermediate layer, so data interaction is required between intermediate nodes at the same level.

[0081] Figure 3 FIG. 1 is a schematic diagram of a data collection topology structure of another application scenario of the present invention. Figure 3 As shown in the figure, this is a data aggregation topology structure with multiple terminal aggregation nodes. The data aggregation topology structure includes three types of data aggregation nodes, namely, terminal aggregation nodes, intermediate aggregation nodes, and terminal aggregation nodes. In practical applications, data may not necessarily be aggregated to one node for unified processing and storage. Therefore, according to different data processing business requirements, multiple terminal aggregation nodes can be configured for the intermediate aggregation node so that the data filtering module can filter out data of different businesses and upload them to different terminal aggregation nodes. Finally, the system is deployed into a topology structure with multiple terminal aggregation nodes to achieve the purpose of pressure diversion or business relevance division. In the previous paragraph, it is described that the terminal aggregation node and the intermediate aggregation node both have data filtering and storage functions, so the intermediate aggregation node can also be the terminal aggregation node for some data. Through the description of this step, it can be seen that this framework is generally similar to the architecture of a channel filter mode. Through multiple intermediate nodes, the data is processed and filtered layer by layer, and the data is continuously screened and stored. Finally, the data that reaches the terminal node for processing is basically a small amount of data that needs to be aggregated at a high level. This design effectively improves the efficiency of data analysis, enhances the timeliness of abnormal situation alarms, fundamentally shares the pressure of real-time data processing, and controls system abnormalities locally.

[0082] The application of the above method can process data in layers, reduce the amount of high-level data processing, make the data traceable, facilitate data management, and have strong versatility.

[0083] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0084] Figure 4 This is a flow chart of a data collection method provided in Embodiment 1 of the present invention, such as Figure 4 As shown, the method in this embodiment is applied in a system including a terminal aggregation node, an intermediate aggregation node, and a terminal aggregation node, and may include:

[0085] S101. Collect initial data through the terminal aggregation node and upload the initial data to the intermediate aggregation node.

[0086] In this embodiment, the terminal aggregation node collects initial data according to a preset collection frequency; sends a timing request to the intermediate aggregation node; receives the time information sent by the intermediate aggregation node; performs time calibration based on the time information to ensure that the time of the terminal aggregation node is consistent with the time of the intermediate aggregation node; packages the initial data to obtain a first data packet; serializes the first data packet according to the collection time sequence, and uploads the first data packet to the intermediate aggregation node in sequence.

[0087] Specifically, the terminal collection node is a data collection node on the terminal collector. It is located on the data collector computer together with the data collection process and is the closest place to the data in the data collection system. There are generally two deployment forms of terminal collection nodes. One is the multi-threaded data collection form in the same process; the other is the data collection form based on network protocols or inter-process communication protocols between different processes. These two forms have an important common feature, that is, the terminal collection node is the data collection on the same computer. Therefore, the terminal collection node also serves as the data timing and is the origin of the data time. Regarding the timing of data, it can be synchronized and unified. Although the specific data collection terminal collects data from other devices through a certain protocol in most cases, the data collection terminal will collect data to the terminal node in a timely manner. Figure 5 A schematic diagram of the service flow of the terminal aggregation node provided by an embodiment of the present invention, such as Figure 5As shown in the figure, the terminal aggregation node mainly includes four aspects of business, namely data collection and packaging, time calibration, real-time data analysis and data upload. Taking the most common log collection business as an example, the business logic of the terminal aggregation node is introduced. Generally, logs are in lines, and a line of log generally contains time and content. The data collection process of the terminal aggregation node performs polling access to the log at a set frequency to read the key data after time and content decomposition. Then, the collected data is processed by system standardization, for example, standardized into the format of time_data ID_type_value. The time calibration process of the terminal aggregation node refers to calibrating the time part of the data item through the timing service, and calibrating the local time to the standard time of the data collection system. Then, the terminal aggregation node delivers the data to the real-time data analysis process branch (if there is a configuration at this level) and the data upload branch according to the calibrated time. The real-time data analysis process branch of the terminal aggregation node analyzes the data through the configured data recognition and data operation scripts to obtain the results of the data analysis. Then, according to the configuration, an early warning event can be generated when the threshold is exceeded. At the same time, the data analysis results can also be written back to the second packet as a normal data item when needed for data upload. The data upload branch of the terminal aggregation node is to create or obtain the second packet according to the calibrated time in the data item, and then press the data item into the second packet. After the second packet of the current second is created, the second packet of the previous second will be closed and put into the asynchronous queue at the same time. Finally, the second packet is serialized and sent to the upper-level node (intermediate / terminal data aggregation node).

[0088] S102: filtering and processing the initial data according to business requirements through the intermediate aggregation node to obtain intermediate data, and uploading the intermediate data to the terminal aggregation node.

[0089] In this embodiment, the intermediate aggregation node obtains all first data packets uploaded at the same time in chronological order; filters the first data packets to filter out data that is not related to business needs to obtain intermediate data; packages the intermediate data to obtain second data packets; serializes the second data packets according to chronological order, and uploads the second data packets to the terminal aggregation node in sequence.

[0090] Optionally, after filtering the first data packet to remove data irrelevant to business needs and obtaining intermediate data, the method further includes: aggregating the intermediate data and storing it in a relational database; and generating a data report corresponding to the intermediate data according to a first query request input by the user.

[0091] Optionally, when there are multiple terminal aggregation nodes and the number of intermediate aggregation nodes is greater than 1, the method also includes: any intermediate aggregation node transmits the intermediate data to another intermediate aggregation node, so that the other intermediate aggregation node filters the intermediate data and then uploads it to the terminal aggregation node.

[0092] Specifically, the intermediate aggregation node also includes a function of filtering the aggregation results, and determines through filtering whether to upload the received data to the upper-level aggregation node. Figure 6 A schematic diagram of the service flow of the intermediate aggregation node provided by an embodiment of the present invention, such as Figure 6 As shown in the figure, the intermediate aggregation node mainly includes four aspects of business, namely data aggregation process, real-time data analysis process, data storage process and data upload process. The data aggregation process of the intermediate aggregation node refers to receiving the second packets sent by all upstream nodes (end aggregation nodes or intermediate aggregation nodes), arranging all the second packets of the same second, and aggregating the data. Figure 7 A logical schematic diagram of data collection provided by an embodiment of the present invention, such as Figure 7As shown in the figure, the intermediate aggregation node receives the data second packets of collectors A, B, and C. At this time, the aggregated data packet is empty, as shown in the initial data in the figure. The intermediate aggregation node judges the front data items of the multiple data packets that have been sorted in time order, takes out the front data among them, and then enters the next iteration. This process is analogous to the end until all the data in multiple packets are finally taken. One point that needs to be noted in the process is that when there are multiple data items with the same timestamp at the front of all packets, all data items with the same timestamp can be taken out at once, thereby effectively reducing the number of iterations. Therefore, in the first round of circulation, the timestamp of collector A is 10, the timestamp of collector B is 10, and the timestamp of collector B is 11. The timestamp of 10 is ranked first, and the timestamps of collectors A and collector B are both 10. Therefore, in the first round of circulation, the data AA of collector A and the data BA of collector B are put into the aggregated data packet, and the corresponding data is deleted in collectors A and collector B. In the second cycle, the timestamp 11 of collector C is sorted first, the data CA of collector C is put into the aggregated data packet, and the corresponding data is deleted in collector C. In the third cycle, the timestamp 33 of collector C is sorted first, the data CB of collector C is put into the aggregated data packet, and the corresponding data is deleted in collector C. The real-time data analysis process of the intermediate aggregation node is to analyze the data through the configured data identification and data operation scripts to obtain the results of the data analysis. Then, according to the configuration, an early warning event can be generated when the threshold is exceeded. At the same time, the data analysis results can also be written back to the second packet as a normal data item when needed for data upload. The data upload process of the intermediate aggregation node is to create or get the second packet according to the time in the data item, and then press the data item into the second packet. After the second packet of the current second is created, the second packet of the previous second will be closed and put into the asynchronous queue at the same time. The intermediate aggregation node takes out the second packet from the queue in order, unpacks and filters it, and routes the data item according to the system configuration information. Finally, the data is packaged and serialized according to the routing tag, and the data packets are stored in the time series database or sent to the specified upper node according to the routing tag. The data storage process of the intermediate aggregation node is to filter out the data items that need to be processed, continue to calculate, count and store the raster data in the relational database for the display of reports.

[0093] S103: Filter the intermediate data through the terminal aggregation node to obtain target data.

[0094] In this embodiment, all second data packets uploaded at the same time are acquired in chronological order; the second data packets are unpacked and filtered to obtain target data.

[0095] Optionally, after the target data is summarized, it is stored in a relational database; and according to the second query request input by the user, a data report corresponding to the target data is generated.

[0096] Specifically, Figure 8 A schematic diagram of a service flow of a terminal aggregation node provided in an embodiment of the present invention, such as Figure 8 As shown in the figure, the business process of the terminal aggregation node is basically the same as that of the intermediate aggregation node. The only difference is that there is no data upload process in the terminal aggregation node, and there is no need to upload data to other nodes. Therefore, the business process of the terminal aggregation node is detailed in the relevant introduction of the intermediate aggregation node, which will not be repeated here.

[0097] This embodiment is applied in a system including a terminal aggregation node, an intermediate aggregation node, and a terminal aggregation node. Initial data is collected through the terminal aggregation node and uploaded to the intermediate aggregation node; the initial data is filtered and processed according to business requirements by the intermediate aggregation node to obtain intermediate data, and the intermediate data is uploaded to the terminal aggregation node; the intermediate data is filtered and processed by the terminal aggregation node to obtain target data. In this way, data can be processed in layers, the amount of data processing at the high level can be reduced, and the data can be traceable, which is convenient for data management and has strong versatility.

[0098] Fig. 9 A schematic diagram of the structure of a data collection device provided in Embodiment 2 of the present invention, such as Fig. 9 As shown, the data collection device of this embodiment may include:

[0099] The terminal collection node 31 is used to collect initial data and upload the initial data to the intermediate collection node;

[0100] The intermediate aggregation node 32 is used to filter and process the initial data according to business requirements, obtain intermediate data, and upload the intermediate data to the terminal aggregation node;

[0101] The terminal aggregation node 33 is used to filter the intermediate data to obtain the target data.

[0102] In a possible design, the terminal collection node 31 is specifically used for:

[0103] Collect initial data according to the preset collection frequency;

[0104] Send a timing request to the intermediate aggregation node;

[0105] Receive time information sent by the intermediate aggregation node;

[0106] Through the time information, time calibration is performed to make the time of the terminal aggregation node consistent with the time of the intermediate aggregation node;

[0107] Packing the initial data to obtain a first data packet;

[0108] After the first data packets are serialized according to the collection time sequence, the first data packets are uploaded to the intermediate aggregation node in sequence.

[0109] In a possible design, the intermediate collection node 32 is specifically used for:

[0110] Obtain all first data packets uploaded at the same time in chronological order;

[0111] Filtering the first data packet to remove data irrelevant to the business requirements and obtain intermediate data;

[0112] Packing the intermediate data to obtain a second data packet;

[0113] After the second data packets are serialized according to the time sequence, the second data packets are uploaded to the terminal aggregation node in sequence.

[0114] In a possible design, the intermediate collection node 32 is specifically used for:

[0115] After the intermediate data is summarized, it is stored in a relational database;

[0116] According to the first query request input by the user, a data report corresponding to the intermediate data is generated.

[0117] In a possible design, the terminal aggregation node 33 is specifically used for:

[0118] Obtain all second data packets uploaded at the same time in chronological order;

[0119] The second data packet is unpacked and filtered to obtain target data.

[0120] In a possible design, the terminal aggregation node 33 is specifically used for:

[0121] After the target data is summarized, it is stored in a relational database;

[0122] According to the second query request input by the user, a data report corresponding to the target data is generated.

[0123] In a possible design, when there are multiple terminal aggregation nodes and the number of intermediate aggregation nodes is greater than 1, any intermediate aggregation node transmits the intermediate data to another intermediate aggregation node, so that the other intermediate aggregation node filters the intermediate data and then uploads it to the terminal aggregation node.

[0124] The data collection device of this embodiment can execute Figure 2For the technical solution in the method shown, its specific implementation process and technical principles can be found in Figure 2 The relevant descriptions in the method shown will not be repeated here.

[0125] This embodiment is applied in a system including a terminal aggregation node, an intermediate aggregation node, and a terminal aggregation node. Initial data is collected through the terminal aggregation node and uploaded to the intermediate aggregation node; the initial data is filtered and processed according to business requirements by the intermediate aggregation node to obtain intermediate data, and the intermediate data is uploaded to the terminal aggregation node; the intermediate data is filtered and processed by the terminal aggregation node to obtain target data. In this way, data can be processed in layers, the amount of data processing at the high level can be reduced, and the data can be traceable, which is convenient for data management and has strong versatility.

[0126] Fig.10 The structure diagram of the data collection system provided by the third embodiment of the present invention is as follows: Fig.10 As shown, the data collection system 40 of this embodiment may include: a processor 41 and a memory 42 .

[0127] The memory 42 is used to store programs; the memory 42 may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory 42 is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories 42. And the above computer programs, computer instructions, data, etc. can be called by the processor 41.

[0128] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories 42 in partitions. And the above-mentioned computer programs, computer instructions, data, etc. may be called by the processor 41.

[0129] The processor 41 is used to execute the computer program stored in the memory 42 to implement the various steps in the method involved in the above embodiment.

[0130] For details, please refer to the relevant description in the previous method embodiment.

[0131] The processor 41 and the memory 42 may be independent structures or integrated structures. When the processor 41 and the memory 42 are independent structures, the memory 42 and the processor 41 may be coupled and connected via a bus 43 .

[0132] The data collection system of this embodiment can execute Figure 2 For the technical solution in the method shown, its specific implementation process and technical principles can be found in Figure 2 The relevant descriptions in the method shown will not be repeated here.

[0133] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When at least one processor of a user device executes the computer-executable instructions, the user device executes the above-mentioned various possible methods.

[0134] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0135] The present application also provides a program product, which includes a computer program. The computer program is stored in a readable storage medium. At least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server implements any method of the above-mentioned embodiments of the present invention.

[0136] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data collection method, characterized in that: Applied in a system including a terminal aggregation node, an intermediate aggregation node, and a terminal aggregation node, the method includes: Collecting initial data through the terminal aggregation node and uploading the initial data to the intermediate aggregation node; The intermediate aggregation node filters the initial data according to business requirements to obtain intermediate data, and uploads the intermediate data to the terminal aggregation node; Filtering the intermediate data through the terminal aggregation node to obtain target data; When there are multiple terminal aggregation nodes and the number of intermediate aggregation nodes is greater than 1, any intermediate aggregation node transmits the intermediate data to another intermediate aggregation node, so that the other intermediate aggregation node filters the intermediate data and then uploads it to the terminal aggregation node.

2. The method according to claim 1, characterized in that Collecting initial data through the terminal aggregation node and uploading the initial data to the intermediate aggregation node includes: Collecting the initial data according to a preset collection frequency; Sending a timing request to the intermediate aggregation node; Receiving time information sent by the intermediate aggregation node; Perform time calibration using the time information so that the time of the terminal converging node is consistent with the time of the intermediate converging node; Packing the initial data to obtain a first data packet; After serializing the first data packets according to the collection time sequence, the first data packets are uploaded to the intermediate aggregation node in sequence.

3. The method according to claim 2, characterized in that The intermediate aggregation node filters the initial data according to business requirements to obtain intermediate data, and uploads the intermediate data to the terminal aggregation node, including: Obtain all first data packets uploaded at the same time in chronological order; Filtering the first data packet to remove data irrelevant to the business requirement to obtain the intermediate data; Packing the intermediate data to obtain a second data packet; After serializing the second data packets according to the time sequence, the second data packets are uploaded to the terminal aggregation node in sequence.

4. The method according to claim 3, characterized in that After filtering the first data packet to remove data irrelevant to the service requirement and obtaining the intermediate data, the method further includes: After the intermediate data is aggregated, it is stored in a relational database; A data report corresponding to the intermediate data is generated according to the first query request input by the user.

5. The method according to claim 3, characterized in that: The intermediate data is filtered and processed by the terminal aggregation node to obtain target data, including: Obtain all the second data packets uploaded at the same time in chronological order; The second data packet is unpacked and filtered to obtain the target data.

6. The method according to claim 5, characterized in that Also includes: After the target data is aggregated, the target data is stored in a relational database; According to the second query request input by the user, a data report corresponding to the target data is generated.

7. A data collection device, characterized in that: include: The terminal collection node is used to collect initial data and upload the initial data to the intermediate collection node; The intermediate aggregation node is used to filter the initial data according to business requirements to obtain intermediate data, and upload the intermediate data to the terminal aggregation node; The terminal aggregation node is used to filter the intermediate data to obtain target data; When there are multiple terminal aggregation nodes and the number of intermediate aggregation nodes is greater than 1, any intermediate aggregation node transmits the intermediate data to another intermediate aggregation node, so that the other intermediate aggregation node filters the intermediate data and then uploads it to the terminal aggregation node.

8. A data collection system, characterized in that: include: A memory and a processor, wherein executable instructions of the processor are stored in the memory; wherein the processor is configured to execute the data collection method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data collection method according to any one of claims 1 to 6 is implemented.

Images