Data processing method, device and computer-readable storage medium

Abstract

Embodiments of the present invention disclose a data processing method, apparatus, and computer-readable storage medium, including: reading incremental performance data from a performance data queue using a stream processing program; the performance data queue is used to record network performance data collected from at least one network element in a communication network; aggregating the incremental performance data using current resource data in a preset database to obtain an incremental summary result; the preset database is used to record the resource data and performance data summary result corresponding to the at least one network element; and updating the preset database based on the incremental summary result. The present invention improves the accuracy and efficiency of data processing.

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to a data processing method, device, and computer-readable storage medium. Background Art

[0002] The telecommunications industry generates massive amounts of communication performance data every day. The network management system collects more than 400,000 pieces of raw communication performance data every day. Only by summarizing and analyzing the collected raw communication performance data in real time and feeding it back to network personnel can the secure and reliable operation of the communication network be guaranteed.

[0003] Among the related technologies, one method reduces the time loss between the various processes of data collection, reporting and aggregation by introducing a message mechanism. However, in a communication reporting system based on a message mechanism, the series connection of the collection, aggregation and other modules depends on the coordination of the message module, that is, the robustness of the entire system is greatly dependent on the normal operation of the communication module, and the overall robustness deteriorates with the introduction of the message mechanism. Another method presets a timer for the reported data. When the aggregation timer exceeds the previously reported data, the data that was not reported in time is put into the next cycle for processing. However, by setting a timeout time, the delay between the completion of data collection and processing and the completion of aggregation is reduced. Before the timer times out, polling is performed to determine whether the data in the time period has been reported. This consumes a lot of resources, and for the data reported after the timeout, it is necessary to maintain a section of logical judgment and process the data reported with delays in the previous time periods. Summary of the Invention

[0004] The present invention mainly provides a data processing method, device and computer-readable storage medium, which improve the accuracy and efficiency of data processing.

[0005] The technical solution of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a data processing method, including:

[0007] Reading incremental performance data from a performance data queue through a stream processing program; the performance data queue is used to record network performance data collected from at least one network element of the communication network;

[0008] The incremental performance data is aggregated using current resource data in a preset database to obtain an incremental aggregation result; the preset database is used to record the resource data and performance data aggregation result corresponding to the at least one network element;

[0009] The preset database is updated according to the incremental summary result.

[0010] In the above solution, updating the preset database according to the incremental summary result includes:

[0011] Generate an index corresponding to the incremental summary result according to the attribute information of the incremental summary result stored in the preset update queue; the attribute information includes at least one of the network element name, the summary time dimension, the summary space dimension, and the time node of the summary data;

[0012] The index is matched with a preset index corresponding to the at least one network element, and based on the matching result, the preset database is updated.

[0013] In the above solution, the matching results include matching failure and matching success;

[0014] The updating of the performance data based on the matching result includes:

[0015] If the match fails, the incremental summary result and the index are inserted into the preset database, and the index is inserted into the preset index library corresponding to the preset database to complete the update of the preset database; the preset index library is used to provide index information for querying the preset database;

[0016] If the match is successful, the target performance data summary result corresponding to the successfully matched target preset index is obtained;

[0017] The target performance data summary result and the incremental summary result are merged to obtain a merged summary result, and the target performance data summary result is updated using the merged summary result to complete the update of the preset database.

[0018] In the above solution, before aggregating the incremental performance data using the current resource data in the preset database to obtain the incremental summary result, the method further includes:

[0019] Obtaining, through a stream processing program, original resource data corresponding to the at least one network element from the preset database;

[0020] Data extraction is performed on the original resource data to obtain a resource data list; the resource data list contains current resource data corresponding to each network element.

[0021] In the above solution, the incremental performance data is aggregated using the current resource data in the preset database to obtain the incremental aggregation result, including:

[0022] According to each item of current resource data in the resource data list, at least one numerical integration and aggregation process is performed on the incremental performance data to obtain the incremental aggregation result.

[0023] In the above solution, before reading the incremental performance data from the performance data queue through the stream processing program, the method further includes:

[0024] Collecting network performance data corresponding to at least one network element;

[0025] The network performance data is normalized and recorded in the performance data queue.

[0026] In the above solution, the performance data queue is a message queue, and reading incremental performance data from the performance data queue through a stream processing program includes:

[0027] Obtaining the recorded historical read data position through the stream processing program;

[0028] Based on the historical read position, an incremental data portion is determined from the performance data queue and read to obtain the incremental performance data.

[0029] In the above solution, the incremental summary result is the incremental performance data summary result corresponding to the first preset time period; the method further includes:

[0030] Aggregating the incremental performance data according to a second preset time period to obtain an initial incremental aggregation result; the second preset time period is shorter than the first preset time period;

[0031] When each initial incremental summary result is obtained, a summary process is performed based on at least one initial incremental summary result obtained in the first preset time period to obtain the incremental summary result.

[0032] An embodiment of the present invention provides a data processing device, comprising: a reading unit, a summarizing unit, and an updating unit; wherein,

[0033] The reading unit is used to read incremental performance data from a performance data queue through a stream processing program; the performance data queue is used to record network performance data collected from at least one network element of the communication network;

[0034] The summarizing unit is configured to summarize the incremental performance data using the current resource data in the preset database to obtain an incremental summary result; the preset database is configured to record the summary result of the resource data and performance data corresponding to the at least one network element;

[0035] The updating unit is configured to update the preset database according to the incremental summary result.

[0036] An embodiment of the present invention provides a data processing device, comprising:

[0037] a memory for storing executable instructions;

[0038] The processor is configured to execute the executable instructions stored in the memory. When the executable instructions are executed, the processor executes the data processing method.

[0039] An embodiment of the present invention provides a storage medium storing executable instructions. When the executable instructions are executed, the storage medium causes a processor to execute the data processing method according to the embodiment of the present invention.

[0040] An embodiment of the present invention provides a data processing method, device and computer-readable storage medium, the method comprising: reading incremental performance data from a performance data queue through a stream processing program; the performance data queue is used to record network performance data collected from at least one network element of a communication network; using current resource data in a preset database, the incremental performance data is summarized and processed to obtain incremental summary results; the preset database is used to record resource data and performance data summary results corresponding to the at least one network element; and the preset database is updated according to the incremental summary results. In the above scheme, a message queue and a stream processing program are used to summarize and process incremental data uploaded to the file system in batches, and the performance data in the database is updated according to the summary results and the corresponding index files, which can observe the real-time statistical results of the performance data reported on the day in various dimensions, avoid the risk of data accuracy loss caused by delayed data reporting, and improve the accuracy and efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a data processing method provided by an embodiment of the present invention Figure 1 ;

[0042] Figure 2 A flowchart of a near real-time communication report data system solution framework provided by an embodiment of the present invention;

[0043] Figure 3 A flowchart of incremental aggregation provided by an embodiment of the present invention;

[0044] Figure 4 A flowchart of an index cluster update provided by an embodiment of the present invention;

[0045] Figure 5 A flowchart of incremental data extraction provided by an embodiment of the present invention;

[0046] Figure 6 A data flow diagram for summarizing performance data provided by an embodiment of the present invention;

[0047] Figure 7 A schematic structural diagram of a data processing device provided by an embodiment of the present invention;

[0048] Figure 8 A schematic structural diagram of another data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. Figure 1 A schematic diagram of a data processing method provided by an embodiment of the present invention Figure 1 , which will be explained in detail with the following steps.

[0051] S101. Read incremental performance data from a performance data queue through a stream processing program; the performance data queue is used to record network performance data collected from at least one network element of a communication network.

[0052] In an embodiment of the present invention, a data processing device reads incremental performance data from a performance data queue by stream processing; wherein the performance data queue is used to record network performance data collected from at least one network element in a communication network.

[0053] In embodiments of the present invention, in a streaming data processing model, data arrives continuously, and the system promptly processes newly arrived data and continuously generates output. Processed data is generally discarded, but can also be stored. The streaming data processing model emphasizes the speed of data processing. This is partly due to the rapid rate at which data is generated, necessitating timely processing. Because streaming data processing systems can promptly process newly arriving data, they can provide decision makers with the latest trends in developments and changes, enabling them to respond promptly to emergencies and adjust countermeasures. In batch processing, data is first continuously collected and stored in a database, followed by analysis and processing (including SQL queries). Batch processing is suitable for processing large volumes of data. Final results must be obtained only after the entire analysis and processing task is completed. Due to the size of the data set to be processed and the varying computing power of the computer system, the entire process can sometimes take considerable time, resulting in significant delays in obtaining the final analysis and processing results. Batch processing is the most common data processing model. Traditional relational database systems, Hadoop, and Spark big data processing platforms all employ or primarily utilize this data processing model. Since the entire data set needs to be completely saved and analyzed and processed on it, people need to invest more hardware resources than streaming data processing systems.

[0054] In the embodiments of the present invention, a queue is a special linear list, and the data elements in a queue are also called queue elements. Inserting a queue element into a queue is called enqueuing, and deleting a queue element from a queue is called dequeuing. Because a queue only allows insertion at one end and deletion at the other, only the element that enters the queue first can be deleted from the queue first. Therefore, a queue is also called a first-in-first-out linear list.

[0055] In the embodiment of the present invention, the performance data is the current incremental data of the current period relative to the previous period. A network element is composed of one or more disks or frames, and can independently perform certain transmission functions.

[0056] S102. Use the current resource data in the preset database to summarize the incremental performance data to obtain an incremental summary result; the preset database is used to record the resource data and performance data summary results corresponding to the at least one network element.

[0057] In an embodiment of the present invention, the data processing device aggregates the incremental performance data according to the current resource data stored in the preset database to obtain an incremental aggregate result; and then stores the incremental aggregate result in the preset database.

[0058] In the embodiment of the present invention, the current resource data may be resource data collected on the current day, which is not limited by the present invention.

[0059] In an embodiment of the present invention, the summary processing includes at least one of: summing up the data, finding an average value, finding a maximum value, and finding a minimum value.

[0060] In the embodiment of the present invention, compared with the full amount of resource data, the communication performance data is incremental data generated in real time, that is, incremental performance communication data.

[0061] S103: Update the preset database according to the incremental summary result.

[0062] In an embodiment of the present invention, the data processing device merges data in a preset database according to the incremental aggregation result.

[0063] In an embodiment of the present invention, Figure 2 As shown, Figure 2 This is a flowchart of a near real-time communication report data system solution framework provided by an embodiment of the present invention. Figure 2In the data collection module, the resource data is collected and stored in the database. The resource data in the database is periodically stored in the Hadoop distributed file system using the full resource data extraction algorithm. The data collection module stores the collected performance data in the Kafka message queue and uses the Flink incremental aggregation algorithm to aggregate the resource data and the incremental performance data in the Kafka message queue to obtain the aggregation results. SolrCloud is used to index the incremental aggregation results. Finally, the database is updated using the indexed aggregation results. The data calculation module includes the performance data incremental processing module and the resource data full processing module. The performance data incremental processing module uses stream processing to pull and aggregate the performance data pushed to the Kafka message queue in real time. During the aggregation process, the performance data indicators need to be integrated according to the resource data information. The resource data information is relatively stable and has small daily changes. The resource data full processing module will extract the resource data in the database into the Hadoop distributed file system using the full data extraction algorithm for Flink aggregation. The solrCloud in the data processing device solves the problem of inserting multiple incremental summary results obtained by incremental data aggregation into the database. Specifically, a unique index is established for the incremental summary results of each spatial dimension and time period of each network element. Before inserting the incremental summary results into the database, an index conflict query will be performed, and the results without conflicts will be inserted and indexed, and the summary results with conflicts will be updated. The data calculation module in the data processing device includes a performance data incremental processing module and a resource data full processing module. According to these two modules, the summary statistics of the original data are completed to provide the data source for the entire report. The solrCloud in the data processing device associates the established index with the incremental summary results in the database, and completes the management of the data generated by the data calculation module through the creation, modification and deletion of the index. The data acquisition module in the data processing device maintains the source data.

[0064] It can be understood that in an embodiment of the present invention, a message queue and a stream processing program are used to summarize and process the incremental data uploaded to the file system in batches, and the performance data in the database is updated and processed based on the summary results and the corresponding index files. The real-time statistical results of the performance data reported on the day can be observed in various dimensions, avoiding the risk of data accuracy loss caused by delayed data reporting, and improving the accuracy and efficiency of data processing.

[0065] In the embodiment of the present invention, S103 can be implemented through S1031 to S1032, which will be specifically described in conjunction with the following steps.

[0066] S1031. Generate an index corresponding to the incremental summary result according to the attribute information of the incremental summary result stored in the preset update queue; the attribute information includes at least one of the network element name, summary time dimension, summary space dimension and time node of the summary data.

[0067] In an embodiment of the present invention, the data processing device obtains data waiting to be updated from a preset update queue, and generates an index corresponding to the incremental summary result based on at least one of the network element name, the summary time and space dimension, and the time node of the incremental summary data.

[0068] In an embodiment of the present invention, the time dimension may be a 15-minute granularity, an hourly granularity, a daily granularity, or a monthly granularity; the spatial dimension may be a region, a data center, a resource pool, a network element, a province, a country, and the like.

[0069] In some embodiments of the present invention, the solrCloud module in the data processing device associates the network element, time dimension, spatial dimension, time node of the aggregated data and other information of the data according to preset rules and index names. The association relationship between the data report and the index established by the solrCloud module is: each time period data of each table corresponds to an index. The creation rule of the solrCloud module to establish an index is: the solr index is composed of separate or different files. Each file is composed of different fields, and each field has a single attribute. The association relationship between the data report and the solr index is that each table corresponds to a solr index, and each row of data corresponds to a file under the index. Each file is composed of multiple fields, and each field corresponds to each piece of data information, including the corresponding network element, time dimension, spatial dimension, and time node of the aggregated data.

[0070] S1032: Match the index with a preset index corresponding to at least one network element, and update a preset database based on the matching result.

[0071] In an embodiment of the present invention, the data processing device matches the established index with a preset index corresponding to at least one network element, and updates the preset database using the incremental summary result according to the matching result.

[0072] In some embodiments of the present invention, Figure 3 As shown, Figure 3 The flowchart of an incremental summary provided by the embodiment of the present invention is implemented from Sa to Sf. Figure 3 In the process, the incremental data aggregation task begins; Flink reads the full data file from the Hadoop distributed file system and generates a table; reads the performance data from the Kafka message queue and processes it into a table; uses the resource data information in the full data file to aggregate the performance data; the aggregated data is written to the database; and the incremental data aggregation task ends.

[0073] In some embodiments of the present invention, Figure 4 As shown, Figure 4 The present invention provides an index cluster update flow chart, which is implemented from SA to SH. Figure 4 In the process, start the processing program; insert the summary result into the update queue; obtain the data in the waiting update queue; generate the corresponding index from the Solr cluster (SolrCloud) based on the network element name of the summary data and the summary space-time dimension, time node and other information; determine whether there is an index conflict; if there is an index conflict, perform the update database operation; if there is no index conflict, insert the summary result and the corresponding index into the database; the task ends.

[0074] In some embodiments of the present invention, S1033 may be implemented through S201 to S203, which will be specifically described in conjunction with the following steps.

[0075] S201. If the match fails, the incremental summary result and the index are inserted into the preset database, and the index is inserted into the preset index library corresponding to the preset database to complete the update of the preset database; the preset index library is used to provide index information for querying the preset database.

[0076] In an embodiment of the present invention, the matching results include match failures and match successes. When a match fails, that is, when the index query finds that the index corresponding to the incremental summary result does not exist in the preset database, the data processing device inserts the incremental summary result and the generated index into the preset database and simultaneously inserts the generated database into the preset index library corresponding to the preset database, thereby completing the update of the preset database and the preset index library.

[0077] S202: If the match is successful, obtain the target performance data summary result corresponding to the successfully matched target preset index.

[0078] In an embodiment of the present invention, if the match is successful, that is, the index query finds that the index of the incremental summary result corresponds to the same index as a certain data in the preset database, the data processing device obtains the target performance data summary result corresponding to the successfully matched target preset index in the preset database.

[0079] S203: Merge the target performance data summary result and the incremental summary result to obtain a merged summary result, and use the merged summary result to update the target performance data summary result to complete the update of the preset database.

[0080] In an embodiment of the present invention, after the data processing device obtains the target performance summary result, it merges the target performance summary result and the incremental summary result to obtain a merged summary result; uses the merged summary result to update the target performance data summary result to complete the update of the preset database.

[0081] In the embodiment of the present invention, S102 also includes S301 to S302, which will be specifically described through the following steps.

[0082] S301. Obtain original resource data corresponding to at least one network element from a preset database through a stream processing program.

[0083] In an embodiment of the present invention, the data processing device obtains the collected original resource data corresponding to at least one network element from a preset database through a stream processing program.

[0084] S302: Extract the original resource data to obtain a resource data list; the resource data list includes current resource data corresponding to each network element.

[0085] In an embodiment of the present invention, a data processing device extracts data from original resource data to obtain a resource data list including current resource data corresponding to each network element.

[0086] In some embodiments of the present invention, the data processing device may regularly obtain a resource data list from a preset database every day.

[0087] In some embodiments of the present invention, Figure 5 As shown, Figure 5 The flowchart of an incremental data extraction provided by the embodiment of the present invention is implemented through Sg to Sk. Figure 5 In the process, the scheduled task starts; the required full resource data is read from the database; the full resource data is written to the local data file; the file is imported into the Hadoop distributed file system; and the task ends.

[0088] In some embodiments of the present invention, S102 can be implemented through S1021, which will be specifically described in conjunction with the following steps.

[0089] S1021. Perform at least one numerical integration and aggregation process on the incremental performance data according to each item of current resource data in the resource data list to obtain an incremental aggregation result.

[0090] In some embodiments of the present invention, the data processing device determines corresponding incremental performance data based on each current resource data in the resource data list, and performs at least one data integration and aggregation process on the corresponding incremental performance data to obtain an incremental aggregation result.

[0091] In some embodiments of the present invention, at least one numerical integration and aggregation process is to perform data summing, averaging, maximum value calculation, minimum value calculation, etc.

[0092] In some embodiments of the present invention, S101 also includes S401 to S402, which will be specifically described in conjunction with the following steps.

[0093] S401. Collect network performance data corresponding to at least one network element.

[0094] In some embodiments of the present invention, a data processing device collects network performance data corresponding to at least one network element.

[0095] In some embodiments of the present invention, the data processing device may collect network performance data corresponding to at least one network element from an operation and maintenance center.

[0096] S402: Normalize the network performance data and record it in a performance data queue.

[0097] In some embodiments of the present invention, the data processing device normalizes the collected network performance data to obtain normalized performance data, and records the normalized performance data in a performance data queue.

[0098] In some embodiments of the present invention, S101 may be implemented through S1011 to S1012, which will be specifically described in conjunction with the following steps.

[0099] S1011. Obtain the recorded historical read data location through the stream processing program.

[0100] In some embodiments of the present invention, the data processing device obtains recorded historical read data positions.

[0101] In some embodiments of the present invention, the Kafka message queue creates a log file for each partition, which stores the data in the partition. The data processing device periodically appends the acquired performance data to the end of the log file. Each piece of data has its own offset, and the Flink incremental aggregation algorithm records in real time which part of the performance data is aggregated.

[0102] S1012. Based on the historical read position, determine the incremental data portion from the performance data queue and read it to obtain incremental performance data.

[0103] In some embodiments of the present invention, the data processing device determines the incremental data portion from the performance data queue based on the historical performance data reading position and reads it to obtain the incremental performance data.

[0104] In an embodiment of the present invention, when the incremental summary result is the incremental performance data summary result corresponding to the first preset time period, the data processing method also includes S501 to S502, which will be specifically explained in combination with the following steps.

[0105] S501: Summarize incremental performance data according to a second preset time period to obtain an initial incremental summary result; the second preset time period is shorter than the first preset time period.

[0106] In some embodiments of the present invention, after obtaining the incremental performance data summary results corresponding to the first preset time period, the data processing device summarizes the incremental performance data of the second preset time period to obtain an initial incremental summary result; the second preset time period is shorter than the first preset time period.

[0107] S502: When each initial incremental summary result is obtained, perform summary processing based on at least one initial incremental summary result obtained in the first preset time period to obtain an incremental summary result.

[0108] In some embodiments of the present invention, the second preset time period is smaller than the first preset time period, that is, the second preset time period and the first preset time period are different time dimensions. The data processing device can summarize the incremental performance data in at least one time dimension based on the time attribute information in the incremental performance data. Every time the data processing device obtains an initial incremental summary result corresponding to a second preset time period, the data processing device summarizes the initial incremental summary result with other initial incremental summary results corresponding to the second preset time period obtained within the first preset time period in the time dimension as the current incremental summary result corresponding to the first preset time period; when the initial incremental summary result corresponding to a new second preset time period is obtained within the first preset time period, the data processing device continues to perform summary processing based on at least one initial incremental summary result currently obtained in the first preset time period until the summary of the initial incremental summary results corresponding to all second preset time periods in the first preset time period is completed.

[0109] It should be noted that, in embodiments of the present invention, the data processing device may also aggregate the incremental performance data along at least one spatial dimension based on the spatial attribute information in the incremental performance data. In other words, the data processing device may aggregate the incremental performance data along at least one of the temporal and spatial dimensions to obtain incremental aggregation results.

[0110] In some embodiments of the present invention, Figure 6 As shown, Figure 6 A data flow diagram for summarizing performance data provided by an embodiment of the present invention. Figure 6The method of performing aggregation processing on four time dimensions and six spatial dimensions by a data processing device is shown, wherein the four time dimensions include, from low to high, 15-minute granularity ( Figure 6 Indicated by q), hourly granularity ( Figure 6 h in the table), day granularity ( Figure 6 Indicated by d), monthly granularity ( Figure 6 The six spatial dimensions include: space, data center (dc), resource pool / virtualized infrastructure manager (vim), network element (ne), province (province), and country (country). Figure 6 Taking the 15-minute granularity as an example, the data processing device aggregates the six spatial dimensions under the 15-minute time dimension, and can obtain: space_q (15-minute regional aggregation result), dc_q (15-minute data center aggregation result), vim_q (15-minute resource pool aggregation result), ne_q (15-minute network element aggregation result), province_q (15-minute provincial aggregation result), and country_q (15-minute national aggregation result), and other time dimensions, and so on.

[0111] In the related art, it is usually the case that after completing the aggregation of all 15-minute granularities within an hour, the aggregation of the one-hour granularity is performed, and after completing the aggregation task of all hourly granularities within a day, the aggregation of the one-day granularity is performed, and so on. It can be seen that the aggregation method of the related art has a large delay and cannot meet the requirements of real-time queries in high time dimensions. In an embodiment of the present invention, when generating a low-dimensional aggregation result, such as a 15-minute aggregation result, the low-dimensional aggregation result can be stored in a kafka message queue, and the low-dimensional aggregation result can be obtained from the kafka message queue through a stream processing program, and the high-dimensional data aggregation is started in real time to obtain the high-dimensional data aggregation result. Here, the low-dimensional aggregation result can be the initial incremental aggregation result corresponding to the above-mentioned second preset time period, and the high-dimensional aggregation result can be the current incremental aggregation result corresponding to the above-mentioned first preset time period.

[0112] For example, when the first 15-minute network element-wide summary result ne_q is generated, the data processing device stores the 15-minute network element-wide summary result in the Kafka message queue and simultaneously starts the next higher time dimension, that is, the hourly dimension summary process. The data processing device obtains the ne_q from the Kafka message queue through the stream processing program. Every time a new ne_q appears within the hour, the new ne_q is aggregated with the ne_q already obtained within the hour in the time dimension as the current incremental summary result ne_h corresponding to the hour. At the same time, the spatial dimension aggregation can also be started based on ne_q in the spatial scope, such as the resource pool scope, to obtain the vim_q corresponding to the 15 minutes, up to the summary results of the spatial scope such as province_q.

[0113] It can be seen that by starting data aggregation in the resource pool, province, and network element dimensions in real time, it is possible to observe the real-time statistical results of the performance data reported on the day in each dimension in real time, avoiding the risk of data accuracy loss caused by delayed data reporting and improving the accuracy and efficiency of data processing.

[0114] An embodiment of the present invention provides a data processing method, which is specifically described in conjunction with the following steps.

[0115] S1. The data processing device collects resource data and performance data and stores them in a preset database and a preset message queue respectively.

[0116] S2. The data processing device extracts resource data from a preset database using a preset algorithm and stores the data in a preset distributed system.

[0117] In the embodiment of the present invention, S2 can be implemented through S2.1 to S2.4, which will be described in detail in conjunction with the following content.

[0118] S2.1. The data processing device sets a scheduled task and executes it on a scheduled basis.

[0119] S2.2. The data processing device connects to the preset database and reads the latest resource data.

[0120] S2.3. The data processing device writes the resource data information extracted from the preset database into a file.

[0121] S2.4. The data processing device uploads the file containing the resource data to the Hadoop distributed file system.

[0122] S3. The data processing device aggregates the collected performance data and resource data using a preset aggregation algorithm to obtain an aggregation result.

[0123] In some embodiments of the present invention, S3 can be implemented through S3.1 to S3.4, which will be specifically described in conjunction with the following steps.

[0124] S3.1. The data processing device starts a stream processing program.

[0125] S3.2. The data processing device reads all resource data information from the Hadoop distributed file system and registers it into a table.

[0126] S3.3. The data processing device reads performance data from the preset message queue.

[0127] S3.4. The data processing device aggregates the performance data using the resource data to obtain an aggregated result.

[0128] S4. The data processing device creates an index for the summary result.

[0129] In the embodiment of the present invention, S4 can be implemented through S4.1 to S4.4, which will be specifically described in conjunction with the following steps.

[0130] S4.1. After obtaining the summary results, the data processing device starts the indexing program.

[0131] S4.2. The data processing device inserts the summary result into the update queue.

[0132] S4.3. The data processing device obtains data waiting to be updated.

[0133] S4.4. The data processing device generates a corresponding index ID from the preset cluster according to the network element name of the aggregated data and the aggregated spatial and temporal dimensions, time nodes and other information.

[0134] S5. The data processing device determines whether there is an index conflict; if so, execute S6; if not, execute S7.

[0135] S6. The data processing device extracts the original index file and merges the data in the database.

[0136] S7. The data processing device inserts the summary result and the index corresponding to the summary result into the database, and at the same time inserts the index corresponding to the summary result into the index library.

[0137] In an embodiment of the present invention, the data processing device updates the report in the preset database using the summary result after indexing, and saves it into the preset index file according to the index format. When encountering index conflicts, the original index file is extracted and data is merged, that is, the data update operation is performed.

[0138] It can be understood that in an embodiment of the present invention, a message queue and a stream processing program are used to summarize and process the incremental data uploaded to the file system in batches, and the performance data in the database is updated and processed based on the summary results and the corresponding index files. The real-time statistical results of the performance data reported on the day can be observed in various dimensions, avoiding the risk of data accuracy loss caused by delayed data reporting, and improving the accuracy and efficiency of data processing.

[0139] The embodiment of the present invention provides a data processing device, such as Figure 7 As shown, Figure 7 A structural diagram of a data processing device provided by an embodiment of the present invention, the data processing device 7 includes: a reading unit 701, a summarizing unit 702 and an updating unit 703; wherein,

[0140] The reading unit 701 is configured to read incremental performance data from a performance data queue through a stream processing program; the performance data queue is configured to record network performance data collected from at least one network element of a communication network;

[0141] The summarizing unit 702 is configured to summarize the incremental performance data using the current resource data in the preset database to obtain an incremental summary result; the preset database is configured to record the summary result of the resource data and performance data corresponding to the at least one network element;

[0142] The updating unit 703 is configured to update the preset database according to the incremental aggregation result.

[0143] In some embodiments of the present invention, the update unit 703 is also used to generate an index corresponding to the incremental summary result based on the attribute information of the incremental summary result stored in the preset update queue; the attribute information includes at least one of the network element name, summary time dimension, summary space dimension and time node of the summary data; and, match the index with the preset index corresponding to the at least one network element, and update the preset database based on the matching result.

[0144] In some embodiments of the present invention, the matching results include matching failure and matching success; the update unit 703 is also used to insert the incremental summary result and the index into the preset database if the matching fails, and insert the index into the preset index library corresponding to the preset database to complete the update of the preset database; the preset index library is used to provide index information for querying the preset database; and, if the match is successful, obtain the target performance data summary result corresponding to the target preset index that is successfully matched; and, merge the target performance data summary result with the incremental summary result to obtain a merged summary result, and use the merged summary result to update the target performance data summary result to complete the update of the preset database.

[0145] In some embodiments of the present invention, the reading unit 701, before aggregating the incremental performance data using the current resource data in the preset database to obtain the incremental summary result, is also used to obtain the original resource data corresponding to the at least one network element from the preset database through a stream processing program; and, extract data from the original resource data to obtain a resource data list; the resource data list contains the current resource data corresponding to each network element.

[0146] In some embodiments of the present invention, the summarizing unit 702 is further configured to perform at least one numerical integration and summarizing process on the incremental performance data according to each item of current resource data in the resource data list to obtain the incremental summary result.

[0147] In some embodiments of the present invention, the data processing device further includes a recording unit 704; the recording unit 704 is used to collect network performance data corresponding to at least one network element; and normalize the network performance data and record it in the performance data queue.

[0148] In some embodiments of the present invention, the reading unit 701 is also used to obtain the recorded historical reading data position through the stream processing program; and, based on the historical reading position, determine the incremental data part from the performance data queue and read it to obtain the incremental performance data.

[0149] In some embodiments of the present invention, the incremental summary result is the incremental performance data summary result corresponding to the first preset time period; the summary unit 702 is also used to summarize the incremental performance data according to the second preset time period to obtain an initial incremental summary result; the second preset time period is less than the first preset time period; and, when each initial incremental summary result is obtained, summary processing is performed based on at least one initial incremental summary result obtained in the first preset time period to obtain the incremental summary result.

[0150] It can be understood that in the above-mentioned device implementation scheme, message queues and stream processing programs are used to summarize and process the incremental data uploaded to the file system in batches, and the performance data in the database is updated and processed based on the summary results and the corresponding index files. It can observe the real-time statistical results of the performance data reported on the day in various dimensions, avoid the risk of data accuracy loss caused by delayed data reporting, and improve the accuracy and efficiency of data processing.

[0151] Based on the method of the above embodiment, the embodiment of the present invention provides a structural diagram of a data processing device, as shown in FIG. Figure 8 As shown, Figure 8 A structural diagram of a data processing device provided in an embodiment of the present invention includes: a processor 801 and a memory 802; the memory 802 stores one or more programs executable by the processor 801, and when one or more programs are executed, a data processing method corresponding to the embodiment described above is executed by the processor 801.

[0152] An embodiment of the present invention provides a computer-readable storage medium storing executable instructions for causing a processor to execute the instructions to implement the data processing method.

[0153] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0154] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0157] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Read incremental performance data from the performance data queue through the stream processing program; The performance data queue is used to record network performance data collected from at least one network element of the communication network; Using the current resource data in the preset database, the incremental performance data is aggregated to obtain an incremental aggregation result; The preset database is used to record the summary results of resource data and performance data corresponding to the at least one network element; Updating the preset database according to the incremental summary result; Wherein, updating the preset database according to the incremental summary result includes: Generate an index corresponding to the incremental summary result according to the attribute information of the incremental summary result stored in the preset update queue; the attribute information includes at least one of the network element name, the summary time dimension, the summary space dimension, and the time node of the summary data; The index is matched with a preset index corresponding to the at least one network element, and based on the matching result, the preset database is updated.

2. The method according to claim 1, characterized in that The matching results include matching failure and matching success; The updating of the preset database based on the matching result includes: If the match fails, the incremental summary result and the index are inserted into the preset database, and the index is inserted into the preset index library corresponding to the preset database to complete the update of the preset database; the preset index library is used to provide index information for querying the preset database; If the match is successful, the target performance data summary result corresponding to the successfully matched target preset index is obtained; The target performance data summary result and the incremental summary result are merged to obtain a merged summary result, and the target performance data summary result is updated using the merged summary result to complete the update of the preset database.

3. The method according to any one of claims 1-2, characterized in that Before aggregating the incremental performance data using the current resource data in the preset database to obtain the incremental summary result, the method further includes: Obtaining, through a stream processing program, original resource data corresponding to the at least one network element from the preset database; Data extraction is performed on the original resource data to obtain a resource data list; the resource data list contains current resource data corresponding to each network element.

4. The method according to claim 3, characterized in that The incremental performance data is aggregated using the current resource data in the preset database to obtain incremental aggregate results, including: According to each item of current resource data in the resource data list, at least one numerical integration and aggregation process is performed on the incremental performance data to obtain the incremental aggregation result.

5. The method according to claim 1, wherein Before reading the incremental performance data from the performance data queue by the stream processing program, the method further includes: Collecting network performance data corresponding to at least one network element; The network performance data is normalized and recorded in the performance data queue.

6. The method according to claim 1, characterized in that The performance data queue is a message queue, and the process of reading incremental performance data from the performance data queue through a stream processing program includes: Obtaining the recorded historical read data position through the stream processing program; Based on the historical read position, an incremental data portion is determined from the performance data queue and read to obtain the incremental performance data.

7. The method according to claim 1, characterized in that The incremental summary result includes: the current incremental performance data summary result corresponding to the first preset time period; the method further includes: Aggregating the incremental performance data according to a second preset time period to obtain an initial incremental aggregation result; the second preset time period is shorter than the first preset time period; When each initial incremental summary result is obtained, a summary process is performed based on at least one initial incremental summary result currently obtained in the first preset time period to obtain the incremental summary result.

8. A data processing device, characterized in that: include: Reading unit, aggregation unit and update unit; The reading unit is used to read incremental performance data from the performance data queue through a stream processing program; The performance data queue is used to record network performance data collected from at least one network element of the communication network; The summarizing unit is configured to summarize the incremental performance data using the current resource data in the preset database to obtain an incremental summary result; The preset database is used to record the summary results of resource data and performance data corresponding to the at least one network element; The updating unit is configured to update the preset database according to the incremental summary result; The updating unit is further configured to generate an index corresponding to the incremental summary result based on attribute information of the incremental summary result stored in the preset update queue; the attribute information includes at least one of a network element name, a summary time dimension, a summary space dimension, and a time node of the summary data; The updating unit is further configured to match the index with a preset index corresponding to the at least one network element, and update the preset database based on a matching result.

9. A data processing device, characterized in that: include: a memory for storing executable instructions; The processor is configured to implement the data processing method according to any one of claims 1 to 7 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that The storage medium stores executable instructions, which, when executed, cause the processor to execute the data processing method according to any one of claims 1 to 7.

Images