Method and apparatus for processing network operation data, electronic device, and medium

Abstract

The application provides a network operation data processing method and device, electronic equipment and medium. The method comprises: obtaining to-be-processed data, wherein the to-be-processed data comprises network operation data; adopting a stream processing mode to perform initial aggregation on the to-be-processed data to obtain an initial aggregation result, and writing the initial aggregation result into a data detail layer of an offline data warehouse; based on the initial aggregation result in the data detail layer, adopting a batch processing mode to perform batch aggregation to obtain a target aggregation result, and writing the target aggregation result into a data application layer of the offline data warehouse. The method of the application can ensure the accuracy of network operation data processing, reduce redundant calculation, and reduce the consumption of computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the communication technology field, and particularly relates to a network operation data processing method and device, electronic equipment and medium. BACKGROUND

[0002] A communication network operator collects network operation data in the process of network system operation, and through processing the network operation data, the operator can perform further operations such as data mining, data analysis, report display and query according to the processing result.

[0003] In the prior art, the network operation data processing largely adopts a Lambda-like architecture, which works by receiving data and inputting it into independent stream processing systems and batch processing systems in parallel. The stream processing system extracts data in real time, calculates approximate results and writes them into a fast table. The batch processing system processes data periodically, writes accurate results into a batch processing table and deletes the corresponding inaccurate results from the fast table. An application quickly combines the approximate results in the fast table and the accurate results in the batch processing table to obtain the final result, which is written into the application layer.

[0004] In the above scheme, the Lambda-like architecture is used for redundant calculation, which causes a problem of high consumption of computing resources. SUMMARY

[0005] The present application provides a network operation data processing method, device, electronic equipment and medium to solve the problem of high consumption of computing resources in network operation data processing.

[0006] In one aspect, the present application provides a network operation data processing method, comprising: obtaining to-be-processed data, wherein the to-be-processed data comprises network operation data; using a stream processing method to perform initial aggregation on the to-be-processed data to obtain an initial aggregation result, and writing the initial aggregation result into a data detail layer of an offline data warehouse; using a batch processing method to perform batch aggregation based on the initial aggregation result in the data detail layer to obtain a target aggregation result, and writing the target aggregation result into a data application layer of the offline data warehouse.

[0007] In one possible implementation, the obtaining to-be-processed data comprises: performing light splitting collection on an interface link between communication network elements, and performing packet capture and stream splitting processing on the collected light splitting code stream to obtain raw data; and / or collecting recorded data in a network management system operation log to obtain raw data; performing synthesis processing on the raw data to obtain the to-be-processed data; wherein the synthesis processing comprises at least one of the following: field extraction and assignment, relationship backfilling and service identification.

[0008] In a possible implementation manner, the synthesizing processing on the original data to obtain the to-be-processed data comprises: field value taking on a same signaling flow in the original data; correlation back filling on messages that exist in correlation in the original data; and back filling of service types in the original data by using a deep packet detection technology.

[0009] In a possible implementation manner, the batch processing is used to perform batch convergence based on the initial convergence result in the data detail layer to obtain a target convergence result, comprising: performing first-level convergence on the initial convergence result by using the batch processing to obtain an intermediate convergence result, and writing the intermediate convergence result into a data service layer of an offline data warehouse; and performing second-level convergence on the intermediate convergence result in the data service layer by using the batch processing to obtain the target convergence result.

[0010] In a possible implementation manner, the first-level convergence on the initial convergence result by using the batch processing to obtain the intermediate convergence result comprises: based on the initial convergence result, generating a first data table under each first period periodically according to a predetermined first period; and processing an incremental aggregation result of the first data table under a current first period and the first data table under a historical first period by using an incremental aggregation manner to obtain the intermediate convergence result under the data service layer.

[0011] In a possible implementation manner, the second-level convergence on the intermediate convergence result in the data service layer by using the batch processing to obtain the target convergence result comprises: based on the intermediate convergence result, generating a second data table under each second period periodically according to a predetermined second period; processing an incremental aggregation result of the second data table under a current second period and the second data table under a historical second period by using an incremental aggregation manner to obtain the target convergence result; or processing the second data table under each second period by using a one-time aggregation manner to obtain the target convergence result.

[0012] In a possible implementation manner, before the initial convergence result is obtained by using the stream processing manner on the to-be-processed data, the method further comprises: pre-processing the to-be-processed data, and the pre-processing comprises at least one of the following: filtering illegal data, filling null values, filtering abnormal values, correlating dimension table fields, and splicing records.

[0013] In a possible implementation manner, after the pre-processing is performed on the to-be-processed data, before the initial convergence result is obtained by using the stream processing manner on the to-be-processed data, the method further comprises: writing the pre-processing result into a data operation layer of an offline data warehouse.

[0014] In another aspect, the present application provides a network operation data processing apparatus, comprising: an acquisition module configured to acquire to-be-processed data, wherein the to-be-processed data comprises network operation data; a stream processing module configured to perform initial aggregation on the to-be-processed data in a stream processing manner to obtain an initial aggregation result, and write the initial aggregation result into a data detail layer of an offline data warehouse; and a batch processing module configured to perform batch aggregation on the initial aggregation result in the data detail layer in a batch processing manner to obtain a target aggregation result, and write the target aggregation result into a data application layer of the offline data warehouse.

[0015] In a possible implementation, the acquisition module comprises: a collection unit configured to collect interface links between communication network elements, and perform packet capture and stream processing on the collected split stream to obtain raw data; and / or collect record data in a network management system operation log to obtain raw data; and a synthesis unit configured to perform synthesis processing on the raw data to obtain the to-be-processed data; wherein the synthesis processing comprises at least one of the following: field extraction and assignment, association backfilling, and service identification.

[0016] In a possible implementation, the synthesis unit is specifically configured to: perform field value extraction on a same signaling flow in the raw data; perform association backfilling on messages associated in the raw data; and perform backfilling on service types in the raw data by using deep packet inspection technology.

[0017] In a possible implementation, the batch processing module comprises: a first batch processing unit configured to perform first-level aggregation on the initial aggregation result in a batch processing manner to obtain an intermediate aggregation result, and write the intermediate aggregation result into a data service layer of the offline data warehouse; and a second batch processing unit configured to perform second-level aggregation on the intermediate aggregation result in the data service layer in a batch processing manner to obtain the target aggregation result, and write the target aggregation result into the data application layer of the offline data warehouse.

[0018] In a possible implementation, the first batch processing unit is specifically configured to: based on the initial aggregation result, generate a first data table in each first period according to a predetermined first period; and perform processing on an incremental aggregation result of the first data table in a current first period and the first data table in a historical first period in an incremental aggregation manner to obtain the intermediate aggregation result in the data service layer.

[0019] In a possible implementation, the second batch processing unit is specifically configured to: based on the intermediate aggregation result, periodically generate a second data table in each second cycle according to a predetermined second cycle; and process an incremental aggregation result of the second data table in the current second cycle and the second data table in the historical second cycle in an incremental aggregation manner to obtain the target aggregation result; or, process the second data table in each second cycle in a one-time aggregation manner to obtain the target aggregation result.

[0020] In a possible implementation, the apparatus further includes a preprocessing module configured to preprocess the to-be-processed data before the stream processing module performs initial aggregation on the to-be-processed data to obtain an initial aggregation result, the preprocessing including at least one of the following: filtering illegal data, filling in null values, filtering abnormal values, associating dimension table fields, and splicing records.

[0021] In a possible implementation, the preprocessing module is further configured to write the preprocessing result to a data operation layer of an offline warehouse.

[0022] In another aspect, the present application provides an electronic device, including a processor and a memory connected with the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method as described above.

[0023] In another aspect, the present application provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, the computer execution instructions are executed by a processor to implement the method as described above.

[0024] In the network operation data processing method, apparatus, electronic device and medium provided by the present application, the obtained network operation data is first processed in a stream processing manner to obtain an initial aggregation result, and then the initial aggregation result is used to perform batch aggregation to obtain a final processing result provided to an application layer. The present application can reduce the data processing amount by performing initial aggregation, thereby reducing the intermediate state storage occupation, and significantly reducing the data amount of subsequent batch processing, simplifying the subsequent aggregation algorithm, and improving the accuracy of data aggregation processing by performing initial stream aggregation. Compared with the processing accuracy ensured by the redundant processing manner, the processing procedure of the present application is more simple, thereby ensuring the accuracy of network operation data processing while reducing the consumption of computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0026] Figure 1 A Lambda-like architecture is exemplarily shown in FIG. 1;

[0027] Figure 2 A flowchart of a method for processing network operation data is exemplarily shown in FIG. 2;

[0028] Figure 3 A structure diagram of a data warehouse is exemplarily shown in FIG. 3;

[0029] Figure 4 An architecture diagram of the method for processing network operation data is exemplarily shown in FIG. 4;

[0030] Figure 5 A flowchart of batch aggregation of initial aggregation results is exemplarily shown in FIG. 5;

[0031] Figure 6 A structure diagram of a processing device for network operation data is exemplarily shown in FIG. 6;

[0032] Figure 7 A structure diagram of a processing device for network operation data is exemplarily shown in FIG. 7;

[0033] Figure 8 A structure diagram of an electronic device for processing network operation data is exemplarily shown in FIG. 8.

[0034] The above-mentioned figures have shown the embodiments of the present application, which will be described in more detail hereinafter. These figures and the following description are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0035] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements in the several figures. The following description of exemplary embodiments is not representative of all possible embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] The module in the present application refers to a functional module or a logic module. It can be in the form of software, and its functions are realized by executing program codes by a processor; or it can be in the form of hardware. The "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0037] First, the terms involved in the present application are explained:

[0038] Network operation data: refers to the data collected by the communication network operator in the process of network system operation, including basic resource and configuration data, signaling tracking data, service identification data, performance statistics data and monitoring and early warning data, etc. The data sources include drive test (DT) / call quality test (CQT), telecommunications business operation support system, device network management system, interface probe collection system, etc.

[0039] Data warehouse: refers to a data storage collection, which is created for the purpose of enterprise analytical reporting and decision support, and filters and integrates various business data. The data warehouse mainly provides data for decision analysis, and the operation involved is mainly data query.

[0040] Deep packet inspection technology: refers to a traffic detection and control technology based on the application layer. When the Internet Protocol (IP) data packet, Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) data stream passes through the bandwidth management system based on the DPI (Deep Packet Inspection, a deep packet detection technology based on different network application layer loads for deep detection, which determines the legality by detecting the content of the IP packet payload) technology, the system reads the content of the IP packet payload in depth to reorganize the application layer information in the Open System Interconnection (OSI) seven-layer protocol, thereby obtaining the content of the entire application program, and then performs shaping operation on the traffic according to the management policy defined by the system.

[0041] With the development of communication network technology and the growth of communication network operator business, the amount of network operation data collected by the communication network operator has expanded sharply, which poses a challenge to the processing capacity of network operation data.

[0042] In some technologies, the processing of network operation data largely adopts a Lambda-like architecture, Figure 1 A Lambda-like architecture is exemplarily shown in FIG. 1. This architecture works in such a way that data is received and input into an independent stream processing system and a batch processing system in parallel. The stream processing system employs a real-time computing engine such as Storm (an open source distributed real-time computing system) to perform simple stream computing, to calculate approximate results and write them into a fast table in near real time. The batch processing system employs an offline computing engine such as MapReduce (a computing model, framework and platform for parallel processing of big data) to perform batch processing, to periodically process data in batch storage, to write accurate results into a batch processing table and delete corresponding inaccurate results from the fast table. An application quickly combines the approximate results in the fast table and the accurate results in the batch processing table to obtain final results, which are written into an application layer.

[0043] However, the stream processing system and the batch processing system independently process the input data, which causes redundant computing. The batch processing system saves full-amount upstream data files without convergence, and a large amount of data shuffle and redistribution operations exist in the cluster, and the computing resource occupation is usually uneven, which generally causes periodic resource occupation peaks and resource idling within the period. Therefore, the Lambda-like architecture has the problem of high computing resource consumption.

[0044] The technical solutions of the present application are exemplarily described below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments.

[0045] Embodiment One

[0046] Figure 2 A flowchart of a network operation data processing method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the network operation data processing method provided by the embodiment can include: Figure 2

[0047] S201, obtaining to-be-processed data, wherein the to-be-processed data includes network operation data.

[0048] S202, performing initial convergence on the to-be-processed data in a stream processing manner to obtain initial convergence results, and writing the initial convergence results into a data detail layer of an offline data warehouse.

[0049] S203, performing batch convergence on the initial convergence results in the data detail layer in a batch processing manner to obtain target convergence results, and writing the target convergence results into an application layer of the offline data warehouse.

[0050] ​In actual application, the execution subject of the embodiment can be a network operation data processing device, which can be implemented by a computer program, for example, application software, or can be a medium storing the related computer program, for example, a U disk, a cloud disk, or can be an entity device integrated or installed with the related computer program, for example, a chip or a server.

[0051] The following is an example of the architecture of a data warehouse: Figure 3 For an example of the architecture of a data warehouse, as shown in the figure, the data warehouse includes an operation data store (ODS), a data warehouse details (DWD), a data warehouse middle (DWM), a data warehouse service (DWS), and an application data service (ADS). In combination with a scenario example, when the application layer software (for example, a client APP) executes a certain application software function, it usually needs to obtain some data required for executing the function, for example, the call software provides a call duration query function, which may need to obtain call record data. Correspondingly, the ADS layer can send a data acquisition instruction to the network operation data processing device, the network operation data processing device obtains the to-be-processed data from the data producer according to the required data indicated by the instruction, and obtains the data required by the ADS layer by executing the scheme of the embodiment on the to-be-processed data, and puts the data into the ADS layer, so that the ADS layer software can obtain the required data from the ADS layer. The ADS layer can store target data, the ODS layer can store data preprocessing results, the DWD, DWM, and DWS layers can be used as intermediate layers to store data processing intermediate results, and the specific use can be determined according to actual application.

[0052] Figure 4 For an example of the architecture of the network operation data processing method, specifically, after the network operation data processing device obtains the to-be-processed data, it first performs simple stream aggregation, and the optional algorithms include but are not limited to conditional summation, conditional maximum value calculation, and conditional minimum value calculation, to obtain an initial aggregation result, and the initial aggregation result is written into the offline data warehouse DWD layer in a stream manner, and is written into different time partitions according to the data event time (the time when the business generating the data occurs); then the initial aggregation result of the offline data warehouse DWD layer is read, and is aggregated in batches according to the data event time to obtain a target aggregation result, which is written into the ADS layer.

[0053] For example, assuming that the total call duration of the whole country is finally desired, the network operation data processing device acquires the to-be-processed data, such as the call records in the whole country, first performs stream aggregation on the to-be-processed data, accumulates the call time fields of the users in each province, and obtains the call duration data of the users in each province as an initial aggregation result. Still in combination with the foregoing example, the processing device performs batch processing on the call duration data of the users in each province, obtains the target aggregation result, that is, the total call duration data of the whole country, and writes the total call duration data of the whole country into the ADS layer, thereby completing the processing of the network operation data.

[0054] The to-be-processed data can be acquired in various ways. In one example, S201 can specifically include:

[0055] Splitting and collecting the interface links between the communication network elements, and performing packet capturing and stream splitting processing on the collected split code streams to obtain the raw data; and / or collecting the record data in the network management system operation log to obtain the raw data;

[0056] Performing synthetic processing on the raw data to obtain the to-be-processed data; wherein the synthetic processing includes at least one of the following: field extraction and assignment, association backfilling, and service identification.

[0057] Specifically, the to-be-processed data can be obtained by performing synthetic processing on the raw data. In actual application, based on different data processing scenarios, the raw data can be acquired in a corresponding way. As an example, in the scenario where the network management system operation log data is not universal, the raw data can be obtained by splitting and collecting. Specifically, a binary code stream in an optical cable is split by an optical splitter to obtain a raw code stream, data packets are captured, and stream splitting processing is performed to obtain the raw data. In the scenario where the network management system operation log data is universal, the raw data can be obtained by reading the log. Specifically, the record data in the network management system operation log can be directly collected to obtain the raw data.

[0058] The synthetic processing of the raw data is not limited, such as but not limited to field extraction and assignment, association backfilling, and service identification. It should be noted that the above synthetic processing can be performed only one or multiple, and the above-mentioned ways of acquiring raw data can be implemented alone or in combination, which are not limited herein.

[0059] The implementation means of the synthetic data can be various. In one example, field extraction and assignment can specifically include field value extraction on the same signaling flow in the raw data. In another example, association backfilling can specifically include association backfilling on the messages associated in the raw data. In yet another example, service identification can specifically include backfilling the service type in the raw data by using deep packet inspection technology.

[0060] For example, a paging procedure is identified, which has a fixed code of 41, and the procedure type field is assigned a value of 41; the procedure carries a unique code of a machine present in the mobile phone, and according to the association between the machine code and the user number, the user number field is backfilled; message data is generated in the user's online process, and the device can only obtain the opposite end server, a uniform resource locator (URL), and signaling specific transmission content, and cannot obtain the service type, such as video, game, instant messaging, website access, etc., and a deep packet inspection technology is used to match the message with an identification library to identify the service type and backfill to the signaling procedure. One or more of the synthesis processing modes can be used, which are not limited herein. The necessary information fields are extracted to synthesize the original data for subsequent data processing.

[0061] The number of batch aggregations can be one or more, for example, the initial aggregation result can be batch aggregated once to obtain the target aggregation result, or the initial aggregation result can be batch aggregated once to obtain an intermediate aggregation result, and then the intermediate aggregation result can be batch aggregated again, and so on, until the target aggregation result is finally obtained, which is not limited herein.

[0062] In actual applications, the number of batch aggregations can be determined according to actual conditions, for example, for a small amount of data, such as when the target aggregation result to be obtained is the number of Beijing call users, one batch aggregation can be used, and for a large amount of data, such as when the target aggregation result to be obtained is the number of national call users, multiple batch aggregations can be used, so that a more appropriate way of data processing can be used according to the situation, improving the efficiency of data processing while ensuring the accuracy of data processing.

[0063] Figure 5 A flowchart of a network operation data processing method according to an embodiment of the present application is shown in FIG. 2. As shown in FIG. 2, it is assumed that two batch aggregations are performed, and S203 can specifically include: Figure 5

[0064] S501: using a batch processing mode, performing first-level aggregation on the initial aggregation result to obtain an intermediate aggregation result, and writing the intermediate aggregation result to a DWS layer of an offline data warehouse;

[0065] S502: using a batch processing mode, performing second-level aggregation on the intermediate aggregation result in the DWS layer to obtain the target aggregation result.

[0066] ​Specifically, the initial aggregation result obtained by stream aggregation is read from the DWD layer, and first batch aggregation is performed on the initial aggregation result, and the intermediate aggregation result is obtained by processing in batches according to data event time, and the intermediate aggregation result is written into the DWS layer of the offline data warehouse; the intermediate aggregation result is read from the DWS layer, and second batch aggregation is performed on the intermediate aggregation result, and the target aggregation result is obtained by processing in batches according to data event time. For example, assuming that the number of users calling nationwide per hour is currently desired, the network operation data processing device obtains the data to be processed, such as the calling record nationwide per hour; first, stream processing is used to aggregate the calling record of each user to obtain the initial aggregation result, and the initial aggregation result is written into the DWD layer; then, batch processing is used to aggregate the number of users calling per hour according to the province to obtain the number of users calling in each province as the intermediate aggregation result, and the intermediate aggregation result is written into the DWS layer of the offline data warehouse; and then, batch processing is used to aggregate the number of users calling in each province per hour to finally obtain the number of users calling nationwide per hour as the target aggregation result, and the target aggregation result is written into the ADS layer.

[0067] To implement batch aggregation processing, in one example, S501 can specifically include:

[0068] Based on the initial aggregation result, a first data table in each first period is periodically generated according to a predetermined first period;

[0069] The incremental aggregation result of the first data table in the current first period and the first data table in the historical first period is processed in an incremental aggregation manner to obtain the intermediate aggregation result in the data service layer.

[0070] Specifically, batch processing generates data tables periodically according to data event time in batches, and for a coarse-grained period, such as a day granularity, the day granularity is taken as an original period, a first period is set for batch processing, a data aggregation result in the first period is periodically generated, and then the aggregation result in the current first period is aggregated with the aggregation result before the current first period in the original period until the aggregation result in the original period is obtained as the intermediate aggregation result. For example, assuming that the calling data of Guangzhou per day is currently desired, the period of batch aggregation is predetermined as an hour, and the calling data aggregation result of Guangzhou per hour is generated, and then the calling data aggregation result of the current hour is aggregated with the calling data aggregation result before the current hour of the day until the calling data aggregation result of Guangzhou per day is finally obtained.

[0071] In actual application, the batch aggregation manner can be determined according to actual conditions. For example, for the case of obtaining an hour-level data table, the data can be processed hour by hour to obtain the data table; for the case of obtaining a day-level data table, an incremental aggregation manner can be used to obtain the data table, so as to spread the calculation consumption of the full-day data aggregation result to each hour, reduce the resource occupation peak, and reduce the result output delay.

[0072] In another example, S502 can specifically include:

[0073] Based on the intermediate aggregation result, a second data table in each second period is generated periodically according to a predetermined second period;

[0074] An incremental aggregation manner is used to process the incremental aggregation result of the second data table in the current second period and the second data table in the historical second period, to obtain the target aggregation result; or, a one-time aggregation manner is used to process the second data table in each second period, to obtain the target aggregation result.

[0075] Here, the process of S501 incremental aggregation is similar to S502, and is not described herein again. However, for the ADS layer complex business logic which is not suitable for incremental aggregation, such as multi-table association, a one-time aggregation manner can be used to aggregate the second data table in each second period to generate the ADS layer aggregation result. For example, it is assumed that the final goal is to obtain the call data of Guangzhou per day, the batch aggregation period is set to be an hour, the Guangzhou call data aggregation result is generated hour by hour, and the call data aggregation results of each hour in a day are aggregated to obtain the Guangzhou call data aggregation result per day.

[0076] In actual application, the batch aggregation manner can be determined according to actual conditions. The period can be set, and the data table in each period is generated periodically, and the incremental aggregation result of the data table in the current period and the data table in the historical period is incrementally aggregated; or, the period can be set, and the data table in each period is generated periodically, and the second data table in each period is directly aggregated once. Therefore, a more appropriate manner can be used for data processing, the resource occupation peak is reduced, and the calculation pressure is reduced.

[0077] In addition, in order to improve the data quality, the data can be preprocessed before the stream processing of the to-be-processed data. In one example, before S202, the following can also be included:

[0078] The to-be-processed data is preprocessed, and the preprocessing includes at least one of the following: filtering illegal data, filling null values, filtering abnormal values, associating dimension table fields, and splicing records.

[0079] For example, the preprocessing operation can include: filling the null value with non-null value, deleting the record containing the abnormal value, adding the associated field in the dimension table, splicing the log record to remove the data not conforming to the network protocol definition format; using non-null value, filling the null value field; through the associated field, the association reference between multiple dimension table data is carried out; the abnormal value detection is carried out on the data, the abnormal value exceeding the preset dispersion degree is deleted; and the different network management system log records are spliced. Wherein, the preprocessing mode can adopt one or more of them, which is not limited here.

[0080] In the above example, before the data aggregation processing is performed, the to-be-processed data is preprocessed to ensure the accuracy of the data, thereby further ensuring the accuracy of the subsequent data aggregation processing result.

[0081] In actual application, in addition to considering the data demand of the application layer, backup and storage of the basic data can also be considered. In one example, before S202, it further includes:

[0082] The preprocessing result is written into the data operation layer of the offline data warehouse.

[0083] Specifically, the ODS layer is a temporary storage layer, which is a temporary storage area of interface data. In terms of data granularity, the data granularity of the ODS layer is the finest. Writing the preprocessing result into the ODS layer of the offline data warehouse can basically retain the original data, facilitating subsequent realization of original data detailed query, historical record backtracking, etc. For example, the preprocessed call record data is written into the ODS layer, when the subsequent processing result has an error or data loss, the original data can be queried in the ODS layer. In this example, by writing the preprocessing result into the ODS layer of the offline data warehouse, the original data is basically retained, and the reliability of data processing is improved.

[0084] In the network operation data processing method provided by the embodiment, the obtained network operation data is first simply and streamingly aggregated by using the stream processing mode to obtain an initial aggregation result, and based on the initial aggregation result, batch aggregation is performed to obtain a final processing result provided to the application layer. By performing initial aggregation, the present scheme can reduce the data processing amount, thereby reducing the intermediate state storage occupation, and significantly reducing the data amount of subsequent batch processing, simplifying the subsequent aggregation algorithm, and further, by performing the initial stream aggregation, the accuracy of data aggregation processing can be improved. Compared with the way of ensuring processing accuracy by redundancy processing, the processing flow of the present scheme is more simple, thereby ensuring the accuracy of network operation data processing while reducing the consumption of computing resources.

[0085] Embodiment two

[0086] Figure 6A structural schematic diagram of a network operation data processing apparatus provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the network operation data processing apparatus provided by the embodiment can include: Figure 6

[0087] An acquisition module 61 is configured to acquire to-be-processed data, wherein the to-be-processed data includes network operation data.

[0088] A stream processing module 63 is configured to perform initial aggregation on the to-be-processed data in a stream processing manner to obtain an initial aggregation result, and write the initial aggregation result to a data detail layer of an offline data warehouse.

[0089] A batch processing module 64 is configured to perform batch aggregation on the initial aggregation result in the data detail layer in a batch processing manner to obtain a target aggregation result, and write the target aggregation result to an application layer of the offline data warehouse.

[0090] In actual application, the network operation data processing apparatus can be implemented by a computer program, for example, application software, or can be implemented as a medium storing the computer program, for example, a U disk, a cloud disk, or can be implemented by an entity device integrating or installing the computer program, for example, a chip or a server.

[0091] Specifically, after the acquisition module 61 acquires the to-be-processed data, the stream processing module 63 first performs simple stream aggregation, wherein optional algorithms include but are not limited to conditional summation, conditional maximum value calculation, and conditional minimum value calculation, to obtain the initial aggregation result, and writes the initial aggregation result to the offline data warehouse DWD layer in a stream manner, and writes different time partitions according to data event time (time when the business generating the data occurs).

[0092] The acquisition of the to-be-processed data can be performed in multiple ways. In an example, the acquisition module 61 can specifically include:

[0093] A collection unit 611 is configured to perform light splitting collection on interface links between communication network elements, and perform packet capture and stream processing on the collected light splitting code streams to obtain raw data, and / or collect record data in a network management system operation log to obtain raw data.

[0094] A synthesis unit 612 is configured to perform synthesis processing on the raw data to obtain the to-be-processed data, wherein the synthesis processing includes at least one of the following: field extraction and assignment, relationship backfilling, and business identification.

[0095] ​The synthetic unit 612 is not limited to the manner of performing synthetic processing on the original data, such as but not limited to field extraction assignment, association backfilling, and service identification. It should be noted that the above-mentioned synthetic processing can be performed only one of them, or multiple, and the above-mentioned manner of obtaining original data can be implemented alone or in combination, which is not limited here.

[0096] The implementation means of the above-mentioned synthetic processing can be various. In one example, the synthetic unit 612 performing field extraction assignment can specifically include field value extraction on the same signaling flow in the original data. In another example, the synthetic unit 612 performing association backfilling can specifically include association backfilling on the messages associated in the original data. In yet another example, the synthetic unit 612 performing service identification can specifically include backfilling the service type in the original data using deep packet inspection technology.

[0097] The number of batch processing for batch aggregation can be one or more. In actual application, the number of batch aggregation can be determined according to actual conditions, so that a more appropriate way of data processing is used to improve the efficiency of data processing while ensuring the accuracy of data processing.

[0098] As shown in Figure 6 If batch aggregation is performed twice, the batch processing module 64 can specifically include:

[0099] The first batch processing unit 641 is configured to perform first-level batch aggregation on the initial aggregation result using batch processing to obtain an intermediate aggregation result, and write the intermediate aggregation result to the DWS layer of the offline data warehouse.

[0100] The second batch processing unit 642 is configured to perform second-level batch aggregation on the intermediate aggregation result in the DWS layer using batch processing to obtain the target aggregation result, and write the target aggregation result to the data application layer of the offline data warehouse.

[0101] Specifically, the first batch processing unit 641 reads the initial aggregation result obtained by stream aggregation from the DWD layer, performs first-level batch aggregation, processes according to the data event time in batches, obtains the intermediate aggregation result, and writes the intermediate aggregation result to the DWS layer of the offline data warehouse; reads the intermediate aggregation result from the DWS layer, and performs second-level batch aggregation on the intermediate aggregation result, processes according to the data event time in batches, and obtains the target aggregation result.

[0102] In order to realize batch aggregation processing, in one example, the first batch processing unit 641 is specifically configured to:

[0103] Based on the initial aggregation result, a first data table under each first period is generated periodically according to a predetermined first period;

[0104] The incremental aggregation result of the first data table under the current first period and the first data table under the historical first period is processed in an incremental aggregation manner to obtain the intermediate aggregation result under the data service layer.

[0105] Specifically, the first batch processing unit 641 generates data tables periodically according to data event time batches. For a coarse-grained period, such as a day granularity, it is taken as an original period, and a first period is set for batch processing. The data aggregation result under the first period is generated periodically, and the aggregation result under the current first period is aggregated with the aggregation result before the current first period in the original period until the aggregation result under the original period is obtained as the intermediate aggregation result.

[0106] In actual application, the batch aggregation manner can be determined according to actual conditions. For example, for the case of obtaining an hour-level data table, data can be processed to obtain a data table hour by hour; for the case of obtaining a day-granularity data table, an incremental aggregation manner can be used to obtain a data table, so as to spread the calculation and consumption of the full-day data aggregation result to each hour, thereby reducing the resource occupation peak and the result output delay.

[0107] In another example, the second batch processing unit 642 is specifically used for:

[0108] Based on the intermediate aggregation result, a second data table under each second period is generated periodically according to a predetermined second period;

[0109] The incremental aggregation result of the second data table under the current second period and the second data table under the historical second period is processed in an incremental aggregation manner to obtain the target aggregation result; or, the second data table under each second period is processed in a one-time aggregation manner to obtain the target aggregation result.

[0110] Here, the process of the second batch processing unit 642 incremental aggregation is similar to that of the first batch processing unit 641, which is not described here. However, for the ADS layer complex business logic which is not suitable for incremental aggregation, such as multi-table association, the second data table under each second period can be aggregated in a one-time aggregation manner to generate the ADS layer aggregation result.

[0111] In actual application, the batch aggregation manner can be determined according to actual conditions. A period can be set, and a data table in each period is generated regularly, and the incremental aggregation result of the data table in the current period and the data table in the historical period is incrementally aggregated; or, a period can be set, and a data table in each period is generated regularly, and the second data table in each period is directly aggregated at one time. Thus, a more appropriate manner is used for data processing in a targeted manner, resource occupation peak is reduced, and calculation pressure is reduced.

[0112] In addition, in order to improve data quality, in an example, the apparatus further comprises:

[0113] The preprocessing module 62 is configured to, before the stream processing module 63 performs initial aggregation on the to-be-processed data to obtain an initial aggregation result, preprocess the to-be-processed data, and the preprocessing comprises at least one of the following: filtering illegal data, filling null values, filtering abnormal values, associating dimension table fields, and splicing records.

[0114] For example, the preprocessing operation can comprise: screening and removing illegal data that does not conform to input rules or will cause malicious cycles, filling null values with non-null values, deleting records containing abnormal values, adding associated fields in the dimension table, and splicing log records. The preprocessing manner can adopt one or more of the above manners, which is not limited herein.

[0115] In the above example, the to-be-processed data is preprocessed before data aggregation processing, so as to ensure data accuracy, and further ensure the accuracy of the subsequent data aggregation processing result.

[0116] In actual application, in addition to considering the data requirements of the application layer, backup and storage of the basic data can also be considered. In an example, the preprocessing module 62 is further configured to:

[0117] write the preprocessing result to a data operation layer of the offline data warehouse.

[0118] Specifically, the ODS layer is a temporary storage layer, and is a temporary storage area of interface data. In terms of data granularity, the data granularity of the ODS layer is the finest. Writing the preprocessing result to the ODS layer of the offline data warehouse can basically retain the original data, and facilitate subsequent implementation of original data detail query, historical record backtracking, etc. For example, the preprocessed call record data is written to the ODS layer, and when the subsequent processing result has an error or data is lost, the original data can be queried in the ODS layer.

[0119] In the example, by writing the preprocessing result to the ODS layer of the offline data warehouse, the original data is basically retained, and the reliability of data processing is improved.

[0120] The network operation data processing device provided by the embodiment first performs simple stream aggregation on the network operation data obtained by the acquisition module in a stream processing manner to obtain an initial aggregation result. Based on the initial aggregation result, the batch processing module performs batch aggregation to obtain a final processing result provided to the application layer. The scheme can reduce the data processing amount by first performing initial aggregation, thereby reducing the intermediate state storage occupation and significantly reducing the subsequent batch processing data amount, simplifying the subsequent aggregation algorithm. In addition, the initial stream aggregation can improve the accuracy of data aggregation processing. Compared with the redundant processing method for ensuring processing accuracy, the processing flow of the scheme is more simple, thereby reducing the calculation resource consumption while ensuring the accuracy of network operation data processing.

[0121] Embodiment three

[0122] Figure 7 The structure diagram of the network operation data processing device provided by an embodiment of the application is shown.

[0123] In this example, the open source data warehouse component Hive (a data warehouse tool used for data extraction, transformation, and loading) is used to store the ODS, DWD, DWS, and ADS layer data processing results, respectively.

[0124] The acquisition unit is a general Linux (a multi-user, multi-task, multi-thread, and multi-CPU operating system based on POSIX) server with probe programs and analysis programs deployed for each province in China, which is used to obtain network operation data spectral streams or network management original log records.

[0125] The synthesis unit is a general Linux server with data synthesis programs and Kafka (a high-throughput distributed publish-subscribe message system that can process all action stream data of consumers in a website) producer programs, which is used to perform synthesis processing on the network operation data. The synthesis methods include but are not limited to field extraction and assignment, relationship backfilling, and business identification, etc. The obtained data to be processed is the network operation data, and the synthesis records are reported to the Kafka cluster.

[0126] The transmission module uses the open source distributed message system Kafka to transmit data to the data processing center, which is used to receive the synthesized network operation data and provide it to the downstream data processing module for subscription and consumption.

[0127] The stream processing module uses an open source Flink framework (full name Apache Flink, an open source stream processing framework that executes any stream data program in a data parallel and pipelining manner) as a stream processing engine, and deploys a Flink stream processing task through a YARN (a general resource management system that can provide unified resource management and scheduling for upper-layer applications) cluster to pre-process and perform simple aggregation calculation on the network operation data consumed from Kafka. The dimension table data used in the pre-processing and simple aggregation calculation steps is stored in Redis (a high-speed cache service based on memory) or the task node memory. The pre-processing and simple aggregation calculation results are written into the ODS database and the DWD database of the Hive data warehouse, respectively.

[0128] The batch processing uses an open source Spark framework as a batch processing engine, and deploys a Spark batch processing task through a YARN cluster to further periodically aggregate the aggregation results read from the DWD layer of the Hive data warehouse. The dimension table data required for aggregation is stored in Redis or the task node memory. The embodiment performs two batch aggregations, and the aggregation calculation results are written into the DWS database and the ADS database of the Hive data warehouse, respectively.

[0129] In the network operation data processing method provided in the embodiment, the network operation data obtained is first simply aggregated in a stream processing manner to obtain initial aggregation results, and based on the initial aggregation results, batch aggregation is performed to obtain final processing results provided to the application layer. The scheme can reduce the amount of data processing by first performing initial aggregation, thereby reducing the occupation of intermediate state storage and significantly reducing the amount of data for subsequent batch processing, simplifying the subsequent aggregation algorithm. In addition, by performing the initial stream aggregation, the accuracy of data aggregation processing can be improved. Compared with the way of ensuring processing accuracy through redundancy processing, the processing flow of the scheme is more simple, thereby reducing the consumption of computing resources while ensuring the accuracy of network operation data processing.

[0130] Embodiment Four

[0131] Figure 8 A structural schematic diagram of an electronic device provided in the embodiments of the present disclosure is shown in FIG. 1. Figure 8 As shown in FIG. 1, the electronic device includes:

[0132] The electronic device further includes a processor 291 and a memory 292. The electronic device can further include a communication interface 293 and a bus 294. The processor 291, the memory 292, and the communication interface 293 can communicate with each other through the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can invoke the logical instructions in the memory 292 to execute the method of the above-described embodiments.

[0133] In addition, the logical instructions in the memory 292 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0134] The memory 292, as a computer readable storage medium, can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiments of the present disclosure. The processor 291 executes the functions and data processing by running the software programs, instructions and modules stored in the memory 292, that is, implements the method in the above-described method embodiments.

[0135] The memory 292 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function; the data storage area can store data created during use of the terminal device, etc. In addition, the memory 292 can include a high-speed random access memory, and can also include a non-volatile memory.

[0136] The embodiments of the present disclosure provide a non-transitory computer readable storage medium, which stores computer execution instructions. The computer execution instructions are executed by a processor to implement the method described in the foregoing embodiments.

[0137] Embodiment five

[0138] The embodiments of the present disclosure provide a computer program product, which includes a computer program. The computer program is executed by a processor to implement the method provided by any of the above-described embodiments of the present disclosure.

[0139] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0140] It is to be understood that the application is not limited to the precise construction already described above and shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should only be limited by the claims appended hereto.

Claims

1. A method of processing network operation data, characterized by, The method comprises the following steps: acquiring to-be-processed data, wherein the to-be-processed data comprises network operation data; performing initial aggregation on the to-be-processed data in a stream processing mode to obtain an initial aggregation result, and writing the initial aggregation result into a data detail layer of an offline data warehouse, wherein different time partitions are written according to times of business occurrence of generated data; wherein the initial aggregation comprises conditionally summing, conditionally finding a maximum value, and conditionally finding a minimum value; generating a first data table under each first period according to a predetermined first period based on the initial aggregation result; performing processing on an incremental aggregation result of the first data table under a current first period and the first data table under a historical first period in an incremental aggregation mode to obtain an intermediate aggregation result under a data service layer, and writing the intermediate aggregation result into the data service layer of the offline data warehouse; generating a second data table under each second period according to a predetermined second period based on the intermediate aggregation result; performing processing on an incremental aggregation result of the second data table under a current second period and the second data table under a historical second period in an incremental aggregation mode to obtain a target aggregation result, or performing processing on the second data table under each second period in a one-time aggregation mode to obtain the target aggregation result, and writing the target aggregation result into a data application layer of the offline data warehouse.

2. The method of claim 1, wherein, The acquiring to-be-processed data comprises: performing light splitting collection on interface links between communication network elements, and performing packet capturing and flow splitting processing on collected light splitting code streams to obtain raw data; and / or collecting recorded data in network management system operation logs to obtain raw data; performing synthesis processing on the raw data to obtain the to-be-processed data; wherein the synthesis processing comprises at least one of the following: field extraction and assignment, association backfilling, and business identification.

3. The method of claim 2, wherein, The performing synthesis processing on the raw data to obtain the to-be-processed data comprises: performing field value extraction on a same signaling flow in the raw data; performing association backfilling on messages associated in the raw data; performing backfilling on a business type in the raw data by using a deep packet inspection technology.

4. The method according to any one of claims 1 to 3, characterized in that, Before the performing initial aggregation on the to-be-processed data in a stream processing mode to obtain an initial aggregation result, the method further comprises: performing preprocessing on the to-be-processed data, wherein the preprocessing comprises at least one of the following: filtering illegal data, filling empty values, filtering abnormal values, associating dimension table fields, and splicing records.

5. The method of claim 4, wherein, Before the performing initial aggregation on the to-be-processed data in a stream processing mode to obtain an initial aggregation result after the performing preprocessing on the to-be-processed data, the method further comprises: writing the preprocessing result into a data operation layer of an offline data warehouse.

6. A network operation data processing apparatus, characterized by comprising: The method comprises the following steps: an acquiring module, configured to acquire to-be-processed data, wherein the to-be-processed data comprises network operation data; A stream processing module is configured to perform initial aggregation on the to-be-processed data in a stream processing manner to obtain an initial aggregation result, and write the initial aggregation result into a data detail layer of the offline data warehouse according to time partitions of different time periods in which the data is generated; A batch processing module is configured to perform batch aggregation on the initial aggregation result in the data detail layer in a batch processing manner to obtain a target aggregation result, and write the target aggregation result into a data application layer of the offline data warehouse; The batch processing module comprises: A first batch processing unit is configured to perform first-level aggregation on the initial aggregation result in a batch processing manner to obtain an intermediate aggregation result, and write the intermediate aggregation result into a data service layer of the offline data warehouse; A second batch processing unit is configured to perform second-level aggregation on the intermediate aggregation result in the data service layer in a batch processing manner to obtain the target aggregation result; The first batch processing unit is specifically configured to generate a first data table of each first period based on the initial aggregation result according to a predetermined first period. The first batch processing unit is specifically configured to generate a first data table of each first period based on the initial aggregation result according to a predetermined first period. The second batch processing unit is specifically configured to generate a second data table of each second period based on the intermediate aggregation result according to a predetermined second period. The second batch processing unit is specifically configured to generate a second data table of each second period based on the intermediate aggregation result according to a predetermined second period.

7. An electronic device, comprising: The second batch processing unit is specifically configured to generate a second data table of each second period based on the intermediate aggregation result according to a predetermined second period. comprises: A processor, and a memory connected to the processor in communication; The memory stores computer execution instructions; 8. A computer-readable storage medium, characterized in that, The processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1-5.

9. A computer program product comprising a computer program, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1-5. The computer program, when executed by the processor, is used to implement the method of any one of claims 1-5.

Images