System and method for long-term compilation and retrieval of historical data in network performance analysis

Abstract

To store network performance data for later retrieval, performance data files corresponding to a test cycle are retrieved, reformatted into unified data files according to a predetermined unified file format and a set of predefined unified category identifiers, and stored in a query database. The reformatting can match predefined unified category identifiers with data category identifiers in the performance data files, and the corresponding data values ​​can be stored sequentially in the unified data files or used to increment counter values ​​in similarly stored aggregated counter files. In addition, configuration data files for network cells can be stored less frequently than the frequency of the test cycles. Later retrieval can retrieve both the unified data files and the configuration data files corresponding to a selected time frame and network portion, and merge or aggregate the data appropriately for performance indicator calculations.

Description

[Background technology]

[0001] 1. Field Apparatus and methods consistent with example embodiments relate to network performance analysis, and more particularly to efficient long-term storage, compilation, and retrieval of data for analysis of historical and / or longitudinal network performance.

[0002] 2. Description of Related Technology In large networks, such as mobile networks, it is impractical to physically inspect every component. Instead, problematic components are identified by their impact on network behavior. Network-connected computing devices, including both base unit systems and mobile devices, can monitor network behavior and generate data logs. These logs can then be analyzed to identify areas where the network is not functioning as intended.

[0003] Intended performance levels are typically defined according to various quantifiable metrics, referred to in the art as "key performance indicators" or "KPIs." If the measured or calculated values ​​of a KPI do not achieve the intended level, further investigation is required to identify the cause of the problem. KPIs are further useful for evaluating performance improvements resulting from system upgrades and for generally monitoring and forecasting network development. Summary of the Invention

[0004] One objective of the disclosed system and method is to store network performance data so that it can be easily retrieved based on a selected past time frame.

[0005] Another object of the disclosed system and method is to reduce redundant data within stored network performance data.

[0006] Yet another object of the disclosed system and method is to automatically convert data from multiple disparate sources into a unified data file for retrieval thereof.

[0007] According to certain embodiments of the present disclosure, a method for long-term storage of network performance data for later retrieval is provided. The method includes obtaining a plurality of performance data files corresponding to a test cycle. Each of the plurality of performance data files includes data describing network performance. The plurality of performance data files includes at least a first data file in a first format and a second data file in a second format different from the first format. The method further includes reformatting each of the plurality of performance data files according to a predetermined unified file format and a set of predefined unified category identifiers to obtain a plurality of unified data files. The method further includes storing the plurality of unified data files in a query database in memory.

[0008] According to another embodiment of the present disclosure, a method for analyzing historical network performance is provided. The method includes storing a plurality of unified data files in a query database in a memory. The method further includes searching the query database according to a received search request to thereby obtain a set of retrieved unified data files. The method further includes calculating at least one performance indicator based on the set of retrieved unified data files.

[0009] According to yet another embodiment of the present disclosure, a system for long-term storage of network performance data for later retrieval is provided. The system includes at least one non-volatile memory electrically configured to store computer program code. The system further includes at least one processor operably connected to the non-volatile memory and configured to operate as instructed by the computer program code. The computer program code includes file retrieval code configured to cause at least one of the at least one processor to retrieve a plurality of performance data files corresponding to a test cycle. Each of the plurality of performance data files includes data describing network performance. The plurality of performance data files includes at least a first data file in a first format and a second data file in a second format different from the first format. The computer program code further includes formatting code configured to cause at least one of the at least one processor to reformat each of the plurality of performance data files in accordance with a predetermined uniform file format and a predefined set of uniform category identifiers to obtain a plurality of unified data files. The computer program code further includes storage code configured to cause at least one of the at least one processor to store the plurality of unified data files in a query database in the query memory.

[0010] According to yet another embodiment of the present disclosure, there is provided a non-transitory computer-readable storage medium having instructions executable by at least one processor to execute a method for long-term storage of network performance data for later retrieval. The method includes obtaining a plurality of performance data files corresponding to a test cycle. Each of the plurality of performance data files includes data describing network performance. The plurality of performance data files includes at least a first data file in a first format and a second data file in a second format different from the first format. The method further includes reformatting each of the plurality of performance data files in accordance with a predetermined unified file format and a set of predefined unified category identifiers to obtain a plurality of unified data files. The method further includes storing the plurality of unified data files in a query database in memory.

[0011] Additional aspects will be set forth in part in the description that follows, and in part will be apparent from the description, or may be learned by practice of presented embodiments of the present disclosure. [Brief explanation of the drawings]

[0012] Features, aspects, and advantages of certain exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements.

[0013] [Figure 1A] FIG. 1 is a flow diagram illustrating a process flow for efficient storage of data for historical KPI calculations, according to an example embodiment.

[0014] [Figure 1B] FIG. 10 is a flow diagram illustrating a process flow for fast historical KPI calculation in accordance with an example embodiment.

[0015] [Figure 2]FIG. 10 is a flow diagram illustrating a process flow for reformatting raw data files in accordance with an exemplary embodiment.

[0016] [Figure 3] FIG. 10 is a flow diagram illustrating a process flow for validating a reformatted data file according to an example embodiment.

[0017] [Figure 4A] 1 is a block diagram illustrating a system for processing data search queries, according to an example embodiment.

[0018] [Figure 4B] FIG. 1 is a flow diagram illustrating a process flow for processing a data search query according to an example embodiment.

[0019] [Figure 5] FIG. 10 is a flow diagram illustrating a process flow for calculating KPIs in accordance with an exemplary embodiment.

[0020] [Figure 6] FIG. 1 is a diagram of example components of a device in which embodiments of the systems and / or methods described herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0021] The following detailed description of exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements. Specific exemplary embodiments for sample applications are described below with reference to the figures illustratively shown in the drawings, to illustrate the disclosed systems and methods.

[0022] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of implementations. Moreover, one or more features or components of one embodiment may be incorporated into or combined with another embodiment (or one or more features of another embodiment). Furthermore, in the flowcharts and descriptions of operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed concurrently (at least in part), or the order of one or more operations may be interchanged.

[0023] It will be apparent that the systems and / or methods described herein may be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0024] Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0025] No element, act, or instruction used herein should be construed as critical or essential unless expressly stated as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, terms such as "has," "have," "having," "include," and "including" are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include A only, B only, or both A and B.

[0026] It should be noted that the principles disclosed herein are generally applicable to networks of all forms, including, but not limited to, Internet service provider networks such as fiber optic and cable networks, traditional telephone networks, both wired and wireless networks in structures, complexes, or other localized areas, and even non-communication networks such as power grids. However, throughout this disclosure, the networks analyzed and managed by the disclosed system will be primarily referred to as mobile networks for convenience and brevity.

[0027] As briefly discussed in the Background section, periodic reviews of network key performance indicators (KPIs) are a desirable part of quality assurance testing for large-scale networks. These reviews determine whether KPI values ​​meet baseline or target thresholds, either normally or after upgrades, and also monitor the trends of these values ​​over time to identify areas where future development may be needed.

[0028] KPIs can be calculated for the entire network or any portion thereof. A portion of the network can be defined as one or more cells, each of which is defined by the specific cellular tower or other transceiver to which devices considered to be "in" the cell are coupled. Cells do not have defined physical boundaries, so that devices crossing such physical boundaries consistently disconnect from the cell's transceiver and connect to the transceiver of an adjacent cell. However, when selecting a portion of the network according to an area on a map of the physical region, cells "in" the selected portion can be defined according to the transceivers physically located within the corresponding selected area. This style of portion selection is typically performed by "drawing" a polygonal shape on a digital map representation, and the resulting selected network portion is sometimes referred to as the selected "polygon." Polygons may also be defined on diagrams that visually represent the network according to something other than physical area, such as a transceiver interconnection chart or hierarchical chart. Furthermore, network portions may also be defined and selected according to other criteria, such as, for example, all cells managed by or through a particular central network unit or hub, all cells operating on a particular technology standard (e.g., 4G, 5G) or operating system, etc. Testing portions defined according to each of these criteria may be equivalent to testing characteristics common to the portions.

[0029] As briefly discussed in the "Background," data used to measure or calculate the values ​​of KPIs (for brevity, referred to herein as "calculating KPIs") can be periodically collected by various computing systems connected to a network. More specifically, the data can be values ​​of various parameters, such as, for example, identifier values ​​including, but not limited to, the model of a device and the communication protocol it uses, measurements including, but not limited to, ping time and bandwidth, and counter values ​​(sometimes simply referred to as "counters") including, but not limited to, the number of connections released or particular function calls within a specified period of time.

[0030] Mobile network data logs can be voluminous, making it impractical to retain them in the analysis system's local memory for long periods of time, especially as new data continues to arrive. Furthermore, the most recent data is typically most relevant to KPI calculations. Therefore, as new data (which may be referred to as "current data") is retrieved and requires storage space, older data (which may be referred to as "historical data" or "past data") can be moved to medium- or long-term storage memory in a database, such as an EdgeDB® database, or in another searchable format, to be retrieved only when needed. Many data analysis tools suitable as the basis for KPI calculations, such as Apache® Spark, include the ability to store and retrieve data in this manner.

[0031] However, this approach complicates the calculation of KPIs related to past network behavior.

[0032] As an illustrative example, it may be desirable to perform a post-mortem analysis of an event on a network that occurred three days ago. A search can be performed on long-term storage memory to retrieve historical data from that time frame—i.e., from one or more test cycles that occurred at the time of the event, as well as from test cycles immediately preceding or following the event, if relevant—so that KPIs can be calculated for the event, particularly KPIs that were not calculated as part of a standard test cycle (and therefore were not calculated at the time of the event) but are relevant in this case due to the nature of the event. However, the amount of historical data for a single test cycle can be substantial. Furthermore, the amount of historical data in the entire database can make searching even a single data element therein overly exhaustive. Using a traditional database search, such as that provided by Spark SQL, it can take an inordinate amount of time to retrieve all the historical data related to the event three days ago from the database. It may take several hours before everything necessary to calculate the applicable KPIs is gathered. If KPIs are to be calculated for a specific portion of the network, the search will be longer despite retrieving less data because the search must consider the data in the database according to additional parameters other than timestamp or test cycle label to determine which data to retrieve.

[0033] This problem is amplified when retrieving historical data from multiple test cycles. As an illustrative example, it may be desirable to introduce a new KPI into standard analysis. Ideally, a baseline threshold for that KPI should be established at the same time, representing the value of the KPI that indicates the network is operating satisfactorily. Objectives may also be developed, often based on the baseline. However, developing such a baseline often requires calculating multiple historical values ​​for that KPI, each from a different test cycle. Each such historical value requires its own retrieval set of relevant historical data for the selected test cycle before calculations can be made. The scale of such a retrieval process, in terms of processing time and resources, means that in practice, one can start without a baseline and develop one after several cycles of standard testing for the KPI, or one can develop an initial baseline based on only one or two historical test cycles and then adjust it later. While this is not ideal because it is better to have an accurate baseline as soon as possible, it is the only practical solution when using traditional systems.

[0034] Briefly, exemplary embodiments of the present disclosure provide methods and systems that can collect historical data from other sources as an alternative to, or in addition to, long-term storage database solutions. More specifically, embodiments of the disclosed method and system leverage and refine performance monitoring tools to compile and retrieve abridged versions of historical data logs over time that can be more efficiently formatted for reading specific parameters, at the expense of completeness, size efficiency, or other factors that are less important when quick access is required.

[0035] In a more specific embodiment, the improved performance monitoring tool may be configured to copy recently collected log elements from short-term data storage and store these elements in a more efficient format that is easier to query and retrieve.

[0036] Additionally, an improved site configuration tool may be configured to store a history of configuration data for each transceiver and its corresponding cell, including, but not limited to, cell name, location, beam azimuth and tilt, frequency, bandwidth, operating system, and technology standard (e.g., 4G, 5G). While the values ​​of this data may change over time, they are not expected to change with each test cycle. Thus, the site configuration tool can store a history of this data for periods during which any particular value remains stable, which is more efficient than storing different data points for each test cycle. For example, the site configuration tool may store a data log only once a day, as opposed to every 15-minute test cycle. In this manner, data already stored by the site configuration tool can be omitted from the performance monitoring tool data, reducing storage usage by the performance monitoring tool data and facilitating its retrieval. Relevant portions of the site configuration tool data can then be compiled with the performance monitoring tool data according to the specific time frame queried at the time of each query. The site configuration tool data is not necessarily used in the KPI calculation itself, but may be used to correlate the performance monitoring tool data to specific cells and their corresponding configuration characteristics, allowing calculations to be made according to specific polygons or other defined portions of the network.

[0037] Using the data stored by these tools, and following the disclosed process and variations thereof, KPI calculations for older time frames become feasible within a reasonable period of time.

[0038] Hereinafter, KPIs calculated using past data rather than or in addition to current data will be referred to as "historical KPIs."

[0039] FIG. 1A is a flow diagram illustrating a process flow for efficient storage of data for historical KPI calculations, according to an example embodiment.

[0040] At S110, an element management system server (EMS server) retrieves raw data files corresponding to the current test cycle. The raw data files may be stored short-term on the EMS server to make them available for determining current KPIs before being moved to a medium-term storage database on another memory to clear storage space in that memory for new data. While not described in detail herein, the current KPI determination may be performed by a processor on the EMS server or by another system.

[0041] A suitable storage period for both this purpose and the simplified storage process described below is 12 hours, but this is by way of example only. The ideal period for short-term storage on the EMS server may depend on, among other factors, the processor and memory speed and storage capacity used, as well as the rate of incoming raw data.

[0042] At S120, each raw file is stored in a database in memory. This memory can be organized according to any standard suitable for unstructured data, including, but not limited to, MinIO® and Hadoop® distributed file systems. Unstructured data storage may be necessary because raw files do not yet necessarily share a uniform file format, but each may be formatted according to the standards and preferences of respective device vendors. If the EMS server is efficient enough, it can store the raw files in its own memory and perform subsequent operations on its own processor, although a separate system for this storage is also within the scope of this disclosure.

[0043] The data in each raw file is reformatted in S130 according to a unified file format and a set of unified category identifiers, in a manner further described herein. The resulting files are then validated in S140 to ensure that the formatting was successful, also in a manner further described herein. Finally, in S150, the validated files are stored in a query database implemented on a query memory. The query database may be the same database used in S120, a different database on the same memory, or a different database on a different memory.

[0044] If a database different from the query database is used in S120, the database used in S120 may be referred to as a temporary database. Similarly, if a memory different from the query memory is used in S120, the memory used in S120 may be referred to as a temporary memory.

[0045] If the raw file stored in S120 is already of the desired common file format, operation S130 may still be performed to generate a new file or to modify an existing file to include only data of certain predetermined data categories.

[0046] FIG. 1B is a flow diagram illustrating a process flow for fast historical KPI calculation, according to an example embodiment.

[0047] At S160, the query database is searched for data in accordance with the received request to identify data that (a) falls within the defined search parameters and (b) is relevant to the selected KPIs to be calculated. The search query may be applied to the data file by a suitable query engine. Effective architectures and processes for applying queries to this type of data are each described further herein.

[0048] At S170, historical KPIs are calculated using the data retrieved according to the query, and the results are output at S180, after which the process ends.

[0049] As mentioned above, the raw files retrieved in S110 may be in different file formats. Suitable formats for this type of data include Extensible Markup Language (XML) and XML-like formats (collectively referred to herein as "parsed structure files"), as well as comma-separated value (CSV) files and optimized row columnar (ORC) files, among others.

[0050] A single, unified format is generally preferred for file formatting prior to KPI calculation. Therefore, reformatting raw data files into a unified file can be part of the storage process. Furthermore, reformatting can filter out unwanted categories of data, leaving only the data necessary for historical KPI calculation. Desired categories of data can be defined according to a set of predefined unified category identifiers, which can be used to define the framework for the unified file's content.

[0051] Testing has shown that the ORC format is particularly effective for many of the computational operations described herein, both in terms of read time during queries and storage size in the query database. In particular, the ORC format allows for rapid location and retrieval of specific "stripes" of data from within each data file, rather than the entire file, when a query seeks data from a particular category. However, it has been found that "counter data" is more efficient to store in parsed structure files. Therefore, certain embodiments of the present disclosure can use a hybrid approach to data formatting.

[0052] 2 is a flow diagram illustrating a process flow for reformatting a raw data file according to an exemplary embodiment. This exemplary flow is suitable for formatting operation S130 of FIG. 1A, although other formatting operations are within the scope of the present invention.

[0053] At S210, the raw data file is parsed to identify all data category identifiers according to the file's existing format. As two illustrative examples, a CSV file places its category identifiers in the first line of the file, while an XML file or other parsed structure file uses element names as categories. An appropriate algorithm for parsing such category identifiers can be prepared for each expected file format.

[0054] In S220, it is checked whether the selected one of the unified category identifiers matches any category identifier in the raw file according to the mapping. If so (YES in S220), the flow proceeds to S230. If not (NO in S220), the flow proceeds to S225.

[0055] The matching may be performed according to a predetermined category mapping of the source of the raw files, which may map each uniform category identifier of the predetermined set to one of the expected data category identifiers of the source of the raw files. For example, if the source of the raw files is a vendor, definitions of the raw file categories may be available from the vendor in an inventory file. This inventory can be used prior to execution of the current flow of the process to create an appropriate category mapping from each relevant category to a corresponding uniform category identifier. The category mapping of a given source may also be prepared in advance by other means, and may optionally include a review by the direct administrator of example files to determine or intuit which uniform category identifiers, if any, correspond to a given raw file category identifier.

[0056] However, the source may be newly encountered or may otherwise lack an existing category mapping. Furthermore, the source may have modified the category identifiers of their raw files. Therefore, unless the mapping in S220 allows the selected unified category identifier to match the identifier in the raw file, a comparison of the unified category identifiers to each raw file category identifier in the raw file may be performed in S225 to see if a best match can be identified. This may be done by a suitable text comparison algorithm from the unified category identifier to each raw file category identifier. One or more sample data points corresponding to the raw file category identifiers may also be checked to determine whether they are in an expected format. For example, it may be determined whether there are recognizable numbers in the data corresponding to a suspect "transmission frequency" category identifier and whether the numbers are within the range used for mobile phone transmissions.

[0057] If comparison S225 is configured to output both the best match and the likelihood of the match, a threshold likelihood may be applied. This threshold may be predetermined according to any suitable system requirements, in particular the acceptability or unacceptability of misclassified data, which may vary between unified categories. Best matches that do not meet the threshold likelihood may be discarded, resulting in a "no match."

[0058] If the comparison S225 produces a "no match" result ("No" at S225), it can be assumed that the selected unified category identifier is not present in the raw data file. Thus, flow can proceed to S240 with the value of the selected unified category identifier set to an appropriate null value.

[0059] If the comparison S225 is a match ("Yes" at S225), the flow proceeds to S230. Optionally, before proceeding to S230, an existing category mapping can be automatically updated or a new category mapping can be generated according to this and other determined mapping data.

[0060] Note that in implementations where category mapping is not available, operation S220 may be omitted and process flow can proceed directly from S210 to S225.

[0061] If proceeding to S230, data corresponding to the selected uniform category identifier may be identified within the raw files, and this data may be reformatted as necessary for placement into at least one of two files according to the selected uniform category identifier. That is, the category in question may be used as a direct value, reflected in one or more counter values, or both, as may be desired.

[0062] Thus, in S230, it is determined whether a counter value is predefined that corresponds to the selected unified category identifier and its value in the raw data file. This can be determined according to a table, mapping, or other suitable counter configuration file for defining such a correspondence. The value in this definition can be a single value, a range of values, or an enumerated set of values. Alternatively, the value may be omitted from consideration so that only the selected unified category identifier is important.

[0063] If there is no correspondence (NO in S230), the flow proceeds to S240. If there is a corresponding counter value (YES in S230), the flow proceeds to S235.

[0064] In S235, it is assumed that an aggregate counter file for the test cycle exists. If not, it may be generated as part of operation S235 or before the first iteration of operation S235. The generated counter file includes data representing each of a predetermined set of counter values, each of which may be initialized to 0. The counter file may also store data identifying the test cycle or the corresponding time period. The counter file may be an XML file or other parsed structure file, although the invention is not limited thereto.

[0065] At S235, each counter value corresponding to the selected unified category identifier and its value in the raw data file may be incremented or otherwise increased in value in the counter file.

[0066] As an example, raw data files obtained from a particular device may have a category consistent with the "Device Technology" unified category identifier. The value of this category may be a text value and may be set to "5G" in a particular raw file, which may be understood to indicate that the device operates on "5G" (fifth generation standard) technology. A "Number of 5G Devices" counter may be defined to correspond to a "Device Technology" value of "5G." Accordingly, at S235, the "Number of 5G Devices" counter may be incremented by 1 to indicate that the system has counted the occurrence of this value in one of the raw data files. As subsequent raw data files are similarly processed, it will be apparent that the "Number of 5G Devices" counter continues to increase according to the number of raw data files that exhibit the "5G" value.

[0067] Instead, a particular combination of selected unified category identifier and value can be counted by adding the value to a corresponding counter. As an example, a raw data file obtained from a particular device may have a value of "100" for a category matching the "Megabytes Downloaded" unified category identifier, which may be understood to indicate that 100 megabytes of data have been downloaded to the device since the last test cycle. A "Total Download Throughput" counter may be defined to correspond to any non-zero value in the "Megabytes Downloaded" category. Thus, this value "100" is added to the existing value of the "Total Download Throughput" counter. As subsequent raw data files are similarly processed, it should be apparent that the "Total Download Throughput" counter continues to increment according to the individual "Megabytes Downloaded" values ​​for each individual device.

[0068] Whether a particular counter value should be incremented or the entire value should be added may be indicated as part of the counter configuration file.

[0069] It should be noted that a given combination of uniform category identifier and value may correspond to more than one counter value. It should be further noted that such a correspondence may result in a value being added to one counter value but triggering the increment of another counter value.

[0070] It will be apparent that when all raw data files for a test cycle are so processed, the counter values ​​in the resulting counter file will reflect the activity over the course of the test cycle as represented in the raw data files in the aggregate.

[0071] After S235, the flow continues to S240.

[0072] At S240, a determination is made as to whether to directly store the value of the selected unified category identifier. This may be determined according to a table, mapping, or other suitable storage configuration file indicating this, which may be the same file as the counter configuration file or a different file. This determination need not take into account the value of the category, although a determination that uses the value as a factor is within the scope of this disclosure.

[0073] As previously mentioned, if the process flow reaches this point via the "No at S225" branch, the value in question is not from a raw data file, but is an appropriate "null" value.

[0074] If the value is not to be stored (No in S240), the flow proceeds to S250. If the value is to be stored (Yes in S240), the flow proceeds to S245.

[0075] At S245, the values ​​are arranged according to a predefined sequence in a temporary storage file, which may be called a "data frame," corresponding to the raw data file. The arrangement within the sequence may be based on a uniform category identifier, and the sequence may be defined in a storage configuration file.

[0076] After S245, the flow continues to S250.

[0077] In S250, it is determined whether there is a unified category identifier in the predetermined set that has not yet been selected. If so ("Yes" in S250), the flow returns to S220 to select another unified category identifier and perform another iteration of the loop from S220 to S250. If not ("No" in S250), the flow proceeds to S260.

[0078] At S260, the data in the data frame is converted into a query file, which may have ORC format or another suitable format. Each value from the data frame is stored in the query file according to a sequence and labeled according to its unified category identifier, which may be determined according to the sequence. The entire file may also store data identifying the test cycle or corresponding period, and the data source (e.g., device) of the original raw data file. The file is then output, and the process ends.

[0079] Note that a given value in the raw file may be represented as both an individual stored value in the generated query file and an aggregate of counter values ​​in the aggregate counter file as a result of this process.

[0080] 2 may be repeated for each individual raw data file, using the same aggregated counter file each time. Thus, each iteration of the process flow may end with the generation of another query file corresponding to the raw data file and the appropriate addition of the data in the raw data file to the existing counter values ​​in the aggregated counter file. Note that the resulting query file and aggregated counter file may be referred to as "performance monitoring tool data" as previously described in this disclosure.

[0081] 3 is a flow diagram illustrating a process flow for validating a query data file according to an exemplary embodiment. This exemplary flow is suitable for validation operation S140 of FIG. 1A, although other validation operations are within the scope of the present invention.

[0082] At S310, the number of category identifiers in the data file is checked against a predetermined number of unified category identifiers. If there is a mismatch ("No" at S310), flow proceeds to S370 where the process outputs a "Fail" status and then terminates. If the two numbers match ("Yes" at S310), flow proceeds to S320.

[0083] At S320, each unified category identifier in the file is compared to its order in the sequence. For example, if an identifier does not correspond to a sequence number defined in the storage configuration file used at S245 ("No" at S310), the flow proceeds to S370 where the process outputs a "failure" status and then terminates. If all identifiers correspond to sequence numbers ("Yes" at S320), the flow proceeds to S330.

[0084] The total number of category identifiers can become so large that it becomes impractical for storage, analysis, or both. Therefore, in S330, a check is made to see if the number of category identifiers exceeds a predefined threshold N. If not ("No" in S330), flow proceeds to S360. However, if the number of category identifiers exceeds N ("Yes" in S330), flow proceeds to S340.

[0085] At S340, because the number of category identifiers exceeds N, the file is divided into partitions, each containing N or fewer category identifiers. For example, in many file formats, including the ORC format, this corresponds to dividing the file into N columns of data. Then, at S350, each individual partition is converted into a separate file. Flow then proceeds to block S360.

[0086] At S360, the process outputs one or more files and a "valid" status, and then terminates. The validated file(s) may now be stored in a query database for later retrieval.

[0087] Before discussing the search operation S170 in detail, it may be useful to discuss the architecture of the subsystems that perform the search operation. Figure 4A is a block diagram illustrating a system for processing data search queries, according to an example embodiment.

[0088] The query system 40 may be organized as a distributed cluster, such as a Trino cluster. The distributed cluster includes a coordinator unit 41 and a set of workers 45, each of which may be a computing device having at least one computer processor and a communication module for connecting to other devices in the cluster. The coordinator unit 41 includes a parser 42, a planner 43, and a scheduler 44, each of which may be a software module executing on the processor of the coordinator unit 41. The entire query system is communicatively coupled to a file storage 30, which includes the query database populated in operation S150 of FIG. 1A. A "meta-storage" unit 48 stores a directory of the file storage 30 in a meta-information database 49.

[0089] A user query, including a set of parameters, is received by the coordinator unit 41 from a user or client 20. A parser 42 parses the query according to its structure to determine the meaning of the query in computer instructions. A planner 43 determines a search strategy for dividing the query load into subtasks. A scheduler 44 requests address ranges of specific data via a metastorage unit 48, then schedules and distributes the subtasks among worker units 45, which each perform a search on the file storage 30. By working simultaneously, each worker can manage a part of the search, improving query speed.

[0090] Once the worker 45 provides the complete query results to the parser 42, the parser can return those results to the user.

[0091] 4B is a flow diagram illustrating a process flow for processing a data search query according to an exemplary embodiment. This exemplary flow is suitable for search operation S160 of FIG. 1B, although other query operations are within the scope of the present invention.

[0092] At S410, cell-level data is retrieved from the query database according to the query parameters. This retrieval may be performed, for example, by the query architecture described with reference to FIG. 4A. The query defines, among other factors, a time frame of the data, which may be defined with respect to a corresponding test cycle. Both a counter file for the time frame and a query file are retrieved. As previously mentioned, these files may be referred to as "performance monitoring tool data."

[0093] At S420, site configuration tool data is retrieved from a database according to the same time frame and other query factors. This database may be the query database or a separate database associated with the site configuration tool. S420 may occur simultaneously with S410.

[0094] As previously mentioned, site configuration tool data may be recorded less frequently than once per test cycle. Thus, while a site configuration tool data file may be larger than a performance monitoring tool data query file or counter file, fewer site configuration tool data files may be used to represent the same period. Thus, one site configuration tool data file may be retrieved at S420 to cover the same time frame as several query files.

[0095] At S430, an internal join is performed between the retrieved performance monitoring tool data and the site configuration tool data to remove redundant information and generate an analysis data file for each test cycle. For the reasons described above, several query files and counter files can correspond to a single site configuration tool data file, and therefore, data from the single site configuration tool data file can be combined with each of the above-mentioned query files to generate an analysis data file.

[0096] At S440, the analytical data files are aggregated to form analytical data for the selected portion of the network. For example, for measurements resulting from a query file, one or more of an average, minimum, and maximum value for the selected portion of the network may be determined. For example, for counter values ​​resulting from a counter file, the individual values ​​may be summed to obtain a total count for the entire selected portion. If only a single cell is selected, or if the selected portion otherwise includes only one associated data file for each test cycle, operation S440 may be omitted.

[0097] In S450, the data is again aggregated over the selected period of time. If only a single test cycle or equivalent period of time is selected, operation S450 may be omitted.

[0098] At this point, the analytical data is retrieved and ready for KPI calculation, which is described in more detail here.

[0099] 5 is a flow diagram illustrating a process flow for calculating KPIs according to an exemplary embodiment. This exemplary flow is suitable for calculation operation S170 of FIG. 1B, although other calculation operations are within the scope of the present invention.

[0100] For example, data retrieved via query operation S160 of Figure 1B is collected at S510. At S520, one or more KPI calculations are performed on this data.

[0101] The calculation may require "time shifting," i.e., data from multiple specific cycles or groups of cycles. This differs from KPI calculations over a period encompassing multiple test cycles because individual data from a shorter period is preserved for the calculation process. This may be desirable, for example, for comparison of KPI values ​​across cycles. Another reason may be that a given KPI may be calculated only over individual test cycles and then summed or otherwise mathematically combined after the fact.

[0102] Thus, in S530, a check is made to see if this option was selected. If not ("No" in S530), the process simply outputs the results in S580 and terminates. However, if time-shifting was selected ("Yes" in S530), the results are temporarily stored in S540. If it is then determined that more such data sets are needed ("Yes" in S550), the process returns to S510 to repeat the data collection, this time for another data set for a different test cycle or group of cycles. This flow is repeated until all such data sets have been analyzed and all results have been generated ("No" in S550). The results are then merged in S560, top-level calculations are performed on the merged results as needed in S570, and the final results are output in S580.

[0103] It will be appreciated that the above process may be modified to include "portion shifting" in place of, or in addition to, "time shifting," in which data from multiple designated portions of the network are considered.

[0104] These and related processes, as well as other necessary instructions, are preferably encoded as executable instructions on one or more non-transitory computer-readable media, such as a hard disk drive or optical disk, and executed using one or more computer processors in cooperation with an operating system or other suitable means.

[0105] In a software implementation, the software comprises a plurality of computer-executable instructions embodied on a computer system. Prior to being loaded onto a computer system, the software preferably resides as coded information on a suitable non-transitory computer-readable tangible medium, such as a magnetic, optical, or other suitable coded or recorded medium. Particular media may include, but are not limited to, magnetic floppy disks, magnetic tape, CD-ROMs, DVD-ROMs, solid-state disks, or flash memory devices, and in certain embodiments, takes the form of existing data storage (such as "cloud storage") accessible via operably coupled network means (such as the Internet).

[0106] In certain embodiments, a system includes a dedicated processor, or a processing portion of a system on chip (SOC), a portion of a field programmable gate array (FPGA), or other such suitable means, that executes processor instructions to perform the functions described herein or to emulate the specific structures defined herein. Suitable circuits using discrete logic gates, such as, for example, an Application Specific Integrated Circuit (ASIC), a Programmable Logic Array (PLA), or a Field Programmable Gate Array (FPGA), are also developed to perform these functions in certain embodiments.

[0107] 6 is a diagram of example components of a device 600. Device 600 may correspond to either a coordinator unit 41 and / or a worker 45. As shown in FIG. 6, device 600 may include a bus 610, a processor 620, a memory 630, a storage component 640, an input component 650, an output component 660, and a communication interface 670.

[0108] The bus 610 includes components that enable communication between the components of the device 600. The processor 620 may be implemented in hardware, firmware, or a combination of hardware and software. The processor 620 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 620 includes one or more processors that can be programmed to perform functions. The memory 630 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 620.

[0109] The storage component 640 stores information and / or software related to the operation and use of the device 600. For example, the storage component 640 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive. The input component 650 includes components (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone) that enable the device 600 to receive information, such as via user input. Additionally or alternatively, the input component 650 may include sensors (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator) for sensing information. Output components 660 include components that provide output information from device 600 (eg, a display, a speaker, and / or one or more light-emitting diodes (LEDs)).

[0110] The communication interface 670 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 600 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections, etc. The communication interface 670 may enable the device 600 to receive information from another device and / or provide information to another device. For example, the communication interface 670 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0111] Device 600 may perform one or more processes described herein. Device 600 may perform these processes in response to processor 620 executing software instructions stored by a non-transitory computer-readable medium, such as memory 630 and / or storage component 640. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0112] The software instructions may be loaded into memory 630 and / or storage component 640 from another computer-readable medium or from another device via communication interface 670. When executed, the software instructions stored in memory 630 and / or storage component 640 may cause processor 620 to perform one or more processes described herein.

[0113] Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to implement one or more processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0114] The number and arrangement of components shown in Figure 6 are provided as an example. In practice, device 600 may include additional, fewer, different, or differently arranged components than those shown in Figure 6. Additionally or alternatively, a set of components (e.g., one or more components) of device 600 may perform one or more functions that are described as being performed by another set of components of device 600.

[0115] In an embodiment, any one of the operations or processes of Figures 1, 2, 3, 4B, and 5 may be implemented by or using any one of the elements shown in Figure 6.

[0116] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of implementations.

[0117] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail. Furthermore, one or more of the above components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include computer-readable non-transitory storage medium(s) having computer-readable program instructions for causing a processor to perform operations.

[0118] A computer-readable storage medium may be any tangible device capable of retaining and storing instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved ridge structures with instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0119] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0120] The computer-readable program code / instructions for carrying out operations may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, integrated circuit configuration data, or object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects or operations.

[0121] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, whereby the computer-readable storage medium having instructions stored therein includes an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0122] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0123] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than those shown in the figures. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be executed concurrently or substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0124] It will be apparent that the systems and / or methods described herein may be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. 1. A method for long-term storage of network performance data for later retrieval, comprising: obtaining a plurality of performance data files corresponding to the test cycle, each of the plurality of performance data files including data describing network performance, the plurality of performance data files including at least a first data file in a first format and a second data file in a second format different from the first format; reformatting each of the plurality of performance data files in accordance with a predetermined uniform file format and a predefined set of uniform category identifiers to obtain a plurality of uniform data files; storing the plurality of unified data files in a query database in memory; A method comprising:

2. Each performance data file is parsing the performance data file according to a format of the performance data file to identify a plurality of data category identifiers within the performance data file, each of the plurality of data category identifiers having a corresponding data value within the performance data file; matching each of the set of predefined uniform category identifiers to a respective one of the plurality of data category identifiers of the performance data file; incrementing a counter value of at least one counter in an aggregate counter file based on a uniform category identifier having a predefined correspondence to the counter and a data value of a data category identifier that matches the uniform category identifier having a predefined correspondence to the counter; storing data values ​​of at least one data category identifier at a sequence position within the temporary data frame based on a sequence position having a predefined correspondence with a unified category identifier that matches the data category identifier; converting the temporary data frames into a unified data file having the predetermined unified file format; The method of claim 1 , wherein storing the plurality of unified data files in the query database comprises storing the aggregate counter file.

3. The method of claim 2 , wherein the matching is based on a predetermined category mapping corresponding to the source of the performance data file.

4. The collation a textual comparison of the uniform category identifier with the data category identifier; comparing the expected format and expected range corresponding to the uniform category identifier with sample data points corresponding to the data category identifier in the performance data file; The method of claim 2, based on

5. 10. The method of claim 1, further comprising validating each of the plurality of unified data files, wherein each of the plurality of unified data files is divided into partitions during validation according to a predefined category identifier threshold.

6. The method of claim 1 , further comprising: storing at least one configuration data file for each of a plurality of network cells, wherein the frequency of storage of the configuration data files is less than the frequency of the test cycles.

7. 1. A method for analyzing historical network performance, comprising: Storing a plurality of unified data files in a query database in memory according to the method of claim 1; searching the query database according to the received search request, thereby obtaining a set of retrieved unified data files; calculating at least one performance indicator based on the retrieved set of unified data files; A method comprising:

8. the received search request defines a selected time frame; each of the set of retrieved unified data files corresponds to a test cycle within the selected time frame; aggregating the values ​​in the retrieved set of unified data files to determine a representative value for the selected time frame; the at least one performance indicator is calculated based on the representative values ​​for the selected time frame; The method of claim 7.

9. the received search request defines a selected network portion; each of the set of retrieved unified data files corresponds to a cell within the selected network portion; aggregating the values ​​in the retrieved set of unified data files to determine a representative value for the selected network portion; the at least one performance indicator is calculated based on the representative values ​​of the selected network portions; The method of claim 7.

10. 1. A method for analyzing historical network performance, comprising: Storing a plurality of unified data files in a query database in memory according to the method of claim 6; searching the query database according to a received search request, thereby obtaining a set of retrieved unified data files and a set of retrieved configuration data files, wherein the received search request defines a selected time frame, and each of the set of retrieved unified data files and each of the set of retrieved configuration data files corresponds to the selected time frame; merging data from the set of retrieved unified data files with data from the set of retrieved configuration data files to generate a set of analysis data files; calculating at least one performance indicator based on the set of analytical data files; A method comprising:

11. 1. A system for long-term storage of network performance data for later retrieval, comprising: at least one non-volatile memory electrically configured to store computer program code; at least one processor operatively connected to said at least one non-volatile memory, said at least one processor configured to operate as instructed by said computer program code, said computer program code comprising: file retrieval code configured to cause at least one of the at least one processor to obtain a plurality of performance data files corresponding to a test cycle, each of the plurality of performance data files including data describing network performance, the plurality of performance data files including at least a first data file in a first format and a second data file in a second format different from the first format; formatting code configured to cause at least one of the at least one processor to reformat each of the plurality of performance data files in accordance with a predetermined uniform file format and a predefined set of uniform category identifiers to obtain a plurality of unified data files; storage code configured to cause at least one of the at least one processor to store the plurality of unified data files in a query database in a query memory; Including, the system.

12. Each performance data file is parsing the performance data file according to a format of the performance data file to identify a plurality of data category identifiers within the performance data file, each of the plurality of data category identifiers having a corresponding data value within the performance data file; matching each of the set of predefined uniform category identifiers to a respective one of the plurality of data category identifiers of the performance data file; incrementing a counter value of at least one counter in an aggregate counter file based on a uniform category identifier having a predefined correspondence to the counter and a data value of a data category identifier that matches the uniform category identifier having a predefined correspondence to the counter; storing data values ​​of at least one data category identifier at a sequence position within the temporary data frame based on a sequence position having a predefined correspondence with a unified category identifier that matches the data category identifier; converting the temporary data frames into a unified data file having the predetermined unified file format; Reformatted by the storage code is further configured to cause at least one of the at least one processor to store the aggregate counter file. The system of claim 11.

13. The system of claim 12 , wherein the matching is based on a predetermined category mapping corresponding to the source of the performance data file.

14. The collation a textual comparison of the uniform category identifier with the data category identifier; comparing the expected format and expected range corresponding to the uniform category identifier with sample data points corresponding to the data category identifier in the performance data file; The system of claim 12, based on

15. 12. The system of claim 11, wherein the computer program code further comprises validation code configured to cause at least one of the at least one processor to validate each of the plurality of unified data files, wherein each of the plurality of unified data files is divided into partitions during validation according to a predefined category identifier threshold.

16. 12. The system of claim 11, wherein the storage code is further configured to cause at least one of the at least one processor to store at least one configuration data file for each of a plurality of network cells, wherein storage of the configuration data files is less frequent than a frequency of a test cycle.

17. 1. A non-transitory computer-readable storage medium having stored thereon instructions executable by at least one processor for performing a method for long-term storage of network performance data for later retrieval, the method comprising: obtaining a plurality of performance data files corresponding to the test cycle, each of the plurality of performance data files including data describing network performance, the plurality of performance data files including at least a first data file in a first format and a second data file in a second format different from the first format; reformatting each of the plurality of performance data files in accordance with a predetermined uniform file format and a predefined set of uniform category identifiers to obtain a plurality of uniform data files; storing the plurality of unified data files in a query database in memory; A non-transitory computer-readable recording medium comprising:

18. Each performance data file is parsing the performance data file according to a format of the performance data file to identify a plurality of data category identifiers within the performance data file, each of the plurality of data category identifiers having a corresponding data value within the performance data file; matching each of the set of predefined uniform category identifiers to a respective one of the plurality of data category identifiers of the performance data file; incrementing a counter value of at least one counter in an aggregate counter file based on a uniform category identifier having a predefined correspondence to the counter and a data value of a data category identifier that matches the uniform category identifier having a predefined correspondence to the counter; storing data values ​​of at least one data category identifier at a sequence position within the temporary data frame based on a sequence position having a predefined correspondence with a unified category identifier that matches the data category identifier; converting the temporary data frames into a unified data file having the predetermined unified file format; 20. The storage medium of claim 17, wherein storing the plurality of unified data files in the query database includes storing the aggregate counter file.

19. 20. The storage medium of claim 17, wherein the method further comprises validating each of the plurality of unified data files, wherein each of the plurality of unified data files is divided into partitions during validation according to a predefined category identifier threshold.

20. 20. The storage medium of claim 17, wherein the method further comprises storing at least one configuration data file for each of a plurality of network cells, wherein the storage of the configuration data files is less frequent than the frequency of the test cycles.

Images