Systems and methods for long-term compilation and retrieval of historical data in network performance analysis.

Abstract

To store network performance data for later retrieval, performance data files corresponding to a test cycle are retrieved, reformatted into unified data files according to a predetermined unified file format and a set of predefined unified category identifiers, and stored in a query database. The reformatting can match predefined unified category identifiers with data category identifiers in the performance data files, and the corresponding data values ​​can be stored sequentially in the unified data files or used to increment counter values ​​in similarly stored aggregated counter files. In addition, configuration data files for network cells can be stored less frequently than the frequency of the test cycles. Later retrieval can retrieve both the unified data files and the configuration data files corresponding to a selected time frame and network portion, and merge or aggregate the data appropriately for performance indicator calculations.

Description

Background Art

[0001] 1. Field Apparatuses and methods consistent with exemplary embodiments relate to network performance analysis, and more particularly, to efficient long - term storage, compilation, and retrieval of data for analysis of past and / or time - varying network performance.

[0002] 2. Description of Related Art In large - scale networks such as mobile networks, it is not practical to physically inspect all components. Instead, problematic components are identified by the impact they have on network behavior. Computing devices connected to the network, including both base - unit systems and mobile devices, can monitor network behavior and generate data logs. These logs can then be analyzed to identify where the network is not functioning as intended.

[0003] The intended performance level is typically defined according to various quantifiable metrics, typically referred to in the art as "primary performance indicators" or "key performance indicators: KPIs". If the measured or calculated value of a KPI does not reach the intended level, further investigation is required to identify the cause of the problem. KPIs are more useful for evaluating performance improvements resulting from system upgrades and generally for monitoring and predicting network development.

Summary of the Invention

[0004] One objective of the disclosed systems and methods is to store network - performance data so that it can be easily retrieved based on a selected past time frame.

[0005] Another purpose of the disclosed systems and methods is to reduce redundant data in stored network performance data.

[0006] Another further purpose of the disclosed system and method is to automatically convert data from multiple different sources into a unified data file for retrieval.

[0007] According to certain embodiments of the present disclosure, a method is provided for long-term storage of network performance data for later retrieval. The method includes obtaining a plurality of performance data files corresponding to a test cycle. Each of the plurality of performance data files contains data describing network performance. The plurality of performance data files includes at least a first data file in a first format and a second data file in a second format different from the first format. The method further includes reformatting each of the plurality of performance data files according to a predetermined unified file format and a predefined set of unified category identifiers in order to obtain a plurality of unified data files. The method further includes storing the plurality of unified data files in an in-memory query database.

[0008] Other embodiments of the present disclosure provide a method for analyzing historical network performance. The method includes storing a plurality of unified data files in an in-memory query database. The method further includes searching the query database in accordance with an incoming search request and obtaining a set of unified data files retrieved therefrom. The method further includes calculating at least one performance indicator based on the retrieved set of unified data files.

[0009] Further embodiments of the present disclosure provide a system for long-term storage of network performance data for later retrieval. The system includes at least one non-volatile memory electrically configured to store computer program code. The system further includes at least one processor operably connected to the non-volatile memory, configured to operate as instructed by the computer program code. The computer program code includes file retrieval code configured to cause at least one of the at least one processor to retrieve a plurality of performance data files corresponding to a test cycle. Each of the plurality of performance data files contains data describing network performance. The plurality of performance data files include at least a first data file in a first format and a second data file in a second format different from the first format. The computer program code further includes formatting code configured to cause at least one of the at least one processor to reformat each of the plurality of performance data files according to a predetermined unified file format and a predefined set of unified category identifiers in order to retrieve the plurality of unified data files. The computer program code further includes storage code configured to cause at least one of at least one processors to store multiple unified data files in a query database within query memory.

[0010] Further embodiments of the present disclosure relate to a non-temporary computer-readable recording medium recording instructions executable by at least one processor for performing a method for long-term storage of network performance data for later retrieval. The method comprises obtaining a plurality of performance data files corresponding to a test cycle. Each of the plurality of performance data files contains data describing network performance. The plurality of performance data files comprises at least a first data file in a first format and a second data file in a second format different from the first format. The method further comprises reformatting each of the plurality of performance data files according to a predetermined unified file format and a predefined set of unified category identifiers in order to obtain a plurality of unified data files. The method further comprises storing the plurality of unified data files in a query database in memory.

[0011] Additional embodiments are some of which are described below, some of which are evident from the description, and some may be realized through the practice of the embodiments presented in this disclosure. [Brief explanation of the drawing]

[0012] Features, aspects, and advantages of certain exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which similar reference numerals indicate similar elements.

[0013] [Figure 1A] This flowchart illustrates the process flow for efficient storage of historical KPI calculation data, according to an exemplary embodiment.

[0014] [Figure 1B] This flowchart illustrates the process flow for high-speed historical KPI calculation using an exemplary embodiment.

[0015] [Figure 2]A flowchart showing a process for reformatting raw data files according to an exemplary embodiment.

[0016] [Figure 3] A flowchart showing a process for verifying the validity of a reformatted data file according to an exemplary embodiment.

[0017] [Figure 4A] A block diagram showing a system for processing data search queries according to an exemplary embodiment.

[0018] [Figure 4B] A flowchart showing a process for processing data search queries according to an exemplary embodiment.

[0019] [Figure 5] A flowchart showing a process for calculating KPIs according to an exemplary embodiment.

[0020] [Figure 6] A diagram of exemplary components of a device in which embodiments of the systems and / or methods described herein may be implemented.

Best Mode for Carrying Out the Invention

[0021] The following detailed description of exemplary embodiments refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Hereinafter, specific exemplary embodiments for sample applications will be described with reference to the figures illustratively shown in the drawings to explain the disclosed systems and methods.

[0022] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementation to the exact forms disclosed. Modifications and variations are possible in light of the above disclosure, or may be obtained from practice of the implementation forms. Further, one or more features or components of one embodiment may be incorporated into or combined with another embodiment (or one or more features of another embodiment). Further, in the flowcharts and descriptions of operations provided below, it will be understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be executed simultaneously (at least partially), or the order of one or more operations may be exchanged.

[0023] It will be apparent that the systems and / or methods described herein may be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limiting of the implementation forms. Thus, the operations and behaviors of the systems and / or methods are described herein without reference to specific software code, and it is understood that the software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0024] Although specific combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed herein. Each of the dependent claims listed below may directly depend on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0025] Any elements, actions, or instructions used herein should not be construed as important or essential unless expressly stated otherwise. Furthermore, where used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” When only one item is intended, the term “one” or similar language should be used. Also, where used herein, terms such as “has,” “have,” “having,” “include,” and “including” are intended to be non-restrictive. Additionally, the phrase “based on” should mean “at least partially based on” unless otherwise specified. Furthermore, expressions such as “at least one of [A] and [B]” or “at least one of [A] or [B]” should be understood to include only A, only B, or both A and B.

[0026] It should be noted that the principles disclosed herein are generally applicable to all forms of networks, including but not limited to Internet service provider networks such as fiber optic and cable networks, conventional telephone networks, structures, complexes, or both wired and wireless networks in other localized areas, as well as non-communication networks such as power grids. However, throughout this disclosure, the networks analyzed and managed by the disclosed systems will be primarily referred to as mobile networks for convenience and brevity.

[0027] As briefly explained in "Background Technology," regular reviews of key performance indicators (KPIs) in a network are a desirable part of quality assurance testing for large-scale networks. These reviews determine whether KPI values ​​meet baseline or target thresholds under normal circumstances or after upgrades, and monitor trends in these values ​​over time to identify areas where future development may be needed.

[0028] KPIs can be calculated for the entire network or any part thereof. A portion of the network can be defined as one or more cells, each defined by a specific cellular tower or other transceiver to which devices considered to be "inside" the cell are coupled. Cells do not have defined physical boundaries such that devices crossing such physical boundaries consistently disconnect from the cell's transceiver and connect to the transceiver of an adjacent cell. However, when selecting a portion of the network according to an area on a map of physical regions, the cells "inside" the selected portion can be defined according to the transceivers physically located within the corresponding selected area. This style of partial selection is usually done by "drawing" polygonal shapes on a digital map representation, so the resulting selected portion of the network is sometimes called a selected "polygon." Polygons may also be defined on diagrams that visually represent the network according to something other than physical areas, such as a chart of transceiver interconnections or a hierarchical chart. Furthermore, network components may also be defined and selected according to other criteria, such as all cells managed by or via a specific central network unit or hub, all cells operating under a specific technical standard (e.g., 4G, 5G) or operating system. Testing a component defined according to each of these criteria may be equivalent to testing the common characteristics of that component.

[0029] As briefly explained in "Background Technology," the data used to measure or calculate KPI values ​​(referred to herein as "KPI calculation" for brevity) can be collected periodically by various network-connected computing systems. More specifically, the data may be values ​​for various parameters, such as, for example, identifier values ​​including but not limited to the device model and the communication protocol it uses; measured values ​​including but not limited to ping time and bandwidth; and counter values ​​including but not limited to the number of connections released or specific function calls within a specified period (sometimes simply called "counters").

[0030] Mobile network data logs can be enormous, and it's impractical to retain them in the local memory of an analytics system for extended periods, especially if new data keeps arriving. Furthermore, the most recent data is usually the most relevant to KPI calculations. Therefore, when new data (which could be called "current data") is retrieved and requires storage space, older data (which could be called "historical data" or "past data") can be moved to medium- or long-term storage memory in a database, such as an EdgeDB® database, or in another searchable format, so that it is retrieved only when needed. Many data analytics tools suitable as the basis for KPI calculations, such as Apache® Spark, include the ability to store and retrieve data in this way.

[0031] However, this method complicates the calculation of KPIs related to past network behavior.

[0032] As an illustrative example, it may be desirable to perform a post-event analysis on a network event that occurred three days prior. To calculate KPIs for the event, particularly those not calculated as part of a standard test cycle (and therefore not calculated at the time of the event), but relevant due to the nature of the event, a search can be performed on long-term storage memory to retrieve historical data from its timeframe—that is, from one or more test cycles that occurred at the time of the event, and, where relevant, from test cycles immediately preceding or following the event. However, the amount of historical data for a single test cycle can be quite substantial. Furthermore, the sheer volume of historical data in the entire database can make even searching for a single data element excessively thorough. Using traditional database searches, such as those provided by Spark SQL, retrieving all historical data related to an event from three days prior can take an enormous amount of time. It may take several hours to gather everything necessary to calculate applicable KPIs. If KPIs are calculated for a specific network segment, the search will be longer, even with less data retrieved, because it must consider data in the database according to additional parameters beyond timestamps or test cycle labels to determine which data to retrieve.

[0033] This problem is amplified when retrieving historical data from multiple test cycles. As an illustrative example, it may be desirable to introduce a new KPI into a standard analysis. Ideally, a baseline threshold for that KPI, representing a value indicating that the network is functioning correctly, should be established simultaneously. Goals may also be developed, often based on the baseline. However, developing such a baseline often requires calculating multiple historical values ​​of that KPI, each from a different test cycle. Each such historical value requires its own retrieval set of relevant historical data from the selected test cycle before calculation can be performed. The scale of such a retrieval process, in terms of processing time and resources, means that it is practically possible to start without a baseline and develop one after several cycles of standard testing of the KPI, or to develop an initial baseline based on only one or two historical test cycles and adjust it later. While this is not ideal, as it is better to have an accurate baseline as early as possible, these are the only practical solutions when using conventional systems.

[0034] In short, exemplary embodiments of this disclosure provide methods and systems for collecting historical data from other sources as an alternative to, or in addition to, long-term storage database solutions. More specifically, embodiments of the disclosed methods and systems leverage and improve performance monitoring tools to compile and retrieve, over time, a simplified version of historical data logs that can be formatted more efficiently for reading specific parameters, at the expense of integrity, size efficiency, or other factors that are less critical when rapid access is required.

[0035] In a more specific embodiment, the improved performance monitoring tool may be configured to copy recently collected log elements from short-term data storage and store these elements in a more efficient format that is easier to query and retrieve.

[0036] Furthermore, the improved site configuration tool may be configured to store a history of configuration data for each transceiver and its corresponding cell, including but not limited to cell name, location, beam azimuth and inclination, frequency, bandwidth, operating system, and technical standard (e.g., 4G, 5G). While the values ​​of this data may change over time, they are not expected to change in each test cycle; therefore, the site configuration tool can store a history of this data for periods during which any particular value remains stable, which is more efficient than storing different data points for each test cycle. For example, the site configuration tool may store a data log only once a day, in contrast to test cycles every 15 minutes. Thus, data already stored by the site configuration tool can be omitted from the performance monitoring tool data, reducing storage usage by the performance monitoring tool data and facilitating its retrieval. The relevant portions of the site configuration tool data can then be compiled together with the performance monitoring tool data according to the specific time frame queried at the time of query. Site configuration tool data is not necessarily used in the KPI calculation itself; rather, performance monitoring tool data may be used to correlate specific cells and their corresponding configuration features, thereby allowing the calculation to be performed according to a specific polygon or other defined part of the network.

[0037] Using the data stored by these tools, KPI calculations for older time frames can be performed within a reasonable timeframe, according to the disclosed processes and their variations.

[0038] In the following, KPIs calculated using either current data or historical data in addition to current data will be referred to as "historical KPIs."

[0039] Figure 1A is a flowchart illustrating the process flow for efficient storage of historical KPI calculations, according to an exemplary embodiment.

[0040] In S110, the element management system server (EMS server) retrieves the raw data file corresponding to the current test cycle. The raw data file may be briefly stored in the EMS server to make it available for determining the current KPI before being moved to a medium-term storage database in another memory location to clear its storage space in memory for new data. Although not detailed herein, the determination of the current KPI may be performed by the processor of the EMS server or by another system.

[0041] A suitable storage duration for both this purpose and the simplified storage process described below is 12 hours, but this is just one example. The ideal duration for short-term storage on an EMS server may depend on other factors, including the speed and storage capacity of the processor and memory used, as well as the rate of incoming raw data.

[0042] In S120, each raw file is stored in a database in memory. This memory can be organized according to any standard suitable for unstructured data, including but not limited to MinIO® and Hadoop® distributed file systems. Unstructured data storage may be necessary because raw files do not yet necessarily share a unified file format, and each can be formatted according to the standards and preferences of its respective device vendor. If the EMS server is sufficiently efficient, it can store the raw files in its own memory and perform subsequent operations on its own processor, although a separate system for this storage is also within the scope of this disclosure.

[0043] The data in each raw file is reformatted in S130 according to a unified file format and a set of unified category identifiers, in a manner further described herein. The resulting files are then validated in S140 to confirm that the formatting was successful, in a manner further described herein. Finally, in S150, the validated files are stored in a query database implemented on query memory. The query database may be the same database used in S120, a different database on the same memory, or a different database on different memory.

[0044] If a database different from the query database is used in S120, the database used in S120 may be called a temporary database. Similarly, if a memory different from query memory is used in S120, the memory used in S120 may be called temporary memory.

[0045] If the raw file stored in S120 is already in the desired common file format, operation S130 may still be performed to generate a new file or to modify an existing file to contain only data of a specific predetermined data category.

[0046] Figure 1B is a flowchart illustrating the process flow for high-speed historical KPI calculation according to an exemplary embodiment.

[0047] In S160, the query database retrieves data according to the received request to identify data that (a) is included in the defined search parameters and (b) is related to the selected KPI to be calculated. The search query may be applied to the data file by an appropriate query engine. Effective architectures and processes for queries applied to this form of data are described further herein.

[0048] In S170, historical KPIs are calculated using the data retrieved according to the query, and in S180, the results are output. The process then terminates.

[0049] As mentioned above, the raw files recovered in S110 may be in different file formats. Suitable formats for this type of data include, among others, Extensible Markup Language (XML) and XML-like formats (collectively referred to herein as "parsed structure files"), as well as comma-separated value (CSV) files and Optimized Row Columnar (ORC) files.

[0050] The format of files before KPI calculation is generally preferred to be a single, unified format. Therefore, reformatting raw data files to a unified file may be part of the storage process. Furthermore, reformatting can filter and exclude unnecessary categories of data so that only the data necessary for past KPI calculations remains. Data in desired categories can be defined according to a predefined set of unified category identifiers, which can be used to define the framework of the unified file's content.

[0051] Tests have shown that the ORC format is particularly effective for many of the computational operations described herein in terms of both read time during queries and storage size within the query database. In particular, the ORC format allows for the rapid location and retrieval of specific “stripes” of data from each data file rather than the entire file when a query requests data from a particular category. However, it has been found that “counter data” is more efficient when stored in a parsed structure file. Thus, certain embodiments of this disclosure may use a hybrid approach to the data format.

[0052] Figure 2 is a flowchart illustrating the process flow for reformatting a raw data file according to an exemplary embodiment. This exemplary flow is suitable as the formatting operation S130 in Figure 1A, but other formatting operations are also within the scope of the present invention.

[0053] In S210, the raw data file is parsed to identify all data category identifiers according to the file's existing format. As two illustrative examples, a CSV file places its category identifiers on the first line of the file, while an XML file or other parsed structured file uses element names as categories. A suitable algorithm for parsing such category identifiers can be prepared for each expected file format.

[0054] In S220, according to the mapping, it is checked whether one of the selected unified category identifiers matches any of the category identifiers in the raw file. If so (yes in S220), the flow proceeds to S230. Otherwise (no in S220), the flow proceeds to S225.

[0055] Matching may also be performed according to a predetermined category mapping of the raw file source, where each unified category identifier in a predetermined set may be mapped to one of the expected data category identifiers of the raw file source. For example, if the raw file source is a vendor, the definition of the raw file categories may be available from the vendor in an inventory file. This inventory can be used before the execution of the current flow of the process to create appropriate category mappings from each relevant category to the corresponding unified category identifiers. The category mapping of a given source may also be prepared in advance by other means, and may include, if necessary, a review by an exemplary file direct manager to determine or intuitively understand which unified category identifiers (if any) correspond to a given raw file category identifier.

[0056] However, the source may be newly encountered and otherwise lack existing category mappings. Furthermore, the source may have modified the category identifiers of those raw files. Therefore, unless the mapping in S220 can match the selected unified category identifier with an identifier in the raw file, in S225 a comparison of the unified category identifier with each raw file category identifier in the raw file can be performed to see if a best match can be identified. This can be done by a suitable text comparison algorithm from the unified category identifier to each raw file category identifier. It is also possible to check one or more sample data points corresponding to the raw file category identifiers to determine whether they are in the expected format. For example, it is possible to determine whether there is a recognizable number of data corresponding to a suspicious "transmission frequency" category identifier and whether that number is within the range used for mobile phone transmission.

[0057] If comparison S225 is configured to output both the best match and the likelihood of a match, a threshold likelihood may be applied. This threshold may be predetermined according to any appropriate system requirements, in particular, the acceptability or unacceptability of misclassified data, which may differ between unified categories. Best matches that do not meet the threshold likelihood may be discarded, resulting in a "no match" result.

[0058] If comparison S225 produces a result of "No Match" (i.e., "No" in S225), it can be assumed that the selected unified category identifier does not exist in the raw data file. Therefore, the flow can proceed to S240 with the value of the selected unified category identifier set to an appropriate null value.

[0059] If comparison S225 matches ("Yes" in S225), the flow proceeds to S230. Optionally, before proceeding to S230, existing category mappings can be automatically updated or new category mappings can be generated based on this determined mapping data and other determined mapping data.

[0060] Note that in implementations where category mapping is not available, operation S220 may be omitted, and the process flow can proceed directly from S210 to S225.

[0061] If the process proceeds to S230, the data corresponding to the selected unified category identifier may be identified in the raw file. This data is reformatted as needed to be placed in at least one of the two files according to the selected unified category identifier. That is, the category may be used as a direct value, reflected in one or more counter values, or preferably both.

[0062] Therefore, in S230, it is determined whether counter values ​​corresponding to the selected unified category identifiers and their values ​​in the raw data file are predefined. This can be determined according to a table, mapping, or other appropriate counter configuration file for defining such correspondences. The values ​​in this definition may be a single value, a range of values, or an enumerated set of values. Alternatively, these values ​​may be omitted from consideration so that only the selected unified category identifiers are important.

[0063] If there is no correspondence ("No" in S230), the flow proceeds to S240. If there is a corresponding counter value ("Yes" in S230), the flow proceeds to S235.

[0064] In S235, assume that an aggregated counter file for the test cycle exists. If it does not exist, it may be generated as part of operation S235 or before the first iteration of operation S235. The generated counter file contains data representing each of a predetermined set of counter values, each of which may be initialized to 0. The counter file may also store data identifying the test cycle or the corresponding period. The counter file may be an XML file or another parsed structure file, but the present invention is not limited thereto.

[0065] In S235, each counter value corresponding to the selected unified category identifier and its value in the raw data file may be incremented or otherwise increased in the counter file.

[0066] For example, a raw data file obtained from a particular device may have a category that matches the "device technology" unified category identifier. The value of this category may be a text value, and in a particular raw file, it may be set to "5G," which can be understood as indicating that the device is operating with "5G" (fifth-generation standard) technology. The "5G device count" counter may be defined to correspond to the "device technology" value of "5G." Thus, in S235, the "5G device count" counter may be incremented by 1 to indicate that the system has counted the occurrence of this value in one of the raw data files. As subsequent raw data files are processed similarly, it will be clear that the "5G device count" counter will continue to increase according to the number of raw data files that show the "5G" value.

[0067] Alternatively, specific combinations of selected unified category identifiers and values ​​can be counted by adding the values ​​to the corresponding counters. For example, a raw data file retrieved from a particular device might have a value of "100" for the category matching the "Megabytes Downloaded" unified category identifier, which can be understood as indicating that 100 megabytes of data have been downloaded to the device since the last test cycle. The "Total Download Throughput" counter can be defined to correspond to any non-zero value within the "Megabytes Downloaded" category. Thus, this value "100" is added to the existing value of the "Total Download Throughput" counter. As subsequent raw data files are processed similarly, it will be clear that the "Total Download Throughput" counter will continue to increase according to the individual "Megabytes Downloaded" values ​​for each individual device.

[0068] Whether a specific counter value should be incremented or the total value should be added may be indicated as part of the counter configuration file.

[0069] Note that a given combination of a unified category identifier and a value can correspond to two or more counter values. Furthermore, note that as a result of such correspondence, a value is added to one counter value, but this can trigger an increment in another counter value.

[0070] If all raw data files in a test cycle are processed in this manner, it will be clear that the counter values ​​in the resulting counter files will reflect the activity throughout the test cycle, as represented in the raw data files within the aggregate.

[0071] After S235, the flow continues to S240.

[0072] In S240, it is determined whether to directly store the value of the selected unified category identifier. This may be determined according to a table, mapping, or other appropriate storage configuration file that represents it, which may be the same file as the counter configuration file or a different file. This determination does not need to consider the category value, but using the value as a factor in the determination is within the scope of this disclosure.

[0073] As mentioned above, if the process flow reaches this point via the "No in S225" branch, the value in question is not from the raw data file, but rather the appropriate "null" value.

[0074] If the value is not to be remembered ("No" in S240), the flow proceeds to S250. If the value is to be remembered ("Yes" in S240), the flow proceeds to S245.

[0075] In S245, values ​​are arranged according to a predefined sequence within a temporary storage file, which may be called a "data frame" corresponding to the raw data file. The arrangement within the sequence can be based on a unified category identifier, and the sequence can be defined within a storage configuration file.

[0076] After S245, the flow continues to S250.

[0077] In S250, it is determined whether there is a unified category identifier in the predetermined set that has not yet been selected. If so (yes in S250), the flow returns to S220, selects another unified category identifier, and performs another iteration of the loop from S220 to S250. Otherwise (no in S250), the flow proceeds to S260.

[0078] In S260, the data within the data frame is converted into a query file, which can have ORC format or another suitable format. Each value from the data frame is stored in the query file according to its sequence and labeled according to its unified category identifier, which may be determined according to the sequence. The entire file can also store data identifying the test cycle or corresponding period, and the data source (e.g., device) of the original raw data file. The file is then output, and the process terminates.

[0079] It should be noted that a given value in a raw file can be represented both as an aggregate of individual stored values ​​in the generated query file and as an aggregate of counter values ​​in the aggregated counter file, as a result of this process.

[0080] The process flow shown in Figure 2 may be repeated for each individual raw data file, using the same aggregated counter file each time. Thus, each iteration of the process flow can end with the generation of another query file corresponding to the raw data file and the appropriate addition of the data in the raw data file to the existing counter value in the aggregated counter file. Note that the resulting query file and aggregated counter file may be referred to as “performance monitoring tool data” as previously stated in this disclosure.

[0081] Figure 3 is a flowchart showing the process flow for verifying the validity of a query data file according to an exemplary embodiment. This exemplary flow is suitable as the validity verification operation S140 in Figure 1A, but other validity verification operations are also within the scope of the present invention.

[0082] In S310, the number of category identifiers in the data file is checked against a predetermined number of unified category identifiers. If there is a mismatch ("No" in S310), the flow proceeds to S370, where the process outputs a "Failed" status and then terminates. If the two numbers match ("Yes" in S310), the flow proceeds to S320.

[0083] In S320, each unified category identifier in the file is compared to its order in the sequence. For example, if an identifier does not correspond to a sequence number defined in the storage configuration file used in S245 ("No" in S310), the flow proceeds to S370, where the process outputs a "Failed" status and then terminates. If all identifiers correspond to a sequence number ("Yes" in S320), the flow proceeds to S330.

[0084] The total number of category identifiers can be quite large, making it impractical for storage, analysis, or both. Therefore, in S330, it is checked whether the number of category identifiers exceeds a predefined threshold N. If not ("No" in S330), the flow proceeds to S360. However, if the number of category identifiers exceeds N ("Yes" in S330), the flow proceeds to S340.

[0085] In S340, since the number of category identifiers exceeds N, the file is divided into partitions, each containing N or fewer category identifiers. For example, in many file formats, including the ORC format, this is equivalent to dividing the file into N columns of data. Next, in S350, each individual partition is converted into a separate file. The flow then proceeds to block S360.

[0086] In S360, the process outputs one or more files and a "valid" status, and then terminates. The file(s) whose validity has been confirmed may be stored in the query database for later retrieval.

[0087] Before discussing the search operation S170 in detail, it may be useful to discuss the architecture of the subsystem that performs the search operation. Figure 4A is a block diagram showing a system for processing data search queries according to an exemplary embodiment.

[0088] The query system 40 may be organized as a distributed cluster, such as a Trino cluster. The distributed cluster includes a coordinator unit 41 and a set of workers 45, each of which may be a computing device having at least one computer processor and a communication module for connecting to other devices in the cluster. The coordinator unit 41 includes a parser 42, a planner 43, and a scheduler 44, each of which may be a software module running on the processor of the coordinator unit 41. The entire query system is communicatively coupled to a file storage 30 containing the query database captured in operation S150 of Figure 1A. A "meta-storage" unit 48 stores the directory of the file storage 30 in a metadata database 49.

[0089] A user query containing a set of parameters is received by the coordinator unit 41 from the user or client 20. The parser 42 parses the query according to its structure to determine the meaning of the query in computer instructions. The planner 43 determines a search strategy to divide the query load into subtasks. The scheduler 44 requests an address range of specific data via the metastorage unit 48, and then schedules and distributes the subtasks among the worker units 45, each worker 45 performing the search on the file storage 30. By operating simultaneously, each worker can manage a portion of the search and improve query speed.

[0090] Once worker 45 provides parser 42 with the complete query results, the parser can return those results to the user.

[0091] Figure 4B is a flowchart illustrating the process flow for processing a data retrieval query according to an exemplary embodiment. This exemplary flow is suitable as the retrieval operation S160 in Figure 1B, but other query operations are also within the scope of the present invention.

[0092] In S410, cell-level data is retrieved from the query database according to the query parameters. This retrieval can be performed, for example, by the query architecture described with reference to Figure 4A. The query defines a time frame for the data, which may be defined with respect to the corresponding test cycle, among other factors. Both the time frame counter file and the query file are retrieved. As mentioned earlier, these files may be called "performance monitoring tool data".

[0093] In S420, site configuration tool data is retrieved from the database according to the same time frame and other query factors. This database may be the query database or a separate database related to the site configuration tool. S420 may occur simultaneously with S410.

[0094] As mentioned earlier, site configuration tool data may be recorded less than once per test cycle. Therefore, the site configuration tool data file may be larger than the query or counter file for performance monitoring tool data, but fewer site configuration tool data files may be used to represent the same period. Thus, one site configuration tool data file may be retrieved in S420 to cover the same time frame as several query files.

[0095] In S430, an internal join is performed between the recovered performance monitoring tool data and site configuration tool data, redundant information is removed, and an analysis data file is generated for each test cycle. For the reasons mentioned above, several query files and counter files can correspond to a single site configuration tool data file, and therefore, data from a single site configuration tool data file can be joined with each of the aforementioned query files to generate an analysis data file.

[0096] In S440, the analysis data files are aggregated to form the analysis data for the selected portion of the network. For example, in the case of measurements resulting from a query file, one or more of the mean, minimum, and maximum values ​​for the selected portion of the network can be determined. For example, in the case of counter values ​​resulting from a counter file, the individual values ​​can be summed to obtain the total count for the entire selected portion. If only a single cell is selected, or if the selected portion otherwise contains only one relevant data file for each test cycle, operation S440 may be omitted.

[0097] In S450, the data is similarly aggregated again over the entire selected period. If only a single test cycle or an equivalent period is selected, operation S450 may be omitted.

[0098] At this point, the analysis data has been retrieved and is ready for KPI calculations, which will be explained in more detail here.

[0099] Figure 5 is a flowchart showing the process flow for calculating KPIs according to an exemplary embodiment. This exemplary flow is suitable as calculation operation S170 in Figure 1B, but other calculation operations are also within the scope of the present invention.

[0100] For example, the data retrieved via the query operation S160 in Figure 1B is collected in S510. In S520, one or more KPI calculations are performed on this data.

[0101] Calculations may require a "time shift," meaning data from multiple specific cycles or groups of cycles is needed. This differs from KPI calculations over periods encompassing multiple test cycles, where individual data from shorter periods is stored for the calculation process. This may be desirable, for example, for comparing KPI values ​​across cycles. Another reason might be that a given KPI is calculated only within individual test cycles and then summed up, or otherwise mathematically combined retrospectively.

[0102] Therefore, in S530, it is checked whether this option has been selected. If not ("No" in S530), the process simply outputs the results in S580 and terminates. However, if time shift is selected ("Yes" in S530), the results are temporarily stored in S540. Next, if it is determined that more such datasets are needed ("Yes" in S550), the process returns to S510 and repeats data collection for another dataset, this time for a different test cycle or group of cycles. This flow is repeated until all such datasets have been analyzed and all results have been generated ("No" in S550). Then, in S560 the results are merged, in S570 top-level calculations are performed on the merged results as needed, and in S580 the final results are output.

[0103] It will be recognized that the above process may be modified to include a "partial shift," which considers data from multiple specified parts of the network, instead of, or in addition to, a "time shift."

[0104] These and related processes, as well as other necessary instructions, are preferably encoded as executable instructions on one or more non-temporary computer-readable media, such as a hard disk drive or optical disc, and executed using one or more computer processors in cooperation with an operating system or other suitable means.

[0105] In a software implementation, the software includes a set of computer-executable instructions that are executed on a computer system. Before being loaded onto the computer system, the software preferably resides as encoded information on a suitable non-temporary computer-readable tangible medium, such as a magnetic, optical, or other appropriately encoded or recorded medium. The specific medium may include, but is not limited to, magnetic floppy disks, magnetic tapes, CD-ROMs, DVD-ROMs, solid-state disks, or flash memory devices, and in certain embodiments, may take the form of existing data storage (such as "cloud storage") accessible via an operable-coupled network means (such as the Internet).

[0106] In certain embodiments, the system includes a dedicated processor, a processing portion of a system on a chip (SOC), a portion of a field programmable gate array (FPGA), or other suitable means for performing the functions described herein or for executing processor instructions to emulate certain structures defined herein. For example, suitable circuits using discrete logic gates, such as application-specific integrated circuits (ASICs), programmable logic arrays (PLAs), or field programmable gate arrays (FPGAs), are also developed in certain embodiments to perform these functions.

[0107] Figure 6 shows an exemplary component of device 600. Device 600 can correspond to either a coordinator unit 41 and / or a worker 45. As shown in Figure 6, device 600 may include a bus 610, a processor 620, memory 630, a storage component 640, an input component 650, an output component 660, and a communication interface 670.

[0108] Bus 610 includes components that enable communication between components of device 600. Processor 620 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 620 may be a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, processor 620 includes one or more processors that can be programmed to perform functions. Memory 630 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 620.

[0109] The storage component 640 stores information and / or software related to the operation and use of device 600. For example, the storage component 640 may include, along with a corresponding drive, a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, magnetic tape, and / or another type of non-temporary computer-readable media. The input component 650 includes components that enable device 600 to receive information via user input, etc. (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, and / or microphone). In addition, or instead, the input component 650 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or actuators). The output component 660 includes components that provide output information from the device 600 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)).

[0110] The communication interface 670 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable device 600 to communicate with other devices via wired connections, wireless connections, or a combination of wired and wireless connections. The communication interface 670 may also enable device 600 to receive information from and / or provide information to other devices. For example, the communication interface 670 may include Ethernet interfaces, optical interfaces, coaxial interfaces, infrared interfaces, radio frequency (RF) interfaces, universal serial bus (USB) interfaces, Wi-Fi interfaces, cellular network interfaces, and the like.

[0111] Device 600 may perform one or more processes described herein. Device 600 may perform these processes in response to the processor 620 executing software instructions stored in a non-temporary computer-readable medium such as memory 630 and / or storage component 640. A computer-readable medium is defined herein as a non-temporary memory device. A memory device includes a memory space within a single physical storage device or a memory space that extends across multiple physical storage devices.

[0112] Software instructions may be read into memory 630 and / or storage component 640 from another computer-readable medium or from another device via the communication interface 670. When executed, the software instructions stored in memory 630 and / or storage component 640 may cause the processor 620 to execute one or more processes described herein.

[0113] In addition, or instead, hardwired circuits may be used in place of, or in combination with, software instructions to perform one or more of the processes described herein. Therefore, the implementations described herein are not limited to any particular combination of hardware circuits and software.

[0114] The number and arrangement of components shown in Figure 6 are provided as an example. In practice, device 600 may include additional components, fewer components, different components, or components arranged differently than those shown in Figure 6. In addition, or instead, a set of components of device 600 (e.g., one or more components) may perform one or more functions that are described as being performed by another set of components of device 600.

[0115] In the embodiment, any one of the operations or processes shown in Figures 1, 2, 3, 4B, and 5 may be implemented by or using any one of the elements shown in Figure 6.

[0116] The foregoing disclosures are illustrative and explanatory, but are not intended to be exhaustive or to limit implementations to the exact forms disclosed. Modifications and variations are possible in light of the foregoing disclosures or may be derived from the practice of the implementations.

[0117] Some embodiments may relate to systems, methods, and / or computer-readable media in integration at any possible level of technical detail. Furthermore, one or more of the components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include computer-readable non-temporary storage media (or more) having computer-readable program instructions for causing a processor to perform an operation.

[0118] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction-executing device. A computer-readable storage medium may, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital multipurpose discs (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or grooved structures on which instructions are stored, and any preferred combination thereof. The computer-readable storage media used herein should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0119] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network, transfers the computer-readable program instructions, and stores them in a computer-readable storage medium within each computing / processing device.

[0120] The computer-readable program code / instructions for performing an operation may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the C programming language or similar programming languages. Computer-readable program instructions can run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (for example, via the Internet using an Internet Service Provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by personalizing the electronic circuit using state information of computer-readable program instructions in order to perform an action or operation.

[0121] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to generate a machine, thereby generating means for instructions executed via the processor of the computer or other programmable data processing device to implement functions / operations specified in one or more blocks of a flowchart and / or block diagram, or both. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device, and / or other device to function in a particular manner, thereby including a product in which the computer-readable storage medium internally storing the instructions includes instructions that implement modes of functions / operations specified in one or more blocks of a flowchart and / or block diagram, or both.

[0122] Computer-readable program instructions can also be loaded into a computer, another programmable device, or another device to generate a computer implementation process that, as a result, is executed on the computer, another programmable device, or another device, performing functions / operations specified in one or more blocks of a flowchart and / or block diagram or both.

[0123] The flowcharts and block diagrams in the figures illustrate the architecture, functions, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function(s). Methods, computer systems, and computer-readable media may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in the figures. In some alternative implementations, the functions described in the blocks may be performed in a different order than shown in the figures. For example, two consecutively shown blocks may actually be executed simultaneously or substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0124] It will be apparent that the systems and / or methods described herein may be implemented in various forms, such as hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to the implementation form. Therefore, it is understood that the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and that software and hardware may be designed to implement the systems and / or methods based on the descriptions herein.

Claims

1. A method for long-term storage that enables the later retrieval of network performance data, wherein for at least one test cycle of the network, Obtaining a plurality of performance data files corresponding to the test cycle, wherein each of the plurality of performance data files contains data describing network performance, and the plurality of performance data files include a first data file in at least a first format and a second data file in a second format different from the first format. To obtain multiple unified data files, each of the multiple performance data files is reformatted according to a predetermined unified file format and a predefined set of unified category identifiers. The aforementioned multiple unified data files are stored in a query database in memory, The system involves storing at least one configuration data file for each of the multiple network cells, with the frequency of storage of the configuration data files being lower than the frequency of the test cycles. Methods that include...

2. Each performance data file, The process involves parsing the performance data file according to the performance data file format to identify a plurality of data category identifiers within the performance data file, wherein each of the plurality of data category identifiers has a corresponding data value within the performance data file. Matching each of the aforementioned predefined set of unified category identifiers to each of the aforementioned multiple data category identifiers in the performance data file, The counter value of at least one counter in the aggregated counter file is increased based on a unified category identifier having a predefined correspondence to the counter and the data value of a data category identifier that matches the unified category identifier having a predefined correspondence to the counter. The data value of at least one data category identifier is stored at the sequence location within a temporary data frame, based on the sequence location having a predefined correspondence with a unified category identifier that matches the data category identifier. The temporary data frame is converted into a unified data file having the predetermined unified file format, and is reformatted by this process. The method according to claim 1, wherein storing the plurality of unified data files in the query database includes storing the aggregated counter file.

3. The method according to claim 2, wherein the matching is based on a predetermined category mapping corresponding to the source of the performance data file.

4. The aforementioned verification, Text comparison between the aforementioned unified category identifier and the aforementioned data category identifier, A comparison between the expected format and range corresponding to the aforementioned unified category identifier and the sample data points corresponding to the aforementioned data category identifier in the performance data file, The method according to claim 2, based on the present invention.

5. The method according to claim 1, further comprising verifying the validity of each of the plurality of unified data files, wherein each of the plurality of unified data files is partitioned during the validity verification according to a predefined category identifier threshold.

6. A method for analyzing past network performance, The method described in claim 1 involves storing multiple unified data files in a query database in memory, The query database is searched according to the received search request, thereby obtaining a set of retrieved unified data files. Calculate at least one performance indicator based on the recovered set of unified data files, Methods that include...

7. The received search request defines the selected time frame, Each of the recovered unified data files corresponds to a test cycle within the selected time frame, The values ​​in the recovered set of unified data files are aggregated to determine representative values ​​for the selected time frames. The at least one performance indicator is calculated based on the representative value of the selected time frame. The method according to claim 6.

8. The received search request defines the selected network portion, Each of the recovered unified data file sets corresponds to a cell within the selected network portion, The values ​​in the recovered set of unified data files are aggregated to determine representative values ​​for the selected network portion. The at least one performance indicator is calculated based on the representative value of the selected network portion. The method according to claim 6.

9. A method for analyzing past network performance, The method described in claim 1 involves storing multiple unified data files in a query database in memory, The process involves searching the query database according to the received search request, thereby obtaining a set of recovered unified data files and a set of recovered configuration data files, wherein the received search request defines a selected time frame, and each of the recovered unified data files and each of the recovered configuration data files corresponds to the selected time frame. The data from the recovered set of unified data files is merged with the data from the recovered set of configuration data files to generate a set of analysis data files. Calculate at least one performance indicator based on the set of analysis data files, Methods that include...

10. A system for long-term storage that enables the later retrieval of network performance data, Obtain multiple performance data files corresponding to a test cycle, wherein each of the multiple performance data files contains data describing network performance, and the multiple performance data files include a first data file in at least a first format and a second data file in a second format different from the first format. To obtain multiple unified data files, each of the multiple performance data files is reformatted according to a predetermined unified file format and a predefined set of unified category identifiers. The aforementioned multiple unified data files are stored in a query database within the query memory. For each of the multiple network cells, at least one configuration data file is stored, where the frequency of storage of the configuration data file is lower than the frequency of the test cycle. A system configured in such a way.

11. Each performance data file, The process involves parsing the performance data file according to the performance data file format to identify a plurality of data category identifiers within the performance data file, wherein each of the plurality of data category identifiers has a corresponding data value within the performance data file. Matching each of the aforementioned predefined set of unified category identifiers to each of the aforementioned multiple data category identifiers in the performance data file, The counter value of at least one counter in the aggregated counter file is increased based on a unified category identifier having a predefined correspondence to the counter and the data value of a data category identifier that matches the unified category identifier having a predefined correspondence to the counter. The data value of at least one data category identifier is stored at the sequence location within a temporary data frame, based on the sequence location having a predefined correspondence with a unified category identifier that matches the data category identifier. Converting the aforementioned temporary data frame into a unified data file having the predetermined unified file format, Reformatted by, The system is further configured to store the aggregated counter file. The system according to claim 10.

12. The system according to claim 11, wherein the matching is based on a predetermined category mapping corresponding to the source of the performance data file.

13. The aforementioned verification, Text comparison between the aforementioned unified category identifier and the aforementioned data category identifier, A comparison between the expected format and range corresponding to the aforementioned unified category identifier and the sample data points corresponding to the aforementioned data category identifier in the performance data file, The system according to claim 11, based on the above.

14. The system according to claim 10, further configured to verify the validity of each of the plurality of unified data files, wherein each of the plurality of unified data files is partitioned during validity verification according to a predefined category identifier threshold.

15. A computer program recording instructions executable by at least one processor for performing a method for long-term storage that enables the later retrieval of network performance data, wherein the method comprises at least one test cycle of a network. Obtaining a plurality of performance data files corresponding to the test cycle, wherein each of the plurality of performance data files contains data describing network performance, and the plurality of performance data files include a first data file in at least a first format and a second data file in a second format different from the first format. To obtain multiple unified data files, each of the multiple performance data files is reformatted according to a predetermined unified file format and a predefined set of unified category identifiers. The aforementioned multiple unified data files are stored in a query database in memory, The system involves storing at least one configuration data file for each of the multiple network cells, with the frequency of storage of the configuration data files being lower than the frequency of the test cycles. A computer program that includes [this].

16. Each performance data file, The process involves parsing the performance data file according to the performance data file format to identify a plurality of data category identifiers within the performance data file, wherein each of the plurality of data category identifiers has a corresponding data value within the performance data file. Matching each of the aforementioned predefined set of unified category identifiers to each of the aforementioned multiple data category identifiers in the performance data file, The counter value of at least one counter in the aggregated counter file is increased based on a unified category identifier having a predefined correspondence to the counter and the data value of a data category identifier that matches the unified category identifier having a predefined correspondence to the counter. The data value of at least one data category identifier is stored at the sequence location within a temporary data frame, based on the sequence location having a predefined correspondence with a unified category identifier that matches the data category identifier. The temporary data frame is converted into a unified data file having the predetermined unified file format, and is reformatted by this process. The computer program according to claim 15, wherein storing the plurality of unified data files in the query database includes storing the aggregated counter file.

17. The computer program according to claim 15, wherein the method further comprises verifying the validity of each of the plurality of unified data files, wherein each of the plurality of unified data files is partitioned during the validity verification according to a predefined category identifier threshold.

18. The computer program according to claim 15, further comprising storing at least one configuration data file for each of a plurality of network cells, wherein the frequency of storage of the configuration data file is less than the frequency of the test cycle.

Images