A traffic engineering monitoring data compression storage method

Abstract

This invention relates to the field of data encoding and compression technology, specifically to a method for compressing and storing traffic engineering monitoring data. The invention divides traffic data into segments to be encoded based on the data acquisition frequency, ensuring the regularity of data within each segment. It obtains the initial allocation space for each character type based on their frequency within the segments. Based on the number of character types in each segment and the initial allocation space for each character type, it determines the necessity of merging each initial allocation space. Iteratively merging is performed based on the initial allocation space size and merging necessity to obtain the final allocation space sequence. An index dictionary for each segment is obtained from the final allocation space sequence. The LZW algorithm is then used to encode the segments, resulting in more efficient compressed encoding for storage. This invention optimizes the data encoding process, leading to higher efficiency in compressing and storing traffic data.

Description

Technical Field

[0001] This invention relates to the field of data encoding and compression technology, specifically to a method for compressing and storing traffic engineering monitoring data. Background Technology

[0002] With the continuous expansion of transportation infrastructure, the safety and durability of critical infrastructure such as highways, tunnels, slopes, and bridges require timely monitoring. Modern monitoring systems rely on numerous sensors to collect various types of data in real time, including strain, acceleration, displacement, and temperature, and use this data to monitor potential risks to transportation infrastructure. The diverse types of sensors generate massive amounts of sensor data; for example, a monitoring system for a major bridge deployed nearly 4,000 sensors, with the annual data volume for a single bridge typically reaching terabytes. The high frequency and continuous sampling of monitoring data place enormous pressure on wireless transmission bandwidth and storage resources, especially in critical locations such as bridges and tunnels in remote areas, where it is necessary to minimize the transmission burden while ensuring data integrity.

[0003] Traditional data compression techniques, such as feature reduction and principal component analysis, are applicable in certain scenarios, but they suffer from severe information loss. When using the LZW algorithm to compress and store traffic monitoring data, the large amount of data and the variety of data types lead to an increasing number of strings during compression, making the time for matching and searching in the compression dictionary longer and longer, resulting in extremely low compression and storage efficiency. Summary of the Invention

[0004] To address the extremely low efficiency of compressing and storing traffic data using the LZW algorithm in existing technologies, this invention aims to provide a method for compressing and storing traffic engineering monitoring data. The specific technical solution adopted is as follows: This invention provides a method for compressing and storing traffic engineering monitoring data, the method comprising: Acquire traffic data; divide the traffic data into segments to be encoded based on the data collection frequency; Based on the frequency of each type of character in each data segment to be encoded and the initial dictionary size, the initial allocation space corresponding to each type of character in each data segment to be encoded is obtained; based on the number of character types in each data segment to be encoded, the initial dictionary size, and the initial allocation space for each type of character, the necessity of merging each initial allocation space is obtained; based on the size and merging necessity of all initial allocation spaces in each data segment to be encoded, the initial allocation spaces are iteratively merged to obtain the final allocation space sequence for each data segment to be encoded. The index dictionary for each data segment to be encoded is obtained based on the final allocation space sequence of each data segment to be encoded; the LZW algorithm is used to encode the data segment to be encoded using the index dictionary to obtain compressed codes; all compressed codes in the traffic data are stored.

[0005] Furthermore, the method for obtaining the initial allocation space includes: Obtain the preset maximum encoding length and use the preset maximum encoding length to calculate the initial dictionary size; For any given data segment to be encoded, count the number of character types contained in the data segment to be encoded, and the frequency of each type of character appearing in the data segment to be encoded. The difference between the initial dictionary size and the number of categories is used as a dynamic expansion space; The frequency of occurrence of each type of character is multiplied by the dynamic expansion space to obtain a first product. The sum of the first product and the value 1 is used as the initial allocation space for each type of character in the data segment to be encoded.

[0006] Furthermore, the method for obtaining the necessity of merging includes: For any data segment to be encoded, the number of character types in the data segment to be encoded is taken as the number of encoding types of the data segment to be encoded. The ratio of the initial dictionary size to the number of encoding types is used as the average allocation space of the data segment to be encoded; the ratio of the average allocation space to the size of each initial allocation space in the data segment to be encoded is used as the necessity of merging each initial allocation space in the data segment to be encoded.

[0007] Further, the step of iteratively merging the initial allocation spaces to obtain the final allocation space sequence for each data segment to be encoded, based on the size of all initial allocation spaces in each data segment to be encoded and the necessity of merging, includes: Sort the initial allocation space of all characters in the data segment to be encoded in descending order to obtain the initial allocation space sequence; The index value of the first initial allocation space in the initial allocation space sequence is used as the head index, and the index value of the last initial allocation space is used as the tail index. Perform iterative merging operation: If the merging necessity corresponding to the head index is less than or equal to the preset merging threshold, and the merging necessity corresponding to the tail index is greater than the preset merging threshold, then merge the initial allocation space corresponding to the tail index into the initial allocation space corresponding to the head index to obtain a new initial allocation space; rearrange all the current initial allocation spaces in descending order to update the initial allocation space sequence and the corresponding merging necessity, and reacquire the head index and the tail index, and return to execute the iterative merging operation; If the preset stopping condition is met, the merging stops, and the current initial allocation space sequence is determined as the final allocation space sequence. The preset stopping condition includes any one of the following: the merging necessity corresponding to the head index is greater than the preset merging threshold; the merging necessity corresponding to the tail index is less than or equal to the preset merging threshold; and the initial allocation space sequence contains an initial allocation space.

[0008] Furthermore, the method for obtaining the index dictionary includes: Obtain the initial dictionary for each data segment to be encoded, with the size of the initial dictionary being the same as the initial dictionary size; For any data segment to be encoded, round down each initial allocation space in the final allocation space sequence of the data segment to obtain each final allocation space in the final allocation space sequence; according to the size of the final allocation space in the final allocation space sequence of the data segment to be encoded, divide the index interval corresponding to each final allocation space in the initial dictionary in turn. Obtain the character group corresponding to each initial allocation space in the final allocation space of the data segment to be encoded; store the character group corresponding to each final allocation space into the corresponding index range to obtain the index dictionary of the data segment to be encoded.

[0009] Further, obtaining the character group corresponding to each initial allocation space in the final allocation space of the data segment to be encoded includes: For any final allocation space, during the merging process of that final allocation space, the initial allocation space of each type of character to be merged will be used as the merging allocation space of that final allocation space. Each character in the final allocation space is combined into a character group corresponding to that final allocation space.

[0010] Furthermore, the step of encoding the data segment to be encoded using the LZW algorithm through an index dictionary to obtain compressed encoding includes: For any data segment to be encoded, obtain the character index table based on the index value of each type of character in the index dictionary; During the encoding and compression process of the data segment to be encoded using the LZW algorithm, the index value of the first character in the string is used to search and store the corresponding index position in the index dictionary, and the index dictionary is updated until the data segment to be encoded is fully encoded and the compressed code of the data segment is obtained.

[0011] Furthermore, the method for retrieving and storing includes: During the process of searching the string backwards from the index position, when an empty space is found in the index dictionary, the corresponding string is stored in the corresponding empty space in the index dictionary.

[0012] Furthermore, the step of dividing traffic data into segments to be encoded based on the data collection frequency includes: Obtain the data acquisition frequency, calculate all data acquisition frequencies within a preset time period as a frequency sequence; calculate and normalize the variance of the data acquisition frequencies in the frequency sequence to obtain the frequency fluctuation. When the frequency fluctuation is less than or equal to the preset fluctuation threshold, the adjacent data collection frequency after the preset time period is assigned to the frequency sequence, and the frequency fluctuation is iteratively calculated; the iteration stops when the frequency fluctuation is greater than the preset fluctuation threshold or there is no new data collection frequency; the traffic data corresponding to all data collection frequencies in the final frequency sequence are taken as the data segments to be encoded.

[0013] Furthermore, the rounding method is to round down.

[0014] The present invention has the following beneficial effects: Traditional LZW algorithms struggle to effectively utilize data patterns when processing globally non-stationary traffic data. This invention dynamically divides a continuous data stream into multiple highly regular data segments to be encoded. This ensures that the statistical characteristics (such as character distribution and repetition patterns) within each data segment are highly consistent, greatly enhancing the LZW algorithm's ability to capture and utilize local data features. Simultaneously, initial space is allocated to each character category based on its frequency of occurrence, reflecting the true distribution of the data. By calculating the necessity of merging, low-frequency and inefficient fragmented spaces are intelligently identified and merged. Through iterative optimization, the resulting final allocation space sequence and the index dictionary constructed accordingly achieve on-demand, efficient, and fault-tolerant allocation of dictionary space. This significantly improves the utilization rate of the limited dictionary space, reduces space waste, and enhances the algorithm's adaptability to local data fluctuations through reserved buffers and merging strategies, avoiding the defect of triggering a complete dictionary reset due to the rapid exhaustion of space for individual characters. Furthermore, to address the issue of linearly increasing retrieval time caused by the linear expansion of the LZW algorithm's dictionary, a character index table was constructed to establish a direct mapping between characters and specific retrieval intervals within the dictionary. During encoding, only the first character of the string needs to be queried in this table, which drastically narrows the retrieval range from traversing the entire dictionary to a local interval. This reduces the average time complexity of string matching operations from O(n) in traditional methods to nearly O(1), solving the bottleneck of increasingly longer matching and retrieval times in traditional algorithms. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a traffic engineering monitoring data compression and storage method provided in one embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a traffic engineering monitoring data compression and storage method proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for a traffic engineering monitoring data compression and storage method provided by the present invention.

[0020] Please see Figure 1 The diagram illustrates a flowchart of a traffic engineering monitoring data compression and storage method according to an embodiment of the present invention, which includes the following steps: S1: Acquire traffic data; divide the traffic data into segments to be encoded according to the data collection frequency.

[0021] The main purpose of this invention is to optimize the encoding and compression process of the LZW algorithm, thereby improving the compression and storage of large-scale intelligent transportation data. This is mainly achieved by optimizing the initial compression dictionary of the LZW algorithm and classifying and storing the strings in the compression dictionary by constructing an index. This reduces the retrieval time of the data in the compression dictionary during the encoding process, thereby improving the compression and storage efficiency of traffic data. Therefore, traffic data is obtained first.

[0022] In this embodiment of the invention, initial traffic data is primarily collected by sensors on different road sections to facilitate traffic planning. The collected initial traffic data includes, but is not limited to, traffic flow data collected by traffic flow sensors, such as the number of vehicles on the road, vehicle flow, and traffic congestion information; vehicle location and speed information collected by onboard equipment; and traffic light status, road condition information, and traffic events collected by the traffic management system. It should be noted that the main process for collecting initial traffic data uses techniques well-known to those skilled in the art and will not be elaborated upon here.

[0023] To facilitate encoding and compression of the LZW algorithm and reduce storage costs, this embodiment of the invention unifies the data format of the collected initial traffic data to obtain traffic data. This improves the consistency and usability of the traffic data. It should be noted that data format unification can be achieved using data standardization methods. Data standardization is a well-known technique among those skilled in the art, primarily utilizing standards and specifications to define the data structure, naming rules, and encoding methods, such as using the common data standard GTFS, which will not be elaborated upon here. In other embodiments of the invention, a data model can also be defined to model and define the initial traffic data, for example, through a unified modeling language. The method of data modeling using a unified modeling language is also a well-known technique among those skilled in the art and is not limited here.

[0024] While traffic data exhibits certain recurring patterns, these patterns vary depending on road traffic conditions. For instance, traffic flow differs between morning and evening rush hours, and similarly, traffic flow differs between midnight and noon. Traffic flow reflects road conditions, and to collect data more effectively, different data collection frequencies are typically used for different traffic periods. For example, the data collection frequency is lower at midnight when traffic is less, while a higher frequency is needed during peak hours for more accurate traffic analysis.

[0025] It should be noted that the data collection frequency of the traffic data is set according to conventional strategies in the field of traffic monitoring. For example, the monitoring system can adaptively switch the sampling frequency (such as 1 time / second at low frequency or 10 times / second at high frequency) based on the comparison result of real-time traffic flow and preset threshold. The innovation of the present invention lies in the subsequent data compression and storage method, which does not have any special dependence on or limitation on the specific value of the front-end data collection frequency.

[0026] Therefore, different road conditions, i.e., different data collection frequencies, can reflect different patterns in traffic information data. The LZW algorithm's compression effect is also poor for traffic data with different patterns, thus requiring a certain segmentation of the traffic data. In one embodiment of this invention, the data collection frequency is obtained, and all data collection frequencies within a preset time period are calculated as a frequency sequence. The variance of the data collection frequencies in the frequency sequence is calculated and normalized to obtain the frequency fluctuation. The normalization process uses the max-min normalization method, limiting the value to the [0,1] interval. The frequency fluctuation reflects the difference in data collection frequencies; the larger the frequency fluctuation, the greater the difference in data collection frequencies. When the frequency fluctuation is less than or equal to a preset fluctuation threshold, the adjacent data collection frequency after the preset time period is assigned to the frequency sequence. The frequency fluctuation is iteratively calculated until the frequency fluctuation exceeds the preset fluctuation threshold or no new data collection frequency is found. The iteration stops then, and the traffic data corresponding to all data collection frequencies in the final frequency sequence is taken as a data segment to be encoded.

[0027] During the segmentation process, all unencoded data is sequentially divided, meaning all unencoded and compressed traffic data is divided into multiple data segments to be encoded. Extensive traffic data analysis revealed that when the frequency fluctuation is below 0.5, the data exhibits strong regularity and is suitable for being segmented into the same data segment; when the frequency fluctuation is above 0.5, the data regularity changes significantly, requiring the segmentation into new data segments. Therefore, in this embodiment of the invention, the preset time period is set to 1 minute, and the preset fluctuation threshold is set to 0.5. In practical applications, this threshold can be adjusted within the range of [0.3, 0.7] according to the specific scenario.

[0028] S2: Based on the frequency of each type of character in each data segment to be encoded and the initial dictionary size, obtain the initial allocation space corresponding to each type of character in each data segment to be encoded; based on the number of character types in each data segment to be encoded, the initial dictionary size, and the initial allocation space of each type of character, obtain the merging necessity of each initial allocation space; based on the size and merging necessity of all initial allocation spaces in each data segment to be encoded, iteratively merge the initial allocation spaces to obtain the final allocation space sequence of each data segment to be encoded.

[0029] An initial compression dictionary is set for each data segment to be encoded. To reduce retrieval time, block indexing is performed based on character types, which reduces the time spent on character matching. In the data segment to be encoded, the higher the frequency of a certain type of character, the more space the string starting with that type of character is likely to occupy, requiring more allocation space. Therefore, based on the frequency of each type of character in each data segment to be encoded and the initial dictionary size, the initial allocation space corresponding to each type of character in each data segment to be encoded is obtained.

[0030] The LZW algorithm achieves compression by creating a string table, i.e., a compression dictionary, and replacing long strings with shorter codes. Therefore, an initial compression dictionary needs to be constructed. In this embodiment of the invention, different initial compression dictionaries are set according to the types of characters in different data segments to be encoded. When setting the initial compression dictionary, in order to store characters reasonably in the dictionary, the initial dictionary size needs to be set to partition the space in the dictionary, and at the same time, limit the total length of the string encoding and set an upper limit on the retrieval time for each time. It should be noted that the LZW algorithm is a well-known technique to those skilled in the art, and will not be described in detail here.

[0031] Preferably, the initial dictionary size is obtained by counting the number of character types in the traffic data. The number of types is positively correlated with the initial dictionary size, and the initial dictionary size must be greater than the number of types to ensure that the strings can be stored. In this embodiment of the invention, the positive correlation is reflected using exponential and logarithmic forms. The specific expression for the initial dictionary size is: In the formula, L is the preset maximum number of encoding bits, which is 16 in this embodiment to ensure that the initial dictionary size S is much larger than the number of character types n.

[0032] Furthermore, by allocating dynamic expansion space based on the frequency of each character type, for any given data segment to be encoded, the frequency of each character type in that data segment is counted. The product of the frequency of each character type and the dynamic expansion space is added to the number 1 to obtain the initial allocation space for each character type in the data segment to be encoded. The specific calculation formula is as follows: In the formula, Initially allocate space for the i-th character class. Let S be the frequency of occurrence of the i-th character class, S be the initial dictionary size, and n be the number of character classes. Dynamic expansion space is allocated based on the frequency of each character class. By calculating the initial allocation space for each character class, it ensures that each basic character has at least one fixed basic index position, while also ensuring that high-frequency characters receive more expansion space, thus avoiding the LZW algorithm's failure due to the dictionary being filled with basic characters.

[0033] The initial allocation space is the size of the corresponding partition in the dictionary for each type of character. However, in the data segment to be encoded, there are some characters with extremely low frequency. The space allocation for these characters is also relatively small, resulting in low space utilization and the possibility that the space may be filled, requiring the dictionary to be updated, which will increase the amount of computation during storage. Therefore, in order to improve the rationality of the dictionary partitioning and increase fault tolerance, the allocation space for these low-frequency characters is merged.

[0034] For the character types in the data segment to be encoded, the allocation space for each character type under the standard allocation condition can be obtained, where the standard allocation space is evenly distributed. If the allocation space for some characters is larger than the space under the standard allocation condition, it indicates that there may be many strings for this type of character, resulting in a long retrieval time, making merging unsuitable. Therefore, the necessity of merging is calculated for each character type. In this embodiment of the invention, the necessity of merging each initial allocation space is obtained based on the number of character types in each data segment to be encoded, the initial dictionary size, and the initial allocation space for each character type.

[0035] Preferably, for any given data segment to be encoded, the number of character types in the segment is taken as the number of encoding types K. If K is 0, the segment is marked as invalid and not subjected to subsequent compression. If K is greater than 0, the ratio of the initial dictionary size to the number of encoding types is taken as the average allocation space of the segment. The average allocation space is the size of the allocation space corresponding to each character type under standard allocation conditions. The ratio of the average allocation space to the size of each initial allocation space in the segment to be encoded is taken as the merging necessity of each initial allocation space in the segment, i.e., each initial allocation space corresponds to a merging necessity. In this embodiment, the expression for merging necessity is: In the formula, Represented as the first The nth data segment to be encoded The necessity of merging the initial allocated spaces Represented as the first The nth data segment to be encoded The size of the initial allocated space, Represented as the first The average size of the space allocated to each data segment to be encoded.

[0036] When the merging necessity is less than or equal to the preset merging threshold, it indicates that the space allocated to this type of character has increased according to frequency while satisfying the average space allocation, and the corresponding initial allocation space is already large enough. When the merging necessity is greater than the preset merging threshold, it indicates that the allocation space is small, and merging can increase the allocation space to ensure sufficient string filling. When the initial allocation space of a certain type of character is less than the average allocation space (i.e., merging necessity > 1), it indicates that its space allocation is insufficient and it is easily filled quickly during the encoding process, causing dictionary update overhead, so it is suitable for merging. Threshold 1 is the dividing point for judging whether the space is insufficient. In this embodiment of the invention, the preset merging threshold is set to 1. In other embodiments, this threshold can be finely adjusted within the range of [0.8, 1.2] to balance space utilization and merging frequency. The specific value can be adjusted by the implementer.

[0037] Further, based on the size of all initial allocation spaces in each data segment to be encoded and the necessity of merging, the initial allocation spaces are iteratively merged to obtain the final allocation space sequence for each data segment to be encoded. During merging, in order to ensure that each merged allocation space is not too large, it is necessary to judge the size of the initial allocation space and merge smaller allocation spaces with larger allocation spaces as much as possible.

[0038] Preferably, for any data segment to be encoded, the initial allocation spaces of all characters in the data segment to be encoded are sorted in descending order to obtain an initial allocation space sequence. By sorting, the larger and smaller initial allocation spaces can be merged more accurately, so that the allocated spaces are relatively close and do not produce large differences.

[0039] The index of the first initially allocated space in the initial allocation space sequence is used as the head index, which is the character group with the largest current allocation space. The necessity of merging is extremely low. The index of the last initially allocated space is used as the tail index, which is the character group with the smallest current allocation space. The merging is urgently needed and is highly necessary.

[0040] During the iterative merging process: when the merging necessity corresponding to the head index is less than or equal to the preset merging threshold, and the merging necessity corresponding to the tail index is greater than the preset merging threshold, it indicates that the head index space is ample while the tail index space is extremely scarce. In this case, the initial allocation space corresponding to the tail index is directly merged into the initial allocation space corresponding to the head index. After merging, a new initial allocation space is obtained. All initial allocation spaces are rearranged in descending order, the merging necessity is updated, and the next iteration is performed.

[0041] As smaller spaces are continuously absorbed by larger spaces, the space pointed to by the tail index continues to grow. Merging stops when the necessity of merging the initial allocation space corresponding to the tail index is less than or equal to the preset merging threshold, or when only one initial allocation space remains in the sequence. The initial allocation space sequence at this point is then taken as the final allocation space sequence.

[0042] Each initial allocation space in the final allocation space sequence is the size of each interval into which the initial compressed dictionary is divided.

[0043] S3: Obtain the index dictionary for each data segment to be encoded based on the final allocation space sequence of each data segment to be encoded; use the LZW algorithm to encode the data segment to be encoded using the index dictionary to obtain compressed codes; store all compressed codes in the traffic data.

[0044] By using the final allocation space sequence of each data segment to be encoded, an initial compression encoding dictionary is set. According to the final allocation space sequence, each type of character is stored in the corresponding position of the dictionary. In one embodiment of the present invention, the initial dictionary of each data segment to be encoded is obtained. The size of the initial dictionary is the same as the initial dictionary size. The initial dictionary is used as a blank dictionary for each data segment to be encoded, and strings are filled in.

[0045] Preferably, for any data segment to be encoded, each initial allocation space in the final allocation space sequence of the data segment to be encoded is rounded down to obtain each final allocation space in the final allocation space sequence. Since the initial allocation space obtained when calculating the initial allocation space may not be an integer in order to ensure the accuracy of the space calculation, but an integer space size is required when dividing the dictionary, it is necessary to round down the partition. Therefore, in this embodiment of the invention, in order to ensure the integrity of the space allocation, the rounding method is to choose the rounding down method.

[0046] It is important to note that since the computational space for extremely low-frequency characters may be less than 1, to prevent the allocated space from becoming 0 after rounding down, which could cause such characters to completely lose their index position in the dictionary and trigger subsequent encoding failures, this embodiment, during the rounding down process, if the rounding result is 0, forces it to be modified and assigned a value of 1. This ensures that the final allocated space is an integer, guaranteeing that each type of legal character can obtain at least one basic index interval position in the dictionary.

[0047] Furthermore, based on the size of the final allocation space in the final allocation space sequence of the data segment to be encoded, the index range corresponding to each final allocation space is sequentially divided in the initial dictionary. For example, when the final allocation space in the final allocation space sequence is {7, 6, 4, 3}, and the initial dictionary size is 20, the index range corresponding to the final allocation space 7 in the initial dictionary is [1-7], the index range corresponding to the final allocation space 6 is [8-13], the index range corresponding to the final allocation space 4 is [14-17], and the index range corresponding to the final allocation space 3 is [18-20].

[0048] Further, the character groups corresponding to each initial allocation space in the final allocation space of the data segment to be encoded are obtained. Since the final allocation space is obtained by merging the initial allocation spaces corresponding to each type of character, each final allocation space corresponds to one or more types of characters. In this embodiment of the invention, for any final allocation space, during the merging process of the final allocation space, the initial allocation space of each type of character being merged is used as the merged allocation space of the final allocation space, and the characters in each merged allocation space in the final allocation space are combined to form the character group corresponding to the final allocation space. For example, if the final allocation space is obtained by merging the initial allocation spaces of character a and character b, then the initial allocation spaces of character a and character b are the merged allocation spaces of the final allocation space, and character a and character b form the character group (a, b).

[0049] Finally, the character groups corresponding to each final allocation space are stored in the corresponding index interval to obtain the index dictionary of the data segment to be encoded. The character groups in the final allocation space and the index intervals also have a one-to-one correspondence. For example, in the example above, the index interval corresponding to the final allocation space of 6 is [8-13]. When the character group corresponding to the final allocation space is (a, b), characters a and b are filled into the index interval [8-13] in sequence. After filling each type of character in the data segment to be encoded into the initial dictionary, the index dictionary is obtained.

[0050] After obtaining the index dictionary, the index dictionary becomes the initial compression code for each data segment to be encoded. The LZW algorithm is used to encode the data segment to obtain the compressed code. Preferably, for any data segment to be encoded, a character index table is obtained based on the index value of each type of character in the index dictionary. During the compression encoding process using the index dictionary, the character index table can locate the search position to the first character interval of the string, greatly reducing the search time.

[0051] The process of constructing the character index table is as follows: Initialize a mapping table from characters to index values, preferably implemented using a hash table; traverse each index interval of the index dictionary; for each independent character stored within an interval (i.e., a character not merged with other characters), use the character as the key and its specific index position in the index dictionary as the value, storing it in the mapping table; for character groups stored within the same index interval due to merging operations, use all characters in the character group as keys and their starting index in the index interval as values, storing them in the mapping table; after traversing all index intervals, the character index table is obtained. In subsequent LZW encoding, for any string to be encoded, a starting retrieval index can be quickly obtained from the character index table based on its first character, thereby locking the retrieval scope within a specific area of ​​the index dictionary, avoiding global traversal, and significantly improving retrieval efficiency.

[0052] During the encoding and compression of the data segment to be encoded using the LZW algorithm, based on the index value of the first character in the string in the character index table, the corresponding index position in the index dictionary is searched and stored, and the index dictionary is updated until the data segment to be encoded is fully encoded and the compressed code of the data segment is obtained. Here, the index position is the position of the character corresponding to the index value in the index dictionary. Specifically, the retrieval and storage are performed as follows: Within a search interval: Starting from the initial search index, within the corresponding search interval, each position is checked sequentially (in the direction of increasing index value): If an empty space is encountered, the current string will be stored in that position, and the current search will end. If a previously existing string is encountered that is identical to the current string, the index of that position is output as the compression code, and the current search ends. If an existing string is encountered that is different from the current string, the process continues to check the next position.

[0053] Outside-range processing: If no empty space or complete match is found after traversing the entire search range in step 1, then processing is performed according to a preset strategy. The preset strategy includes: Strategy 1: Continue searching within the remaining space of the index dictionary until a space or a complete match is found; Strategy 2: Trigger dictionary reset or indicate that the dictionary is full, and process the current string according to the rules of the classic LZW algorithm.

[0054] By localizing global retrieval, the average retrieval length is significantly reduced, thereby improving the overall efficiency of compression encoding.

[0055] Finally, all data segments to be encoded are compressed to obtain a compressed code for each segment. All compressed codes in the traffic data are then stored. It should be noted that to ensure the compressed codes can be decoded, a corresponding index dictionary needs to be stored for each code. By optimizing the obtained index dictionary, the compression efficiency is improved while maintaining the traffic data compression rate, saving storage time costs.

[0056] In summary, this invention, considering the regularity of traffic data, divides traffic data into segments to be encoded based on data collection frequency, ensuring the regularity of data within each segment and achieving a high compression ratio using the LZW algorithm. Furthermore, an initial allocation space is obtained from a dictionary for each character type, enabling block analysis of each character type and the string starting with each character type, reducing retrieval time for other information. Since the initial allocation space is based on the frequency of each character type in the segment to be encoded, the utilization rate of the initial allocation space for some low-frequency characters is low and the fault tolerance is small. When the string exceeds the limit, it increases the computational load. Therefore, based on the number of character types in each segment to be encoded, the initial dictionary size, and the initial allocation space for each character type, the necessity of merging each initial allocation space is determined. By merging some initial allocation spaces based on the necessity, the reliability of space allocation is improved. Based on the initial allocation space size and merging necessity, the final allocation space sequence is obtained through iterative merging. Based on the final allocation spatial sequence, an index dictionary can be obtained for each data segment to be encoded. Using the index dictionary and the LZW algorithm to encode the data segment improves the retrieval efficiency of characters and strings, and allows for more efficient acquisition of compressed codes for storage. This invention optimizes the dictionary in the LZW algorithm, resulting in higher efficiency for compressed storage of traffic data.

[0057] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0058] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A traffic engineering monitoring data compression storage method, characterized by, The method includes: Acquire traffic data; divide the traffic data into segments to be encoded based on the data collection frequency; Based on the frequency of each type of character in each data segment to be encoded and the initial dictionary size, the initial allocation space corresponding to each type of character in each data segment to be encoded is obtained; based on the number of character types in each data segment to be encoded, the initial dictionary size, and the initial allocation space for each type of character, the necessity of merging each initial allocation space is obtained; based on the size and merging necessity of all initial allocation spaces in each data segment to be encoded, the initial allocation spaces are iteratively merged to obtain the final allocation space sequence for each data segment to be encoded. The index dictionary for each data segment to be encoded is obtained based on the final allocation space sequence of each data segment to be encoded; the LZW algorithm is used to encode the data segment to be encoded using the index dictionary to obtain compressed codes; all compressed codes in the traffic data are stored.

2. The method for compressing and storing monitoring data of traffic engineering according to claim 1, wherein, The method for obtaining the initial allocation space includes: Obtain the preset maximum encoding length and use the preset maximum encoding length to calculate the initial dictionary size; For any given data segment to be encoded, count the number of character types contained in the data segment to be encoded, and the frequency of each type of character appearing in the data segment to be encoded. The difference between the initial dictionary size and the number of categories is used as a dynamic expansion space; The frequency of occurrence of each type of character is multiplied by the dynamic expansion space to obtain a first product. The sum of the first product and the value 1 is used as the initial allocation space for each type of character in the data segment to be encoded.

3. The method of claim 1, wherein The method for obtaining the necessity of merging includes: For any data segment to be encoded, the number of character types in the data segment to be encoded is taken as the number of encoding types of the data segment to be encoded. The ratio of the initial dictionary size to the number of encoding types is used as the average allocation space of the data segment to be encoded; the ratio of the average allocation space to the size of each initial allocation space in the data segment to be encoded is used as the necessity of merging each initial allocation space in the data segment to be encoded.

4. The method for compressing and storing traffic engineering monitoring data according to claim 1, characterized in that, The step of iteratively merging the initial allocation spaces to obtain the final allocation space sequence for each data segment to be encoded, based on the size of all initial allocation spaces in each data segment to be encoded and the necessity of merging, includes: Sort the initial allocation space of all characters in the data segment to be encoded in descending order to obtain the initial allocation space sequence; The index value of the first initial allocation space in the initial allocation space sequence is used as the head index, and the index value of the last initial allocation space is used as the tail index. Perform iterative merging operation: If the merging necessity corresponding to the head index is less than or equal to the preset merging threshold, and the merging necessity corresponding to the tail index is greater than the preset merging threshold, then merge the initial allocation space corresponding to the tail index into the initial allocation space corresponding to the head index to obtain a new initial allocation space; rearrange all the current initial allocation spaces in descending order to update the initial allocation space sequence and the corresponding merging necessity, and reacquire the head index and the tail index, and return to execute the iterative merging operation; If the preset stopping condition is met, the merging stops, and the current initial allocation space sequence is determined as the final allocation space sequence. The preset stopping condition includes any one of the following: the merging necessity corresponding to the head index is greater than the preset merging threshold; the merging necessity corresponding to the tail index is less than or equal to the preset merging threshold; and the initial allocation space sequence contains an initial allocation space.

5. The method for compressing and storing traffic engineering monitoring data according to claim 4, characterized in that, The methods for obtaining the index dictionary include: Obtain the initial dictionary for each data segment to be encoded, with the size of the initial dictionary being the same as the initial dictionary size; For any data segment to be encoded, round down each initial allocation space in the final allocation space sequence of the data segment to obtain each final allocation space in the final allocation space sequence; according to the size of the final allocation space in the final allocation space sequence of the data segment to be encoded, divide the index interval corresponding to each final allocation space in the initial dictionary in turn. Obtain the character group corresponding to each initial allocation space in the final allocation space of the data segment to be encoded; store the character group corresponding to each final allocation space into the corresponding index range to obtain the index dictionary of the data segment to be encoded.

6. The method for compressing and storing traffic engineering monitoring data according to claim 5, characterized in that, The step of obtaining the character group corresponding to each initial allocation space in the final allocation space of the data segment to be encoded includes: For any final allocation space, during the merging process of that final allocation space, the initial allocation space of each type of character to be merged will be used as the merging allocation space of that final allocation space. Each character in the final allocation space is combined into a character group corresponding to that final allocation space.

7. The method for compressing and storing traffic engineering monitoring data according to claim 1, characterized in that, The process of encoding the data segment to be encoded using the LZW algorithm through an index dictionary to obtain compressed encoding includes: For any data segment to be encoded, obtain the character index table based on the index value of each type of character in the index dictionary; During the encoding and compression process of the data segment to be encoded using the LZW algorithm, the index value of the first character in the string is used to search and store the corresponding index position in the index dictionary, and the index dictionary is updated until the data segment to be encoded is fully encoded and the compressed code of the data segment is obtained.

8. The method for compressing and storing traffic engineering monitoring data according to claim 7, characterized in that, The retrieval and storage methods include: During the process of searching the string backwards from the index position, when an empty space is found in the index dictionary, the corresponding string is stored in the corresponding empty space in the index dictionary.

9. The method for compressing and storing traffic engineering monitoring data according to claim 1, characterized in that, The process of dividing traffic data into segments to be encoded based on the data collection frequency includes: Obtain the data acquisition frequency, calculate all data acquisition frequencies within a preset time period as a frequency sequence; calculate and normalize the variance of the data acquisition frequencies in the frequency sequence to obtain the frequency fluctuation. When the frequency fluctuation is less than or equal to the preset fluctuation threshold, the adjacent data collection frequency after the preset time period is assigned to the frequency sequence, and the frequency fluctuation is iteratively calculated; the iteration stops when the frequency fluctuation is greater than the preset fluctuation threshold or there is no new data collection frequency; the traffic data corresponding to all data collection frequencies in the final frequency sequence are taken as the data segments to be encoded.

10. The method for compressing and storing traffic engineering monitoring data according to claim 5, characterized in that, The rounding method is to round down.

Images