A data compression method, device, system, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data compression method, device, system, electronic equipment and storage medium, wherein the data compression method comprises: receiving log data to be processed of at least one base station managed by the network management device; screening uncompressed content from the log data to be processed according to a pre-constructed log compression data dictionary; updating the log compression data dictionary according to the uncompressed content to obtain an updated data dictionary; and compressing the uncompressed content according to the updated data dictionary to obtain compressed data.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a data compression method, apparatus, system, electronic device, and storage medium. Background Technology

[0002] With the development of mobile networks, base stations have evolved from single sites in the 2G era to distributed sites in 5G, and the number of network elements in a single base station has increased several times. For example, a 2G macro base station is a single network element, while a 5G macro base station is divided into CU network elements, DU network elements, and RRU network elements. 5G pico base stations can be expanded to more than 100 network elements. For example, the service volume of a single site has evolved from 8 voice users carried by a single carrier in the 2G era to 1200 data users carried by a single carrier in 5G. Correspondingly, the base station logs used to record various information during the daily operation of the base station need to record an exponential increase in content. This leads to the following problems: (1) Network management needs to spend a lot of storage resources; taking 10,000 base stations in a city as an example, 2G, 3G, 4G and 5G together are 40,000 base stations. Assuming that each base station has 100M logs per hour and the network management needs to keep them for at least 7 days, it will require 672,000TB of storage resources. (2) Limited upload bandwidth; based on each network administrator managing 5000 base stations, at least 139M of effective upload bandwidth is required. The above problem can be avoided by forcibly reducing the base station log size. However, once the base station log size is forcibly reduced, a lot of base station operation information will be missed or even lost, resulting in the inability to effectively locate base station faults and poor communication quality. Summary of the Invention

[0003] This invention provides a data compression method, apparatus, system, electronic device, and storage medium to reduce base station log space usage while improving compatibility among multiple devices.

[0004] In a first aspect, embodiments of the present invention provide a data compression method, applied to a network management device, comprising:

[0005] Receive pending log data from at least one base station managed by the network management device;

[0006] Based on a pre-built log compression data dictionary, uncompressed content is filtered out from the log data to be processed;

[0007] Update the log compressed data dictionary based on the uncompressed content to obtain the updated data dictionary;

[0008] Based on the updated data dictionary, the uncompressed content is compressed to obtain compressed data.

[0009] In one possible implementation, after obtaining the updated data dictionary, the method further includes:

[0010] Store the updated data dictionary and the uncompressed content.

[0011] In one possible implementation, after compressing the uncompressed content according to the updated data dictionary to obtain compressed data, the method further includes:

[0012] The compressed data is stored so that when the network management device receives a query request from a user, it can determine the uncompressed content corresponding to the compressed data based on the updated data dictionary, wherein the query request includes an online read request or a download request.

[0013] In one possible implementation, after obtaining the updated data dictionary, the method further includes:

[0014] The updated data dictionary is sent to the at least one base station so that the at least one base station compresses incremental data according to the updated data dictionary, wherein the incremental data is data obtained by parsing the log files of each base station.

[0015] In one possible implementation, compressing the uncompressed content according to the updated data dictionary to obtain compressed data includes:

[0016] Extract the target string from the uncompressed content;

[0017] Based on the updated data dictionary, determine the target index value corresponding to the target string;

[0018] The target string is replaced with the target index value to obtain target data including the target index value, and the target data is used as the compressed data.

[0019] Secondly, embodiments of the present invention provide a data compression device, applied at a network management end, comprising:

[0020] A receiving unit is configured to receive log data to be processed from at least one base station managed by the network management device;

[0021] The filtering unit is used to filter out uncompressed content from the log data to be processed based on a pre-built log compression data dictionary;

[0022] The update unit is used to update the log compressed data dictionary according to the uncompressed content to obtain the updated data dictionary;

[0023] The obtaining unit is used to compress the uncompressed content according to the updated data dictionary to obtain compressed data.

[0024] In one possible implementation, the apparatus further includes a first storage unit, which is used to store the updated data dictionary and the uncompressed content after the update unit updates the log compressed data dictionary according to the uncompressed content and obtains the updated data dictionary.

[0025] In one possible implementation, the device further includes a second storage unit, which is used to store the compressed data after the obtaining unit compresses the uncompressed content according to the updated data dictionary to obtain compressed data, so that when the network management device receives a query request from a user, it can determine the uncompressed content corresponding to the compressed data according to the updated data dictionary, wherein the query request includes an online reading request or a download request.

[0026] In one possible implementation, the apparatus further includes a sending unit, which is configured to send the updated data dictionary to the at least one base station after the updating unit updates the log compression data dictionary according to the uncompressed content and obtains the updated data dictionary, so that the at least one base station compresses incremental data according to the updated data dictionary, wherein the incremental data is data obtained by parsing the log files of each base station.

[0027] In one possible implementation, the obtaining unit is specifically used for:

[0028] Extract the target string from the uncompressed content;

[0029] Based on the updated data dictionary, determine the target index value corresponding to the target string;

[0030] The target string is replaced with the target index value to obtain target data including the target index value, and the target data is used as the compressed data.

[0031] Thirdly, embodiments of the present invention provide a data compression system, including:

[0032] Network management equipment and at least one base station managed by said network management equipment;

[0033] The network management device is configured as follows:

[0034] Receive log data to be processed from at least one base station;

[0035] Based on a pre-built log compression data dictionary, uncompressed content is filtered out from the log data to be processed;

[0036] Update the log compressed data dictionary based on the uncompressed content to obtain the updated data dictionary;

[0037] Based on the updated data dictionary, the uncompressed content is compressed to obtain compressed data.

[0038] Fourthly, embodiments of the present invention provide an electronic device, the electronic device including a processor, the processor being configured to execute a computer program stored in a memory to implement the steps of the data compression method as described in any of the preceding claims.

[0039] Fifthly, embodiments of the present invention provide a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data compression method as described in any of the preceding claims.

[0040] The beneficial effects of this invention are as follows:

[0041] This invention provides a data compression method, apparatus, system, electronic device, and storage medium. For a network management device, it first receives log data to be processed from at least one base station managed by the device, such as operation logs, fault logs, and other base station logs. Then, based on a pre-built log compression data dictionary, uncompressed content is filtered from the log data to be processed. Next, the log compression data dictionary is updated based on the uncompressed content, resulting in an updated data dictionary. Finally, the uncompressed content in the log data to be processed is compressed based on the updated data dictionary to obtain compressed data. On one hand, the entire data compression process involves the network management device managing at least one base station constructing the log compression data dictionary. Correspondingly, the log compression data dictionary is generated by the network management device through analysis. This allows for full consideration of the different characteristics of each base station managed by the network management device, thereby improving compatibility among multiple base stations. On the other hand, the log compression data dictionary can be dynamically updated in real time based on the uncompressed content in the log data to be processed received by the network management device. Subsequent compression of the log data to be processed based on the updated data dictionary to obtain compressed data reduces the space occupancy rate of base station logs. In this way, the space occupied by base station logs is reduced while the compatibility between multiple devices is improved. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of a mobile wireless access network topology to which a data compression method provided in an embodiment of the present invention is applicable;

[0043] Figure 2A flowchart illustrating one method of a data compression method provided in an embodiment of the present invention;

[0044] Figure 3 for Figure 2 Flowchart of one method for step S104;

[0045] Figure 4 A flowchart illustrating the interaction between a network management device and the various base stations it manages in a data compression method provided in an embodiment of the present invention.

[0046] Figure 5 for Figure 4 Flowchart of one method for updating the pre-built log compression data dictionary in step S03;

[0047] Figure 6 This is a structural block diagram of a data compression device provided in an embodiment of the present invention;

[0048] Figure 7 This is a structural block diagram of a data compression system provided in an embodiment of the present invention. Detailed Implementation

[0049] In the specification, claims, and accompanying drawings of this invention, the terms "first," "second," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0050] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0051] To better understand the above technical solutions, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0052] In related technologies, log compression can be performed at the base station using a pre-prepared index table. However, since mobile network base station equipment is often from multiple vendors, and different vendors have different log designs, it is impossible to establish a unified index table, resulting in low compression efficiency. Moreover, reading logs requires translating the logs back from the index table, which necessitates dedicated translation tools. Furthermore, even for the same base station equipment from the same vendor, continuous upgrades and iterations require different index tables for different versions, necessitating different translation tools for each version. With each base station equipment update, the dedicated translation tools also need to be updated, leading to poor universality.

[0053] In view of this, embodiments of the present invention provide a data compression method, apparatus, system, electronic device, and storage medium, which can reduce the space occupied by base station logs while improving compatibility among multiple devices.

[0054] Before introducing the data compression method provided in the embodiments of the present invention, in conjunction with Figure 1 The illustrated mobile radio access network topology diagram explains the application scenarios applicable to this data compression method. Figure 1 The topology diagram shown includes a network management system, multiple base stations, and a core network. The base stations connect to the core network via service interfaces (i.e., user plane and control plane data). The base stations connect to the network management system via southbound interfaces. Furthermore, the number of base stations is not limited to a fixed set. Figure 1 The number shown is not specified here. The number and location of base stations can be deployed according to actual application needs, and are not limited here.

[0055] like Figure 2 As shown, an embodiment of the present invention provides a data compression method, the method comprising:

[0056] S101: Receive log data to be processed from at least one base station managed by the network management device;

[0057] In practical implementation, the network management device can manage one or more base stations. The specific number of base stations can be set according to actual application needs and is not limited here. Each base station managed by the same network management device can collect corresponding log data and upload the collected log data to the corresponding network management device. The network management device then receives the log data to be processed from the at least one base station it manages. In one exemplary embodiment, the log data to be processed can be a data set of information recorded over a period of time. In practical implementation, the log data to be processed is specifically the base station log from the corresponding base station, which can be at least one of operation logs, fault logs, and logs of signaling and protocol interaction processes. Of course, the type of log data to be processed received by the network management device can also be set according to actual application needs and is not limited here.

[0058] S102: Based on the pre-built log compression data dictionary, filter out uncompressed content from the log data to be processed;

[0059] In practical implementation, after receiving log data to be processed from at least one base station under its management, the network management device can filter out uncompressed content from the log data based on a pre-built log compression data dictionary. In one exemplary embodiment, the pre-built log compression data dictionary can be a dictionary pre-built through natural language processing. In another exemplary embodiment, the pre-built log compression data dictionary can also be a dictionary pre-built through incremental learning. Of course, the required log compression data dictionary can also be pre-built according to the actual application needs, which is not limited here.

[0060] In one exemplary embodiment, the log compression data dictionary can be a mapping table including strings and index values. In this exemplary embodiment, after the network management device receives the log data to be processed from the base station it manages, it can determine the string corresponding to each piece of information in the log data. Then, based on the mapping table between strings and index values ​​in the log compression data dictionary, it can determine whether the corresponding piece of information matches the data in the log compression data dictionary. If they match, it indicates that the piece of information is compressed. If there is no matching data in the log compression data dictionary, it indicates that the piece of information is uncompressed.

[0061] S103: Update the log compression data dictionary according to the uncompressed content to obtain the updated data dictionary;

[0062] In practical implementation, after the network management device filters out uncompressed content from the log data to be processed, it can update the pre-built log compression data dictionary in real time based on this uncompressed content, thus obtaining the updated data dictionary. For example, a machine learning algorithm can be used to update the pre-built log compression data dictionary based on the uncompressed content. In one exemplary embodiment, a self-organizing map (SOM) neural network machine learning algorithm can be used to update the pre-built log compression data dictionary in real time, thereby ensuring the compression accuracy of the log compression data dictionary.

[0063] S104: Based on the updated data dictionary, compress the uncompressed content to obtain compressed data.

[0064] In practice, after obtaining the updated data dictionary, the uncompressed content can be compressed based on the updated data dictionary to obtain compressed data. This reduces the space usage of base station logs.

[0065] In practice, the entire data compression process involves a network management device managing at least one base station constructing the log compression data dictionary. This dictionary is generated by the network management device through analysis, allowing for full consideration of the different characteristics of each base station managed by the device, thus improving compatibility among multiple base stations. Furthermore, the log compression data dictionary can be dynamically updated in real-time based on the uncompressed content in the log data received by the network management device. Subsequent compression of the uncompressed content is then performed based on the updated dictionary, resulting in compressed data and reducing the space occupancy of base station logs. In this way, while reducing base station log space usage, compatibility among multiple devices is improved.

[0066] In this embodiment of the invention, after obtaining the updated data dictionary in step S103, the method further includes:

[0067] Store the updated data dictionary and the uncompressed content.

[0068] In practice, the data dictionary can be compressed based on the uncompressed content update log. After obtaining the updated data dictionary, both the updated data dictionary and the uncompressed content can be stored. This makes it possible for subsequent network management devices to compress other uncompressed data.

[0069] In this embodiment of the invention, after step S104: compressing the uncompressed content according to the updated data dictionary to obtain compressed data, the method further includes:

[0070] The compressed data is stored so that when the network management device receives a query request from a user, it can determine the uncompressed content corresponding to the compressed data based on the updated data dictionary, wherein the query request includes an online read request or a download request.

[0071] In practical implementation, after compressing the uncompressed content according to the updated data dictionary to obtain compressed data, the compressed data can be stored. This way, when the network management device receives a query request from a user, it can determine the uncompressed content corresponding to the compressed data based on the updated data dictionary. In one exemplary embodiment, if the query request is an online read request, the user can directly read the uncompressed data corresponding to the compressed data online through the network management device. The entire process does not require a dedicated translation tool to read logs, thereby improving the efficiency of users reading log data online. In another exemplary embodiment, if the query request is a download request, the user can directly download the uncompressed data corresponding to the compressed data through the network management device. The entire process does not require a dedicated translation tool, thereby improving the efficiency of users downloading log data.

[0072] In this embodiment of the invention, after step S103: updating the log compression data dictionary according to the uncompressed content to obtain the updated data dictionary, the method further includes:

[0073] The updated data dictionary is sent to the at least one base station so that the at least one base station compresses incremental data according to the updated data dictionary, wherein the incremental data is data obtained by parsing the log files of each base station.

[0074] In the specific implementation process, after updating the data dictionary based on the compressed content of the log and obtaining the updated data dictionary, the updated data dictionary is sent to at least one base station managed by the network management equipment. Upon receiving the updated data dictionary, at least one base station can compress the incremental data accordingly. This reduces the effective upload bandwidth of each base station. It should be noted that the incremental data is obtained by parsing the log files of each base station. After receiving the updated data dictionary, at least one base station can also update its original data dictionary, thus achieving real-time updates of the base station-side data dictionary and ensuring log compression efficiency. Furthermore, since the base station can directly collect log data, users can directly view uncompressed log data at the base station without the need for specialized translation tools, thereby improving the user experience.

[0075] In embodiments of the present invention, such as Figure 3As shown, step S104: Based on the updated data dictionary, compress the uncompressed content to obtain compressed data, including:

[0076] S201: Extract the target string from the uncompressed content;

[0077] S202: Determine the target index value corresponding to the target string based on the updated data dictionary;

[0078] S203: Replace the target string with the target index value to obtain target data including the target index value, and use the target data as the compressed data.

[0079] In the specific implementation process, steps S201 to S203 are implemented as follows:

[0080] After obtaining the updated data dictionary, the network management device can extract the target string from the uncompressed content. Then, based on the updated data dictionary, it can determine the target index value corresponding to that target string. The target string can then be replaced with the target index value to obtain the target data including the target index value. This target data is the compressed data, and its space usage is significantly smaller than the uncompressed data to be processed. In this way, the network management device achieves compression of the log data to be processed, reducing the space usage of the log data.

[0081] In one exemplary embodiment, the log data to be processed includes five pieces of information as shown in ①-⑤ below:

[0082] ①:[24448:0828 / 132829.627:W:profile_data.cpp(135)]profile data initvid:1688852038150851;

[0083] ②:[24448:0828 / 132829.642:W:profile_data.cpp(135)]profile data initvid:1688854270367847;

[0084] ③:[24448:0828 / 132829.642:W:profile_data.cpp(135)]profile data initvid:0;

[0085] ④:[24448:0828 / 132829.642:W:profile_data.cpp(135)]profile data initvid:1688851899522554;

[0086] ⑤:[24448:0828 / 132829.642:W:profile_data.cpp(135)]profile data initvid:0.

[0087] If we cluster the string "W:profile_data.cpp(" and the string ")]profiledata init vid:" according to SOM clustering, the first entry of the updated data dictionary will be:

[0088] {key:”profile_data”,value:”W:profile_data.cpp(%d)]profile data initvid:%d“}. Accordingly, the updated data dictionary includes a key value to represent the index value, and a string value to represent the corresponding key value.

[0089] After compressing the five pieces of information shown in ①-⑤ above according to the entries of the data dictionary, the compressed data are as follows: ①' represents the compressed content of information ① under the updated data dictionary entry; ②' represents the compressed content of information ② under the updated data dictionary entry; ③' represents the compressed content of information ③ under the updated data dictionary entry; ④' represents the compressed content of information ④ under the updated data dictionary entry; and ⑤' represents the compressed content of information ⑤ under the updated data dictionary entry.

[0090] ①':[24448:0828 / 132829.627:profile_data(135,1688852038150851);

[0091] ②':[2829.642:profile_data(135,1688854270367847);

[0092] ③':[24448:0828 / 132829.642:profile_data(135,0);

[0093] ④':[24448:0828 / 132829.642:profile_data(135,1688851899522554);

[0094] ⑤':[24448:0828 / 132829.642:profile_data(135,0).

[0095] Comparing information ① and information ①', the string before compression is W:profile_data.cpp(135)]profiledata init vid:1688852038150851. After compression of the first entry of the updated data dictionary, information ① becomes profile_data(135,1688852038150851). It is clear that the space occupied is significantly reduced after data dictionary compression. Furthermore, other exemplary embodiments will not be elaborated further.

[0096] In one exemplary embodiment, taking the log messages shown in ①-⑤ as an example, if the string "W:profile_data.cpp(135)]profile datainit vid:" is clustered according to SOM, the second entry of the updated data dictionary is:

[0097] {key:”p_135”,value:”W:profile_data.cpp(135)]profile data init vid:%d”}.

[0098] Accordingly, the updated data dictionary includes key values ​​to represent index values ​​and string value values ​​to represent the corresponding key values.

[0099] After compressing the five pieces of information ①-⑤ mentioned above according to the data dictionary, the compressed data are as follows: ⑥' represents the compressed content of information ① under entry two of the updated data dictionary; ⑦' represents the compressed content of information ② under entry two of the updated data dictionary; ⑧' represents the compressed content of information ③ under entry two of the updated data dictionary; ⑨' represents the compressed content of information ④ under entry two of the updated data dictionary; and ⑩' represents the compressed content of information ⑤ under entry two of the updated data dictionary.

[0100] ⑥':[24448:0828 / 132829.627:p_135(1688852038150851);

[0101] ⑦':[24448:0828 / 132829.642:p_135(1688854270367847);

[0102] ⑧':[24448:0828 / 132829.642:p_135(0);

[0103] ⑨':[24448:0828 / 132829.642:p_135(1688851899522554);

[0104] ⑩':[24448:0828 / 132829.642:p_135(0).

[0105] Comparing information ① and information ⑥', the string before compression is W:profile_data.cpp(135)]profiledata init vid:1688852038150851. After compression of entry two of the updated data dictionary, information ① becomes p_135(1688852038150851). It can be clearly seen that the space occupied is significantly reduced after data dictionary compression. In addition, other exemplary embodiments will not be described in detail.

[0106] The inventors have discovered that regardless of the type of data dictionary used, compressing log data reduces space usage to some extent. Of course, besides the data dictionaries mentioned above, other types of data dictionaries can be used to compress log data according to actual application needs; this is not limited here.

[0107] The following is combined Figure 4 The flowchart illustrating the interaction between the network management device and the various base stations it manages explains the data compression method provided in this embodiment of the invention. The network management device also includes a log analysis module and a log storage module.

[0108] Step S01: The base station uploads the corresponding logs to the network management device through the southbound interface;

[0109] Step S02: The network management device transfers the received logs to its log analysis module;

[0110] Step S03: The log analysis module processes the log files using methods such as natural language processing and incremental learning to generate a log compressed data dictionary;

[0111] In one exemplary embodiment, log files can be processed using machine learning models such as natural language processing and incremental learning to generate a log compression data dictionary. For example, unsupervised learning methods such as SOM (Structured Object Model) can be used to generate the log compression data dictionary through clustering. In practical applications, a machine learning model can be used to pre-build the log compression data dictionary, and the network management device can then continuously update the pre-built log compression data dictionary based on the collected log files.

[0112] Step S04: The log analysis module transfers the generated log compressed data dictionary and the collected base station logs to the log storage module;

[0113] Step S05: The log storage module compresses and stores the corresponding base station logs according to the log compression data dictionary;

[0114] Step S06: The log analysis module sends the data dictionary to the base station through the network management device and the southbound interface;

[0115] Step S07: The base station compresses subsequent incremental logs based on the received data dictionary;

[0116] Step S08: Repeat steps S01 to S07.

[0117] The specific implementation process of steps S01 to S07 can be referred to the description in the relevant sections above, and will not be repeated here.

[0118] It should be noted that the update process of the pre-built log compression data dictionary in step S03 can be implemented using... Figure 5 The flowchart of the method is shown.

[0119] S10: Log collection from each base station;

[0120] S11: Beginning;

[0121] S12: The network management device reads data from each log;

[0122] S13: Obtain the pre-built log compression data dictionary;

[0123] S14: Filter out uncompressed content from the read logs;

[0124] S15: Calculate and update the pre-built log compression data dictionary based on the uncompressed content using machine learning;

[0125] It should be noted that uncompressed content can be filtered out from the read logs based on a pre-built log compression data dictionary. In one exemplary embodiment, uncompressed content can be filtered out by determining whether the data in the read logs is uncompressed. For example, the read log data can be matched with the log data in the pre-built log compression data dictionary. If they match, it indicates that the corresponding data in the read logs is compressed; if they do not match, it indicates that the corresponding data in the read logs is uncompressed. If the data in the read logs is already compressed, this part can be ignored, and there is no need to update the pre-built log compression dictionary.

[0126] In the specific implementation process, the specific implementation process of steps S10 to S15 can be referred to the description of the relevant parts mentioned above, and will not be repeated here.

[0127] Based on the same inventive concept, such as Figure 6 As shown, this embodiment of the invention provides a data compression device applied to a network management terminal, comprising:

[0128] The receiving unit 10 is used to receive log data to be processed from at least one base station managed by the network management device;

[0129] The filtering unit 20 is used to filter out uncompressed content from the log data to be processed based on a pre-built log compression data dictionary;

[0130] Update unit 30 is used to update the log compressed data dictionary according to the uncompressed content to obtain the updated data dictionary;

[0131] The obtaining unit 40 is used to compress the uncompressed content according to the updated data dictionary to obtain compressed data.

[0132] In this embodiment of the invention, the device further includes a first storage unit, which is used to store the updated data dictionary and the uncompressed content after the update unit 30 updates the log compressed data dictionary according to the uncompressed content and obtains the updated data dictionary.

[0133] In this embodiment of the invention, the device further includes a second storage unit, which is used to store the compressed data after the obtaining unit 40 compresses the uncompressed content according to the updated data dictionary to obtain compressed data, so that when the network management device receives a query request from a user, it can determine the uncompressed content corresponding to the compressed data according to the updated data dictionary, wherein the query request includes an online reading request or a download request.

[0134] In this embodiment of the invention, the apparatus further includes a sending unit, which is configured to send the updated data dictionary to the at least one base station after the update unit 30 updates the log compression data dictionary according to the uncompressed content and obtains the updated data dictionary, so that the at least one base station compresses incremental data according to the updated data dictionary, wherein the incremental data is data obtained by parsing the log files of each base station.

[0135] In one possible implementation, the obtaining unit 40 is used to:

[0136] Extract the target string from the uncompressed content;

[0137] Based on the updated data dictionary, determine the target index value corresponding to the target string;

[0138] The target string is replaced with the target index value to obtain target data including the target index value, and the target data is used as the compressed data.

[0139] Based on the same inventive concept, such as Figure 7 As shown, an embodiment of the present invention provides a data compression system, including:

[0140] Network management device 100 and at least one base station 200 managed by said network management device 100;

[0141] The network management device 100 is configured as follows:

[0142] Receive log data to be processed from at least one base station;

[0143] Based on a pre-built log compression data dictionary, uncompressed content is filtered out from the log data to be processed;

[0144] Update the log compressed data dictionary based on the uncompressed content to obtain the updated data dictionary;

[0145] Based on the updated data dictionary, the uncompressed content is compressed to obtain compressed data.

[0146] In this embodiment of the invention, after obtaining the updated data dictionary, the network management device 100 is further configured to:

[0147] Store the updated data dictionary and the uncompressed content.

[0148] In this embodiment of the invention, after compressing the uncompressed content according to the updated data dictionary to obtain compressed data, the network management device 100 is further configured to:

[0149] The compressed data is stored so that when the network management device receives a query request from a user, it can determine the uncompressed content corresponding to the compressed data based on the updated data dictionary, wherein the query request includes an online read request or a download request.

[0150] In this embodiment of the invention, after obtaining the updated data dictionary, the network management device 100 is further configured to:

[0151] The updated data dictionary is sent to the at least one base station so that the at least one base station compresses incremental data according to the updated data dictionary, wherein the incremental data is data obtained by parsing the log files of each base station.

[0152] In this embodiment of the invention, the network management device is configured as follows:

[0153] Extract the target string from the uncompressed content;

[0154] Based on the updated data dictionary, determine the target index value corresponding to the target string;

[0155] The target string is replaced with the target index value to obtain target data including the target index value, and the target data is used as the compressed data.

[0156] In this embodiment of the invention, if the data to be processed is compressed content, the log compression data dictionary is kept unchanged.

[0157] Based on the same inventive concept, embodiments of the present invention provide an electronic device, the electronic device including a processor, the processor being configured to execute a computer program stored in a memory to implement the steps of the data compression method as described in any of the above claims.

[0158] Based on the same inventive concept, embodiments of the present invention provide a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data compression method as described in any of the above claims.

[0159] This invention provides a data compression method, apparatus, system, electronic device, and storage medium. For a network management device, it first receives log data to be processed from at least one base station managed by the device, such as operation logs, fault logs, and other base station logs. Then, based on a pre-built log compression data dictionary, uncompressed content is filtered from the log data to be processed. Next, the log compression data dictionary is updated based on the uncompressed content, resulting in an updated data dictionary. Finally, the uncompressed content in the log data to be processed is compressed based on the updated data dictionary to obtain compressed data. On one hand, the entire data compression process involves the network management device managing at least one base station constructing the log compression data dictionary. Correspondingly, the log compression data dictionary is generated by the network management device through analysis. This allows for full consideration of the different characteristics of each base station managed by the network management device, thereby improving compatibility among multiple base stations. On the other hand, the log compression data dictionary can be dynamically updated in real time based on the uncompressed content in the log data to be processed received by the network management device. Subsequent compression of the log data to be processed based on the updated data dictionary to obtain compressed data reduces the space occupancy rate of base station logs. In this way, the space occupied by base station logs is reduced while the compatibility between multiple devices is improved.

[0160] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0162] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0164] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data compression method, applied to network management equipment of a mobile wireless access network, characterized in that, include: Receive pending log data from at least one base station managed by the network management device; The log data to be processed is at least one of operation logs, fault logs, and logs of signaling and protocol interaction processes; the network management device is connected to the at least one base station through a southbound interface; Based on a pre-built log compression data dictionary, uncompressed content is filtered out from the log data to be processed; The log compressed data dictionary is updated based on the uncompressed content to obtain the updated data dictionary, and the updated data dictionary is stored. Based on the updated data dictionary, the uncompressed content is compressed to obtain compressed data, and the compressed data is stored. The updated data dictionary is sent to the at least one base station, so that the at least one base station compresses incremental data according to the updated data dictionary, wherein the incremental data is data obtained by parsing the log files of each base station; When a query request is received from a user, the uncompressed content corresponding to the compressed data is determined according to the updated data dictionary, wherein the query request includes an online read request or a download request.

2. The method as described in claim 1, characterized in that, The step of compressing the uncompressed content according to the updated data dictionary to obtain compressed data includes: Extract the target string from the uncompressed content; Based on the updated data dictionary, determine the target index value corresponding to the target string; The target string is replaced with the target index value to obtain target data including the target index value, and the target data is used as the compressed data.

3. A data compression device, applied to the network management terminal of a mobile wireless access network, characterized in that, include: A receiving unit is configured to receive log data to be processed from at least one base station managed by the network management device; The log data to be processed is at least one of operation logs, fault logs, and logs of signaling and protocol interaction processes; the network management device is connected to the at least one base station through a southbound interface; The filtering unit is used to filter out uncompressed content from the log data to be processed based on a pre-built log compression data dictionary; An update unit is used to update the log compressed data dictionary according to the uncompressed content, obtain the updated data dictionary, and store the updated data dictionary; The obtaining unit is configured to compress the uncompressed content according to the updated data dictionary, obtain compressed data, and store the compressed data; A sending unit is configured to send the updated data dictionary to the at least one base station, so that the at least one base station compresses incremental data according to the updated data dictionary, wherein the incremental data is data obtained by parsing the log files of each base station; The network management terminal is used to determine the uncompressed content corresponding to the compressed data based on the updated data dictionary when receiving a query request from a user, wherein the query request includes an online read request or a download request.

4. The apparatus as described in claim 3, characterized in that, The obtaining unit is specifically used for: Extract the target string from the uncompressed content; Based on the updated data dictionary, determine the target index value corresponding to the target string; The target string is replaced with the target index value to obtain target data including the target index value, and the target data is used as the compressed data.

5. A data compression system applied to a mobile wireless access network, characterized in that, include: Network management equipment and at least one base station managed by said network management equipment; The network management device is configured as follows: The network management device receives log data to be processed from at least one base station; the log data to be processed is at least one of operation logs, fault logs, and logs of signaling and protocol interaction processes; the network management device is connected to the at least one base station through a southbound interface. Based on a pre-built log compression data dictionary, uncompressed content is filtered out from the log data to be processed; The log compressed data dictionary is updated based on the uncompressed content to obtain the updated data dictionary, and the updated data dictionary is stored. Based on the updated data dictionary, the uncompressed content is compressed to obtain compressed data, and the compressed data is stored. The updated data dictionary is sent to the at least one base station, so that the at least one base station compresses incremental data according to the updated data dictionary, wherein the incremental data is data obtained by parsing the log files of each base station; When a query request is received from a user, the uncompressed content corresponding to the compressed data is determined according to the updated data dictionary, wherein the query request includes an online read request or a download request.

6. An electronic device, characterized in that, The electronic device includes a processor that executes a computer program stored in a memory to implement the steps of the data compression method as described in claim 1 or 2.

7. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data compression method as described in claim 1 or 2.

Images