Call record data compression transmission method, device and equipment and storage medium

Abstract

The application discloses a kind of call bill data compression transmission method, device, equipment and storage medium, belong to data transmission technical field.The application is generated by obtaining current call bill data after service processing;Load target hoffman coding dictionary of word dimension from distributed cache;The current call bill data is compressed based on the target hoffman coding dictionary, and compressed call bill data is obtained;The compressed call bill data is transmitted, by constructing target hoffman coding dictionary of word dimension, and using it to the current call bill data is compressed processing, target hoffman coding dictionary is stored in distributed cache, without passing through encoding dictionary in message, online transmission call bill data is greatly reduced, improve the efficiency of call bill data compression.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, and in particular to a bill data compression transmission method, device, equipment and storage medium. BACKGROUND

[0002] Traditional cloud services are charged for resource-based products, such as user renting cloud host services. During use, the service platform outputs fixed billing bills to the billing system in modes such as per hour and per day, and the billing system calculates the cost according to the use time period, use product category, and billing dimension information provided in the bill, and charges the user. Since a bill is output for a relatively long period of time, the total number of billing bills is controllable, and the pressure of supporting system online data flow processing is not large, so the current system uses no compression or some common compression methods in the industry. The single bill is compressed and transmitted independently before transmission. With the output of real-time service usage behavior of users, the number of bills will increase explosively, and a large amount of data transmission across the east and west will require efficient compression of transmitted data to reduce network pressure.

[0003] The current common compression method is to perform dictionary encoding on the character level. That is, the probability of occurrence of each character in the text to be compressed is counted, and a coding dictionary of each character mapping coding value is created based on the self-encoding method, and then the text is encoded and compressed based on the coding dictionary. However, in the billing bill scenario, a single bill is less than 500 bytes, the text is short, and the repeated part is also less, so the compression effect is relatively limited. SUMMARY

[0004] The main purpose of the present application is to provide a bill data compression transmission method, device, equipment and storage medium, which aims to solve the technical problem of low compression efficiency of bill data in the prior art.

[0005] To achieve the above purpose, the present application provides a bill data compression transmission method, which comprises the following steps:

[0006] Obtaining current bill data generated after service processing;

[0007] Loading a target Huffman coding dictionary of word dimension from a distributed cache;

[0008] Compressing the current bill data based on the target Huffman coding dictionary to obtain compressed bill data;

[0009] Transmitting the compressed bill data.

[0010] Optionally, the compression of the current bill data based on the target Huffman coding dictionary to obtain compressed bill data comprises:

[0011] perform word segmentation processing on the current bill data using a preset word segmentation strategy to obtain current bill word data;

[0012] map each bill word in the current bill word data to a corresponding encoding value based on the target Huffman encoding dictionary to obtain a mapping result;

[0013] merge the mapping result to obtain compressed bill data.

[0014] Optionally, before the mapping each bill word in the current bill word data to a corresponding encoding value based on the target Huffman encoding dictionary to obtain a mapping result, the method further comprises:

[0015] query whether there is an uncoded word in the target Huffman encoding dictionary through the current bill word data;

[0016] when there is an uncoded word in the target Huffman encoding dictionary, compress the current bill word data using a preset compression algorithm to obtain compressed bill data.

[0017] Optionally, before the loading the target Huffman encoding dictionary of word dimension from the distributed cache, the method further comprises:

[0018] obtain historical bill data of a preset time period;

[0019] divide and count the historical bill data based on a preset word length to obtain a number of words in the historical bill words that meet a preset condition, and obtain a statistical distribution set;

[0020] obtain single-character data in the statistical distribution set, and obtain a first word set according to the single-character data;

[0021] take the first word set as a current optimal word set, and obtain a second word set according to the statistical distribution set and the single-character data;

[0022] obtain an initial word weight distribution through the first word set and the statistical distribution set;

[0023] construct an initial Huffman encoding dictionary based on the initial word weight distribution;

[0024] construct a target Huffman encoding dictionary through the first word set, the second word set, the statistical distribution set, and the initial Huffman encoding dictionary;

[0025] store the target Huffman encoding dictionary to a distributed cache.

[0026] Optionally, the constructing a target Huffman encoding dictionary through the first word set, the second word set, the statistical distribution set, and the initial Huffman encoding dictionary comprises:

[0027] compressing the historical bill data by the initial Huffman coding dictionary to obtain a current maximum compression ratio;

[0028] obtaining a target selection word from the second word set;

[0029] adding the target selection word to the first word set to obtain an updated first word set, and deleting the target selection word from the second word set;

[0030] calculating a reference word weight distribution according to the statistical distribution set and the updated first word set;

[0031] constructing a reference Huffman coding dictionary based on the reference word weight distribution;

[0032] compressing the historical bill data by the reference Huffman coding dictionary to obtain a reference compression ratio;

[0033] when the reference compression ratio is greater than the current maximum compression ratio, taking the reference Huffman coding dictionary as a target Huffman coding dictionary.

[0034] Optionally, the obtaining a target selection word from the second word set comprises:

[0035] performing word segmentation processing on each word in the second word set by the current optimal word set to obtain a word segmentation result set comprising all words in the second word set;

[0036] taking the initial Huffman coding dictionary as a current optimal coding dictionary;

[0037] calculating the length of the encoding value of each element in the word segmentation result set in the current optimal coding dictionary;

[0038] calculating a first statistical value of each element in the word segmentation result set in the statistical distribution set, and calculating a second statistical value of each word in the second word set in the statistical distribution set;

[0039] calculating an encoding estimated value of each word in the second word set by the first statistical value and the second statistical value and the length of the encoding value;

[0040] obtaining a maximum encoding estimated value from the encoding estimated value;

[0041] taking the word corresponding to the maximum encoding estimated value in the second word set as a target selection word.

[0042] Optionally, the calculating a reference word weight distribution according to the statistical distribution set and the updated first word set comprises:

[0043] Obtain the statistical number of all words in the updated first word set in the statistical distribution set, and obtain an updated initial word weight distribution;

[0044] Traverse the words in the updated initial word weight distribution according to the word length from large to small, and obtain a current word;

[0045] Obtain the sub-word of the current word, and obtain the weight value of the current word, the weight value of the sub-word, and the number of occurrences of the sub-word in the current word;

[0046] Calculate the updated weight value of the sub-word through the weight value of the current word, the weight value of the sub-word, and the number of occurrences;

[0047] Obtain a reference word weight distribution through the updated weight value of the sub-word.

[0048] In addition, to achieve the above object, the application further provides a call data compression transmission device, which comprises:

[0049] An acquisition module, configured to acquire current call data generated after service processing;

[0050] A loading module, configured to load a target Huffman coding dictionary of word dimension from a distributed cache;

[0051] A compression module, configured to compress the current call data based on the target Huffman coding dictionary, and obtain compressed call data;

[0052] A transmission module, configured to transmit the compressed call data.

[0053] In addition, to achieve the above object, the application further provides a call data compression transmission device, which comprises a memory, a processor, and a call data compression transmission program stored in the memory and executable on the processor, and the call data compression transmission program is configured to implement the steps of the call data compression transmission method as described above.

[0054] In addition, to achieve the above object, the application further provides a storage medium, which stores a call data compression transmission program, and the call data compression transmission program implements the steps of the call data compression transmission method as described above when executed by a processor.

[0055] The present invention obtains current call record data generated after business processing; loads a target Huffman coding dictionary of word dimension from a distributed cache; compresses the current call record data based on the target Huffman coding dictionary to obtain compressed call record data; transmits the compressed call record data, constructs a target Huffman coding dictionary of word dimension, and uses it to compress the current call record data. The target Huffman coding dictionary is stored in the distributed cache, and there is no need to transmit the coding dictionary in the message. The amount of call record data transmitted online is greatly reduced, thereby improving the efficiency of call record data compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a structural diagram of a call record data compression and transmission device in a hardware operating environment involved in an embodiment of the present invention;

[0057] Figure 2 This is a flow chart of a first embodiment of a method for compressing and transmitting call log data according to the present invention;

[0058] Figure 3 The figure is a schematic diagram of an exemplary process of compressing and encoding online stream messages;

[0059] Figure 4 This is a flow chart of a second embodiment of the method for compressing and transmitting call log data according to the present invention;

[0060] Figure 5 This is a flow chart of a third embodiment of the method for compressing and transmitting call log data according to the present invention;

[0061] Figure 6 A schematic diagram of the system architecture involved in the method for compressing and transmitting call log data of the present invention;

[0062] Figure 7 This is a structural block diagram of the first embodiment of the call bill data compression and transmission device of the present invention.

[0063] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0065] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a call record data compression and transmission device in the hardware operating environment involved in an embodiment of the present invention.

[0066] like Figure 1As shown, the bill data compression transmission device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection communication between the components. The user interface 1003 can include a display, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 can be a high-speed random access memory (RAM), and can also be a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0067] Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the bill data compression transmission device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0068] As Figure 1 As shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a bill data compression transmission program.

[0069] In Figure 1 In the bill data compression transmission device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the bill data compression transmission device can be arranged in the bill data compression transmission device, and the bill data compression transmission device calls the bill data compression transmission program stored in the memory 1005 through the processor 1001, and executes the bill data compression transmission method provided by the embodiment of the application.

[0070] The embodiment of the application provides a bill data compression transmission method, which refers to Figure 2 , Figure 2 The flowchart of the first embodiment of the bill data compression transmission method of the application is shown.

[0071] In this embodiment, the bill data compression transmission method includes the following steps:

[0072] Step S10: obtaining current bill data generated after service processing.

[0073] It should be noted that the execution subject of the embodiment can be a bill data compression transmission device, and can also be other devices that can achieve the same or similar functions. The embodiment does not limit this, and the embodiment is described by taking the bill data compression transmission device as an example.

[0074] For example, in the billing bill scenario, a single bill is less than 500 bytes, the text is short, and the repeated part is also less. The compression effect is relatively limited. At the same time, the encoding dictionary generated during compression needs to be transmitted together with the message for decompression at the other end. That is, the data after text compression includes two parts: the encoding dictionary and the compressed data. When the overall text is short, the size ratio of the dictionary data is also relatively higher. From the perspective of the total bill data, the fields in the bill, such as the billing time, the billing dimension code, and the product code, have obvious repetitive rules. Especially in the computing network billing bill business scenario, the values of the dimension code fields in the bill are in a smaller range, and the dimension code field values of different bills are more likely to be repeated. However, these rules are difficult to effectively utilize in the currently commonly used compression encoding method that takes a single bill as a unit, thereby affecting the final compression effect. For example, as shown in Figure 3 Figure 3 For example, the processing schematic diagram of online flow message compression encoding, the bill content in the billing system is transmitted between application programs in the form of a message. The message sending end program generates 3 original bill messages at different times, respectively. Each message constructs an encoding dictionary and compresses the current bill content. However, the repetitive rules in the content, which are underlined, are reflected among multiple messages. The encoding rule constructed by a single message cannot apply to such rules.

[0075] Therefore, the present application designs a Huffman coding compression method based on global data distribution. The target Huffman coding dictionary of the word dimension can be constructed in advance, and the target Huffman coding dictionary is saved in the distributed cache of the system according to the version number. Therefore, when real-time bill data compression is performed, the constructed target Huffman coding dictionary can be read from the distributed cache for compression and decompression. The repetitive rules of global data distribution are fully utilized to improve the coding compression ratio. At the same time, the time consumption of constructing an encoding dictionary for each bill is saved, and the encoding dictionary does not need to be transmitted in the compressed data, which further improves the compression speed and compression ratio.

[0076] It should be understood that the current bill data generated after service processing can be obtained by the server of the message sending end. The current bill data is real-time generated bill message data.

[0077] Step S20: loading the target Huffman coding dictionary of the word dimension from the distributed cache.

[0078] ​In a specific implementation, when the current bill data is acquired, the current bill data can be compressed, and the compressed data is transmitted. Therefore, the target Huffman coding dictionary of the word dimension can be loaded from the distributed cache. The target Huffman coding dictionary is a dictionary constructed in advance. The system can assign a version number to the constructed Huffman coding dictionaries in advance. The version number starts from 1, and is incremented by 1 each time. When the maximum value 255 is reached, the version number starts from 1 again. The Huffman coding dictionary with the version number is loaded into the distributed cache. The distributed cache retains the last three versions of the Huffman coding dictionary in the form of a dictionary.

[0079] The version number is used to manage the Huffman coding dictionary, considering that the billing bill is processed continuously at a time. During the period of updating the coding dictionary every day, there may be two versions of the coding dictionary, and the bill is used. Therefore, the coding dictionary version number is written into the compressed content during compression. When decompressing, the target Huffman coding dictionary can be queried in the distributed cache according to the read version number to decompress the current bill data.

[0080] Step S30: Compress the current bill data based on the target Huffman coding dictionary to obtain compressed bill data.

[0081] The currently used Huffman coding method usually only encodes in the single character dimension, and cannot encode the repeated values such as “10000059006” in the dimension and other fields of the algorithm billing bill.

[0082] The present embodiment optimizes the existing coding method, adds a word dimension coding method based on coding value estimation, and can further improve the compression ratio of the algorithm. The current bill data can be compressed by the target Huffman coding dictionary. The compression specifically includes word segmentation processing of the current bill data, and mapping the words processed by the word segmentation to corresponding coding values according to the target Huffman coding dictionary, so as to realize compression of the current bill data and obtain compressed bill data.

[0083] Step S40: Transmit the compressed bill data.

[0084] It can be understood that after obtaining the compressed bill data, the message sending end adds a byte in front of the compressed bill data to identify the version number of the target Huffman coding dictionary used in this coding, and sends the compressed bill data and the version number of the target Huffman coding dictionary used to the message receiving end.

[0085] Since the target Huffman coding dictionary is pre-constructed, there can be a situation that an uncoded word appears in the newly generated message. Therefore, when there is an uncoded word after the new message data is segmented, a compression algorithm commonly used in the industry is used for compression. When the compression method used is the word dimension target Huffman coding dictionary compression coding, the byte value is the version number of the target Huffman coding dictionary, and when the compression algorithm used is the rest of the coding, the byte value is filled with 0.

[0086] It should be noted that after the compressed message data transmitted is received at the message receiving end, the compressed message data can be decompressed. The steps of decompressing the compressed message data are: first reading the first byte of the message, if the value is 0, then using the corresponding compression algorithm to decompress the subsequent byte data; otherwise, according to the coding dictionary version number identified by the first byte value, reading the corresponding Huffman coding dictionary from the distributed cache for decompression, restoring the received compressed message data to the original message content for business processing, and after the processing is completed, the message is stored in the database.

[0087] The embodiment obtains the current message data generated after business processing; loads the word dimension target Huffman coding dictionary from the distributed cache; compresses the current message data based on the target Huffman coding dictionary to obtain compressed message data; and transmits the compressed message data. By constructing a word dimension target Huffman coding dictionary and using it to compress the current message data, the target Huffman coding dictionary is stored in the distributed cache, and there is no need to transmit the coding dictionary in the message. The amount of online transmission message data is greatly reduced, and the efficiency of message data compression is improved.

[0088] Reference Figure 4 , Figure 4 The flowchart of the second embodiment of the message data compression transmission method of the present application is shown.

[0089] Based on the above first embodiment, the step S40 of the message data compression transmission method of the present embodiment specifically includes:

[0090] Step S401: using a preset segmentation strategy to segment the current message data to obtain current message word data.

[0091] It should be noted that the preset segmentation strategy can be a forward maximum matching segmentation algorithm (FMM) to segment the current message data to obtain the current message word data.

[0092] As an example, since the word dimension Huffman coding dictionary is pre-constructed, there can be a situation that it is not comprehensive, so after obtaining the current message word data, it further includes:

[0093] Querying whether there is an uncoded word in the target Huffman coding dictionary through the current call word data; when there is an uncoded word in the target Huffman coding dictionary, compressing the current call word data using a preset compression algorithm to obtain compressed call data.

[0094] It should be understood that whether there is an uncoded word in the target Huffman coding dictionary can be queried through the current call word data, for example, the current call word data is a, b, c, and there is no c in the target Huffman coding dictionary, so there is an uncoded word c in the target Huffman coding dictionary.

[0095] When there is an uncoded word in the target Huffman coding dictionary, the current call word data can be directly compressed using a preset compression algorithm to obtain compressed call data. The preset compression algorithm can be a Zstd algorithm, and can also be other compression algorithms.

[0096] When there is no uncoded word in the target Huffman coding dictionary, step S402 is performed, and the current call word data is compressed and coded using the target Huffman coding dictionary.

[0097] Step S402: mapping each call word in the current call word data to a corresponding coding value based on the target Huffman coding dictionary to obtain a mapping result.

[0098] It should be noted that each word in the current call word data after word segmentation can be mapped to a corresponding coding value according to the target Huffman coding dictionary to obtain a mapping result, for example, the current call word data is abcabc, the target Huffman coding dictionary is {'a': 00, 'b': 01, 'c': 1}, the current call word data obtained by using a forward maximum matching segmentation algorithm (FMM) for segmentation is {'a', 'b', 'c', 'a', 'b', 'c'}, and the current call word data is mapped to a coding value through the target Huffman coding dictionary to obtain a mapping result 0001100011.

[0099] Step S403: merging the mapping result to obtain compressed call data.

[0100] In a specific implementation, when the mapping result is obtained, the mapping result can be merged to obtain a compression result, that is, compressed call data.

[0101] As shown in Table 1, Table 1 is a compression effect comparison table of compression methods, and Table 1 shows comparison results of using different compression methods to process a part of billing call data collected in batches in a production system. It can be seen that the compression ratio of directly using the ordinary Huffman coding method is relatively small compared with the compression ratio of the zstd algorithm commonly used in the industry, but the compression and decompression time is long, and the practical application value is not high in the billing scene of the calculation network. After the Huffman coding method provided by the application is used, the compression ratio is greatly improved, compared with the zstd algorithm, the compression ratio is improved by about 70%, and the compression speed is also relatively close. Although the decompression speed is still relatively long, it generally meets the real-time processing time length requirement of the billing scene of the calculation network. When the decompression speed requirement is high, the parallel decompression processing method can be used to improve the speed, which is especially suitable for scenes with tight bandwidth resources or high bandwidth cost.

[0102] Table 1

[0103]

[0104] The embodiment uses a preset word segmentation strategy to perform word segmentation processing on the current call data to obtain current call word data; maps each call word in the current call word data to a corresponding encoding value based on the target Huffman coding dictionary to obtain a mapping result; and merges the mapping result to obtain compressed call data. By using the target Huffman coding dictionary constructed in advance, the time for constructing the coding dictionary in real time is saved, the compression speed is improved, and compared with the current compression algorithm, the compression ratio of the target Huffman coding dictionary constructed in advance is improved, and the amount of online transmission call stream data is greatly reduced.

[0105] Reference Figure 5 , Figure 5 FIG. 3 is a flowchart of a third embodiment of the call data compression transmission method of the application.

[0106] Based on the first embodiment, the call data compression transmission method of the embodiment further includes, before step S20:

[0107] Step S11: Obtain historical call data of a preset time period.

[0108] It should be noted that the preset time period can be set by demand, for example, the data distribution statistics is performed at 0:15 every morning, and the historical call data of the recent 30 days saved in the database is read as a data set, that is, the historical call data.

[0109] Step S12: Divide and count the historical call data based on a preset word length to obtain the number of words in the historical call word data that meet the preset condition, and obtain a statistical distribution set.

[0110] It should be understood that the preset word length can be a maximum word length, and the number of all words less than or equal to the maximum word length in the historical session data can be counted according to the maximum word length, so as to obtain the number of occurrences of each word and the corresponding word, and the number of occurrences of each word is the word quantity, and the preset condition is the word less than or equal to the preset word length. After obtaining the word and the word quantity satisfying the preset condition, they are summarized to obtain the statistical distribution set P.

[0111] The statistical distribution set P is stored in the form of a dictionary, the key is the word content, and the value is the word statistical quantity. For example, the historical session data is abcabc, and the preset word length is set to 3, and the statistical distribution set P is {'a': 2, 'b': 2, 'c': 2, 'ab': 2, 'bc': 2, 'ca': 1, 'abc': 2, 'bca': 1, 'cab': 1}.

[0112] Step S13: Obtain single character data in the statistical distribution set, and obtain a first word set according to the single character data.

[0113] In a specific implementation, the single character data in the statistical distribution set can be taken as the first word set.

[0114] Step S14: Take the first word set as a current optimal word set, and obtain a second word set according to the statistical distribution set and the single character data.

[0115] In a specific implementation, the first word set is an initial word set for constructing a target Huffman coding dictionary. When the first word set is obtained, the first word set is taken as the current optimal word set, and the remaining characters not selected are taken as the second word set. For example, the statistical distribution set P is {'a': 2, 'b': 2, 'c': 2, 'ab': 2, 'bc': 2, 'ca': 1, 'abc': 2, 'bca': 1, 'cab': 1}, the first word set G0 is {'a', 'b', 'c'}, and the second word set Ut is {'ab', 'bc', 'ca', 'abc', 'bca', 'cab'}.

[0116] Step S15: Obtain an initial word weight distribution by using the first word set and the statistical distribution set.

[0117] In a specific implementation, the statistical quantity of a single character word in the statistical distribution set can be directly obtained by using the first word set as the initial word weight distribution W0. For example, the statistical distribution set P is {'a': 2, 'b': 2, 'c': 2, 'ab': 2, 'bc': 2, 'ca': 1, 'abc': 2, 'bca': 1, 'cab': 1}, the first word set G0 is {'a', 'b', 'c'}, and the initial word weight distribution W0 is {'a': 2, 'b': 2, 'c': 2}.

[0118] Step S16: constructing an initial Huffman coding dictionary based on the initial word weight distribution.

[0119] In a specific implementation, an initial Huffman coding dictionary D0 can be constructed based on the initial word weight distribution. Specifically, a basic Huffman coding method is used, a Huffman coding binary tree is constructed, and the coding value corresponding to each character is output through the binary tree, so that the initial Huffman coding dictionary D0 {'a':00, 'b':01, 'c':1} is obtained. The value of the dictionary is represented in bit, and the initial Huffman coding dictionary is taken as the current optimal coding dictionary.

[0120] Step S17: constructing a target Huffman coding dictionary based on the first word set, the second word set, the statistical distribution set, and the initial Huffman coding dictionary.

[0121] It should be understood that when the initial Huffman coding dictionary is obtained, the target Huffman coding dictionary can be constructed based on the first word set, the second word set, the statistical distribution set, and the initial Huffman coding dictionary. The target Huffman coding dictionary is obtained by continuously updating the initial Huffman coding dictionary.

[0122] Step S18: storing the target Huffman coding dictionary to the distributed cache.

[0123] It should be noted that after the target Huffman coding dictionary is obtained, a version number can be assigned to the target Huffman coding dictionary, the version number starts from 1 and increases by 1 each time, and when the maximum value 255 is reached, the assignment starts from 1 again. The target Huffman coding dictionary with the version number is stored in the distributed cache, and the last three versions of the Huffman coding dictionary are retained in the form of a dictionary, the key is the version number, and the value is the coding dictionary. The data is as follows: {1: {'a':000, 'b':001, 'c':01, 'ab':1}, 2: {'a':00, 'b':01, 'c':1,}, 3: {'a':000, 'b':001, 'c':01, 'ab':1}}

[0124] The embodiment obtains historical call data of a preset time period, divides and counts the historical call data based on a preset word length to obtain a number of words in historical call data that meet a preset condition and obtain a statistical distribution set, obtains single-character data in the statistical distribution set, and obtains a first word set according to the single-character data, takes the first word set as a current optimal word set, and obtains a second word set according to the statistical distribution set and the single-character data, obtains an initial word weight distribution through the first word set and the statistical distribution set, constructs an initial Huffman coding dictionary based on the initial word weight distribution, constructs a target Huffman coding dictionary through the first word set, the second word set, the statistical distribution set, and the initial Huffman coding dictionary, and stores the target Huffman coding dictionary in a distributed cache, thereby directly saving the overhead of separately constructing a coding dictionary for each call in real-time call processing and improving the efficiency of real-time message compression. Meanwhile, the pre-construction method can utilize the data distribution characteristics among different calls to improve the coding compression ratio.

[0125] As an example, the step of constructing the target Huffman coding dictionary through the first word set, the second word set, the statistical distribution set, and the initial Huffman coding dictionary includes:

[0126] The historical call data is compressed through the initial Huffman coding dictionary to obtain a current maximum compression ratio;

[0127] It should be noted that the historical call data can be compressed through the constructed initial Huffman coding dictionary, the initial Huffman coding dictionary is taken as a current optimal dictionary, a forward maximum matching word segmentation algorithm is used to perform word segmentation processing on the historical call data, each word after word segmentation is mapped to a corresponding code value according to the initial Huffman coding dictionary to obtain a mapping result, the mapping result is combined to obtain a compression result, a compression ratio R0 of the compressed data is calculated, and this is taken as a current maximum compression ratio Rmax. Since the compression ratio R0 is the compression ratio obtained for the first time, it is temporarily taken as the maximum compression ratio. In the subsequent iteration process, if a larger compression ratio is obtained, it is replaced.

[0128] For example, the mapping result is 0001100011. Since 8 bits are one byte, the length after coding is 2 bytes, the original length is 6 bytes, the compression ratio R0 is 3, and the current maximum compression ratio Rmax is also 3. Compression ratio = size of data before compression / size of data after compression.

[0129] A target selection word is obtained from the second word set;

[0130] In a specific implementation, the second word set can be processed, and a target selection word can be selected from the second word set, the target selection word being a word in the second word set with a maximum estimated encoding value.

[0131] Optionally, the step of obtaining the target selection word from the second word set comprises:

[0132] Each word in the second word set is processed by the current optimal word set to obtain a segmentation result set comprising all words in the second word set;

[0133] For the second word set, the current optimal word set (i.e., the first word set) is used as a dictionary, and a forward maximum matching segmentation algorithm (FMM) is used to process each word in the second word set to obtain a segmentation structure set S comprising all words in the second word set. For example, the second word set Ut is {'ab', 'bc', 'ca', 'abc', 'bca', 'cab'}, and the current optimal word set is {'a', 'b', 'c'}. The segmentation result set obtained is {('a', 'b'), ('b', 'c'), ('c', 'a'), ('a', 'b', 'c'), ('b', 'c', 'a'), ('c', 'a', 'b')}.

[0134] The initial Huffman encoding dictionary is used as the current optimal encoding dictionary;

[0135] The encoding value length of each element in the segmentation result set in the current optimal encoding dictionary is calculated;

[0136] It should be noted that the initial Huffman encoding dictionary can be used as the current optimal encoding dictionary, and the encoding value length length(Dopt(x)) of each element x in the segmentation result set S in the current optimal encoding dictionary Dopt can be calculated. For example, the element x in the segmentation result set S is {'a', 'b'}, and the current optimal encoding dictionary Dopt is {'a':00, 'b':01, 'c':1}. The current optimal encoding dictionary mapping values of 'a' and 'b' are 00 and 01, and the encoding value lengths length(Dopt(x)) of 'a' and 'b' are both 2.

[0137] The first statistical value of each element in the segmentation result set in the statistical distribution set is calculated, and the second statistical value of each word in the second word set in the statistical distribution set is calculated;

[0138] In a specific implementation, a first statistical value x1 of each element in the set of segmentation results in the set of statistical distributions can be calculated, for example, the element x in the set of segmentation results is {a, b}, since the length of the string in the element x is 1, the first statistical value of the element x in the set of statistical distributions P is 2, and the second statistical value x2 of each word in the second word set in the set of statistical distributions is calculated.

[0139] The encoding estimated value of each word in the second word set is calculated by the first statistical value and the second statistical value and the length of the encoding value;

[0140] In a specific implementation, the adjusted word distribution statistical value P1 can be obtained by subtracting the first statistical value from the second statistical value, and the sum of the product of the length of the encoding value and the adjusted word distribution statistical value P1 is calculated to obtain the encoding estimated value. The adjusted word distribution statistical value P1 is calculated as follows in formula 1:

[0141]

[0142] In formula 1, P1 is the adjusted word distribution statistical value, P(U t,i ) is the second statistical value, P(x) is the first statistical value, when the element x is a single character, the second statistical value is equal to the first statistical value, and when P1 is less than 0, it is set to 0.

[0143] In a specific implementation, when P1 is obtained, the encoding estimated value is calculated as follows in formula 2:

[0144] V t,i =∑ x∈S length(Dopt(x))*P1 (formula 2)

[0145] In the above formula 2, V t,i is the encoding estimated value, length(Dopt(x)) is the length of the encoding value, and P1 is the adjusted word distribution statistical value.

[0146] For example, the first word set is ('a', 'b'), and the P1 value of 'a' and 'b' is calculated respectively, because the length of the string is 1, and both are equal to the distribution statistical value of 'ab' in P, which is 2. The current optimal encoding dictionary mapping value of 'a' and 'b' is 00 and 01, and the length(Dopt(x)) value is also 2. Therefore, the encoding estimated value of 'ab' is 2*2+2*2=8. The encoding estimated value of all words in the second word set can be calculated to obtain the encoding estimated value set.

[0147] The maximum encoding estimated value is obtained from the encoding estimated value;

[0148] The word corresponding to the maximum encoding estimated value in the second word set is selected as the target selection word.

[0149] It should be noted that the value with the maximum encoding estimated value can be selected from the set of encoding estimated values, and the word corresponding to the maximum encoding estimated value is taken as the target selection word, and the target selection word is a word that can be added to the current optimal word set.

[0150] The target selection word is added to the first word set to obtain an updated first word set, and the target selection word is deleted from the second word set;

[0151] The reference word weight distribution is calculated according to the set of statistical distributions and the updated first word set.

[0152] In specific implementation, the target selection word can be added to the first word set to form a new word set Gt+1, and the target selection word can be deleted from the second word set to form a new second word set Ut+1.

[0153] For example, the target selection word is 'abc', the updated first word set Gt+1 is {'a', 'b', 'c', 'abc'}, and the updated second word set Ut+1 is {'ab', 'bc', 'ca', 'bca', 'cab'}.

[0154] In specific implementation, when the updated first word set is obtained, the weight distribution of all words in the updated first word set can be recalculated, that is, the reference word weight distribution.

[0155] Optionally, the step of calculating the reference word weight distribution according to the set of statistical distributions and the updated first word set specifically comprises:

[0156] The statistical number of all words in the updated first word set in the set of statistical distributions is obtained to obtain an updated initial word weight distribution.

[0157] It should be noted that the mapping relationship of the updated first word set in the set of statistical distributions P can be initialized, so that the statistical number of all words in the updated first word set in the set of statistical distributions is counted to obtain the updated initial word weight distribution. For example, the updated initial word weight distribution is {'a': 2, 'b': 2, 'c': 2, 'abc': 2}.

[0158] The words in the updated initial word weight distribution are traversed in descending order of word length to obtain a current word.

[0159] In specific implementation, the words in the updated initial word weight distribution can be traversed in descending order of word length, for example, the updated initial word weight distribution is {'a': 2, 'b': 2, 'c': 2, 'abc': 2}, and the current word is abc, that is, the word with the longest length, which is obtained by traversing in descending order of word length.

[0160] obtaining a sub-word of the current word, and obtaining a weight value of the current word, a weight value of the sub-word, and a number of times the sub-word appears in the current word;

[0161] It should be noted that, for example, A is a sub-word of B, which means that the length of A is less than that of B, and A is a middle segment of B. For example, the current word is abc, and 'a', 'b', and 'c' can be considered as sub-words of 'abc'.

[0162] The weight value of the current word, the weight value of the sub-word of the current word, and the number of times the sub-word appears in the current word n are obtained.

[0163] The updated weight value of the sub-word is calculated based on the weight value of the current word, the weight value of the sub-word, and the number of times the sub-word appears in the current word.

[0164] It should be noted that the process of calculating the updated weight value of the sub-word is as follows:

[0165] W' t+1,j =W t+1,j -W t+1,i *n (Formula 3)

[0166] In formula 3, W' t+1,j is the updated weight value of the sub-word, W t+1,j is the weight value of the sub-word, and W t+1,i is the weight value of the current word. The updated weight value of the sub-word is calculated by formula 3. For example, the current word is abc, and 'a', 'b', and 'c' are sub-words in the remaining elements. After updating the weight value of the sub-word according to formula 3, the updated weight value of the sub-word is W' t+1,j is {'a': 0, 'b': 0, 'c': 0, 'abc': 2}.

[0167] The reference word weight distribution is obtained based on the updated weight value of the sub-word.

[0168] After obtaining the updated weight value of all sub-words, the updated weight value of the sub-word is replaced by the weight value of the sub-word, thereby obtaining the reference word weight distribution. For example, after traversing abc, the elements 'a', 'b', and 'c' are traversed. Since no word is a sub-word of them, the final result of the reference word weight distribution is {'a': 0, 'b': 0, 'c': 0, 'abc': 2}.

[0169] A reference Huffman coding dictionary is constructed based on the reference word weight distribution.

[0170] In a specific implementation, when the reference word weight distribution is obtained, the reference Huffman coding dictionary can be constructed based on the reference word weight distribution, and the construction manner is the same as the above-mentioned manner of constructing the initial Huffman coding dictionary.

[0171] Compressing the historical call data by the reference Huffman coding dictionary to obtain a reference compression ratio;

[0172] In a specific implementation, the historical call data can be compressed by the constructed reference Huffman coding dictionary to obtain a compression ratio Rt+1, i.e., a reference compression ratio. The reference compression ratio is compared with a current maximum compression ratio Rmax.

[0173] When the reference compression ratio is greater than the current maximum compression ratio, the reference Huffman coding dictionary is taken as a target Huffman coding dictionary.

[0174] In a specific implementation, if the reference compression ratio is greater than the current maximum compression ratio, the optimal Huffman coding dictionary is updated to the reference Huffman coding dictionary, the optimal word set is updated to the updated first word set, and the current maximum coding estimated value corresponding to the word in the updated second word set is selected, and the updated weight distribution is recalculated, so as to construct the next Huffman coding dictionary again. Through comparison of compression ratios, all words in the second word set are traversed. When the reference compression ratio is less than or equal to the current maximum compression ratio, the optimal coding dictionary before the target selected word is taken as the target Huffman coding dictionary.

[0175] For example, the reference Huffman coding dictionary Dt+1 "{'a': 000, 'b': 001, 'c': 01, 'abc': 1}" and Gt+1 "{'a', 'b', 'c', 'abc'}" are used to compress "abcabc". First, the words are divided according to Gt+1 to obtain the result ('abc', 'abc'), and then compressed according to Dt+1. The compression result is 11, i.e., the length after coding is 1 byte, while the original length is 6 bytes, and the compression ratio Rt+1 is 6.

[0176] As shown in FIG. 6, Figure 6 Figure 6 ​The system architecture schematic involved in the call slip data compression transmission method comprises the following steps: obtaining the call slip data, and performing data distribution statistics at regular time, pre-constructing the Huffman coding dictionary with the version number of the word dimension, and loading the coding dictionary of the target version to the cache, when performing data compression, the message sending end service loads the target Huffman coding dictionary from the distributed cache, performs coding compression, and transmits the compressed data to the message receiving end service, performs decoding and decompression, and thus performs business processing. The system architecture for pre-constructing the Huffman coding dictionary based on batch historical call slip data, loading the distributed cache, and transmitting the call slip data through the compressed data structure with the version number is designed, and the existing Huffman coding algorithm is optimized, the word dimension coding method based on coding value estimation is designed, the compression ratio of the call slip data of the online billing system is improved, the load of the system bandwidth resource is reduced, the hardware resource consumption is reduced, the network congestion phenomenon caused by a large amount of call slip processing in the peak period is reduced, the timeliness of call slip processing is improved, the user experience is also significantly improved, and the application popularization value is high.

[0177] Reference Figure 7 , Figure 7 The structure block diagram of the call slip data compression transmission device according to the first embodiment of the present application is shown.

[0178] As Figure 7 shown, the call slip data compression transmission device according to the embodiment of the present application comprises:

[0179] The obtaining module 10 is configured to obtain the current call slip data generated after business processing.

[0180] The loading module 20 is configured to load the target Huffman coding dictionary of the word dimension from the distributed cache.

[0181] The compression module 30 is configured to compress the current call slip data based on the target Huffman coding dictionary, to obtain the compressed call slip data.

[0182] The transmission module 40 is configured to transmit the compressed call slip data.

[0183] In an embodiment, the compression module 30 is further configured to perform word segmentation processing on the current call slip data using a preset word segmentation strategy, to obtain the current call word data, map each call word in the current call word data to a corresponding coding value based on the target Huffman coding dictionary, to obtain a mapping result, and merge the mapping result, to obtain the compressed call slip data.

[0184] In an embodiment, the compression module 30 is further configured to query whether there is an uncoded word in the target Huffman coding dictionary through the current bill word data; and when there is an uncoded word in the target Huffman coding dictionary, compress the current bill word data using a preset compression algorithm to obtain compressed bill data.

[0185] In an embodiment, the loading module 20 is further configured to obtain historical bill data of a preset time period; divide and count the historical bill data based on a preset word length to obtain a number of words in historical bill words that meet a preset condition, and obtain a statistical distribution set; obtain single character data in the statistical distribution set, and obtain a first word set according to the single character data; take the first word set as a current optimal word set, and obtain a second word set according to the statistical distribution set and the single character data; obtain an initial word weight distribution through the first word set and the statistical distribution set; construct an initial Huffman coding dictionary based on the initial word weight distribution; construct a target Huffman coding dictionary through the first word set, the second word set, the statistical distribution set, and the initial Huffman coding dictionary; and store the target Huffman coding dictionary to a distributed cache.

[0186] In an embodiment, the loading module 20 is further configured to compress the historical bill data through the initial Huffman coding dictionary to obtain a current maximum compression ratio; obtain a target selection word through the second word set; add the target selection word to the first word set to obtain an updated first word set, and delete the target selection word from the second word set; calculate a reference word weight distribution according to the statistical distribution set and the updated first word set; construct a reference Huffman coding dictionary based on the reference word weight distribution; compress the historical bill data through the reference Huffman coding dictionary to obtain a reference compression ratio; and when the reference compression ratio is greater than the current maximum compression ratio, take the reference Huffman coding dictionary as a target Huffman coding dictionary.

[0187] In an embodiment, the loading module 20 is further configured to perform word segmentation on each word in the second word set based on the current optimal word set to obtain a word segmentation result set comprising all words in the second word set; take the initial Huffman coding dictionary as a current optimal coding dictionary; calculate the length of the coding value of each element in the word segmentation result set in the current optimal coding dictionary; calculate the first statistical value of each element in the word segmentation result set in the statistical distribution set, and calculate the second statistical value of each word in the second word set in the statistical distribution set; calculate the coding estimated value of each word in the second word set based on the first statistical value, the second statistical value and the length of the coding value; obtain the maximum coding estimated value from the coding estimated value; and take the word corresponding to the maximum coding estimated value in the second word set as a target selection word.

[0188] In an embodiment, the loading module 20 is further configured to obtain the statistical number of all words in the updated first word set in the statistical distribution set to obtain an updated initial word weight distribution; traverse the words in the updated initial word weight distribution in descending order of word length to obtain a current word; obtain the sub-word of the current word, and obtain the weight value of the current word, the weight value of the sub-word and the number of occurrences of the sub-word in the current word; calculate the updated weight value of the sub-word based on the weight value of the current word, the weight value of the sub-word and the number of occurrences; and obtain a reference word weight distribution based on the updated weight value of the sub-word.

[0189] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium stores a call data compression transmission program, and the call data compression transmission program is executed by a processor to implement the steps of the call data compression transmission method.

[0190] Since the storage medium adopts all the technical solutions of the above embodiments, it at least has all the beneficial effects brought by the technical solutions of the above embodiments, which will not be repeated here.

[0191] It should be understood that the above is only for illustration, and does not constitute any limitation on the technical solutions of the present application. In specific applications, those skilled in the art can set it according to the needs, and the present application does not limit this.

[0192] It should be noted that the above-described workflow is only illustrative and does not limit the scope of protection of the present application. In actual application, those skilled in the art can select part or all of them to achieve the purpose of the embodiment, and this place does not limit it.

[0193] In addition, technical details not described in detail in the present embodiment can be found in the message data compression transmission method provided by any embodiment of the present application, which will not be described here.

[0194] Furthermore, it is to be understood that the terms "including", "comprising", "consisting" or any other variation thereof are intended to cover the non-exclusive inclusion of the elements specified, such that processes, methods, articles, or systems that comprise these elements, but not other elements, are also within the scope of the present application. The term "comprising" does not exclude other elements from being present in the process, method, article, or system, and does not exclude that an additional one or more of these elements can be present.

[0195] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0196] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk), and includes a number of instructions for making a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in the various embodiments of the present application.

[0197] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the present application specification and drawings, is also included in the patent protection scope of the present application.

Claims

1. A method for compressed transmission of a phone bill data, characterized by, The bill data compression transmission method comprises: obtaining current bill data generated after service processing; loading a target Huffman coding dictionary of word dimensions from a distributed cache; compressing the current bill data based on the target Huffman coding dictionary to obtain compressed bill data; transmitting the compressed bill data; Before loading the target Huffman coding dictionary of word dimensions from the distributed cache, the method further comprises: obtaining historical bill data of a preset time period; dividing and counting the historical bill data based on a preset word length to obtain the number of words in the historical bill data that meet a preset condition, thereby obtaining a statistical distribution set; obtaining single-character data in the statistical distribution set and obtaining a first word set according to the single-character data; taking the first word set as a current optimal word set and obtaining a second word set according to the statistical distribution set and the single-character data; obtaining an initial word weight distribution through the first word set and the statistical distribution set; constructing an initial Huffman coding dictionary based on the initial word weight distribution; constructing a target Huffman coding dictionary through the first word set, the second word set, the statistical distribution set, and the initial Huffman coding dictionary; storing the target Huffman coding dictionary to the distributed cache.

2. The method for transmission of compressed call data of claim 1, wherein, The method for compressing the current bill data based on the target Huffman coding dictionary to obtain compressed bill data comprises: performing word segmentation processing on the current bill data using a preset word segmentation strategy to obtain current bill word data; mapping each bill word in the current bill word data to a corresponding code value based on the target Huffman coding dictionary to obtain a mapping result; merging the mapping result to obtain compressed bill data.

3. The method for transmission of compressed call data of claim 2, wherein, Before mapping each bill word in the current bill word data to a corresponding code value based on the target Huffman coding dictionary to obtain a mapping result, the method further comprises: querying whether there is an uncoded word in the target Huffman coding dictionary through the current bill word data; when there is an uncoded word in the target Huffman coding dictionary, compressing the current bill word data using a preset compression algorithm to obtain compressed bill data.

4. The method for transmission of compressed call data of claim 1, wherein, The method for constructing a target Huffman coding dictionary through the first word set, the second word set, the statistical distribution set, and the initial Huffman coding dictionary comprises: compressing the historical bill data through the initial Huffman coding dictionary to obtain a current maximum compression ratio; obtaining a target selection word from the second word set; adding the target selection word to the first word set to obtain an updated first word set and deleting the target selection word from the second word set; calculating a reference word weight distribution according to the statistical distribution set and the updated first word set; constructing a reference Huffman coding dictionary based on the reference word weight distribution; compressing the historical bill data through the reference Huffman coding dictionary to obtain a reference compression ratio; when the reference compression ratio is greater than the current maximum compression ratio, taking the reference Huffman coding dictionary as a target Huffman coding dictionary.

5. The method for transmission of compressed slip data according to claim 4, wherein, The method for obtaining a target selection word from the second word set comprises: perform word segmentation processing on each word in the second word set by using the current optimal word set, to obtain a word segmentation result set including all words in the second word set; use the initial Huffman coding dictionary as a current optimal coding dictionary; calculate the coding value length of each element in the word segmentation result set in the current optimal coding dictionary; calculate a first statistical value of each element in the word segmentation result set in the statistical distribution set, and calculate a second statistical value of each word in the second word set in the statistical distribution set; calculate the coding estimated value of each word in the second word set by using the first statistical value, the second statistical value, and the coding value length; obtain a maximum coding estimated value from the coding estimated values; use the word corresponding to the maximum coding estimated value in the second word set as a target selection word.

6. The method for transmission of compressed call data of claim 4, wherein, The calculating a reference word weight distribution according to the statistical distribution set and the updated first word set comprises: obtain the statistical number of all words in the updated first word set in the statistical distribution set to obtain an updated initial word weight distribution; traverse the words in the updated initial word weight distribution in descending order of word length to obtain a current word; obtain the sub-words of the current word, and obtain the weight value of the current word, the weight value of the sub-words, and the number of occurrences of the sub-words in the current word; calculate the updated weight value of the sub-words by using the weight value of the current word, the weight value of the sub-words, and the number of occurrences; obtain the reference word weight distribution by using the updated weight value of the sub-words.

7. A call data compression transmission apparatus, characterized by comprising: The call data compression transmission device comprises: an obtaining module configured to obtain current call data generated after service processing; a loading module configured to load a target Huffman coding dictionary of a word dimension from a distributed cache; a compression module configured to compress the current call data based on the target Huffman coding dictionary to obtain compressed call data; a transmission module configured to transmit the compressed call data; the loading module is further configured to obtain historical call data in a preset time period; divide and statistically analyze the historical call data based on a preset word length to obtain the number of words in the historical call data that meet a preset condition, and obtain a statistical distribution set; obtain single-character data in the statistical distribution set, and obtain a first word set according to the single-character data; use the first word set as a current optimal word set, and obtain a second word set according to the statistical distribution set and the single-character data; obtain an initial word weight distribution by using the first word set and the statistical distribution set; construct an initial Huffman coding dictionary based on the initial word weight distribution; construct a target Huffman coding dictionary by using the first word set, the second word set, the statistical distribution set, and the initial Huffman coding dictionary; and store the target Huffman coding dictionary in the distributed cache.

8. A call data compression transmission apparatus, characterized by comprising: The call data compression transmission device comprises a memory, a processor, and a call data compression transmission program stored in the memory and executable on the processor, and the call data compression transmission program is configured to implement the call data compression transmission method in any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium stores a call bill data compression transmission program, and the call bill data compression transmission program is executed by the processor to realize the call bill data compression transmission method in any one of claims 1 to 6.

Images