Dictionary updating method, message compression method, system, equipment, product and medium
By filtering and training new dictionaries, the problem of low message compression rate in the existing technology is solved, and higher message compression efficiency and business relevance are achieved, ensuring fast response and efficient compression when business changes are made.
Patent Information
- Application Number
- CN202410985962.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-19
AI Technical Summary
The existing message compression algorithm performs well in the field of file compression, but is not effective in message compression, mainly because a single message is too small and cannot provide sufficient redundancy for effective compression.
By obtaining multiple message data, filtering and fusing the corresponding message compression rate of the dictionary of the target device, training a new dictionary, and issuing it to the target device when the compression performance of the new dictionary is better than that of the old dictionary, in order to improve the message compression rate.
It realizes higher message compression rates and quickly generates business-related dictionaries when business changes, ensuring the robustness and real-timeness of message compression rates.
Smart Images

Figure CN118984339A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of communication technology, and in particular to a dictionary updating method, a message compression method, a system, a device, a product and a medium. Background Art
[0002] In the field of network transmission, compressing the transmitted messages based on message compression technology can effectively save bandwidth consumption. Some traditional compression algorithms (such as Gzip, Brotli) and new compression algorithms (such as Zstd) can be used for message compression. However, these algorithms have good performance in the field of file compression, but the compression effect on messages is not satisfactory. The main reason for this problem is that a single message itself is too small and cannot provide enough redundancy for compression.
[0003] In view of this, a method that can improve the message compression rate is urgently needed. Summary of the invention
[0004] In view of this, embodiments of the present disclosure provide a dictionary updating method, a dictionary updating system, an electronic device, a computer program product, and a computer-readable storage medium, which can improve the message compression rate.
[0005] The present disclosure provides a dictionary updating method, the method comprising:
[0006] Acquire multiple message data, the message data comes from at least one target device, the target device includes at least one dictionary, the message data includes an original message, a message compression rate and a dictionary identifier, the message compression rate is obtained after the target device compresses the original message according to the dictionary represented by the dictionary identifier;
[0007] Based on the dictionary identifier of the first dictionary currently used by the target device, multiple message compression rates corresponding to the first dictionary are screened out from the message data, and the multiple message compression rates are merged to obtain a first compression performance value of the first dictionary;
[0008] training a second dictionary based on the original message, and determining a second compression performance value of the second dictionary;
[0009] If the second compression performance value is greater than the first compression performance value, the second dictionary is sent to the target device.
[0010] In one aspect, the present disclosure provides a dictionary updating system, the system comprising:
[0011] A dictionary acquisition module, configured to, if a second dictionary sent by a compression management server is further acquired after the first dictionary has been acquired, save the first dictionary and the second dictionary at the same time, wherein the first dictionary and the second dictionary are not completely the same;
[0012] The message processing module is used to decompress the received compressed message or compress the original message to be sent according to one of the first dictionary and the second dictionary.
[0013] Another aspect of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method described above is implemented.
[0014] Another aspect of the present disclosure provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method described above is implemented.
[0015] Another aspect of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method as described above is implemented.
[0016] In the technical solutions of some embodiments of the present disclosure, a first compression performance value of the first dictionary is obtained based on multiple message compression rates corresponding to the first dictionary screened out from the message data, and a second dictionary is obtained based on the original message training, and when the second compression performance value of the second dictionary is greater than the first compression performance value, the second dictionary is sent to the target device. On the one hand, since the second dictionary is obtained based on the real-time training of the original message, the second dictionary can maintain a greater correlation with the current business and can greatly improve the message compression rate; on the other hand, since it supports dynamic updating of the dictionary in the target device, when the business changes, the dictionary related to the business can be generated quickly and sent to the target device in time to ensure the message compression rate in the target device. The robustness is good. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:
[0018] Figure 1 A schematic diagram of a network architecture for message compression provided by an embodiment of the present disclosure is shown;
[0019] Figure 2 A schematic diagram showing a flow chart of a message compression method provided by some embodiments of the present disclosure is shown;
[0020] Figure 3A flowchart of a dictionary updating method provided by an embodiment of the present disclosure is shown;
[0021] Figure 4 A module schematic diagram of a dictionary updating system provided by an embodiment of the present disclosure is shown;
[0022] Figure 5 A module schematic diagram of a message compression system provided by an embodiment of the present disclosure is shown;
[0023] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0026] In dictionary-based compression technology, the same content between different HTTP messages can be saved in a dictionary. In this way, when transmitting HTTP messages, only the difference parts need to be transmitted, achieving the purpose of message compression. At present, this technology is mainly aimed at server / client architecture. Since the services provided by the server are often relatively fixed and the data can be predicted, a relatively high compression rate can be achieved. However, for network architectures with unstable services, the data is unpredictable and the compression rate is relatively low. For example, for CDN (Content Delivery Network) edge nodes, since the services on the edge nodes are unstable and the data is unpredictable, the message compression rate on the edge nodes is relatively low.
[0027] In order to solve the above problems, refer to Figure 1 , is a schematic diagram of a network architecture for message compression provided by an embodiment of the present disclosure. Figure 1In the example, at least one target device 11 may be included, and HTTP communication may be performed between the target devices 11. When messages are transmitted between different target devices 11, the transmitted messages may be compressed messages. In order to achieve message compression, each target device 11 may include a compression proxy module 13. The compression proxy module 13 may compress the message sent by the target device or decompress the compressed message received by the target device 11 according to at least one dictionary in the target device 11.
[0028] The compression management server 12 can perform dictionary training based on data obtained from one or more target devices 11, and send the trained dictionary to each target device 11 to update the dictionary in each target device 11, so that the compression agent module 13 in each target device 11 can achieve better message compression effect based on the new dictionary.
[0029] In the embodiment shown in the present disclosure, the target device 11 may be an edge node in a CDN network.
[0030] In combination with the method provided by the present disclosure, Figure 1 The principle of the network architecture shown is explained.
[0031] The present disclosure first provides a message compression method. The message compression method can be applied to a first target device in the target device 11. The first target device can be Figure 1 Any one of the target devices 11. Figure 2 , which is a flow chart of a message compression method provided in some embodiments of the present disclosure. Figure 2 In the method for compressing a message, the method may include the following steps:
[0032] Step S21 , when the first dictionary has been acquired, if the second dictionary of the compression management server 12 is acquired, the first dictionary and the second dictionary are saved simultaneously, wherein the first dictionary and the second dictionary are not completely the same.
[0033] Among them, compared with the first dictionary, the second dictionary can be a dictionary with better compression effect sent by the compression management server 12 to the first target device. In other words, compared with the first dictionary, when the message is compressed according to the second dictionary, the message compression rate can be higher. In some embodiments of the present disclosure, the message compression rate can be shown as expression (1):
[0034]
[0035] Among them, R represents the message compression rate, Origin Bytes represents the size of the message before compression, and CompressedBytes represents the size of the message after compression. The larger R is, the better the compression effect of the message is.
[0036] Since the second dictionary has a better compression effect, after receiving the second dictionary, the second dictionary can be used as a new dictionary for message compression. That is, in the subsequent message compression process, the message compression is no longer performed based on the first dictionary, but based on the second dictionary. However, considering that there may be a conflict period between the receiving process of the second dictionary and the generation / transmission process of the compressed message, some messages being transmitted may still need to be decompressed based on the first dictionary. Therefore, after receiving the second dictionary, the first dictionary needs to be saved at the same time. The so-called conflict period is that the receiving process of the second dictionary overlaps with the generation / transmission process of the compressed message. For example, suppose that the second target device generates a compressed message A to be sent to the first target device based on the first dictionary at 0.3 seconds, and the compressed message A is sent to the first target device at 0.6 seconds. However, between 0.4 seconds and 0.5 seconds, the first target device receives the second dictionary. If the first target device deletes the first dictionary when receiving the second dictionary, then at 0.6 seconds, after the compressed message A is transmitted to the first target device, it cannot be decompressed based on the first dictionary, resulting in problems such as failure or error in decompression of message A. Therefore, it is necessary to save the first dictionary and the second dictionary simultaneously in the first target device.
[0037] Step S22: decompress the received compressed message or compress the original message to be sent according to one of the first dictionary and the second dictionary.
[0038] It can be seen from the description of step S22 that after receiving the second dictionary, the message compression operation performed by the first target device can be performed entirely according to the first dictionary, but the message decompression operation performed by the first target device can be divided into the following two cases:
[0039] 1) If the second target device sends a compressed message to the first target device during the conflict period, the first target device needs to decompress the compressed message according to the first dictionary;
[0040] 2) If the second target device sends a compressed message to the first target device after the conflict period, the first target device needs to decompress the compressed message based on the second dictionary. For example, suppose that between 0.4 seconds and 0.5 seconds, both the first target device and the second target device receive the second dictionary. If the second target device needs to send a message to the first target device at 0.7 seconds (i.e., outside the conflict period), the second target device will inevitably generate a compressed message based on the second dictionary. After the compressed message is transmitted to the first target device, the first target device needs to decompress the compressed message based on the second dictionary.
[0041] It can be seen from the above description that when decompressing different compressed messages, the dictionaries used may not be completely the same. Therefore, how to distinguish the dictionaries used when decompressing each compressed message becomes a problem. To this end, the present disclosure proposes the following solution.
[0042] Specifically, in some embodiments of the present disclosure, the first dictionary may have a first dictionary identifier, and the second dictionary may have a second dictionary identifier. The first dictionary identifier and the second dictionary identifier may be different. When the first target device needs to send a message to the second target device, the first original message to be sent may be compressed according to the second dictionary to obtain a first compressed message. After obtaining the first compressed message, the first compressed message may be identified using the second dictionary identifier, and the identified first compressed message may be sent to the second target device. In this way, the second target device may decompress the first compressed message according to the second dictionary based on the second dictionary identifier carried by the first compressed message.
[0043] Correspondingly, if the first target device receives the second compressed message sent by the second target device, the target dictionary identifier in the second compressed message can be obtained. When the target dictionary identifier is the first dictionary identifier, the second compressed message can be decompressed according to the first dictionary to obtain the second original message; when the target dictionary identifier is the second dictionary identifier, the second compressed message can be decompressed according to the second dictionary to obtain the second original message.
[0044] In some embodiments, the hash value of the first dictionary may be used as the first dictionary identifier, and the hash value of the second dictionary may be used as the second dictionary identifier.
[0045] In the above embodiment, different dictionary identifiers are set for the first dictionary and the second dictionary respectively, and the dictionary identifier is used to identify the dictionary required for message decompression. In this way, the problem of message decompression based on the wrong dictionary when the first dictionary and the second dictionary coexist is effectively avoided.
[0046] In some embodiments, after the first target device obtains the second dictionary, if it continues to obtain the third dictionary sent by the compression management server 12, it can save the second dictionary and the third dictionary at the same time and delete the first dictionary, wherein the third dictionary is not completely the same as the second dictionary.
[0047] Specifically, the third dictionary can be a dictionary with better compression effect than the second dictionary. Therefore, after the first target device receives the third dictionary, the third dictionary can be used as a new dictionary for message compression. In the compression management server 12, the interval between the two new dictionaries can be controlled to ensure that when the third dictionary is sent, all compressed messages that need to be decompressed according to the first dictionary have been fully transmitted and decompressed. Therefore, when the third dictionary is received, the first dictionary can be deleted. In this way, two versions of the dictionary can always be maintained in the first target device, reducing the storage consumption of the first target device.
[0048] This completes the description of the message compression method.
[0049] Corresponding to the message compression method, the present disclosure also provides a dictionary updating method. The dictionary updating method can be applied to the compression management server 12. Figure 3 , which is a flowchart of a dictionary updating method provided by an embodiment of the present disclosure. Figure 3 In the dictionary updating method, the following steps may be included:
[0050] Step S31, obtain multiple message data, the message data comes from at least one target device 11, the target device 11 may include at least one dictionary, the message data includes the original message, the message compression rate and the dictionary identifier, the message compression rate is obtained by the target device 11 compressing the original message according to the dictionary represented by the dictionary identifier.
[0051] The plurality of message data may be historical communication data between the target devices 11. When the target device 11 receives a message, the message data may be HTTP response message data. The original message in the message data may be the content obtained after decompression operation is performed on the compressed message received by the target device 11. When the target device 11 sends a message, the message data may be HTTP request message data. The original message in the message data may be the content obtained before compression operation is performed on the message to be sent by the target device 11.
[0052] In some embodiments, the compression management server 12 may detect whether the target devices 11 send or receive messages at preset time intervals in each target device 11. If it is detected that the target devices 11 send or receive messages, the message data of these messages are collected.
[0053] In other embodiments, when sending or receiving a compressed message, each target device 11 can determine whether the message sampling condition is met at the current time point, and if so, send the original message, dictionary identifier, and message compression rate corresponding to the compressed message to the compression management server 12. The message sampling condition includes but is not limited to whether the current time point is a sampling time point. When the current time point is a sampling time point, the original message, dictionary identifier, and message compression rate corresponding to the compressed message are actively sent to the compression management server 12.
[0054] In summary, the message data acquired in step S31 may include multiple data records, each of which is an original message, a dictionary identifier, and a message compression ratio corresponding to a compressed message. For example, the acquired message data may be similar to the following:
[0055] Original message A, dictionary identifier XX and message compression rate A
[0056] Original message B, dictionary identifier XX and message compression rate B
[0057] Original message C, dictionary identifier YY and message compression ratio C
[0058] …
[0059] Step S32, based on the dictionary identifier of the first dictionary currently used by the target device 11, multiple message compression rates corresponding to the first dictionary are screened out from the message data, and the multiple message compression rates are merged to obtain a first compression performance value of the first dictionary.
[0060] The first dictionary here may be the latest dictionary sent by the compression management server 12 to the target device 11. From the perspective of the target device 11, the first dictionary may also be the dictionary based on which the target device 11 currently performs the message compression operation.
[0061] When the compression management server 12 sends a dictionary to the target device 11, it can record and store the dictionary identifier of the latest dictionary sent. It can be understood that as the compression management server 12 continues to send dictionaries to the target device 11, the dictionary identifier of the latest dictionary recorded and stored by it changes dynamically. In this way, in step S32, the compression management server 12 can directly use the dictionary identifier currently stored locally as the dictionary identifier of the first dictionary (hereinafter referred to as the first dictionary identifier), and filter out the data records including the first dictionary identifier in the message data. The message compression rate in these filtered data records is the message compression rate corresponding to the first dictionary. For example, assuming that the dictionary identifier of the first dictionary is XX, and the acquired message data is similar to the following:
[0062] Original message A, dictionary identifier XX and message compression rate A
[0063] Original message B, dictionary identifier XX and message compression rate B
[0064] Original message C, dictionary identifier YY and message compression ratio C
[0065] Then the message compression rates obtained by screening are message compression rate A and message compression rate B.
[0066] Since the first dictionary is the dictionary based on which the target device 11 currently compresses the message, the message compression rate corresponding to the first dictionary reflects the current message compression effect. The first compression performance value represents the message compression rate performance of the first dictionary. The larger the first compression performance value, the greater the message compression rate can be when the message is compressed according to the first dictionary; the smaller the first compression performance value, the smaller the message compression rate can be when the message is compressed according to the first dictionary.
[0067] In this embodiment, a weighted calculation may be performed on the compression ratios of the multiple messages obtained through screening to obtain a first compression performance value.
[0068] Step S33: obtain a second dictionary based on the original message training, and determine a second compression performance value of the second dictionary.
[0069] In this embodiment, the original message can be divided into a training queue message and a test queue message. The second dictionary can be trained using the training queue message. The test queue message is compressed according to the trained second dictionary, and the second compression performance value of the second dictionary can be determined according to the compressed message size and the message size of the test queue message.
[0070] Specifically, the second dictionary can be trained according to the dictionary training method provided by ZSTD (Zstandard). According to expression (2), the second compression performance value can be calculated:
[0071]
[0072] Among them, R new-test represents the second compression performance value, S test Indicates the message size of the test queue, S test-compressed Indicates the compressed test queue message size.
[0073] The second compression performance value is used to characterize the message compression rate performance of the second dictionary. The larger the second compression performance value, the greater the message compression rate can be when the message is compressed according to the second dictionary; the smaller the second compression performance value, the smaller the message compression rate can be when the message is compressed according to the second dictionary.
[0074] Step S34 , if the second compression performance value is greater than the first compression performance value, the second dictionary is sent to the target device 11 .
[0075] It can be understood that if the second compression performance value is greater than the first compression performance value, it means that the message compression effect based on the second dictionary is better. In this case, the second dictionary can be sent to the target device 11 so that the target device 11 uses the second dictionary as the latest dictionary based on which the message is compressed. In this way, it is ensured that the target device 11 always has a better message compression effect.
[0076] In some embodiments, after obtaining the training queue message in step S33, dictionary training may not be performed immediately. In this case, the message data can continue to be obtained, and the original message in the message data that continues to be obtained can be used to replace the original message in the training queue message. In this way, it can be ensured that the training queue message always has more correlation with future messages, so that the second dictionary obtained by training can have better compression performance. Moreover, based on this correlation, when the amount of data in the training queue message is small, the second dictionary obtained by training can still have good versatility and compression performance.
[0077] In some embodiments, if the second compression performance value is greater than the first compression performance value, sending the second dictionary to the target device 11 may include:
[0078] Obtaining a first time point when a first dictionary is sent to the target device 11;
[0079] If the second compression performance value is greater than the first compression performance value, the second dictionary is sent to the target device 11 at a preset time interval after the first time point, wherein the preset time interval is greater than or equal to the maximum communication time interval between any two target devices.
[0080] In this way, the update frequency of the dictionary is controlled to avoid that during the conflict period, the dictionary required for decompressing the compressed message has been deleted before the compressed message is decompressed. For example, suppose the second target device generates a compressed message A to be sent to the first target device based on the first dictionary at 0.3 seconds, and the compressed message A is sent to the first target device at 0.9 seconds. However, between 0.4 and 0.5 seconds, the first target device receives the second dictionary, and between 0.6 and 0.7 seconds, the first target device receives the third dictionary. Figure 2 Description, when the third dictionary is received, the first dictionary will be deleted. Then, after the compressed message A is transmitted to the first target device, it is impossible to decompress the compressed message A according to the first dictionary, resulting in problems such as failure or error in decompression of message A. Therefore, by sending the second dictionary to the target device 11 after a preset time interval after the first time point, it can be ensured that the first dictionary is deleted after the compressed message A is decompressed. High reliability.
[0081] In summary, in the technical solutions of some embodiments of the present disclosure, a first compression performance value of the first dictionary is obtained based on multiple message compression rates corresponding to the first dictionary screened out from the message data, and a second dictionary is obtained based on the original message training, and when the second compression performance value of the second dictionary is greater than the first compression performance value, the second dictionary is sent to the target device. On the one hand, since the second dictionary is obtained based on the real-time training of the original message, the second dictionary can maintain a greater correlation with the current business, which can greatly improve the message compression rate; on the other hand, since it supports dynamic updating of the dictionary in the target device 11, when the business changes, the dictionary related to the business can be generated quickly and sent to the target device 11 in time to ensure the message compression rate in the target device 11. The robustness is good.
[0082] This completes the description of the method of the present disclosure.
[0083] The present disclosure also provides a dictionary updating system. Figure 4 , which is a module diagram of a dictionary updating system provided by an embodiment of the present disclosure. Figure 4 In , the dictionary update system includes:
[0084] A data acquisition module, used to acquire multiple message data, the message data comes from at least one target device, the target device includes at least one dictionary, the message data includes an original message, a message compression ratio and a dictionary identifier, the message compression ratio is obtained after the target device compresses the original message according to the dictionary represented by the dictionary identifier;
[0085] A first performance value determination module, configured to filter out multiple message compression rates corresponding to the first dictionary from the message data based on a dictionary identifier of the first dictionary currently used by the target device, and fuse the multiple message compression rates to obtain a first compression performance value of the first dictionary;
[0086] A second performance value determination module, used to obtain a second dictionary based on original message training and determine a second compression performance value of the second dictionary;
[0087] The dictionary sending module is used to send the second dictionary to the target device if the second compression performance value is greater than the first compression performance value.
[0088] In some embodiments, the first performance value determination module is specifically used to:
[0089] A weighted calculation is performed on the compression ratios of multiple messages to obtain a first compression performance value.
[0090] In some embodiments, the second performance value determination module is specifically used to:
[0091] Divide the original message into training queue messages and test queue messages;
[0092] Use the training queue message to train to obtain the second dictionary;
[0093] The test queue message is compressed according to the trained second dictionary, and a second compression performance value of the second dictionary is determined according to the compressed message size and the message size of the test queue message.
[0094] In some embodiments, after dividing the original message into the training queue message and the test queue message, the second performance value determination module is specifically used to:
[0095] Continue to obtain message data, and use the original message in the message data that continues to be obtained to replace the original message in the training queue message.
[0096] In some embodiments, the dictionary delivery module is specifically used for:
[0097] Get the first time point when the first dictionary is sent to the target device;
[0098] If the second compression performance value is greater than the first compression performance value, the second dictionary is sent to the target device at a preset time interval after the first time point, wherein the preset time interval is greater than or equal to the maximum communication time interval between any two target devices.
[0099] The present disclosure also provides a message compression system. Figure 5 , which is a module diagram of a message compression system provided by an embodiment of the present disclosure. Figure 5 In the , the message compression system includes:
[0100] A dictionary acquisition module, used for, when the first dictionary has been acquired, if a second dictionary sent by the compression management server is acquired, saving the first dictionary and the second dictionary at the same time, wherein the first dictionary and the second dictionary are not completely the same;
[0101] The message processing module is used to decompress the received compressed message according to one of the first dictionary and the second dictionary, or to compress the original message to be sent.
[0102] In some embodiments, the second dictionary has a second dictionary identifier; and the message processing module is specifically configured to:
[0103] When it is necessary to send a message to a second target device, compressing the first original message to be sent according to the second dictionary to obtain a first compressed message;
[0104] The first compressed message is identified using the second dictionary identifier, and the identified first compressed message is sent to the second target device.
[0105] In some embodiments, the first dictionary has a first dictionary identifier, and the second dictionary has a second dictionary identifier; the message processing module is specifically configured to:
[0106] If a second compressed message sent by a second target device is received, obtaining a target dictionary identifier in the second compressed message;
[0107] When the target dictionary identifier is the first dictionary identifier, decompressing the second compressed message according to the first dictionary to obtain a second original message;
[0108] When the target dictionary identifier is the second dictionary identifier, the second compressed message is decompressed according to the second dictionary to obtain a second original message.
[0109] When sending or receiving a compressed message, the message processing module is specifically used to:
[0110] Determine whether the message sampling condition is met at the current time point, and if so, send the original message, dictionary identifier and message compression ratio corresponding to the compressed message to the compression management server.
[0111] In some embodiments, after acquiring the second dictionary, the dictionary acquisition module is specifically used to:
[0112] If a third dictionary sent by the compression management server is continuously obtained, the second dictionary and the third dictionary are saved at the same time, and the first dictionary is deleted, wherein the third dictionary is not completely the same as the second dictionary.
[0113] Combined with reference Figure 6 , is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device includes a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, the above method is implemented.
[0114] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.
[0115] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.
[0116] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0117] An embodiment of the present disclosure further provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program. When the computer program is executed by a processor, the above method is implemented.
[0118] The present disclosure also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the above method is implemented.
[0119] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A dictionary updating method, characterized in that: The method comprises: Acquire multiple message data, the message data comes from at least one target device, the target device includes at least one dictionary, the message data includes an original message, a message compression rate and a dictionary identifier, the message compression rate is obtained after the target device compresses the original message according to the dictionary represented by the dictionary identifier; Based on the dictionary identifier of the first dictionary currently used by the target device, multiple message compression rates corresponding to the first dictionary are screened out from the message data, and the multiple message compression rates are merged to obtain a first compression performance value of the first dictionary; training a second dictionary based on the original message, and determining a second compression performance value of the second dictionary; If the second compression performance value is greater than the first compression performance value, the second dictionary is sent to the target device.
2. The method according to claim 1, characterized in that The fusing the plurality of message compression rates to obtain a first compression performance value of the first dictionary includes: The first compression performance value is obtained by performing weighted calculation on the multiple message compression rates.
3. The method according to claim 1, characterized in that The step of obtaining a second dictionary based on the original message training and determining a second compression performance value of the second dictionary includes: Dividing the original message into a training queue message and a test queue message; Using the training queue message to train to obtain the second dictionary; The test queue message is compressed according to the second dictionary obtained through training, and the second compression performance value of the second dictionary is determined according to the compressed message size and the message size of the test queue message.
4. The method according to claim 3, characterized in that After dividing the original message into a training queue message and a test queue message, the method further includes: Continue to obtain message data, and use the original message in the message data that continues to be obtained to replace the original message in the training queue message.
5. The method according to claim 1, characterized in that If the second compression performance value is greater than the first compression performance value, sending the second dictionary to the target device includes: Acquire a first time point when the first dictionary is sent to the target device; If the second compression performance value is greater than the first compression performance value, the second dictionary is sent to the target device at a preset time interval after the first time point, wherein the preset time interval is greater than or equal to the maximum communication time interval between any two of the target devices.
6. A message compression method, characterized in that: The method is applied to a first target device among a plurality of target devices; the method comprises: In the case where the first dictionary has been obtained, if a second dictionary sent by the compression management server is further obtained, the first dictionary and the second dictionary are saved at the same time, wherein the first dictionary and the second dictionary are not completely the same; According to one of the first dictionary and the second dictionary, the received compressed message is decompressed, or the original message to be sent is compressed.
7. The method according to claim 6, characterized in that The second dictionary has a second dictionary identifier; The decompressing the received compressed message or compressing the original message to be sent according to one of the first dictionary and the second dictionary includes: When it is necessary to send a message to a second target device, compressing the first original message to be sent according to the second dictionary to obtain a first compressed message; The first compressed message is identified by using the second dictionary identifier, and the identified first compressed message is sent to the second target device.
8. The method according to claim 6, characterized in that The first dictionary has a first dictionary identifier, and the second dictionary has a second dictionary identifier; The decompressing the received compressed message or compressing the original message to be sent according to one of the first dictionary and the second dictionary includes: If a second compressed message sent by a second target device is received, obtaining a target dictionary identifier in the second compressed message; When the target dictionary identifier is the first dictionary identifier, decompressing the second compressed message according to the first dictionary to obtain a second original message; When the target dictionary identifier is the second dictionary identifier, the second compressed message is decompressed according to the second dictionary to obtain the second original message.
9. The method according to claim 7 or 8, characterized in that: When sending or receiving a compressed message, the method further includes: Determine whether the message sampling condition is met at the current time point, and if so, send the original message, dictionary identifier and message compression ratio corresponding to the compressed message to the compression management server.
10. The method according to claim 6, characterized in that After acquiring the second dictionary, the method further includes: If a third dictionary sent by the compression management server is continuously obtained, the second dictionary and the third dictionary are saved at the same time, and the first dictionary is deleted, wherein the third dictionary is not completely the same as the second dictionary.
11. A dictionary updating system, characterized in that: The system comprises: A data acquisition module, used to acquire multiple message data, the message data comes from at least one target device, the target device includes at least one dictionary, the message data includes an original message, a message compression rate and a dictionary identifier, the message compression rate is obtained by the target device compressing the original message according to the dictionary represented by the dictionary identifier; a first performance value determination module, configured to filter out a plurality of message compression rates corresponding to a first dictionary currently used by the target device from the message data based on a dictionary identifier of the first dictionary, and fuse the plurality of message compression rates to obtain a first compression performance value of the first dictionary; A second performance value determination module, configured to obtain a second dictionary based on the original message training, and determine a second compression performance value of the second dictionary; A dictionary sending module is used to send the second dictionary to the target device if the second compression performance value is greater than the first compression performance value.
12. A message compression system, characterized in that: The system comprises: A dictionary acquisition module, configured to, if a second dictionary sent by a compression management server is further acquired after the first dictionary has been acquired, save the first dictionary and the second dictionary at the same time, wherein the first dictionary and the second dictionary are not completely the same; The message processing module is used to decompress the received compressed message or compress the original message to be sent according to one of the first dictionary and the second dictionary.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 5, or implements the method according to any one of claims 6 to 10.
14. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, it implements the method as claimed in any one of claims 1 to 5, or implements the method as claimed in any one of claims 6 to 10.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented, or the method according to any one of claims 6 to 10 is implemented.
Citation Information
Cited By
General data dictionary determination method and device, equipment, medium and program product
CN120017722A
