Data compression methods and related equipment

Abstract

This invention provides a data compression method, comprising: acquiring data to be compressed, the data to be compressed including multiple types of fields; calculating the dispersion of each type of field in the data to be compressed to obtain the dispersion of each type of field; and encoding and compressing target type fields with dispersion below a preset threshold according to a preset compression strategy to obtain compressed data corresponding to the data to be compressed. By calculating the dispersion of each type of field in the data to be compressed and compressing target type fields with dispersion below a preset threshold according to a preset compression strategy, compressed data corresponding to the data to be compressed is obtained, thereby reducing the amount of data to be compressed. This solves the problem of high network I / O pressure and slow I / O speed in the prior art of network gateways. At the same time, it can retain information in non-target type fields with dispersion not lower than the preset threshold, avoiding distortion of the data to be compressed during the compression process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a data compression method and related equipment. BACKGROUND

[0002] In order to manage and record the request initiated by the public security network to the video special network, the public security system introduces a gateway as a middleware, and the principle is as shown in Figure 1 The system of the public security network initiates a request to the gateway, and the request parameter has the target URL and request parameter of the video special network. After the gateway receives the request, the gateway analyzes the request and initiates a request with a params parameter to the target URL of the video special network, and then returns the response result to the request initiator of the public security network. Since the gateway supports the demand of multiple companies and multiple systems to access the video special network from the public security network, the network IO pressure of the gateway is large, and the IO speed is slow. SUMMARY

[0003] The embodiment of the present application provides a data compression method, which calculates the dispersion of each type field in the to-be-compressed data, encodes and compresses the target type field with a dispersion lower than a preset threshold according to a preset compression strategy, obtains the compression data corresponding to the to-be-compressed data, so that the data amount of the to-be-compressed data is reduced, thereby solving the problem of large network IO pressure and slow IO speed of the gateway in the prior art, and meanwhile, the information in the non-target type field with a dispersion not lower than the preset threshold can be reserved, so that the distortion of the to-be-compressed data in the compression process is avoided.

[0004] In a first aspect, the embodiment of the present application provides a data compression method, and the method comprises the following steps:

[0005] obtaining to-be-compressed data, wherein the to-be-compressed data comprises fields of multiple types;

[0006] calculating the dispersion of each type field in the to-be-compressed data, to obtain the dispersion of each type field;

[0007] encoding and compressing the target type field with a dispersion lower than a preset threshold according to a preset compression strategy, to obtain the compression data corresponding to the to-be-compressed data.

[0008] Further, the step of calculating the dispersion of each type field in the to-be-compressed data, to obtain the dispersion of each type field, comprises the following steps:

[0009] extracting the values of each type field in the to-be-compressed data, and determining the value range of each type field in the to-be-compressed data according to the values of each type field in the to-be-compressed data;

[0010] According to a value range of each type field in the to-be-compressed data, a dispersion of each type field in the to-be-compressed data is calculated.

[0011] Further, the target type field with a dispersion lower than a preset threshold is encoded and compressed according to a preset compression strategy, to obtain compressed data corresponding to the to-be-compressed data, including:

[0012] According to the dispersion of the target type field, an encoding range of the target type field is obtained;

[0013] According to the encoding range, the target type field is encoded and compressed to obtain a compressed field of the target type field;

[0014] The non-target type field is combined with the compressed field to obtain the compressed data corresponding to the to-be-compressed data.

[0015] Further, after the non-target type field is combined with the compressed field, the method includes:

[0016] A to-be-sent data sequence is obtained, the to-be-sent data sequence including to-be-sent data obtained by combining the non-target type field with the compressed field;

[0017] According to a preset sorting strategy, the to-be-sent data in the to-be-sent data sequence is sorted to obtain a target data sequence.

[0018] Further, the to-be-sent data in the to-be-sent data sequence is sorted according to the preset sorting strategy to obtain the target data sequence, including:

[0019] According to the encoding range of the compressed field and a position of the compressed field in the to-be-sent data, an ordered array of the to-be-sent data is obtained;

[0020] The to-be-sent data is sorted by the ordered data set to obtain the target data sequence.

[0021] Further, the ordered array of the to-be-sent data is obtained according to the encoding range of the compressed field and the position of the compressed field in the to-be-sent data, including:

[0022] According to the encoding range of the compressed field, a value range of the compressed field is determined;

[0023] According to a position sequence of the compressed field in the to-be-sent data, a position of the compressed field in the ordered array is determined;

[0024] Based on the value range and the position in the ordered array, the ordered array of the to-be-sent data is obtained.

[0025] Further, the sorting the to-be-sent data through the ordered array to obtain a target data sequence comprises:

[0026] initializing the ordered array to obtain an initialized ordered array;

[0027] sorting the to-be-sent data in a preset direction in the initialized ordered array to obtain a target data sequence.

[0028] In a second aspect, an embodiment of the present application provides a data compression device, and the device comprises:

[0029] a first obtaining module, configured to obtain to-be-compressed data, wherein the to-be-compressed data comprises a plurality of types of fields;

[0030] a calculating module, configured to calculate a dispersion degree of each type of field in the to-be-compressed data to obtain the dispersion degree of each type of field;

[0031] a compression module, configured to encode and compress a target type of field with a dispersion degree lower than a preset threshold according to a preset compression strategy to obtain compressed data corresponding to the to-be-compressed data.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements steps in a data compression method provided by an embodiment of the present application when executing the computer program.

[0033] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program implements steps in a data compression method provided by an embodiment of the present application when executed by a processor.

[0034] In an embodiment of the present application, to-be-compressed data is obtained, wherein the to-be-compressed data comprises a plurality of types of fields; a dispersion degree of each type of field in the to-be-compressed data is calculated to obtain the dispersion degree of each type of field; and a target type of field with a dispersion degree lower than a preset threshold is encoded and compressed according to a preset compression strategy to obtain compressed data corresponding to the to-be-compressed data. Through the calculation of the dispersion degree of each type of field in the to-be-compressed data and the compression of the target type of field with the dispersion degree lower than the preset threshold according to the preset compression strategy, the compressed data corresponding to the to-be-compressed data is obtained, so that the data amount of the to-be-compressed data is reduced, thereby solving the problem of large network IO pressure and slow IO speed of a network gateway in the prior art, and meanwhile, information in a non-target type of field with a dispersion degree not lower than the preset threshold can be reserved to avoid distortion of the to-be-compressed data in the compression process. BRIEF DESCRIPTION OF DRAWINGS

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of a conventional message bypass gateway provided by an embodiment of the present invention;

[0037] Figure 2 This is a flowchart of a data compression method provided in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of a data compression device provided in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the structure of a computing module provided in an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of the structure of a compression module provided in an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram of another data compression device provided in an embodiment of the present invention;

[0042] Figure 7 This is a schematic diagram of the structure of a sorting module provided in an embodiment of the present invention;

[0043] Figure 8 This is a schematic diagram of the structure of a second processing submodule provided in an embodiment of the present invention;

[0044] Figure 9 This is a schematic diagram of the structure of a sorting submodule provided in an embodiment of the present invention;

[0045] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Please see Figure 2 ,Figure 2 This is a flowchart of a data compression method provided in an embodiment of the present invention, such as... Figure 2 As shown, the data compression method includes the following steps:

[0048] 201. Obtain the data to be compressed.

[0049] The aforementioned data compression method is used to compress request data when the public network sends request data to the private network. The public network can also be referred to as the external network or public safety network, and the private network can also be referred to as the internal network. The public network and the private network are isolated by a network gateway. When a client in the public network needs to request data from a server in the private network, it first sends a request data, also known as a request message, to the network gateway. The request data contains the target URL (Uniform Resource Locator, URL) of the video private network and request parameters. After receiving the request, the network gateway parses the request and initiates a request with the params parameter, which is sent to the target URL of the server in the private network. Then, the server's response is returned to the client in the public network.

[0050] The data to be compressed can be request data sent to the gateway. Specifically, it can be request data sent by a client on the public network to the gateway, or response data sent by a server on a private network to the gateway.

[0051] In this embodiment of the invention, the data to be compressed can be several messages that satisfy the MQTT protocol. The MQTT protocol has a fixed-length header of 1 byte, used to describe information such as control message type, duplicate distribution flag, and quality of service. The third bit is used to describe the duplicate distribution flag. Due to the restrictions of the network gateway, the duplicate distribution logic is completed by the video network Java reverse proxy and the public security network Java proxy. Therefore, this flag bit has no practical function. In this embodiment of the invention, this flag bit can be used to describe whether it is full data. When this flag bit is 0, it is full data; when this flag bit is 1, it is incremental data.

[0052] In one possible embodiment, an MQTT message body is provided, which treats each difference from the previous data (considered as an incremental change) as incremental data. This incremental data consists of three parts: the first part is 12 bits long and describes the start position of the incremental change in the message body, ranging from 0 to 4096. The second part is 12 bits long and describes the length of the incremental change, also ranging from 0 to 4096. The third part is the content of the incremental change, with a length corresponding to the second part. Therefore, each incremental change consists of a fixed three-byte header (the first and second parts), plus the variable-length incremental change content, effectively reducing the size of the transmitted message. It should be noted that each part in the incremental data can be considered as a field.

[0053] In this embodiment of the invention, the data to be compressed includes fields of multiple types. It can be understood that a piece of data to be compressed is obtained by combining fields of multiple types. For example, the data to be compressed is as follows: {"version":5060,"type":3,"year":2021,"sequence":54980018}. That is, the data to be compressed includes the fields "version", "type", "year", and "sequence", where "version" is the version field, "type" is the type field, "year" is the time field, and "sequence" is the sequence field.

[0054] 202. Calculate the dispersion of each type of field in the data to be compressed to obtain the dispersion of each type of field.

[0055] In this embodiment of the invention, the data to be compressed is request data. Each piece of data to be compressed is composed of fields of multiple types, and the field structure of each piece of data to be compressed is identical. The dispersion of each type of field is used to describe the number of possible values ​​for the corresponding field. The following example of data to be compressed will be used for illustration:

[0056] {"version":5060,"type":3,"year:"2021,"sequence":54980018}

[0057] {"version":5050,"type":3,"year:"2021,"sequence":54980019}

[0058] {"version":5038,"type":2,"year:"2020,"sequence":54980020}

[0059] {"version":5030,"type":1,"year:"2020,"sequence":54980021}

[0060] ...

[0061] The version field has a value range of 5060 / 5050 / 5038 / 5030 and can have 4 possible values. The type field has a value range of 3 / 2 / 1 and can have 3 possible values. The year field has a value range of 2021 / 2020 and can have 2 possible values. The sequence field can have any integer greater than 0.

[0062] Optionally, in the step of calculating the dispersion of each type of field in the data to be compressed, the values ​​of each type of field in the data to be compressed can be extracted, and the value range of each type of field in the data to be compressed can be determined based on the values ​​of each type of field in the data to be compressed; and the dispersion of each type of field in the data to be compressed can be calculated based on the value range of each type of field in the data to be compressed.

[0063] Specifically, the aforementioned dispersion can be equal to the number of possible values ​​within the range. For example, if the number of possible values ​​within the range is 5, then the dispersion is 5. Taking the data to be compressed as an example, the value range of the `version` field is 5060 / 5050 / 5038 / 5030, and the number of possible values ​​for the `version` field is 4, so the dispersion of the `version` field is 4; the value range of the `type` field is 3 / 2 / 1, and the number of possible values ​​for the `type` field is 3, so the dispersion of the `type` field is 3; the value range of the `year` field is 2021 / 2020, and the number of possible values ​​for the `year` field is 2, so the dispersion of the `year` field is 2; the value range of the `sequence` field is any integer greater than 0, and the number of possible values ​​for the `sequence` field is infinite, so the dispersion of the `sequence` field is infinite. It can be seen that the version field has a finite number of possible values; for any piece of data to be compressed, the version field will only take one of 5060, 5050, 5038, or 5030. The type field also has a finite number of possible values; for any piece of data to be compressed, the type field will only take one of 3, 2, or 1. The year field also has a finite number of possible values; for any piece of data to be compressed, the year field will only take one of 2021 or 2020. The sequence field can take any integer greater than 0, and can take an unlimited number of values.

[0064] 203. Encode and compress target type fields with dispersion below a preset threshold according to the preset compression strategy to obtain compressed data corresponding to the data to be compressed.

[0065] In this embodiment of the invention, the target type field is a field in the data to be compressed whose dispersion is lower than a preset threshold. The encoding compression involves encoding the target field so that the data volume of the encoded target field is smaller than that of the unencoded target field. Since the dispersion of the target type field is lower than the preset threshold, the encoding amount of the target field can be reduced. After encoding and compressing the target type field, the compressed target type field is combined with non-target fields to obtain the compressed data corresponding to the data to be compressed. The compressed data corresponding to the data to be compressed is used to send to the network gateway.

[0066] In one possible embodiment, for non-target type fields with a dispersion not lower than a preset threshold, another compression method can be used for compression. This other compression method can be a general compression method and is not limited in this embodiment of the invention. By encoding and compressing the target type field using a preset compression strategy and compressing the non-target type field using another compression method, the compressed target type field and the compressed non-target type field are combined to obtain the compressed data corresponding to the data to be compressed, which can reduce the bandwidth required for the transmission of the requested data.

[0067] Optionally, in the step of compressing target type fields with dispersion below a preset threshold according to a preset compression strategy to obtain compressed data corresponding to the data to be compressed, the encoding range of the target type field can be obtained according to the dispersion of the target type field; the target type field can be encoded and compressed according to the encoding range to obtain the compressed field of the target type field; and the non-target type fields can be combined with the compressed field to obtain the compressed data corresponding to the data to be compressed.

[0068] Specifically, the dispersion of each type of field can be determined based on its value range, and then the encoding range of the target type field can be determined based on its dispersion. The encoding range of the target type field can correspond to its value range, with a coded value within one encoding range corresponding to a value within that range.

[0069] Taking the data to be compressed as an example, the value range of the version field is 5060 / 5050 / 5038 / 5030, and the number of possible values ​​for the version field is 4, so the dispersion of the version field is 4, and the encoding range of the version field can be 0 / 1 / 2 / 3; the value range of the type field is 3 / 2 / 1, and the number of possible values ​​for the type field is 3, so the dispersion of the type field is 3, and the encoding range of the type field can be 0 / 1 / 2; the value range of the year field is 2021 / 2020, and the number of possible values ​​for the year field is 2, so the dispersion of the year field is 2, and the encoding range of the year field can be 0 / 1; the value range of the sequence field is any integer greater than 0, and the number of possible values ​​for the sequence field is infinite, so the dispersion of the sequence field is infinite. Therefore, this embodiment of the invention does not encode and compress the sequence field. It can be seen that the value range of the `version` field is a finite number of values; the encoded value of the `version` field for any piece of data to be compressed will only be one of 0, 1, 2, or 3. Similarly, the value range of the `type` field is also a finite number of values; the encoded value of the `type` field for any piece of data to be compressed will only be one of 0, 1, or 2. The value range of the `year` field is also a finite number of values; the encoded value of the `year` field for any piece of data to be compressed will only be one of 0 or 1. Thus, for the data {"version":5060, "type":3, "year":2021, "sequence":54980018}, "version":5060 can be encoded as 0, "type":3 as 2, and "year":2021 as 0. Here, the `sequence` field is a non-target type field. Combining the non-target type field with the compression field yields {0, 3, 2, "sequence":54980018}.

[0070] Optionally, after combining the non-target type fields with the compressed fields, a data sequence to be sent can be obtained. The data sequence to be sent includes the data to be sent after combining the compressed fields with the non-target type fields. The data to be sent in the data sequence to be sent is sorted according to a preset sorting strategy to obtain the target data sequence.

[0071] Furthermore, the data to be sent is compressed data. Each piece of data to be sent is obtained by combining a compressed field of the target type with a field of a non-target type. It should be noted that when combining the compressed field of the target type with the field of the non-target type, the order of the compressed field of the target type and the field of the non-target type remains unchanged. Multiple pieces of compressed data are arranged according to a preset sorting strategy to obtain a compressed data queue, which serves as the compressed data corresponding to the data to be compressed.

[0072] Optionally, in the step of sorting the data to be sent in the data sequence to be sent according to a preset sorting strategy to obtain the target data sequence, an ordered array of data to be sent can be obtained according to the encoding range of the compressed field and the position of the compressed field in the data to be sent; the ordered data array is then sorted to obtain the target data sequence.

[0073] The ordered array mentioned above can be understood as an array in which elements are arranged in a certain order. The structure of the ordered array is related to the structure of the compressed field in the data to be sent, and each element represents a compressed field.

[0074] Optionally, in the step of obtaining an ordered array of data to be sent based on the encoding range of the compressed field and the position of the compressed field in the data to be sent, the possible value range of the compressed field can be determined based on the encoding range of the compressed field; the position of the compressed field in the ordered array can be determined based on the position order of the compressed field in the data to be sent; and the ordered array of data to be sent can be obtained based on the possible value range and the position in the ordered array.

[0075] The ordered array described above is used to arrange the data to be sent. Taking the data to be compressed as an example, the value range of the version field is 5060 / 5050 / 5038 / 5030, and the number of possible values ​​for the version field is 4, so the dispersion of the version field is 4, and the encoding range of the version field can be 0 / 1 / 2 / 3; the value range of the type field is 3 / 2 / 1, and the number of possible values ​​for the type field is 3, so the dispersion of the type field is 3, and the encoding range of the type field can be 0 / 1 / 2; the value range of the year field is 2021 / 2020, and the number of possible values ​​for the year field is 2, so the dispersion of the year field is 2, and the encoding range of the year field can be 0 / 1. The version field is the first position in the data to be compressed, therefore, the compressed field corresponding to the version field is also the first position in the data to be sent; similarly, the type field is the second position in the data to be compressed, therefore, the compressed field corresponding to the type field is also the second position in the data to be sent; similarly, the year field is the third position in the data to be compressed, therefore, the compressed field corresponding to the year field is also the third position in the data to be sent.

[0076] Furthermore, a first array can be created and maintained. The number of elements in the first array is the same as the number of compressed fields of different types. For example, if there are three compressed fields, then there are also three corresponding elements in the first array. Each element corresponds to a compressed field of a certain type. The order of the elements is the same as the order of the compressed fields in the data to be sent. The maximum encoded value of the corresponding compressed field is taken as the value for each element.

[0077] Furthermore, taking the aforementioned data to be compressed as an example, the value range of the `version` field is 5060 / 5050 / 5038 / 5030, and the number of possible values ​​for the `version` field is 4, so the dispersion of the `version` field is 4, and the encoding range of the `version` field can be 0 / 1 / 2 / 3; the value range of the `type` field is 3 / 2 / 1, and the number of possible values ​​for the `type` field is 3, so the dispersion of the `type` field is 3, and the encoding range of the `type` field can be 0 / 1 / 2; the value range of the `year` field is 2021 / 2020, and the number of possible values ​​for the `year` field is 2, so the dispersion of the `year` field is 2, and the encoding range of the `year` field can be 0 / 1. Therefore, the first array is [3, 2, 1], where the order of 3 in the first array corresponds to the order of the `version` field in the data to be sent, the order of 2 in the first array corresponds to the order of the `type` field in the data to be sent, and the order of 1 in the first array corresponds to the order of the `year` field in the data to be sent. In the first array, 3 is the maximum encoded value in the encoding range 0 / 1 / 2 / 3, 2 is the maximum encoded value in the encoding range 0 / 1 / 2, and 1 is the maximum encoded value in the encoding range 0 / 1.

[0078] Optionally, the ordered array includes a first array. In the step of sorting the data to be sent through the ordered array to obtain the target data sequence, the ordered array can be initialized to obtain an initialized ordered array. The data to be sent is then sorted in the initialized ordered array in a preset direction to obtain the target data sequence.

[0079] Specifically, the first array can be initialized so that the value of each element is the maximum encoded value of the corresponding compressed field. Taking the data to be compressed as an example, the first array is initialized to [3, 2, 1], generating the first piece of data to be sent [0, 0, 0]. The data to be compressed corresponding to the first piece of data to be sent [0, 0, 0] is {"version":5060, "type":3, "year:"2021,……}. The data to be sent is sorted with the goal of minimizing the change in the target type field between two adjacent pieces of data to be sent.

[0080] To minimize the changes between the first arrays, changes can be made from left to right or from right to left, changing only the encoding value of one compressed data at a time, thus minimizing the sum of differences between adjacent data and minimizing the total size of the message body. The first array is then used as the ordered array, and the ordered array is used as the target data sequence. After obtaining the target data sequence, the client or server sends the target data sequence to the network gateway.

[0081] In this embodiment of the invention, the changes are performed from right to left, changing one value at a time, resulting in the following ordered array:

[0082] [0,0,0]

[0083] [0,0,1]

[0084] [0,1,1]

[0085] [0,1,0]

[0086] [0,2,0]

[0087] [0,2,1]

[0088] [1,2,1]

[0089] [1,2,0]

[0090] [1,1,0]

[0091] [1,1,1]

[0092] [1,0,1]

[0093] [1,0,0]

[0094] [2,0,0]

[0095] [2,0,1]

[0096] [2,1,1]

[0097] [2,1,0]

[0098] [2,2,0]

[0099] [2,2,1]

[0100] [3,2,1]

[0101] [3,2,0]

[0102] [3,1,0]

[0103] [3,1,1]

[0104] [3,0,1]

[0105] [3,0,0]

[0106] The corresponding data to be compressed is:

[0107] {"version":5060,"type":3,"year:"2021,……}

[0108] {"version":5060,"type":3,"year:"2020,……}

[0109] {“version”:5060,“type”:2,“year:”2020,……}

[0110] {“version”:5060,“type”:2,“year:”2021,……}

[0111] {“version”:5060,“type”:1,“year:”2021,……}

[0112] {“version”:5060,“type”:1,“year:”2020,……}

[0113] {“version”:5050,“type”:1,“year:”2020,……}

[0114] {“version”:5050,“type”:1,“year:”2021,……}

[0115] {“version”:5050,“type”:2,“year:”2021,……}

[0116] {“version”:5050,“type”:2,“year:”2020,……}

[0117] {“version”:5050,“type”:3,“year:”2020,……}

[0118] {“version”:5050,“type”:3,“year:”2021,……}

[0119] {“version”:5038,“type”:3,“year:”2021,……}

[0120] {“version”:5038,“type”:3,“year:”2020,……}

[0121] {“version”:5038,“type”:2,“year:”2020,……}

[0122] {“version”:5038,“type”:2,“year:”2021,……}

[0123] {“version”:5038,“type”:1,“year:”2021,……}

[0124] {"version":5038,"type":1,"year:"2020,……}

[0125] {"version":5030,"type":1,"year:"2020,……}

[0126] {"version":5030,"type":1,"year:"2021,……}

[0127] {"version":5030,"type":2,"year:"2021,……}

[0128] {"version":5030,"type":2,"year:"2020,……}

[0129] {"version":5030,"type":3,"year:"2020,……}

[0130] {"version":5030,"type":3,"year:"2021,……}

[0131] It can be seen that there is only one change between two adjacent data sets to be compressed, minimizing the difference between them. The maximum difference between two adjacent data sets to be sent is only one item, minimizing the overall size of the compressed data. By modifying the input[], sorting rules can be generated for a data list with any number of fields and any number of values ​​for each field.

[0132] In one possible embodiment, the ordered array further includes a second array as an auxiliary array. The length of the second array is the same as that of the first array, but in the second array, the value of each element is either 1 or -1. The ordered array is initialized by setting the value of each element in the second array to 1.

[0133] Specifically, the first array can be initialized so that the value of each element is the maximum encoded value of the corresponding compressed field. Using the data to be compressed as an example, the first array `input[n]` is initialized to [3, 2, 1], and the second array `direction[n]` is initialized to [1, 1, 1]. The first piece of data to be sent, [0, 0, 0], is generated. The data to be compressed corresponding to the first piece of data to be sent, [0, 0, 0], is {"version":5060, "type":3, "year":2021, ...}. For the previous piece of data to be sent, the last bit is taken from left to right. If the value is equal to the upper limit of the compressed field's value range `input[n]` and `direction[n]` is 1, or if the value is equal to the lower limit of the field's value range `0` and `direction[n]` is -1, then the bit cannot be changed further. This bit is skipped, and the same judgment is performed on the previous bit to the left until a bit `m` that can be changed is selected. The value of the `m`th bit is added to `direction[m]` to form a new message. And multiply the values ​​of the (m+1)th to the last item of the direction array by -1.

[0134] The following explanation uses a portion of the ordered array as an example:

[0135] [0,0,0]

[0136] [0,0,1]

[0137] [0,1,1]

[0138] [0,1,0]

[0139] [0,2,0]

[0140] [0,2,1]

[0141] [1,2,1]

[0142] The previous data to be sent is [0,0,0]. From left to right, it contains the version field, type field, and year field. Taking the last bit from left to right, the value is 0, which means the year field is encoded as 0. In the first array [3,2,1], we know that the encoding range of the year field is 0 / 1, and the maximum value can only be 1. Therefore, the last bit of the previous data to be sent [0,0,0] is not equal to the maximum encoded value of the year field, 1. At this time, the initialized second array is [1,1,1]. Adding the last bit of the previous data to be sent [0,0,0] to the last bit of the second array, we get [0,0,1], which is placed after the previous data to be sent [0,0,0]. At this time, there are no elements after the last bit of the second array, so there is no change.

[0143] When the current data to be sent is [0,0,1], the last bit from left to right is 1, indicating that the year field is encoded as 1. In the first array [3,2,1], the year field's encoding range is 0 / 1, with a maximum value of 1. Therefore, the last bit of the previous data to be sent ([0,0,0]) equals the maximum encoded value of the year field, 1. At this point, the second array is [1,1,1], and the last bit of the second array is 1, fulfilling the skip condition. This is because the sum of the last bit of the previous data to be sent ([0,0,1]) and the last bit of the second array (+1) exceeds the encoding range of the year field, so it needs to be skipped. Skip the element corresponding to the year field, take the previous element to the left, which corresponds to the type field. The encoding range of the type field is 0 / 1 / 2. The encoding value of the type field in the previous data to be sent is 0 in [0,0,1]. Therefore, we can add the value of the middle position of the second array [1,1,1] to the middle position of the previous data to be sent [0,0,1] to get [0,1,1] as the new data to be sent and place it after the previous data to be sent [0,0,1]. At this time, there is an element after the middle position of the second array. Multiply all the values ​​after the middle position in the second array by -1 to get the new second array [1,1,-1].

[0144] When the current data to be sent is [0,1,1], taking the last bit from left to right, the value is 1, indicating that the encoding of the year field is 1. In the first array [3,2,1], we know that the encoding range of the year field is 0 / 1, and the maximum value can only be 1. Therefore, the last bit of the previous data to be sent, [0,0,0], is equal to the maximum encoding value of the year field, 1. At this time, the second array is [1,1,-1]. However, since the second array is [1,1,-1], the last bit of the second array is not 1. In the previous data to be sent... The value of the last element in [0,1,1] plus the value of the last element in the second array (-1) does not exceed the encoding range of the year field, so there is no need to skip the corresponding element of the year field. The value of the last element in the previous data to be sent, [0,1,1], plus the value of the last element in the second array, results in [0,1,0] which is placed after the previous data to be sent, [0,1,1]. At this time, there are no elements after the last element in the second array, so there is no change. The second array is [1,1,-1].

[0145] When the current data to be sent is [0,1,0], since the last element of the first array is 0 and the last element of the second array is -1, 0 + (-1) exceeds the encoding range of the year field. Therefore, it is necessary to skip the element position where the year field is located and take the previous element position to the left, which corresponds to the type field. The encoding range of the type field is 0 / 1 / 2. The encoding value of the type field in the previous data to be sent is 1. Therefore, the value of the middle position of the second array [1,1,-1] can be added to the middle position of the previous data to be sent [0,1,0] to get [0,2,0] as the new data to be sent and placed after the previous data to be sent [0,1,0]. At this time, there is an element position after the middle position of the second array. Multiply all the values ​​after the middle position in the second array by -1 to get the new second array [1,1,1].

[0146] When the current data to be sent is [0,2,0], since the last element of the first array is 0 and the last element of the second array is not -1, the element containing the year field is not skipped. The value of the last element of the previous data to be sent [0,2,0] is added to the value of the last element of the second array, resulting in [0,2,1], which is then placed after the previous data to be sent [0,2,0]. At this time, there is no element after the last element of the second array [1,1,1], so it remains unchanged.

[0147] When the current data to be sent is [0,2,1], since the last element of both the first and second arrays is 1, 1+1=2, which exceeds the encoding range of the year field, it is necessary to skip the element containing the year field and take the previous element to the left, which corresponds to the type field. The encoding range of the type field is 0 / 1 / 2. In the previous data to be sent, [0,2,1], the encoding value of the type field is 2. Furthermore, in the second array [1,1,1], the middle element corresponds to the type field value of 1, and 2+1=3, which exceeds the encoding range of the type field. Therefore, it is necessary to skip the type field again and take the previous element to the left, which corresponds to the version field. The first position in the first array has a version field with an encoding range of 0 / 1 / 2 / 3. The previous data to be sent, [0,2,1], has a version field encoding value of 0. Since the value of the element in the second array that corresponds to the version field is not -1, we can add the value of the first position of the second array [1,1,-1] to the first position of the previous data to be sent, [0,2,1]. This results in [1,2,1], which is placed after the previous data [0,2,1]. At this point, there are elements after the first position in the second array. Multiply all values ​​after the first position in the second array by -1 to get the new second array [1,-1,-1].

[0148] The calculation continues until all element bits are fixed, resulting in the arranged data to be sent. This arranged data is then sent to the gateway as the target data sequence. By sorting the data to be sent, the total difference between adjacent data pairs is minimized, thus minimizing the total data size of the target data sequence and reducing the gateway's I / O burden.

[0149] In this embodiment of the invention, data to be compressed is obtained, which includes fields of multiple types. The dispersion of each type of field in the data to be compressed is calculated to obtain the dispersion of each type of field. Target type fields with dispersion below a preset threshold are encoded and compressed according to a preset compression strategy to obtain compressed data corresponding to the data to be compressed. By calculating the dispersion of each type of field in the data to be compressed and compressing target type fields with dispersion below a preset threshold according to a preset compression strategy, compressed data corresponding to the data to be compressed is obtained, thereby reducing the amount of data to be compressed. This solves the problem of high network I / O pressure and slow I / O speed in existing network gateways. At the same time, information in non-target type fields with dispersion not lower than the preset threshold can be retained, avoiding distortion of the data to be compressed during the compression process.

[0150] It should be noted that the data compression method provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that are capable of data compression.

[0151] Optional, please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a data compression device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the device includes:

[0152] The first acquisition module 301 is used to acquire data to be compressed, the data to be compressed including fields of multiple types;

[0153] The calculation module 302 is used to calculate the dispersion of each type of field in the data to be compressed, and to obtain the dispersion of each type of field.

[0154] Compression module 303 is used to encode and compress target type fields with dispersion below a preset threshold according to a preset compression strategy to obtain compressed data corresponding to the data to be compressed.

[0155] Furthermore, such as Figure 4 As shown, the computing module 302 includes:

[0156] Extraction submodule 3021 is used to extract the values ​​of each type of field in the data to be compressed, and to determine the value range of each type of field in the data to be compressed based on the values ​​of each type of field in the data to be compressed;

[0157] The calculation submodule 3022 is used to calculate the dispersion of each type of field in the data to be compressed based on the value range of each type of field in the data to be compressed.

[0158] Furthermore, such as Figure 5 As shown, the compression module 303 includes:

[0159] The first processing submodule 3031 is used to obtain the encoding range of the target type field based on the dispersion of the target type field;

[0160] The second processing submodule 3032 is used to encode and compress the target type field according to the encoding range to obtain the compressed field of the target type field;

[0161] The third processing submodule 3033 is used to combine the non-target type field with the compression field to obtain the compressed data corresponding to the data to be compressed.

[0162] Furthermore, such as Figure 6 As shown, the device further includes:

[0163] The second acquisition module 304 is used to acquire a data sequence to be sent, the data sequence to be sent including the data to be sent after combining the non-target type field and the compressed field;

[0164] The sorting module 305 is used to sort the data to be sent in the data sequence to be sent according to a preset sorting strategy to obtain the target data sequence.

[0165] Furthermore, such as Figure 7 As shown, the sorting module 305 includes:

[0166] The fourth processing submodule 3051 is used to obtain an ordered array of the data to be sent based on the encoding range of the compressed field and the position of the compressed field in the data to be sent.

[0167] The sorting submodule 3052 is used to sort the data to be sent using the ordered data group to obtain the target data sequence.

[0168] Furthermore, such as Figure 8 As shown, the fourth processing submodule 3051 includes:

[0169] The first determining unit 30511 is used to determine the possible value range of the compressed field based on the encoding range of the compressed field.

[0170] The second determining unit 30512 is used to determine the position of the compressed field in the ordered array according to the position order of the compressed field in the data to be sent;

[0171] The first processing unit 30513 is used to obtain the ordered array of the data to be sent based on the range of possible values ​​and the position in the ordered array.

[0172] Furthermore, such as Figure 9 As shown, the sorting submodule 3052 includes:

[0173] The second processing unit 30521 is used to initialize the ordered array to obtain an initialized ordered array;

[0174] The sorting unit 30522 is used to sort the data to be sent in an initialized ordered array in a preset direction to obtain a target data sequence.

[0175] It should be noted that the data compression device provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that are capable of data compression.

[0176] The data compression apparatus provided in this embodiment of the invention can implement all the processes of the data compression method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, further details are omitted here.

[0177] See Figure 10 , Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 10 As shown, it includes: a memory 1002, a processor 1001, and a computer program for a data compression method stored on the memory 1002 and executable on the processor 1001, wherein:

[0178] The processor 1001 is used to call the computer program stored in the memory 1002 and perform the following steps:

[0179] Obtain the data to be compressed, which includes fields of multiple types;

[0180] The dispersion of each type of field in the data to be compressed is calculated to obtain the dispersion of each type of field;

[0181] The target type fields with a dispersion lower than a preset threshold are encoded and compressed according to a preset compression strategy to obtain the compressed data corresponding to the data to be compressed.

[0182] Furthermore, the processor 1001 performs the calculation of the dispersion of each type of field in the data to be compressed to obtain the dispersion of each type of field, including:

[0183] Extract the values ​​of each type of field in the data to be compressed, and determine the value range of each type of field in the data to be compressed based on the values ​​of each type of field in the data to be compressed;

[0184] Calculate the dispersion of each type of field in the data to be compressed based on the value range of each type of field in the data to be compressed.

[0185] Furthermore, the processor 1001 executes the encoding and compression of the target type field with a dispersion lower than a preset threshold according to a preset compression strategy to obtain the compressed data corresponding to the data to be compressed, including:

[0186] The encoding range of the target type field is obtained based on the dispersion of the target type field;

[0187] The target type field is encoded and compressed according to the encoding range to obtain the compressed field of the target type field;

[0188] The non-target type field is combined with the compression field to obtain the compressed data corresponding to the data to be compressed.

[0189] Furthermore, after the processor 1001 performs the process of combining the non-target type field with the compressed field, the method includes:

[0190] Obtain a data sequence to be sent, the data sequence to be sent including the data to be sent after combining the non-target type field and the compressed field;

[0191] The data to be sent in the data sequence to be sent is sorted according to a preset sorting strategy to obtain the target data sequence.

[0192] Furthermore, the process executed by processor 1001 to sort the data to be sent in the data sequence according to a preset sorting strategy to obtain the target data sequence includes:

[0193] Based on the encoding range of the compressed field and the position of the compressed field in the data to be sent, an ordered array of the data to be sent is obtained;

[0194] The data to be sent is sorted by the ordered data group to obtain the target data sequence.

[0195] Furthermore, the step of processor 1001 executing the step of obtaining an ordered array of the data to be sent based on the encoding range of the compressed field and the position of the compressed field in the data to be sent includes:

[0196] Based on the encoding range of the compressed field, determine the possible value range of the compressed field;

[0197] The position of the compressed field in the ordered array is determined based on the order of the compressed field in the data to be sent;

[0198] Based on the range of possible values ​​and the position in the ordered array, the ordered array of the data to be sent is obtained.

[0199] Furthermore, the step of processor 1001 sorting the data to be sent using the ordered array to obtain the target data sequence includes:

[0200] The ordered array is initialized to obtain an initialized ordered array;

[0201] The data to be sent is sorted in a preset direction in an initialized ordered array to obtain the target data sequence.

[0202] The electronic device provided in this embodiment of the invention can implement all the processes of the data compression method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, further details are omitted here.

[0203] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the data compression method or application-side data compression method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0204] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0205] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A data compression method characterized by, The method comprises the following steps: obtaining to-be-compressed data, the to-be-compressed data comprising a plurality of types of fields; extracting values of the types of fields in the to-be-compressed data, determining value ranges of the types of fields in the to-be-compressed data according to the values of the types of fields in the to-be-compressed data, and calculating dispersions of the types of fields in the to-be-compressed data according to the value ranges of the types of fields in the to-be-compressed data; obtaining an encoding range of a target type of field according to the dispersion of the target type of field, encoding and compressing the target type of field according to the encoding range to obtain a compressed field of the target type of field, and combining non-target types of fields with the compressed field to obtain compressed data corresponding to the to-be-compressed data.

2. The data compression method of claim 1, wherein, After the combining of the non-target types of fields with the compressed field, the method comprises: obtaining a to-be-sent data sequence, the to-be-sent data sequence comprising to-be-sent data obtained by combining the non-target types of fields with the compressed field; sorting the to-be-sent data in the to-be-sent data sequence according to a preset sorting strategy to obtain a target data sequence.

3. The data compression method of claim 2, wherein, The sorting of the to-be-sent data in the to-be-sent data sequence according to the preset sorting strategy to obtain the target data sequence comprises: obtaining an ordered array of the to-be-sent data according to the encoding range of the compressed field and a position of the compressed field in the to-be-sent data; sorting the to-be-sent data by using the ordered array to obtain the target data sequence.

4. The data compression method of claim 3, wherein, The obtaining of the ordered array of the to-be-sent data according to the encoding range of the compressed field and the position of the compressed field in the to-be-sent data comprises: determining a value range of the compressed field according to the encoding range of the compressed field; determining a position of the compressed field in the ordered array according to an order of the position of the compressed field in the to-be-sent data; obtaining the ordered array of the to-be-sent data based on the value range and the position in the ordered array.

5. The data compression method of claim 4, wherein, The sorting of the to-be-sent data by using the ordered array to obtain the target data sequence comprises: initializing the ordered array to obtain an initialized ordered array; sorting the to-be-sent data in the initialized ordered array in a preset direction to obtain the target data sequence.

6. A data compression device, characterized by, The apparatus comprises: a first obtaining module configured to obtain to-be-compressed data, the to-be-compressed data comprising a plurality of types of fields; a calculating module configured to extract values of the types of fields in the to-be-compressed data, determine value ranges of the types of fields in the to-be-compressed data according to the values of the types of fields in the to-be-compressed data, and calculate dispersions of the types of fields in the to-be-compressed data according to the value ranges of the types of fields in the to-be-compressed data; The compression module is used for obtaining an encoding range of the target type field according to a dispersion of the target type field; encoding and compressing the target type field according to the encoding range to obtain a compressed field of the target type field; and combining a field of a non-target type with the compressed field to obtain compressed data corresponding to the to-be-compressed data.

7. An electronic device, comprising: The data compression method comprises the following steps: The memory, the processor and the computer program stored in the memory and executable on the processor are used for realizing the steps in the data compression method according to any one of claims 1 to 5 when the processor executes the computer program.

8. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium and is used for realizing the steps in the data compression method according to any one of claims 1 to 5 when the processor executes the computer program.

Images