Numerical value type data compression method and device, medium and equipment
By mapping the numerical characters and characters of numerical type data into binary data, efficiently utilizing byte space during storage, combined with encryption processing, the problems of big data storage space and sensitive data security are solved, and efficient data compression and Desensitization.
Patent Information
- Application Number
- CN202310808167.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-03
- Publication Date
- 2025-01-03
AI Technical Summary
In the era of big data, numerical string data occupies huge storage space, and sensitive data is easily leaked in an uncompressed state, resulting in insufficient security.
By mapping numerical characters in string data of numerical type into four-bit binary data, and each byte has one binary data at the time of storage, the characters are further mapped into eight-bit binary data. Combined with encryption processing, data compression and desensitization are achieved.
The storage space of numerical type data is significantly reduced, almost half, while improving data security and preventing sensitive information leakage.
Smart Images
Figure CN119254239A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data compression, and in particular to a method, device, medium and equipment for compressing numerical data. Background Art
[0002] In today's big data era, various types of data, including user information, are massive, and numerical string data accounts for a large proportion of them. Numeric string data includes pure numeric data (such as mobile phone numbers, ages, etc.), date data, amount data, ID card number data, etc. The characteristics of these data are that numbers (0 to 9) occupy the vast majority, and also include a small number of symbols. The amount of big data is massive, and the above-mentioned numerical string data will take up a lot of storage space when stored. Summary of the invention
[0003] In order to address the deficiencies of the prior art, the present application provides a method, device, medium and equipment for compressing numerical data, which greatly reduces the storage space of numerical data and desensitizes sensitive data to a certain extent.
[0004] This application provides the following technical solutions:
[0005] In a first aspect, the present application provides a method for compressing numerical data, the method comprising:
[0006] Get string data of numeric type;
[0007] Mapping the numeric characters in the string data into first-type four-bit binary data according to a set mapping relationship;
[0008] Wherein, the first four-bit binary data of each digital character mapping is different;
[0009] The first-type four-bit binary data is stored. During the storage process, the upper four bits and the lower four bits of each byte respectively store one of the first-type four-bit binary data.
[0010] Furthermore, the method further comprises:
[0011] Mapping the first type of characters in the string data into the second type of four-bit binary data according to a set mapping relationship;
[0012] The second-category four-bit binary data of each of the first-category character mappings are all different, and each of the second-category four-bit binary data is different from each of the first-category four-bit binary data;
[0013] The second type of four-bit binary data is stored. During the storage process, the upper four bits and the lower four bits of each byte respectively store one of the second type of four-bit binary data.
[0014] Furthermore, the method further comprises:
[0015] Mapping the second type of characters in the string data into eight-bit binary data according to a set mapping relationship;
[0016] The eight-bit binary data of each of the second-category character mappings are all different, and the upper four bits of each of the eight-bit binary data are all different from each of the first-category four-bit binary data and each of the second-category four-bit binary data;
[0017] The eight-bit binary data is stored, and during the storage process, each byte stores one of the eight-bit binary data.
[0018] Furthermore, the method further comprises:
[0019] When the first bit of the two adjacent bits in the string data is the numeric character or the first type of character, and the second bit is the second type of character; the first type of four-bit binary data or the second type of four-bit binary data mapped from the numeric character or the first type of character is stored in the lower four bits of the selected byte, and the set four-bit binary data is stored in the upper four bits of the selected byte;
[0020] The set four-bit binary data is different from the upper four bits of each of the first-type four-bit binary data, each of the second-type four-bit binary data, and each of the eight-bit binary data.
[0021] Furthermore, the digital characters 0 to 9 are mapped to 0x1 to 0x9 respectively, the number of the first category characters is 4, the 4 first category characters are mapped to 0xA to 0xD respectively, the number of the second category characters is 16, the 16 second category characters are mapped to 0xE0 to 0xEF respectively, and the set four-bit binary data is 0xF.
[0022] Furthermore, the method further comprises:
[0023] The mapped first category four-bit binary data, the second category four-bit binary data and the eight-bit binary data are encrypted and stored.
[0024] Furthermore, the method further comprises:
[0025] Read the stored data in bytes in sequence and judge the upper four bits of the read data;
[0026] When the upper four bits of the read data belong to the first category of four-bit binary data or the second category of four-bit binary data, the upper four bits and the lower four bits of the read data are mapped to the digital characters or the first category of characters according to the set mapping relationship;
[0027] When the upper four bits of the read data belong to the upper four bits of the eight-bit binary data, mapping the read data into the second type of characters according to the set mapping relationship;
[0028] When the upper four bits of the read data belong to the set four-bit binary data, the lower four bits of the read data are mapped to the digital characters or the first type of characters according to the set mapping relationship.
[0029] In a second aspect, the present application provides a numerical data compression device, the device comprising:
[0030] The data acquisition module is used to obtain string data of numerical type;
[0031] A first mapping module, used for mapping the numeric characters in the string data into first-type four-bit binary data according to a set mapping relationship;
[0032] Wherein, the first four-bit binary data of each digital character mapping is different;
[0033] The first storage module is used to store the first-category four-bit binary data. During the storage process, the upper four bits and the lower four bits of each byte respectively store one of the first-category four-bit binary data.
[0034] Furthermore, the device also includes:
[0035] A second mapping module, used for mapping the first type of characters in the string data into second type of four-bit binary data according to a set mapping relationship;
[0036] The second-category four-bit binary data of each of the first-category character mappings are all different, and each of the second-category four-bit binary data is different from each of the first-category four-bit binary data;
[0037] The second storage module is used to store the second-category four-bit binary data. During the storage process, the upper four bits and the lower four bits of each byte respectively store one of the second-category four-bit binary data.
[0038] Furthermore, the device also includes:
[0039] A third mapping module, used for mapping the second type of characters in the string data into eight-bit binary data according to a set mapping relationship;
[0040] The eight-bit binary data of each of the second-category character mappings are all different, and the upper four bits of each of the eight-bit binary data are all different from each of the first-category four-bit binary data and each of the second-category four-bit binary data;
[0041] The third storage module is used to store the eight-bit binary data. During the storage process, each byte stores one eight-bit binary data.
[0042] Furthermore, the device also includes:
[0043] A fourth storage module is used for storing the first category four-bit binary data or the second category four-bit binary data mapped from the numeric character or the first category character into the lower four bits of the selected byte, and storing the set four-bit binary data in the upper four bits of the selected byte, when the first bit of the two adjacent bits in the string data is the numeric character or the first category character and the second bit is the second category character;
[0044] The set four-bit binary data is different from the upper four bits of each of the first-type four-bit binary data, each of the second-type four-bit binary data, and each of the eight-bit binary data.
[0045] Furthermore, the digital characters 0 to 9 are mapped to 0x1 to 0x9 respectively, the number of the first category characters is 4, the 4 first category characters are mapped to 0xA to 0xD respectively, the number of the second category characters is 16, the 16 second category characters are mapped to 0xE0 to 0xEF respectively, and the set four-bit binary data is 0xF.
[0046] Furthermore, the device also includes:
[0047] The encryption module is used for encrypting and storing the mapped first category four-bit binary data, the second category four-bit binary data and the eight-bit binary data.
[0048] Furthermore, the device also includes:
[0049] A judgment module is used to read the stored data in bytes in sequence and judge the upper four bits of the read data;
[0050] A first restoration module is used for mapping the upper four bits and lower four bits of the read data to the digital characters or the first type of characters according to a set mapping relationship respectively when the upper four bits of the read data belong to the first type of four-bit binary data or the second type of four-bit binary data;
[0051] A second restoration module is used for mapping the read data into the second type of characters according to a set mapping relationship when the upper four bits of the read data belong to the upper four bits of the eight-bit binary data;
[0052] The third restoration module is used to map the lower four bits of the read data to the digital characters or the first type of characters according to the set mapping relationship when the upper four bits of the read data belong to the set four-bit binary data.
[0053] In a third aspect, the present application provides a computer-readable storage medium for numerical type data, comprising a memory for storing processor-executable instructions, wherein the instructions, when executed by the processor, implement the steps of the numerical type data method described in the first aspect.
[0054] In a fourth aspect, the present application provides a device for numerical type data, characterized in that it includes at least one processor and a memory storing computer executable instructions, and when the processor executes the instructions, the steps of the numerical type data method described in the first aspect are implemented.
[0055] This application has the following beneficial effects:
[0056] This application maps the digital characters in the string data of the numerical type to the first-class four-bit binary data according to the set mapping relationship, and compresses the digital characters. When storing, the upper four bits and the lower four bits of each byte each store a first-class four-bit binary data. This application reduces the storage space of the digital characters by half, greatly reducing the storage space of the string data of the numerical type. The compressed size is only about half of the original data, almost reducing the storage space of the string data of the numerical type by half. While compressing the data, it also plays a certain role in encryption and desensitization, greatly protecting the user's sensitive data, and greatly improving security. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 This is a flowchart of the numerical data compression method of the present application;
[0058] Figure 2 A schematic diagram of a byte in a computer storage system;
[0059] Figure 3 is a schematic diagram for demonstrating Example 1;
[0060] Figure 4 is a schematic diagram for demonstrating Example 2;
[0061] Figure 5 It is a schematic diagram of a numerical data compression device of the present application. DETAILED DESCRIPTION
[0062] In order to make the technical problems, technical solutions and advantages to be solved by the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0063] The present application embodiment provides a method for compressing numerical data. Figure 1 As shown, the method includes:
[0064] S100: Obtain string data of numeric type.
[0065] String data of numerical type includes data of pure numerical type (such as mobile phone number, age, etc.), data of date type, data of amount type, data of ID card number, etc. The characteristics of these data are that the majority of them are numeric characters (0 to 9), and a small number of symbols are also included. For example, space (a separator for data items), minus sign (a separator between year, month, and day in date type), colon (a separator between hour, minute, and second in date type), decimal point (used in amount value), carriage return line feed, comma, x (or capital X) at the end of ID card, and currency unit (such as ￥, $, etc.).
[0066] The smallest unit of computer storage information is bit (also known as bit). The number of storage units contained in the memory is called storage capacity. Its basic unit of measurement is byte (B for short). A byte consists of 8 bits. Figure 2 In addition, there are KB, MB, GB, TB, etc. The conversion relationship between them is 1Byte=8bit, 1KB=1024B, 1MB=1024KB, 1GB=1024MB, 1TB=1024GB.
[0067] Numerical data types are stored in the form of strings, and each character in the string (numbers are also considered as a type of character, which is a numeric character) is stored in one byte. For example, a string data such as "0123456789" contains 10 numeric characters and needs 10 bytes to be stored in the computer.
[0068] According to ASCII encoding, the codes of the numeric characters 0 to 9 are 0x30 to 0x39. For convenience, the codes are expressed in hexadecimal (i.e. 0x30 to 0x39). If expressed in binary, the binary codes of the numeric characters 0 to 9 are 00110000 to 00111001. Then the ten bytes of the string "0123456789" in the storage space are: 0x30 0x31 0x32 0x33 0x34 0x35 0x36 0x37 0x38 0x39. Corresponding to binary, it is 0011000000110001 00110010 00110011 00110100 00110101 001101100011011100111000 00111001.
[0069] Since numeric characters (0 to 9) account for the vast majority of the string data of the numeric type, if the storage of numeric characters (0 to 9) can be compressed, the amount of stored data can be significantly reduced.
[0070] S200: Map the numeric characters in the character string data into first-type four-bit binary data according to a set mapping relationship.
[0071] The first four-bit binary data of each digital character mapping is different.
[0072] A byte consists of 8 bits and can store 2 8 = 256 numbers, while there are only ten digital characters (0 to 9). For the number 9 (pure number, not a digital character), its binary mathematical representation is: 0000 1001. It can be seen that the number 9 only uses four binary digits, corresponding to a byte, which only occupies the lower 4 bits of the 8 bits of a byte, and the upper 4 bits are 0. The 10 numbers 0-9, the largest 9, only needs to occupy 4 binary bits, so only four bits in a byte can be used to store 0-9.
[0073] Based on the above foundation, the present application maps the digital characters 0 to 9 into the form of first-category four-bit binary data according to a set mapping relationship, and the first-category four-bit binary data mapped to each digital character is different.
[0074] In one example, the numeric characters 0 to 9 can be mapped from their ASCII codes 0x30 to 0x39 to 0x1 to 0x9, which are mapped to 0000 0001 0010 00110100 0101 0110 0111 1000 1001. In this way, each numeric character that originally occupies one byte (8 bits) is mapped to occupy only half a byte (4 bits).
[0075] In other examples, the digital characters 0 to 9 may not be mapped to 0000 to 1001 in sequence, but may be mapped according to a set relationship, which can achieve a certain encryption effect.
[0076] S300: storing the first type of four-bit binary data. During the storage process, the upper four bits and the lower four bits of each byte respectively store one first type of four-bit binary data.
[0077] After mapping the numeric characters 0 to 9 to 0000 to 1001, the numeric characters are represented by four bits of binary data. Therefore, when storing, one byte has 8 bits, and one numeric character can be stored in the upper 4 bits and the lower 4 bits respectively, that is, one byte can store two numeric characters. Through the above method of the present application, the string data of the numerical type can be stored in one byte for every 2 numeric characters, which can reduce the storage space by half.
[0078] Taking the aforementioned string "0123456789" as an example, the traditional method requires 10 bytes to store (0x30 0x31 0x32 0x33 0x34 0x35 0x36 0x37 0x38 0x39). Through the method of the present application, after every 2 numeric characters are stored in 1 byte, only 5 bytes of storage space are required (0x01 0x23 0x45 0x67 0x89).
[0079] Since the numeric characters (0-9) occupy the vast majority of the string data of the numeric type, and other characters are only a small number, after reducing the storage space of the numeric characters by half, the storage space of the overall string data of the numeric type can be almost reduced by half, and the compression rate almost reaches a fixed 50%. For example, in a large system database (such as the banking industry, insurance industry, large online shopping platform, etc.), if the storage space occupied by the string data of the numeric type is 100TB, after compression using the method described in the present application, it can be reduced to about 50% of the original, that is, only about 50TB of storage space is needed, which greatly saves storage space.
[0080] This application maps the numeric characters in the string data of the numerical type to the first-class four-bit binary data according to the set mapping relationship, and compresses the numeric characters. When storing, the upper four bits and the lower four bits of each byte each store a first-class four-bit binary data. This application reduces the storage space of the numeric characters by half, greatly reducing the storage space of the string data of the numerical type. The compressed size is only about half of the original data, almost reducing the storage space of the string data of the numerical type by half.
[0081] For numeric string data, especially sensitive data, such as the user's age, mobile phone number, ID number, amount of money deposited in the bank, etc. If it is saved in plain text, once the data is leaked, the person who obtains the information can easily know other people's age, mobile phone number, ID number, amount of money deposited in the bank and other sensitive data.
[0082] This application maps digital characters to first-class four-bit binary data, compresses the data and saves it, which not only greatly reduces the storage space, but also plays the role of encryption and desensitization. Even if the data is leaked, the person who obtains the information can only see the compressed data after mapping. According to the compressed data after mapping, it is impossible to get what the data before mapping was. Therefore, even if the data is leaked, the person who obtains the information cannot understand the data, and it is impossible to know the sensitive data of others. The user's sensitive data is greatly protected, and the security is greatly improved.
[0083] Since the majority of the string data of the numerical type are digital characters (0 to 9), and there are only a few other characters, after the digital characters are mapped and compressed, other characters may not be compressed. Of course, in order to further improve the compression effect, other characters may also be compressed. Based on this, the method of the present application may also include:
[0084] S400: Mapping the first type of characters in the character string data into the second type of four-bit binary data according to the set mapping relationship.
[0085] The second-category four-bit binary data of each first-category character mapping is different, and each second-category four-bit binary data is different from each first-category four-bit binary data.
[0086] The maximum value that can be represented by four-bit binary data is: 1111, which is 15 in decimal. Thus, four-bit binary data can represent up to 16 different data (0 to 15). However, the aforementioned digital characters 0 to 9 only use 10 data of the four-bit binary data, so there are 6 data vacant in the four-bit binary data, which can be used to compress other characters besides digital characters.
[0087] For example, the four commonly used symbols of space, minus sign, colon and decimal point can be used as the first type of characters, and mapped to the second type of four-bit binary data (i.e., 6 idle data) different from the first type of four-bit binary data of 0 to 9 according to the set mapping relationship, and stored in 4-bit binary. For example: 0xA represents space, 0xB represents minus sign, 0xC represents decimal point, and 0xD represents colon.
[0088] It should be noted that the first type of characters in this application is not limited to the aforementioned space, minus sign, colon and decimal point. Any 1 to 4 other non-numeric characters can be selected according to the actual application. The first type of characters are not limited to 0xA to 0xD after mapping, and other data that is not mapped by 0 to 9 can be used. After mapping, similar to digital characters, the same compression method is used for compression storage.
[0089] S500: storing the second type of four-bit binary data. During the storage process, the upper four bits and the lower four bits of each byte respectively store one second type of four-bit binary data.
[0090] Taking the aforementioned space, minus sign, colon and decimal point, the four first-class characters are mapped to 0xA to 0xD as an example, a total of 14 values are occupied in the 4-bit binary, which is 0xD in hexadecimal. In a byte, the maximum value used is 0xDD. And a byte can be a maximum of 0xFF. Therefore, there is still some storage space left in a byte, and some special characters can be added. Based on this, the method of the present application can also include:
[0091] S600: Map the second type of characters in the character string data into eight-bit binary data according to the set mapping relationship.
[0092] The eight-bit binary data of each second-category character mapping is different, and the upper four bits of each eight-bit binary data are different from each first-category four-bit binary data and each second-category four-bit binary data.
[0093] For some uncommon special characters, this application treats them as second-class characters and maps them to eight-bit binary data, using a whole byte for storage. In addition, to prevent the eight-bit binary data of the second-class characters from being confused with the previous four-bit binary data of the first class and the four-bit binary data of the second class, this application uses some storage space left in a byte for storage. Specifically, the upper four bits of the eight-bit binary data should not be the same as the aforementioned four-bit binary data of the first class and the four-bit binary data of the second class.
[0094] For example, special characters such as carriage return, line feed, comma, and the last x (or capital X) of an ID card can be used as second-category characters. It should be noted that this is only an example, and the present application does not limit the number and mapping relationship of first-category characters and second-category characters. However, since the storage space of first-category characters is half a byte and the storage space of second-category characters is one byte, commonly used characters can be classified as first-category characters, and uncommon characters can be classified as second-category characters.
[0095] Exemplarily, one byte of 0xE0-0xEF data can be used as the mapping of the second type of characters. In this way, 16 second type of characters can be accommodated, and they will not be confused with two characters (i.e., numeric characters and first type of characters) stored in a normal one byte.
[0096] When the number of special characters is large and cannot be fully represented by 16 data positions, n first-class characters can be reduced and 16n second-class characters can be added. For example, when there are 3 first-class characters, 0xD0~0xDF and 0xE0~0xEF can be used to provide 32 data as the mapping of the second-class characters, and a maximum of 32 second-class characters can be represented.
[0097] S700: Eight bits of binary data are stored. During the storage process, each byte stores an eight-bit binary data.
[0098] After the eight-bit binary data is mapped, it can be stored. The storage method is different from that of digital characters and first-class characters. Instead, each byte stores an eight-bit binary data.
[0099] The present application expands the availability and applicability of the compression method of the present application by setting up compression mapping and storage methods for first-category characters and second-category characters. The application is not limited to compressing character strings of pure digits, but can also compress data such as date types, amount types, and ID card numbers, with a high compression rate.
[0100] After testing, using the numbers of 0-9, 10000 digital strings are randomly generated, in the form of 100 rows and 100 columns, and the size after being saved as a text file is 10200 bytes (including carriage return line feed characters). The size after being compressed with rar software is 5241 bytes, and the size after being compressed with the method described in the application is 5200 bytes. Since rar software is already a very mature and professional compression software, its compression rate is already very high, and the method of the application can also be further compressed on this basis, indicating that the application has a good compression rate.
[0101] During the compression process of a string data of a numeric type, there may be two adjacent characters, the first of which may be a numeric character or a first-category character, and the second of which may be a second-category character. In this case, the first numeric character or the first-category character can only be stored in one byte. When decompressing, this numeric character or the first-category character stored in one byte cannot be distinguished from the two numeric characters (or first-category characters) stored in one byte normally, resulting in decompression errors.
[0102] To solve the above problems, the method of the present application also includes:
[0103] S800: When the first bit of the two adjacent bits in the character string data is a numeric character or a first-category character, and the second bit is a second-category character; the first-category four-bit binary data or the second-category four-bit binary data mapped from the numeric character or the first-category character is stored in the lower four bits of the selected byte, and the set four-bit binary data is stored in the upper four bits of the selected byte.
[0104] The set four-bit binary data is different from the upper four bits of each first-type four-bit binary data, each second-type four-bit binary data, and each eight-bit binary data.
[0105] That is, when a byte is used to store a numeric character (or a first-class character) alone, it is stored in the lower four bits of a byte, and a set four-bit binary data is stored in the upper four bits of the byte as an identifier. The set four-bit binary data is required to be different from the upper four bits of the four-bit binary data mapped to the numeric character and the first-class character and the eight-bit binary data mapped to the second-class character, so as to distinguish them.
[0106] Exemplarily, the four-bit binary data is set to 0xF, 0xF is fixedly stored in the upper four bits, and the four-bit binary data after the numeric character (or the first type of character) is mapped is stored in the lower four bits, that is, 0xF0-0xFF is used for storage. This can be distinguished from the normal situation of storing two numeric characters (or the first type of characters) in one byte. Because when one byte stores two numeric characters (or the first type of characters), the maximum number used is 0xDD, and 0xE0-0xEF is used to store a single second type of character in one byte, and 0xF0-0xFF is used to store a single byte to store numeric characters (or the first type of characters), there will be no confusion.
[0107] In order to further improve the security of sensitive data and achieve better compression and desensitization effects, the method of this application also includes:
[0108] The mapped first-category four-bit binary data, second-category four-bit binary data and eight-bit binary data are encrypted and stored.
[0109] This application does not impose any restrictions on the encryption method. For example, any set of random numbers can be used as a key to encrypt the mapped data, or any other encryption method can be used to encrypt the mapped data.
[0110] When encrypting, the mapped first-category four-bit binary data, second-category four-bit binary data and eight-bit binary data need to be encrypted. The set four-bit binary data (0xF) serves as an identifier, not specific data, and may not be encrypted, but may also be encrypted.
[0111] The compression method of this application is lossless compression, which can restore the original data without loss. The restoration method of this application is as follows:
[0112] S900: Read the stored data in bytes in sequence, and judge the upper four bits of the read data.
[0113] Since the basic unit of computer data processing is byte, the compressed and stored data needs to be read in bytes. The read data needs to be restored to the original data. When restoring, the situation is judged based on the high four bits of the read data, and the restoration is performed according to different situations. If encryption is performed during compression, it needs to be decrypted first and then restored.
[0114] When the upper four bits of the read data belong to the first category of four-bit binary data or the second category of four-bit binary data, the upper four bits and the lower four bits of the read data are mapped to digital characters or first category characters according to a set mapping relationship.
[0115] Exemplarily, when the upper four bits of the read data are 0x0 to 0xD, it indicates that one byte stores two numeric characters or first-category characters, so the upper four bits and lower four bits of the read byte are restored respectively to obtain two data (numeric characters or first-category characters).
[0116] When the upper four bits of the read data belong to the upper four bits of the eight-bit binary data, the read data is mapped to the second type of characters according to the set mapping relationship.
[0117] Exemplarily, when the upper four bits of the read data are 0xE, it indicates that one byte stores a second-category character, and therefore the read byte is restored as a whole to obtain one data (second-category character).
[0118] When the upper four bits of the read data belong to the set four-bit binary data, the lower four bits of the read data are mapped to digital characters or first-category characters according to the set mapping relationship.
[0119] For example, when the upper four bits of the read data are 0xF, it indicates that a byte stores a numeric character or a first-class character, and the upper four bits of the byte are used for identification data. It is only necessary to restore the lower four bits of the read byte to obtain a data (numeric character or first-class character).
[0120] The compression method of the present application is illustrated below with a specific demonstration example.
[0121] Demonstration Example 1:
[0122] like Figure 3 As shown, Figure 3 This is a screenshot of an MFC test program sample written in VS2015 C++. Figure 3 It can be seen that the data string of 61 characters consisting of pure numbers, dates, amounts, ID numbers, etc., occupies 61 bytes of storage space in its original data, and 84 bytes after being directly converted to Base64 encoding without compression. After compression by the method described in this application, it occupies 32 bytes, and after compression and conversion to Base64 encoding, it is 44 bytes. It can be restored losslessly after decompression.
[0123] Demonstration Example 2:
[0124] like Figure 4 As shown, Figure 4 In, label 1 is the original string data, including pure digital type, date type, amount type, ID number, a total of 124 characters, occupying 124 bytes. Label 2 is the binary data directly compressed by the method described in this application, and the occupied space becomes 66 bytes, which almost reduces the storage space by half. It can be seen from the hexadecimal data of label 2 that while reducing the storage space, it also plays a certain desensitization effect, but some information of the original data can still be seen. In order to increase the security of sensitive data, label 3 is after compression, after a simple encryption operation, so that it is completely impossible to see what the original data is. The space occupied after encryption is also 66 bytes, and the space occupied is only about half of the original data.
[0125] If you do not want to store it in binary format and want to store it as a visible string type, you can perform Base64 encoding. The encoded length becomes 88 bytes, which is much smaller than the 124 bytes of the original data and is only 2 / 3 of the original data. For example, the user's mobile phone number, an 11-digit number, is stored in the database data table as a string type, occupying 11 bytes. If it is compressed and then converted to Base64 encoding, it occupies 8 bytes, which reduces the space occupied, improves the desensitization of the data, and makes it impossible to intuitively see the mobile phone number.
[0126] Demonstration Example 3:
[0127] The original data is the string "25", which is represented by x32 0x35 in hexadecimal and 0011 00100011 0101 in binary. It occupies 2 bytes.
[0128] The method of the present application is used to map the digital character 2 to 0x2 and the digital character 5 to 0x5, where 0x2 occupies the upper 4 bits of a byte and 0x5 occupies the lower 4 bits of a byte. The compressed data is represented as 0x25 in hexadecimal and 0010 0101 in binary, occupying 1 byte.
[0129] Demonstration Example 4:
[0130] The original data is a string of "1.23". The decimal point "." is represented by 0x0C, and the numeric characters 123 are 0x31, 0x32, and 0x33 respectively. The binary representation of the original data is 0011 0001 0000 11000011 0010 0011 0011. It occupies 4 bytes.
[0131] When compressed, the decimal point "." is mapped to 0xC. The compressed data is represented as 0x1C0x23 in hexadecimal and 0001 1100 0010 0011 in binary. It occupies 2 bytes.
[0132] The present application also provides a numerical data compression device. Figure 5 As shown, the device comprises:
[0133] The data acquisition module 100 is used to acquire string data of a numerical type.
[0134] The first mapping module 200 is used to map the numeric characters in the string data into first-type four-bit binary data according to a set mapping relationship.
[0135] The first four-bit binary data of each digital character mapping is different.
[0136] The first storage module 300 is used to store first-category four-bit binary data. During the storage process, the upper four bits and the lower four bits of each byte respectively store one first-category four-bit binary data.
[0137] This application maps the digital characters in the string data of the numerical type to the first-class four-bit binary data according to the set mapping relationship, and compresses the digital characters. When storing, the upper four bits and the lower four bits of each byte each store a first-class four-bit binary data. This application reduces the storage space of the digital characters by half, greatly reducing the storage space of the string data of the numerical type. The compressed size is only about half of the original data, almost reducing the storage space of the string data of the numerical type by half. While compressing the data, it also plays a certain role in encryption and desensitization, greatly protecting the user's sensitive data, and greatly improving security.
[0138] As an improvement of the embodiment of the present application, the aforementioned device may further include:
[0139] The second mapping module is used to map the first type of characters in the string data into the second type of four-bit binary data according to a set mapping relationship.
[0140] The second-category four-bit binary data of each first-category character mapping is different, and each second-category four-bit binary data is different from each first-category four-bit binary data.
[0141] The second storage module is used to store the second type of four-bit binary data. During the storage process, the upper four bits and the lower four bits of each byte respectively store one second type of four-bit binary data.
[0142] As another improvement of the embodiment of the present application, the aforementioned device may further include:
[0143] The third mapping module is used to map the second type of characters in the character string data into eight-bit binary data according to a set mapping relationship.
[0144] The eight-bit binary data of each second-category character mapping is different, and the upper four bits of each eight-bit binary data are different from each first-category four-bit binary data and each second-category four-bit binary data.
[0145] The third storage module is used to store eight-bit binary data. During the storage process, each byte stores an eight-bit binary data.
[0146] As another improvement of the embodiment of the present application, the aforementioned device may further include:
[0147] The fourth storage module is used to store the first-category four-bit binary data or the second-category four-bit binary data mapped from the numeric character or the first-category character into the lower four bits of the selected byte, and store the set four-bit binary data in the upper four bits of the selected byte, when the first bit of the two adjacent bits in the string data is a numeric character or a first-category character and the second bit is a second-category character.
[0148] The set four-bit binary data is different from the upper four bits of each first-type four-bit binary data, each second-type four-bit binary data, and each eight-bit binary data.
[0149] As a specific example, the digital characters 0 to 9 are mapped to 0x1 to 0x9 respectively, the number of first-category characters is 4, the 4 first-category characters are mapped to 0xA to 0xD respectively, the number of second-category characters is 16, the 16 second-category characters are mapped to 0xE0 to 0xEF respectively, and the set four-bit binary data is 0xF.
[0150] To further improve data security, the device of the present application may also include:
[0151] The encryption module is used for encrypting and storing the mapped first-category four-bit binary data, second-category four-bit binary data and eight-bit binary data.
[0152] When data restoration is required, the device of the present application may further include:
[0153] The judgment module is used to read the stored data in bytes in sequence and judge the upper four bits of the read data.
[0154] The first restoration module is used to map the upper four bits and lower four bits of the read data into digital characters or first-category characters according to a set mapping relationship when the upper four bits of the read data belong to the first-category four-bit binary data or the second-category four-bit binary data.
[0155] The second restoration module is used for mapping the read data into the second type of characters according to a set mapping relationship when the upper four bits of the read data belong to the upper four bits of eight-bit binary data.
[0156] The third restoration module is used to map the lower four bits of the read data into digital characters or first-category characters according to a set mapping relationship when the upper four bits of the read data belong to the set four-bit binary data.
[0157] The device provided in the above embodiment, its implementation principle and the technical effect produced correspond to the embodiment of the above method one by one. For the sake of brief description, the part not mentioned in the embodiment of the device can refer to the corresponding content in the embodiment of the above method. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the modules and units described in the device can refer to the corresponding process in the embodiment of the above method, and will not be repeated here.
[0158] The numerical type data compression method described in the above embodiments provided by the present application can implement the business logic through a computer program and record it on a storage medium, and the storage medium can be read and executed by a computer to achieve the effect of the scheme described in the method embodiment of this specification. Therefore, the embodiment of the present application also provides a computer-readable storage medium for numerical type data compression, including a memory for storing processor-executable instructions, and when the instructions are executed by the processor, the steps of the numerical type data compression method including the above embodiments are implemented.
[0159] The storage medium may include a physical device for storing information, which is usually a medium that digitizes the information and then stores it in an electrical, magnetic or optical manner. The storage medium may include: a device that stores information in an electrical energy manner, such as various memories, such as RAM, ROM, etc.; a device that stores information in a magnetic energy manner, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a magnetic bubble memory, a USB flash drive; a device that stores information in an optical manner, such as a CD or a DVD. Of course, there are other readable storage media, such as quantum memory, graphene memory, etc.
[0160] The storage medium described above may also include other implementation methods according to the description of the method embodiment. The implementation principle and technical effects produced by this embodiment are the same as those of the aforementioned method embodiment. For details, please refer to the description of the relevant method embodiment, and no further description will be given here.
[0161] The embodiment of the present application also provides a device for compressing numerical data, which may be a separate computer, or may include an actual operating device using one or more of the methods or one or more of the embodiments of the present specification. The device for compressing numerical data may include at least one processor and a memory storing computer executable instructions, and when the processor executes the instructions, any one or more steps of the numerical data compression method are implemented.
[0162] The device described above may also include other implementation methods according to the description of the method embodiment. The implementation principle and technical effects produced by this embodiment are the same as those of the aforementioned method embodiment. For details, please refer to the description of the relevant method embodiment, and no further description will be given here.
[0163] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for compressing numerical data, characterized in that: The method comprises: Get string data of numeric type; Mapping the numeric characters in the string data into first-type four-bit binary data according to a set mapping relationship; Wherein, the first four-bit binary data of each digital character mapping is different; The first-type four-bit binary data is stored. During the storage process, the upper four bits and the lower four bits of each byte respectively store one of the first-type four-bit binary data.
2. The method for compressing numerical data according to claim 1, characterized in that: The method further comprises: Mapping the first type of characters in the string data into the second type of four-bit binary data according to a set mapping relationship; The second-category four-bit binary data of each of the first-category character mappings are all different, and each of the second-category four-bit binary data is different from each of the first-category four-bit binary data; The second type of four-bit binary data is stored. During the storage process, the upper four bits and the lower four bits of each byte respectively store one of the second type of four-bit binary data.
3. The method for compressing numerical data according to claim 2, characterized in that: The method further comprises: Mapping the second type of characters in the string data into eight-bit binary data according to a set mapping relationship; The eight-bit binary data of each of the second-category character mappings are all different, and the upper four bits of each of the eight-bit binary data are all different from each of the first-category four-bit binary data and each of the second-category four-bit binary data; The eight-bit binary data is stored, and during the storage process, each byte stores one of the eight-bit binary data.
4. The method for compressing numerical data according to claim 3, characterized in that: The method further comprises: When the first bit of the two adjacent bits in the string data is the numeric character or the first type of character, and the second bit is the second type of character; the first type of four-bit binary data or the second type of four-bit binary data mapped from the numeric character or the first type of character is stored in the lower four bits of the selected byte, and the set four-bit binary data is stored in the upper four bits of the selected byte; The set four-bit binary data is different from the upper four bits of each of the first-type four-bit binary data, each of the second-type four-bit binary data, and each of the eight-bit binary data.
5. The method for compressing numerical data according to claim 4, characterized in that: The digital characters 0 to 9 are mapped to 0x1 to 0x9 respectively, the number of the first category characters is 4, the 4 first category characters are mapped to 0xA to 0xD respectively, the number of the second category characters is 16, the 16 second category characters are mapped to 0xE0 to 0xEF respectively, and the set four-bit binary data is 0xF.
6. The method for compressing numerical data according to claim 4, characterized in that: The method further comprises: The mapped first category four-bit binary data, the second category four-bit binary data and the eight-bit binary data are encrypted and stored.
7. The method for compressing numerical data according to claim 4, characterized in that: The method further comprises: Read the stored data in bytes in sequence and judge the upper four bits of the read data; When the upper four bits of the read data belong to the first category of four-bit binary data or the second category of four-bit binary data, the upper four bits and the lower four bits of the read data are mapped to the digital characters or the first category of characters according to the set mapping relationship; When the upper four bits of the read data belong to the upper four bits of the eight-bit binary data, mapping the read data into the second type of characters according to the set mapping relationship; When the upper four bits of the read data belong to the set four-bit binary data, the lower four bits of the read data are mapped to the digital characters or the first type of characters according to the set mapping relationship.
8. A numerical data compression device, characterized in that: The device comprises: The data acquisition module is used to obtain string data of numerical type; A first mapping module, used for mapping the numeric characters in the string data into first-type four-bit binary data according to a set mapping relationship; Wherein, the first four-bit binary data of each digital character mapping is different; The first storage module is used to store the first-category four-bit binary data. During the storage process, the upper four bits and the lower four bits of each byte respectively store one of the first-category four-bit binary data.
9. A computer-readable storage medium for compressing numerical data, characterized in that: It comprises a memory for storing processor executable instructions, and when the instructions are executed by the processor, the steps of the numerical type data compression method according to any one of claims 1 to 7 are implemented.
10. A device for compressing numerical data, characterized in that: The method comprises at least one processor and a memory storing computer executable instructions, wherein when the processor executes the instructions, the steps of the numerical data compression method according to any one of claims 1 to 7 are implemented.