A data processing method, system and related devices

Abstract

The application provides a data processing method, system and related equipment, which can be applied to the field of computer technology. The method comprises the following steps: acquiring a first group of data comprising first data and second data, and outputting a second group of data comprising third data and fourth data. The first data and the second data each occupy a first number of bit positions, the third data and the fourth data are data obtained by compressing the first data and the second data respectively, the third data occupies a second number of bit positions, the fourth data occupies a fourth number of bit positions, the sum of the second number and the third number is less than or equal to the first number, the sum of the fourth number and the third number is less than or equal to the first number, and the third number is the number of bit positions occupied by the sign bit and the index bit of the floating-point type data. Therefore, the method can make the number of bit positions occupied by the compressed data much smaller than the number of bit positions occupied by the data before compression, reduce the compression rate of the data, and thus reduce the data storage cost of the database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a data processing method, system and related equipment. BACKGROUND

[0002] A time series database is a database specially designed for time series data. The floating point type data is one of the commonly used data types in the time series database. Some time series databases even only support floating point type data import.

[0003] The data volume of the floating point type data in the time series database is usually large. In order to reduce the data storage cost, the time series database can compress a large amount of floating point type data and only store the compressed result. In this case, the data storage cost can be measured by the compression rate of the data. The smaller the compression rate is, the lower the data storage cost is. The larger the compression rate is, the higher the data storage cost is. However, the compression rate of the general compression algorithm is high. If it is applied to the time series database, it cannot meet the demand of reducing the data storage cost. SUMMARY

[0004] To solve the above technical problems, the present application provides a data processing method, system and related equipment, which can reduce the compression rate of the data and thus reduce the data storage cost of the database.

[0005] In a first aspect, a data processing method is provided. The method includes: obtaining a first group of data including first data and second data, and outputting a second group of data including third data and fourth data. The first data and the second data are used to represent floating point type data, and each of the first data and the second data occupies a first number of bits. The third data is data obtained by compressing the first data, and the third data occupies a second number of bits. The sum of the second number and a third number is less than or equal to the first number. Based on the third data and the floating point type data represented by a first prefix belonging to a first numerical interval, the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating point type data in the first numerical interval, and the first prefix includes the sign bit and the exponent bit of the floating point type data in the first numerical interval. The fourth data is data obtained by compressing the second data, and the fourth data occupies a fourth number of bits. The sum of the fourth number and the third number is less than or equal to the first number, and based on the fourth data and the floating point type data represented by the first prefix belonging to the first numerical interval.

[0006] Understandably, floating-point data can be categorized into single-precision and double-precision floating-point data based on data precision. Computers store floating-point data according to preset rules. Taking the Institute of Electrical and Electronics Engineers (IEEE) 754 standard (also known as ANSI / IEEE Std 754-1985) as an example, computers use 32 bits to store single-precision floating-point numbers. The first bit is the sign bit (0 for positive, 1 for negative), bits 2 to 9 are the exponent bits, and bits 10 to 32 are the mantissa bits (the leftmost bit is the first bit). Computers use 64 bits to store double-precision floating-point numbers, where the first bit is also the sign bit, bits 2 to 12 are the exponent bits, and bits 13 to 64 are the mantissa bits.

[0007] In the above scheme, the difference between the number of bits occupied by the first data (i.e., the first quantity) and the number of bits occupied by the data obtained by compressing the first data (i.e., the third data) (i.e., the second quantity) is greater than or equal to the fourth quantity. The fourth quantity is the sum of the number of bits occupied by the sign bit and exponent bit of the floating-point data storing the first numerical range. Therefore, the number of bits occupied by the third data is much smaller than the number of bits occupied by the first data. Similarly, the number of bits occupied by the fourth data obtained after compressing the second data is also much smaller than the number of bits occupied by the second data. It can be understood that the more data in the first group is compressed, the larger the proportion of compressed data contained in the second group. This ensures that the total number of bits occupied by the second group is much smaller than the total number of bits occupied by the first group, and the difference between the two increases with the amount of compressed data. Therefore, the above scheme can significantly reduce the data compression rate. Subsequently, storing only the compressed data (i.e., the second group of data) in the database can reduce the database's data storage cost.

[0008] Optionally, in this article, "data occupies XX number of bits" can be understood as the binary form of the data occupying XX number of bits. For example, "the third data occupies the second number of bits" can be understood as the binary form of the third data occupying the second number of bits. Similarly, "the floating-point data represented by XX belongs to the first numerical range" can be understood as the value of the floating-point data represented by XX belonging to the first numerical range. For example, "the floating-point data represented by the third data and the first prefix belong to the first numerical range" can be understood as the values ​​of the floating-point data represented by the third data and the first prefix belonging to the first numerical range.

[0009] Optionally, the data in the first set of data are either all single-precision floating-point numbers or all double-precision floating-point numbers.

[0010] Optionally, the first quantity is 32, or the first quantity is 64.

[0011] Optionally, if the first quantity is 32, the third quantity is 9. If the first quantity is 64, the third quantity is 12.

[0012] Optionally, the first numerical interval is [1,2), or the first numerical interval is [2,3).

[0013] In some possible implementations, the first quantity is A, the second quantity is B, and the third quantity is H. Before outputting the second set of data, the method further includes: normalizing the first data based on the first set of data to obtain the first target data; performing an XOR operation on the second target data and the third target data to obtain the first operation result, the first K bits of the first operation result being 0; setting bits K+2 to A in the second target data to 1 as the first setting result, or setting bits K+2 to A in the third target data to 0 as the first setting result; determining the number of bits H+S to (H+S)+B-1 in the first setting result as the third data. The second target data and the third target data both belong to the first numerical range. The second target data is the sum of the first target data and the first preset value. The third target data is the sum of the second target data and the second preset value. Both the first and second preset values ​​are positive numbers. S is a positive integer, K≥H, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A.

[0014] In the above scheme, the first target data is obtained by normalizing the first set of data. The sum of the first target data and the first preset value is the second target data, which belongs to the first numerical range. The second target data is XORed with the third target data, which also belongs to the first numerical range, to obtain the first operation result where the first K bits are 0. That is, the first K bits of the second target data and the third target data are the same. K≥H can be understood as the sign bit and exponent bit of the second target data and the third target data being completely identical. However, the second target data and the third target data are different at bit K+1. Since the second target data is less than the third target data (the second preset value is positive), the (K+1)th bit of the second target data is 0, and the (K+1)th bit of the third target data is 1. Setting bits K+2 to A in the second target data to 1 is used as the first setting result, or setting bits K+2 to A in the third target data to 0 is used as the first setting result. This ensures the first setting result is greater than or equal to the second target data and less than or equal to the third target data, meaning the value represented by the first setting result also belongs to the first value range. Furthermore, the first H bits of the first setting result are the same as the first H bits of any floating-point number in the first value range. The K+1 bit of the first setting result is 0, and the bits after the K+1 bit are 1. Therefore, determining the number of bits from H+S to (H+S)+B-1 in the first setting result as the third data (KH < B < AH) accurately represents the first data. On the other hand, determining the number of bits from H+S to (H+S)+B-1 in the first setting result as the third data results in the third data occupying significantly less storage space than the first data, greatly saving storage space occupied by the first data (i.e., having an extremely low compression rate), thereby significantly reducing the cost of storing the first set of data. Furthermore, the process of determining the third data only involves normalization and XOR operations, so the above implementation scheme also has high compression efficiency.

[0015] Optionally, the first preset value is a positive integer, for example, the first preset value is 1, 2, 3 or 4, etc.

[0016] In some possible implementations, S = 1.

[0017] The above scheme provides two specific implementations for determining the third data. The first implementation is based on (H+S)+B-1=A, where S=1 and B=HA, meaning the third data consists of all the mantissa bits in the first setting result. The second implementation is based on (H+S)+B-1<A, where S=1 and B<HA, meaning the third data consists of a portion of the mantissa bits in the first setting result. It can be seen that both implementations can make the number of bits occupied by the third data much smaller than the number of bits occupied by the first data, thus significantly reducing the data compression ratio. Furthermore, in both implementations, the first prefix consists of the sign bit and exponent bit of any floating-point number within the first numerical range.

[0018] Optionally, the second set of data includes the first prefix.

[0019] In some possible implementations, the first data is the data with the smallest absolute value in the first group of data, and the first S bits of the second target data are 1.

[0020] In the above scheme, the first target data is the normalized result of the first data. Since the first data is the data with the smallest absolute value in the first group of data, the first target data is also the smallest among the normalized results of all data in the first group. Based on this, the normalized results of all data are added to the first preset value. The sum of the first target data and the first preset value (i.e., the second target data) is the smallest among all the sums, meaning that all sums are greater than or equal to the second target data. When the first S bits of the last digit of the second target data are 1, the first S bits of the last digit of all sums are also 1. Therefore, the first S bits of the last digit of the second target data can be used as the common part of all data in the second group. This is equivalent to using the 1st to (H+S)-1st bits of the second target data as the first prefix, which can further reduce the total number of bits occupied by the second group of data and further reduce the data compression rate.

[0021] Optionally, the first preset value is a positive integer. For example, the first preset value is 1, 2, 3, or 4.

[0022] Optionally, any data in the first set of data can be normalized to obtain positive numbers. For example, any data in the first set of data can be normalized to obtain positive numbers in the range [0,1] or [1,2].

[0023] Optionally, S is a positive integer, for example, S is 1, 2, 3 or 4, etc.

[0024] Optionally, the data in the first set of data can be normalized in the following way: For example, take the absolute value of the data in the first set of data and divide it by M to obtain the normalized value. Here, M is greater than or equal to the largest absolute value of the first set of data.

[0025] Optionally, M equals max + L, where max is the largest absolute value of the first set of data, and L is a positive number.

[0026] Optionally, N is related to the precision of the first set of data, which is related to the maximum number of decimal places in the first set of data. For example, if the maximum number of decimal places in the first set of data is N, then L and N satisfy the following relationship: L = (1 + x) / pow(10, N), where x is a positive number, for example, x = 0.05, 0.1, or 0.2.

[0027] In some possible implementations, the product of the first data and the first precision value is equal to the rounded result of the product of the third preset value and the first verification data. The first precision value is determined based on the data with the most decimal places in the first set of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the first verification data is the difference between the first setting result and the first preset value.

[0028] In the above scheme, the first verification data is the difference between the first setting result and the first preset value. The first setting result is determined based on the first data, and the first setting result is used to determine the third data (i.e., the data obtained by compressing the first data). Therefore, by making the product of the first data and the first precision value equal to the rounded result of the product of the third preset value and the first verification data, the third data can be restored and the first data can be obtained.

[0029] In some possible implementations, the first set of data includes the fifth data, which occupies a first number of bits. If the product of the fifth data and the first precision value is not equal to the floor of the product of the third preset value and the second check data, then the second set of data includes the fifth data, wherein the second check data is derived based on the fifth data. If the fifth data is 0, then the second set of data includes the sixth data, which occupies a fifth number of bits, less than the first number.

[0030] In the above scheme, the decision to compress the fifth data is made by determining whether the product of the fifth data and the first precision value equals the rounded result of the product of the third preset value and the second check data. This ensures the accuracy of decompression and avoids decompression errors. Furthermore, if the fifth data is 0, the second group of data includes a sixth data, which occupies the same number of bits as the fifth data, and this number is less than the number of bits in the first group. This also helps reduce the number of bits required to store '0'. Optionally, the sixth data can be a special combination of bits, for example, the sixth data could be "0000000000".

[0031] In some possible implementations, the second set of data includes a first symbol, which is used to indicate whether the first data is positive or negative. The first symbol and the third data are used together to decompress and obtain the first data.

[0032] In the above scheme, using a first symbol to indicate whether the first data is positive or negative provides more information for reconstructing the first data using the third data, ensuring the correctness of the reconstructed first data. Furthermore, storing the symbol information of the first data through the first symbol also saves storage space and reduces data storage costs.

[0033] Secondly, a data processing method is provided, comprising: acquiring a second set of data including third data and fourth data, and outputting a first set of data including first data and second data. The third data occupies a second number of bits, the sum of the second and third numbers is less than or equal to the first number, and the floating-point data represented by the third data and a first prefix belongs to a first numerical range. The third number is the sum of the number of bits occupied by the sign and exponent bits of the floating-point data storing the first numerical range, and the first prefix includes the sign and exponent bits of the floating-point data in the first numerical range. The fourth data occupies a fourth number of bits, the sum of the fourth and third numbers is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical range. The first data is the data obtained by decompressing the third data, and the second data is the data obtained by compressing the fourth data. Both the first and second data are used to represent floating-point data, and both the first and second data occupy a first number of bits.

[0034] Therefore, this technical solution can use data occupying fewer bits (i.e., the third and fourth data) to restore data occupying more bits (i.e., the first and second data). This ensures the correctness of decompression and saves the data storage cost of the database, because the database can store only the second set of data and not the first set of data, thus achieving the effect of storing the first set of data.

[0035] In some possible implementations, the above method also includes the following before outputting the first set of data:

[0036] Obtain the first verification data; divide the product of the third preset value and the first verification data by the first precision value to obtain the third calculation result; determine the first data based on the third calculation result. The first verification data is the difference between the first setting result and the first preset value. The first setting result includes the first prefix and the third data. The first preset value is a positive number. The first precision value is determined based on the data with the most decimal places in the first set of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The third preset value is a positive number.

[0037] In the above scheme, the third calculation result is obtained by dividing the product of the third preset value and the first verification data by the first precision value, and then the first data is obtained based on the third calculation result. The first preset value, the third preset value, and the first precision value can all be pre-stored in the computing device, so that the first preset value, the third preset value, and the first precision value can be quickly retrieved when decompressing the third data, thus accelerating the calculation speed of the third calculation result. Therefore, the above decompression process is simple and efficient, enabling the compressed data to be decompressed quickly, and the above scheme has high decompression performance.

[0038] In some possible implementations, the second set of data includes a first symbol. The determination of the first data based on the third operation result includes: when the first symbol indicates that the first data is positive, using the third operation result as the first data; when the first symbol indicates that the first data is negative, setting the first bit of the third operation result to 1 and using it as the first data.

[0039] In the above scheme, the first symbol can accurately decompress the third data into the first data, avoiding symbol errors in the decompressed data.

[0040] Optionally, the first symbol is also used to indicate that the fifth data is uncompressed data, and in some cases, the fifth data can be directly extracted as the decompressed data.

[0041] Optionally, the first symbol is also used to indicate that the sixth data is 0, so the decompression result of the sixth data can be set to 0.

[0042] Thirdly, a data processing system is provided, comprising a first acquisition unit and a first output unit. The first acquisition unit acquires a first set of data, including first data and second data. The first output unit outputs a second set of data, including third data and fourth data. Both the first and second data represent floating-point data. Both the first and second data occupy a first number of bits. The third data is data obtained by compressing the first data. The third data occupies a second number of bits. The sum of the second and third numbers is less than or equal to the first number. The floating-point data represented by the third data and a first prefix belongs to a first numerical range. The third number is the sum of the number of bits occupied by the sign bit and exponent bit of the floating-point data storing the first numerical range. The first prefix includes the sign bit and exponent bit of the floating-point data in the first numerical range. The fourth data is data obtained by compressing the second data. The fourth data occupies a fourth number of bits. The sum of the fourth and third numbers is less than or equal to the first number. The floating-point data represented by the fourth data and the first prefix belongs to the first numerical range.

[0043] In some possible implementations, the first quantity is A, the second quantity is B, and the third quantity is H. The data processing system also includes a compression unit. The compression unit is used to: normalize the first data based on the first set of data to obtain the first target data before the first output unit outputs the second set of data; perform an XOR operation on the second target data and the third target data to obtain a first operation result where the first K bits are 0; set bits K+2 to A of the second target data to 1 as the first setting result, or set bits K+2 to A of the third target data to 0 as the first setting result; and determine the number of bits H+S to (H+S)+B-1 of the first setting result as the third data. Both the second and third target data belong to a first numerical range. The second target data is the sum of the first target data and a first preset value. The third target data is the sum of the second target data and a second preset value. Both the first and second preset values ​​are positive numbers. S is a positive integer, K≥H, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A.

[0044] In some possible implementations, S = 1.

[0045] In some possible implementations, the first data mentioned above is the data with the smallest absolute value in the first group of data. The first S bits of the last S bits of the second target data mentioned above are 1s.

[0046] In some possible implementations, the product of the first data and the first precision value is equal to the rounded result of the product of the third preset value and the first verification data. The first precision value is determined based on the data with the most decimal places in the first set of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The first verification data is the difference between the first setting result and the first preset value.

[0047] In some possible implementations, the first set of data mentioned above includes the fifth data. The fifth data occupies a first number of bits. If the product of the fifth data and the first precision value is not equal to the rounded result of the product of the third preset value and the second check data, then the second set of data includes the fifth data. The second check data is derived based on the fifth data. If the fifth data is 0, then the second set of data includes the sixth data. The sixth data occupies a fifth number of bits. The fifth number is less than the first number.

[0048] In some possible implementations, the second set of data includes a first symbol. The first symbol indicates whether the first data is positive or negative. The first symbol and the third data are used together to decompress and obtain the first data.

[0049] Fourthly, a data processing system is provided, comprising a second acquisition unit and a second output unit. The second acquisition unit acquires a second set of data, including third and fourth data. The second output unit outputs a first set of data, including first and second data. The third data occupies a second number of bits. The sum of the second and third numbers is less than or equal to the first number. Floating-point data represented by the third data and a first prefix belongs to a first numerical range. The third number is the sum of the number of bits occupied by the sign and exponent bits of the floating-point data storing the first numerical range. The first prefix includes the sign and exponent bits of the floating-point data in the first numerical range. The fourth data occupies a fourth number of bits. The sum of the fourth and third numbers is less than or equal to the first number. Floating-point data represented by the fourth data and the first prefix belongs to the first numerical range. The first data is data obtained by decompressing the third data. The second data is data obtained by compressing the fourth data. Both the first and second data are used to represent floating-point data. Both the first and second data occupy a first number of bits.

[0050] In some possible implementations, the aforementioned data processing system further includes a decompression unit. This decompression unit is used to: acquire first verification data before the second output unit outputs the first set of data; divide the product of a third preset value and the first verification data by a first precision value to obtain a third calculation result; and determine the first data based on the third calculation result. The first verification data is the difference between the first setting result and the first preset value. The first setting result includes a first prefix and the third data. The first preset value is a positive number. The first precision value is determined based on the data with the most decimal places in the first set of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The third preset value is a positive number.

[0051] In some possible implementations, the second set of data mentioned above includes a first symbol. Specifically, the decompression unit is used to take the third operation result as the first data when the first symbol indicates that the first data is positive; and to take the first bit of the third operation result as the first data when the first symbol indicates that the first data is negative.

[0052] Fifthly, a chip system is provided, including a processor and a power supply circuit for supplying power to the processor, which is used to perform the method of any of the first aspects or the method of any of the second aspects.

[0053] In a sixth aspect, a computing device cluster is provided, including at least one computing device, each computing device including a processor and memory.

[0054] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the cluster of computing devices performs the method of any one of the first aspects, or performs the method of any one of the second aspects.

[0055] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed by a computing device, cause the computing device to perform the method of any of the first aspects or the method of any of the second aspects.

[0056] Eighthly, a computer-readable storage medium is provided, characterized in that it includes computer program instructions, which, when executed by a computing device, perform a method as described in any of the first aspects, or a method as described in any of the second aspects. Attached Figure Description

[0057] Figure 1 This is an architecture diagram of a data processing system provided in an embodiment of this application;

[0058] Figure 2This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0059] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0060] Figure 4 This is a schematic diagram of the data composition of a first setting result provided in an embodiment of this application;

[0061] Figure 5 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0062] Figure 6 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0063] Figure 7 This is a schematic diagram of the structure of a computing device provided in this application;

[0064] Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in this application;

[0065] Figure 9 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0067] To reduce the data storage cost of a database, this application provides a data processing system that can compress data such that the difference in the number of bits occupied by the data before and after compression is greater than or equal to the number of bits occupied by the sign bit and exponent bit of floating-point data. In other words, the system can make the difference in the number of bits occupied by the data before and after compression very large. Therefore, the system can significantly reduce the data compression rate. Subsequently, only the compressed data is stored in the database, which can reduce the data storage cost of the database.

[0068] See Figure 1 , Figure 1 This is an architecture diagram of a data processing system provided in an embodiment of this application. Figure 1As shown, the architecture includes a client 100, a data processing system 200, and a database 300. Communication between the client 100 and the data processing system 200, and between the data processing system 200 and the database 300, can be achieved via wired or wireless means.

[0069] Client 100 generates a first set of data including first data and second data, and sends the first set of data to data processing system 200. The data in the first set represents floating-point data, and each data point occupies a first number of bits. The first set of data includes first data and second data.

[0070] The data processing system 200 compresses the data in the first set of data, uses the compressed data as the data in the second set of data, and sends the second set of data to the database 300. Specifically, the data processing system 200 compresses the first data into the third data in the second set of data, and compresses the second data into the fourth data in the second set of data. The third data occupies a second number of bits, the fourth data occupies a fourth number of bits, the sum of the second and third numbers is less than or equal to the first number, and the sum of the fourth and third numbers is less than or equal to the first number. The third number is the sum of the number of bits occupied by the sign bit and exponent bit for storing floating-point data.

[0071] Database 300 is used to store the second set of data. Database 300 can be a time-series database, relational database, graph database, or any other database that supports storing floating-point data.

[0072] The data processing system 200 is also used to retrieve a second set of data from the database 300, decompress the data in the second set of data to obtain the data in the first set of data, and send the first set of data to the client 100. Specifically, the data processing system 200 is used to decompress the third set of data into the first set of data and decompress the fourth set of data into the second set of data.

[0073] In some possible implementations, the client 100, data processing system 200, and database 300 can all be deployed on computing devices. Computing devices are electronic devices used for computing, processing, and storing data, including servers, supercomputers, personal computers, workstations, mobile devices, and so on.

[0074] The client 100, data processing system 200, and database 300 can be deployed on different computing devices, or on the same computing device. When the client 100, data processing system 200, and database 300 are deployed on different computing devices, at least two of them can be deployed on different computing devices within the same computing device cluster, or on different computing devices within different computing clusters. The computing device cluster can include multiple of the aforementioned computing devices. The specific deployment can be determined based on the actual application scenario, and this application does not impose specific limitations.

[0075] In one specific implementation, the client 100 can also be deployed on a terminal device. The terminal device is an electronic device used to access computing devices, including personal computers, smartphones, handheld devices, tablets, mobile laptops, all-in-one handheld consoles, smart conferencing devices, smart learning machines, and so on.

[0076] Alternatively, the data processing system 200 may be software, a plugin, or a cloud platform console.

[0077] When the data processing system 200 is software, it can be a standalone software specifically designed for data compression. Users can use it by downloading and installing data compression software. Figure 1 Data processing system 200.

[0078] When the data processing system 200 is a plug-in, it can be built into the application tool. For example, it can function as a data compression plug-in for database 300, compressing the data in database 300. Users can use it by upgrading the application tool. Figure 1 The data processing system 200 in China.

[0079] When the data processing system 200 is a console, it can access cloud services that provide data compression / decompression capabilities, including World Wide Web (WWW) clients, application clients, application programming interfaces (APIs), and so on. Users can access these cloud services by purchasing them. Figure 1 Data processing system 200.

[0080] In some possible implementations, the data processing system 200 includes multiple units. For example, in... Figure 1In the data processing system 200, there are a first acquisition unit 210, a compression unit 220, a first output unit 230, a second acquisition unit 240, a decompression unit 250, and a second output unit 260.

[0081] The first acquisition unit 210, compression unit 220, first output unit 230, second acquisition unit 240, decompression unit 250, and second output unit 260 can be implemented in software or hardware. For example, the implementation of the first acquisition unit 210 will be described below. Similarly, the implementation of the compression unit 220, first output unit 230, second acquisition unit 240, decompression unit 250, and second output unit 260 can refer to the implementation of the first acquisition unit 210.

[0082] The first acquisition unit 210, as an example of a software functional unit, may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first acquisition unit 210 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0083] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0084] The first acquisition unit 210, as an example of a hardware functional unit, may include at least one computing device, such as a server. Alternatively, the first acquisition unit 210 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0085] The multiple computing devices included in the first acquisition unit 210 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first acquisition unit 210 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first acquisition unit 210 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0086] The functions of each unit in the data processing system 200 are described below.

[0087] The first acquisition unit 210 receives a first set of data input by the user through the client 100 and inputs the first set of data into the compression unit 220. The data in the first set represents floating-point data, and each data point occupies a first number of bits. The first set of data includes first data and second data. This process can be seen in the following... Figure 2 The execution process of step S101 in the data processing method.

[0088] Compression unit 220 is used to compress the first set of data to obtain the second set of data, and then sends the second set of data to the first output unit 230. Specifically, it compresses the data in the first set of data and uses the compressed data as the data in the second set of data, thereby obtaining the second set of data.

[0089] Specifically, in order to compress the first data in the first group of data into the third data in the first group of data, the compression unit 220 is specifically used to: normalize the first data based on the first group of data to obtain the first target data; take the sum of the first target data and the first preset value as the second target data, and take the sum of the second target data and the second preset value as the third target data; perform an XOR operation on the second target data and the third target data to obtain the first operation result with the first K bits being 0, set the K+2 to A bits in the second target data to 1 as the first setting result, or set the K+2 to A bits in the third target data to 0 as the first setting result; determine the number of bits from the H+S to (H+S)+B-1 bits in the first setting result as the third data. Wherein, the first preset value and the second preset value are both positive numbers, A represents the number of bits occupied by the first data, B represents the number of bits occupied by the third data, H represents the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical range, S is a positive integer, K≥H, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A.

[0090] The process of compressing the first set of data to obtain the second set of data, and the process of compressing the first set of data to obtain the third set of data, can be found in the following: Figure 3 The execution process of steps S201-S205 in the data processing method, or see the following Figure 5 The execution process of steps S301-S305 in the data processing method.

[0091] The first output unit 230 is used to send the second set of data to the database 300. This process can be seen below. Figure 2 The execution process of step S103 in the data processing method.

[0092] The second acquisition unit 240 is used to acquire the second set of data from the database 300 and input the second set of data into the decompression unit 250.

[0093] The decompression unit 250 decompresses the second set of data to obtain the first set of data, and sends the first set of data to the second output unit 260. Specifically, it decompresses the data in the second set of data, uses the decompression result as the data in the first set of data, and thus obtains the first set of data.

[0094] Specifically, in order to decompress the third data in the second set of data into the first data in the first set of data, the decompression unit 250 is specifically used to: obtain the first verification data; divide the integer result of the product of the third preset value and the first verification data by the first precision value to obtain the third calculation result; and determine the first data based on the third calculation result. Wherein, the first verification data is the difference between the first setting result and the first preset value; the first precision value is determined based on the data with the most decimal places in the first set of data; the third preset value is determined based on the first precision value; the third preset value is different from the first precision value; and the third preset value is a positive number.

[0095] The process of decompressing the second set of data to obtain the first set of data, and the process of decompressing the third set of data to obtain the first set of data, can be found in the following: Figure 6 The execution process of step S402 in the data processing method.

[0096] The second output unit 260 is used to send the first set of data to the client 100.

[0097] It should be understood that the units in the aforementioned data processing system 200 are based on... Figure 1 The first acquisition unit 210 to the second output unit 260 of the data processing system 200 are used as examples for illustration. In practical applications, the data processing system 200 may include more or fewer units, and this application does not make specific limitations.

[0098] It should be understood that the architecture of the aforementioned data processing system is based on... Figure 1 The client 100 and database 300 communicating with the data processing system 200 are described using only one example. In practical applications, the number of client 100 and database 300 can be one or more, and this application does not impose a specific limitation. When there are multiple clients 100, one client 100 can generate a set of data to be compressed and input that set of data into the data processing system 200, while another client 100 generates another set of data to be compressed and inputs that set of data into the data processing system 200. When there are multiple databases 300, the data processing system 200 can store one set of compressed data in one database 300 and another set of compressed data in another database 300.

[0099] In some possible application scenarios, the data processing system 200 can also be used to compress a set of data to be compressed that has been stored in the database 300, to obtain a set of compressed data, and then use the compressed data to replace the data to be compressed in the database 300, thereby reducing the data storage cost of the database 300.

[0100] In summary, the data processing system 200 provided in this application ensures that the difference between the number of bits occupied by the data before compression and the number of bits occupied by the data after compression is greater than or equal to the sum of the number of bits occupied by the sign bit and exponent bit of floating-point data. This means that the number of bits occupied by the compressed data is significantly less than the number of bits occupied by the data before compression. It can be understood that the more data in the first group is compressed, the larger the proportion of compressed data in the second group. This ensures that the total number of bits occupied by the second group is significantly less than the total number of bits occupied by the first group, and the difference increases with the amount of compressed data. Therefore, the data processing system 200 can significantly reduce the data compression rate, and by storing only the compressed data (i.e., the second group of data) in the database 300, the data storage cost of the database 300 can be reduced.

[0101] See Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 As shown in the embodiments of this application, the data processing method includes:

[0102] S101: The client sends the first set of data to the data processing system.

[0103] Accordingly, the data processing system receives the first set of data from the client.

[0104] In one specific implementation, the first set of data is received by the first acquisition unit in the data processing system. The data processing system can be as described above. Figure 1 The data processing system 200 in the middle, the first acquisition unit can be the aforementioned Figure 1 The first acquisition unit 210 in the data processing system 200.

[0105] In practical applications, the first acquisition unit can also acquire the first set of data by reading the storage medium or by acquiring the first set of data from the database. This application does not specifically limit the acquisition method of the first set of data.

[0106] The first set of data includes multiple floating-point data. For example, the first set of data includes first data and second data, which are floating-point data. Specifically, the first data is 3.1, the second data is 3.2, etc.

[0107] The number of data points in the first set of data is not limited in this application. For example, the first set of data may contain 10, 20, or 30 data points.

[0108] Optionally, the number of data points in the first set of data can be preset.

[0109] Optionally, the first set of data is a set of data obtained by truncating the data stream, and the data in the first set of data is arranged in the same order as the data in the data stream before truncating.

[0110] Based on data precision, floating-point data is divided into single-precision floating-point data and double-precision floating-point data. The first acquisition unit stores floating-point data according to preset rules. Taking the IEEE 754 standard as an example, the first acquisition unit uses 32 bits to store single-precision floating-point numbers. The first bit is the sign bit, where 0 represents a positive number and 1 represents a negative number. Bits 2 to 9 are the exponent bits, and bits 10 to 32 are the mantissa bits (the leftmost bit is the first bit). The first acquisition unit uses 64 bits to store double-precision floating-point numbers. The first bit is also the sign bit, bits 2 to 12 are the exponent bits, and bits 13 to 64 are the mantissa bits. For ease of description, the leftmost bit of the data will be considered the first bit, i.e., the sign bit is the first bit of the data.

[0111] In the first set of data, each data point occupies a first number of bits. For example, both the first and second data points occupy a first number of bits. Specifically, if the first data point is stored as a single-precision floating-point number, the first number is 32; if the first data point is stored as a double-precision floating-point number, the first number is 64.

[0112] For ease of description, unless otherwise specified below, the data in the first set of data and the second set of data that will appear below can be understood as double-precision floating-point data.

[0113] S102: The compression unit in the data processing system obtains the second set of data based on the first set of data.

[0114] The compression unit can be one of the aforementioned Figure 1 The compression unit 220 in the data processing system 200.

[0115] The second set of data is obtained by compressing the first set of data by the compression unit. The second set of data includes multiple compressed data items, such as the third and fourth sets of data. Specifically, the third set of data is obtained by compressing the first set of data by the compression unit, and the fourth set of data is obtained by compressing the second set of data by the compression unit.

[0116] In this data structure, the third data occupies the second number of bits, and the sum of the second and third numbers is less than or equal to the first number. The third number stores the sum of the bits occupied by the sign and exponent bits of the first data. For example, if the first data is single-precision floating-point data, then the first number is 32, the third number is 9, and the second number can be calculated to be less than or equal to 23. The third data saves at least 9 bits of storage space compared to the first data. If the first data is double-precision floating-point data, then the first number is 64, the third number is 12, and the second number can be calculated to be less than or equal to 52. The third data saves at least 9 bits of storage space compared to the first data.

[0117] The fourth data occupies a fourth number of bits. The sum of the fourth and third numbers is less than or equal to the first number. The third number stores the sum of the bits occupied by the sign and exponent bits of the second data. For example, if the second data is single-precision floating-point data, then the first number is 32, the third number is 9, and the fourth number can be calculated to be less than or equal to 23. The fourth data saves at least 9 bits of storage space compared to the second data. If the second data is double-precision floating-point data, then the first number is 64, the third number is 12, and the fourth number can be calculated to be less than or equal to 52. The fourth data saves at least 9 bits of storage space compared to the second data.

[0118] Therefore, the compression scheme provided in this application has good compression performance.

[0119] The compression unit determines whether the floating-point data it represents belongs to the first numerical range based on the third data and the first prefix, or based on the fourth data and the first prefix. The third quantity is the sum of the number of bits occupied by the sign bit and the exponent bit for storing the floating-point data in the first numerical range. For details on how the compression unit determines the floating-point data it represents based on the third data and the first prefix, or based on the fourth data and the first prefix, please refer to the following text. Figure 3 The details of the relevant content will not be elaborated here. It should be noted that the number of bits occupied by floating-point data represented by the third data and the first prefix, or by floating-point data represented by the fourth data and the first prefix, is the same as the number of bits occupied by the data in the first group. Therefore, the sum of the number of bits occupied by the sign bit and exponent bit of the floating-point data in the first numerical range is the same as the sum of the number of bits occupied by the sign bit and exponent bit of the first data.

[0120] Optionally, the first numerical interval is [1,2), or the first numerical interval is [2,3).

[0121] S103: The data processing system sends the second set of data to the database.

[0122] The data processing system sends the second set of data to the database so that the database can store the second set of data. This allows the database to store only the second set of data without storing the first set of data, thus achieving the same effect as storing the first set of data and saving the database's data storage costs.

[0123] The following is a detailed introduction to the above. Figure 2 The data processing method step S102 is the process of the compression unit compressing the first set of data to obtain the second set of data.

[0124] See Figure 3 , Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. It should be noted that... Figure 3 This can be used as a standalone embodiment, or as described above. Figure 2 The data processing method shown is a supplement to the one described, but this application does not limit it. For example... Figure 3 As shown, the data processing method provided in this application includes:

[0125] S201: The compression unit normalizes the first data in the first group of data to obtain the first target data.

[0126] Normalization is a data preprocessing technique used to transform the feature values ​​of data to a specific range, typically [0,1] or [-1,1]. Its purpose is to eliminate differences between different features caused by factors such as units and orders of magnitude, making the data comparable and enabling analysis and processing at the same scale. The normalization method provided in this application specifically includes the following steps:

[0127] Step 1: The compression unit takes the absolute value of the first set of data and determines the maximum absolute value of the first set of data.

[0128] For example, the first set of data includes: 3.12 -3.18 3.89 -3.91 3.27 -3.55 3.66 -3.30, then the maximum absolute value of the first set of data is 3.91.

[0129] For example, if the first set of data includes: 6.24, 7.11, -8.98, 9.22, then the maximum absolute value of the first set of data is 9.22.

[0130] For ease of description, Max can be used to represent the maximum absolute value of the first set of data, and similarly, Min can be used to represent the minimum absolute value of a set of data.

[0131] Step 2: The compression unit determines the dividend M in the normalization process.

[0132] Where M is a value greater than Max, for example, M = Max + L, and L is a positive number. For example, L is 0.05, 0.1, or 0.2, etc.

[0133] In one possible implementation, L depends on the first precision value of the first set of data, which is determined based on the data with the most decimal places in the first set of data.

[0134] Taking the first set of data, including 3.12 -3.18, 3.89 -3.911, 3.27 -3.55, 3.66 -3.30, as an example, the data with the most decimal places in the first set of data is -3.911, which has 3 decimal places. It should be noted that in this application, the number of decimal places refers to the number of significant decimal places. For example, -3.910 has 2 significant decimal places, and 3.201 has 3 significant decimal places. For ease of description, the symbol N will be used below to represent the maximum number of significant decimal places in the first set of data.

[0135] The first precision value P is positively correlated with the maximum number of significant decimal places N in the first set of data. For example, P = pow(10,N), if N = 3, then P = 1000; another example is P = 100 * N; yet another example is P = pow(N,5). For ease of description, the following text will use P = pow(10,N) as an example for related introduction.

[0136] L is negatively correlated with P. For example, L = (1 + x) / P, where x is a positive number, such as 0.1, 0.2, or 0.3. 2 For example, x can be 0.1, 0.2, or 0.3. For ease of description, the following text will use L = (1 + 0.1) / P as an example for further explanation.

[0137] M = Max + L, where L and Max are both positive numbers, so M is a positive number greater than Max.

[0138] This application also provides a specific implementation method for determining the maximum number N of significant decimal places in the first group of data using the compression unit. Taking the first data v in the first group of data as an example, and setting the initial m to 0, the specific calculation process is as follows:

[0139] 1. Calculate v = v * pow(10, n)

[0140] 2. Convert vv to IEEE 754 binary representation and store it as uint64 (an unsigned 64-bit integer data type), denoted by u. For example, u = float64bits(vv).

[0141] 3. Calculate the exponent e of u, for example, e = u >> 52 – 1023.

[0142] 4. Remove the integer part, for example, u = u & (1 << (52-e) - 1).

[0143] 5. Determine if u is 0. If u is 0, then N = m, and the calculation ends. If u is not equal to 0, then increment the value of m by 1, and repeat steps 1-4 above. Optionally, if the output of u < 8 || u == (1 << (52-e) - 1) is "yes", then N = m can also be determined, and the calculation ends. This is because of floating-point precision issues; directly determining if u is 0 may introduce errors. For example, the compression unit might output the calculation result of 1.001 * 1000 as 1000.9999999999999; or, for another example, the compression unit might output the calculation result of 1.09 * 100 as 109.00000010000001.

[0144] Step 3: The compression unit normalizes the first data to obtain the first target data.

[0145] In one possible implementation, the compression unit divides the absolute value of each data point in the first set of data by M, and the resulting value is the normalized result of that data. Therefore, the compression unit divides the absolute value of the first data by M to obtain the normalized result of the first data, which is then used as the first target data.

[0146] For example, continuing with the first set of data including: 3.12 -3.18 3.89 -3.911 3.27 -3.55 3.66 -3.3, for ease of description, please refer to Table 1 for the normalization process.

[0147] Table 1

[0148]

[0149] Where M = Max + 1.1 / pow(10, N), Max = 3.911, N = 3, we get M = 3.9121.

[0150] The compression unit normalizes the first set of data based on the first set of data to obtain the first target data. Referring again to Table 1, if the first set of data is, for example, the original value in Table 1, and the first data is, for example, 3.12 in Table 1, then the first target data is 0.797526. Since M is a positive number greater than Max, all normalization results are values ​​between 0 and 1.

[0151] S202: The compression unit determines the second target data and the third target data based on the first target data.

[0152] The second target data is the sum of the first target data and the first preset value, and the third target data is the sum of the second target value and the second preset value. Both the first preset value and the second preset value are positive numbers. For example, the first preset value is 1 or 2, and the second preset value is 1 or 2.

[0153] In one possible implementation, the second preset value F, the first precision value P, and the dividend M have the following relationship: F ​​= 1 / (P*M).

[0154] For example, taking the first target data as 0.797526, the first preset value as 1, P = 1000 and M = 3.9121 as an example, the second target data is 0.797526 + 1 = 1.797526, and the third target data is 1.797526 + 1 / (1000 * 3.9121) = 1.797781617187699.

[0155] S203: The compression unit performs an XOR operation on the second target data and the third target data to obtain the first operation result.

[0156] Taking the second target data as 1.797526 and the third target data as 1.797781617187699 as an example, when the computing device stores the second target data and the third target data, the sign bit is 0 and the exponent bit is 1. The first few bits of the mantissa are the same. Therefore, when the computing device performs an XOR operation on the second target data and the third target data, the first K bits of the first operation result are all 0. The size of K depends on the difference between the second target data and the third target data. The smaller the difference, the larger the value of K, and the larger the difference, the smaller the value of K. For example, the mantissa of 1.797526 is “11001100001010…”, and the mantissa of 1.797781617187699 is “11001100001110…”. It can be seen that when storing 1.797526 and 1.797781617187699 in binary, the first 11 bits of the mantissa are the same. Adding the 1 sign bit and the 11 exponent bits, the first 23 bits of the binary form of 1.797526 and 1.797781617187699 are the same. Therefore, K equals 23.

[0157] S204: The compression unit determines the first setting result based on the first calculation result.

[0158] The first setting result is obtained by setting bits K+2 to A in the binary form of the second target data to 1, or by setting bits K+2 to A in the binary form of the third target data to 0.

[0159] Taking the binary representation of the second target data as "01111111111111001100001010..." as an example, setting bits K+2 to A of the second target data to 1 results in "0111111111111100110000101111111...". Similarly, taking the binary representation of the third target data as "01111111111111001100001110..." as an example, setting bits K+2 to A of the third target data to 0 results in "0111111111111100110000110000000...". Based on the above description, it can be seen that the value range of the first setting result is from the second target data to the third target data; that is, the second target data is less than or equal to the first setting result, and the first setting result is less than or equal to the third target data.

[0160] Taking the first numerical interval as [1,2) as an example, the second target data, the third target data, and the first setting result all belong to the first numerical interval.

[0161] K≥H, where H is the third quantity. The third quantity is the number of bits occupied by the sign bit and exponent bit in the binary form of the floating-point data in the first numerical range. For example, taking a double-precision floating-point number as an example, H is 12. Combining the above description, K = 23. Therefore, K≥H.

[0162] S205: The compression unit determines the third data based on the first setting result.

[0163] In one possible implementation, the compression unit determines the number of bits from the H+Sth bit to the (H+S)+B-1th bit in the first setting result as the third data. Here, S is a positive integer, for example, S = 1, 2, 3, or 4. (H+S)+B-1 < A or (H+S)+B-1 = A, KH < B < AH, which means that the third data can be represented by at least K-H+1 bits.

[0164] Taking H=12 and K=23 as an example, the value range of B is [12, 52]. Combining the first setting result as "0111111111111100110000101111111...", and taking S=1 and B=12, we can obtain the third data as the 13th to 27th bits of the first result, that is, the third data is "100110000101111". Combining the first setting result as "0111111111111100110000101111111...", and taking S=2 and B=15, we can obtain the third data as the 14th to 30th bits of the first result, that is, the third data is "00110000101111111".

[0165] In another example, when the first set of data includes 6.24, 7.11, 9.89, 9.22, then N = 2, P = 100, M = 9981. Taking 6.24 as the first data, the second target data can be obtained as 0.63087, the third target data as 0.63188, and the first setting result can be determined therefrom as 0.631835375. According to the ieee754 rule, the third data can be represented as 1010000111.

[0166] It should be noted that since after the Kth bit of the first setting result, all are 1 or all are 0, and the sign bit and exponent bit of the first setting result are also known (both are "011111111111"), therefore, the third data determined by the above method can accurately restore the first setting result, and then decompress to obtain the first data. The decompression process will be introduced in the data processing method below Figure 6 and will not be elaborated here for the time.

[0167] Combined with the above description, it can be known that Figure 3 the compression scheme shown has an extremely low compression ratio, can save a large amount of storage space, and thus reduce the data storage cost.

[0168] This application also provides a specific implementation manner for the above compression unit to determine the third data. The calculation process is as follows:

[0169] 1. The compression unit calculates the second target data f1 and the third target data f2 corresponding to the first data v. For example, f1 = v / M + 1, f1 = (v + 1 / P) / Max + 1. It can be understood that the first preset value is taken as 1 here.

[0170] 2. The compression unit determines the first target data. For example, the compression unit selects a value as the first target data from before the second target data to the third target data. Exemplarily, the computing device converts f1 and f2 into binary representations. For example, the computing device obtains the binary forms of f1 and f2 through u1 = float64bits(f2), u2 = float64bits(f2) respectively.

[0171] 3. The compression unit calculates the number of leading zeros after the exclusive OR of u1 and u2, denoted as Z = LeadingZeros64(u1^u2).

[0172] 4. Since 1 <= f1 < f2 < 2, under the IEEE754 standard: the high 12 bits of u1 and u2 must be the same, then Z >= 12, and the number of common prefixes of the fractional parts can be represented by Z - 12.

[0173] 5. Take u1, set the (Z + 1)-th bit of u1 to 1 and set all the bits after the (Z + 1)-th bit to 0, or, set all the bits after the (Z + 2)-th bit of u2 to 1, and convert it to a new floating-point number f3. Then f3 satisfies f1 <= f3 < f2, that is, f3 is the first setting result.

[0174] 6. The compression unit determines the bits from the (H + 1)-th bit to the (Z + 1)-th bit in the binary form of f3 as the third data.

[0175] Optionally, before the compression unit determines the third data, it needs to check the determined compressed data to determine whether it can be losslessly decompressed. If the check passes, it is used as the third data, otherwise, the data is marked as unable to be compressed. Exemplarily, the compression unit determines whether the product of the first data and the first precision is equal to the floor of the product of the third preset value and the first check data. If the result is "yes", it can be losslessly decompressed, and if the result is "no", it cannot be losslessly decompressed. Among them, the first precision value is determined based on the data with the largest number of decimal places in the first group of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the first check data is the difference between the first setting result and the first preset value. Among them, the first precision value can be the above-mentioned first precision value P, and the third preset value can be M * P. Taking P = 1000, M = 3.9121, the first preset value as 1, and the first setting result as approximately 1.797563 as an example, then the first precision value is 1000, the third preset value is 3912.1, the first setting result is 1.797563, the product of the first data and the first precision is 3.12 * 1000 = 3120, and the floor of the product of the third preset value and the first check data is floor(3912.1 * (1.797563 - 1)) = 3120. Therefore, 3.12 can be compressed to 1.797563.

[0176] The above scheme is equivalent to v == floor(M * F * P) / P, where v is the first data, F is the first setting result, M is the dividend, and P is the first precision value.

[0177] In a possible implementation, the first group of data further includes a fifth data, and the fifth data occupies the first number of bit positions. If the product of the fifth data and the first precision value is not equal to the integer result of the product of the third preset value and the second check data, then the second group of data includes the fifth data, where the second check data is obtained based on the fifth data. If the fifth data is 0, then the second group of data further includes a sixth data, and the sixth data occupies the fifth number of bit positions, and the fifth number is less than the first number.

[0178] If the product of the fifth data and the first precision value is not equal to the rounded result of the product of the third preset value and the second check data, it can be understood that the fifth data cannot pass the above verification process. Therefore, it is determined that the fifth data cannot be compressed. In this case, the fifth data can be directly stored in the first set of data. The description of the second check data can be found in the description of the first check data above, and will not be repeated here. If the fifth data is 0, it usually does not need to be compressed. The position of the fifth data and that the fifth data is 0 can be indicated by storing special characters. A special queue can also be maintained specifically to record the index of data '0'. In this queue, both the special character and one data item in the special queue can be the sixth data item, and the sixth data item occupies the same number of bits as the fifth data item. The number of bits in the fifth data item is less than the number in the first data item; for example, the number of bits in the fifth data item is 1-63.

[0179] In another possible implementation, the second set of data also includes a first symbol, which indicates whether the first data is positive or negative. For example, when the first data is positive, the first symbol is determined to be positive; when the first data is negative, the first symbol is determined to be negative, so that the electronic device can correctly restore the first data during the decompression process.

[0180] Optionally, the second set of data also includes a second symbol, which indicates that the fifth data could not be compressed, or indicates that the fifth data is 0. This helps the electronic device to correctly restore the data during decompression.

[0181] For example, an extended bitmap can be used to record the index of each data in a set of data, with each index occupying 2 bits. For example, "00" indicates a compressible positive number, "01" indicates a value of 0, "10" indicates a compressible negative number, and "11" indicates an incompressible number.

[0182] In one possible implementation, to further reduce the data compression ratio, the compression ratio can be further reduced by extracting common prefixes.

[0183] The above Figure 3 The method shown uses the first data as an example to illustrate the data compression process. Clearly, the compression unit can... Figure 3The scheme shown compresses each data point in the first set of data and generates a corresponding setting result for each data point. With a fixed first preset value, each generated setting result belongs to a first numerical range. Therefore, the sign and exponent bits of the multiple setting results generated by the compression unit (including the first setting result) are identical, and the first S bits of the mantissa are all 1s. Thus, the first H+S bits of the first setting result can be determined as a common prefix (also called the first prefix). Since the first H bits of multiple setting results are both sign and exponent bits, and the first H bits of multiple setting results are identical, the data in the first H bits can be determined. Therefore, the common prefix can be further limited to the (H+1)th to (H+S-1)th bits of the first setting result.

[0184] See Figure 4 , Figure 4 This is a schematic diagram illustrating the data composition of a first setting result provided in an embodiment of this application. For example... Figure 4 As shown, bits 1 to H are the sign and exponent bits, bits H+1 to H+S-1 are the common prefix, bits H+S to H+S+B-1 are the third data, and bits H+S+B to A are all 1s or all 0s. Since the sign, exponent, and the number of bits after H+S+B are known, the first setup result can be reconstructed using the common prefix and the third data.

[0185] For example, taking a first setting result of "0111111111111100110000101111111...", S=2, B=12 as an example, the sign bit and exponent bits are bits 1 to H: "011111111111", the common prefix is ​​bits H+1 to H+S-1: "11", and the third data is bits H+S to H+S+B-1: "001100001011". Since the sign bit, exponent bit, and bits after H+S+B-1 are all known, the first setting result can be determined based on the common prefix and the third data.

[0186] This application also provides a data processing method for determining the number of bits S occupied by the public prefix, as exemplarily shown below:

[0187] 1. The compression unit is determined as K = Min / Max + 1, where Min is the smallest absolute value in the first set of data and Max is the largest absolute value in the first set of data.

[0188] 2. The compression unit converts K to binary. For example, the compression unit determines the binary representation of K as uk by using uk = float64bits(K).

[0189] 3. The compression unit removes the sign bit of uk. For example, the compression unit removes the sign bit by using uk = uk << 1.

[0190] 4. The compression unit determines the number of common prefixes in the first group of data, S = the number of leading 1s in uk - 1.

[0191] Optionally, the second set of data includes the following parts: a common prefix (first prefix), a sign bit (e.g., "00", "01", "10", "11"), and compressed data (e.g., third data, etc.). The order of these parts is not limited in this application. Optionally, the second set of data also includes the sign bit and exponent bit data of the floating-point number in the first numerical range. For example, when the first numerical range is [0,1), the sign bit and exponent bit of the floating-point number in the first numerical range are "0111111111111". Optionally, the second set of data may also include a first preset value. Of course, the sign bit and exponent bit data of the floating-point number in the first numerical range and the first preset value can all be set to default values, so that the second set of data does not need to include this type of data.

[0192] To better understand the compression method provided in this application, this application also provides, for example... Figure 5 The data processing method shown.

[0193] See Figure 5 , Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 5 As shown, the data processing method provided in this application includes:

[0194] S301: The compression unit preprocesses the first set of data.

[0195] For example, the compression unit determines the maximum absolute value Max, the minimum absolute value Min, the maximum significant digit N of the decimal places in the first set of data, and further determines the first precision value P = pow(10, N).

[0196] S302: The compression unit encodes the first group of data. This process can be referenced above. Figure 3 The relevant content will not be repeated here.

[0197] S303: The compression unit verifies the encoded data. This process can be referenced above. Figure 3 The relevant content will not be repeated here.

[0198] S304: The compression unit determines the first prefix. This process can be referenced above. Figure 3 The relevant content will not be repeated here.

[0199] S305: The compression unit extracts a common prefix again from the data after extracting the first prefix. Extracting the common prefix again can be understood as identifying multiple data points that do not contain the first prefix based on the first set of data, and then performing the above process on a portion of these data points that do not contain the first prefix. Figure 3 The operation of extracting the first prefix is ​​used to further compress the data.

[0200] In summary, by employing this compression scheme, the difference between the number of bits occupied by the first data (i.e., the first quantity) and the number of bits occupied by the data obtained after compressing the first data (i.e., the third data) (i.e., the second quantity) is greater than or equal to a fourth quantity. Here, the fourth quantity is the sum of the number of bits occupied by the sign bit and exponent bit of the floating-point data storing the first numerical range. Therefore, the number of bits occupied by the third data is much smaller than the number of bits occupied by the first data. Similarly, the number of bits occupied by the fourth data obtained after compressing the second data is also much smaller than the number of bits occupied by the second data. It can be understood that the more data in the first group is compressed, the larger the proportion of compressed data in the second group. This ensures that the total number of bits occupied by the second group is much smaller than the total number of bits occupied by the first group, and the difference increases with the amount of compressed data. Therefore, the above scheme can significantly reduce the data compression rate. Subsequently, storing only the compressed data (i.e., the second group of data) in the database can reduce the database's data storage costs.

[0201] Specifically, in this technical solution, the first target data is obtained by normalizing the first set of data. The sum of the first target data and the first preset value is the second target data, which belongs to the first numerical range. The second target data is XORed with the third target data, which also belongs to the first numerical range, to obtain the first operation result where the first K bits are 0. That is, the first K bits of the second target data and the third target data are the same. K≥H can be understood as the sign bit and exponent bit of the second target data and the third target data being completely identical. However, the second target data and the third target data are different at bit K+1. Since the second target data is less than the third target data (the second preset value is positive), the (K+1)th bit of the second target data is 0, and the (K+1)th bit of the third target data is 1. Setting bits K+2 to A in the second target data to 1 is used as the first setting result, or setting bits K+2 to A in the third target data to 0 is used as the first setting result. This ensures the first setting result is greater than or equal to the second target data and less than or equal to the third target data, meaning the value represented by the first setting result also belongs to the first value range. Furthermore, the first H bits of the first setting result are the same as the first H bits of any floating-point number in the first value range. The K+1 bit of the first setting result is 0, and the bits after the K+1 bit are 1. Therefore, determining the number of bits from H+S to (H+S)+B-1 in the first setting result as the third data (KH < B < AH) accurately represents the first data. On the other hand, determining the number of bits from H+S to (H+S)+B-1 in the first setting result as the third data results in the third data occupying significantly less storage space than the first data, greatly saving storage space occupied by the first data (i.e., having an extremely low compression rate), thereby significantly reducing the cost of storing the first set of data. Furthermore, the process of determining the third data only involves normalization and XOR operations, so the above implementation scheme also has high compression efficiency.

[0202] Furthermore, this technical solution provides two specific implementation methods for determining the third data. The first implementation method is based on (H+S)+B-1=A, where B=HA when S=1, meaning the third data consists of all the mantissa bits in the first setting result. The second implementation method is based on (H+S)+B-1<A, where B<HA when S=1, meaning the third data consists of a portion of the mantissa bits in the first setting result. It can be seen that both implementation methods can make the number of bits occupied by the third data much smaller than the number of bits occupied by the first data, thus significantly reducing the data compression ratio. Moreover, in both implementation methods, the first prefix is ​​composed of the sign bit and exponent bit of any floating-point number in the first numerical range.

[0203] Furthermore, this technical solution also provides a third specific implementation method for determining the third data. Specifically, the first target data is the normalized result of the first data. Since the first data is the data with the smallest absolute value in the first group of data, the first target data is also the smallest among the normalized results of all data in the first group of data. Based on this, the normalized results of all data are each added to a first preset value. Among all the summation results, the summation result of the first target data and the first preset value (i.e., the second target data) is the smallest, that is, all summation results are greater than or equal to the second target data. When the first S bits of the last digit of the second target data are 1, the first S bits of the last digit of all summation results are also 1. Therefore, the first S bits of the last digit of the second target data can be used as the common part of all data in the second group of data. This is equivalent to using the 1st to (H+S)-1st bits of the second target data as the first prefix. This can further reduce the total number of bits occupied by the second group of data and further reduce the data compression rate.

[0204] In addition, in this technical solution, the first verification data is the difference between the first setting result and the first preset value. The first setting result is determined based on the first data, and the first setting result is used to determine the third data (i.e., the data obtained by compressing the first data). Therefore, by making the product of the first data and the first precision value equal to the rounded result of the product of the third preset value and the first verification data, the third data can be restored and the first data can be obtained.

[0205] On the other hand, this technical solution determines whether to compress the fifth data by judging whether the product of the fifth data and the first precision value is equal to the rounded result of the product of the third preset value and the second check data. This ensures the accuracy of decompression and avoids decompression errors. In addition, if the fifth data is 0, the second group of data includes the sixth data, which occupies the same number of bits as the fifth data. The number of bits in the fifth group is less than the number in the first group, which also helps to reduce the number of bits occupied by storing '0'.

[0206] Furthermore, this technical solution uses a first symbol to indicate whether the first data is positive or negative, which provides more information for reconstructing the first data using third data, ensuring the correctness of the reconstructed first data. In addition, storing the symbol information of the first data through the first symbol can also save storage space and reduce data storage costs.

[0207] This application also provides a decompression scheme for decompressing the above-mentioned... Figure 3 The second set of output data.

[0208] See Figure 6 , Figure 6 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 6As shown, the data processing method provided in this application includes:

[0209] S401: The data processing system retrieves the second set of data from the database.

[0210] In one specific implementation, the second set of data is received by the second acquisition unit in the data processing system. The data processing system can be as described above. Figure 1 In the data processing system 200, the second acquisition unit can be the aforementioned Figure 1 The second acquisition unit 240 in the data processing system 200.

[0211] In practical applications, the second acquisition unit can also receive a second set of data sent from the client. This application does not specifically limit the acquisition method of the second set of data.

[0212] For an introduction to the second set of data, please refer to the preceding text. Figure 2 The content related to the second set of data in step S102 of the data processing method is introduced.

[0213] The second set of data includes multiple compressed data sets, such as the third and fourth sets.

[0214] S402: The decompression unit in the data processing system obtains the first set of data based on the second set of data.

[0215] The decompression unit can be one of the aforementioned Figure 1 The decompression unit 250 in the data processing system 200.

[0216] The first set of data includes multiple floating-point data. For example, the first set of data includes first data and second data, which are floating-point data. Specifically, the first data is 3.1, the second data is 3.2, etc.

[0217] As described above, the third data occupies the second number of bits, and the sum of the second and third numbers is less than or equal to the first number. The floating-point data represented by the third data and the first prefix belongs to the first numerical range. The third number is the sum of the number of bits occupied by the sign and exponent bits of the floating-point data storing the first numerical range. The first prefix includes the sign and exponent bits of the floating-point data in the first numerical range. The fourth data occupies the fourth number of bits, and the sum of the fourth and third numbers is less than or equal to the first number. The floating-point data represented by the fourth data and the first prefix belongs to the first numerical range. The first data is the data obtained by decompressing the third data, and the second data is the data obtained by compressing the fourth data. Both the first and second data are used to represent floating-point data, and both occupy the first number of bits. For a further explanation of the first set of data, please refer to the aforementioned S101.

[0218] In one possible implementation, before outputting the first set of data, the method further includes: obtaining first verification data, wherein the first verification data is the difference between a first setting result and a first preset value, the first setting result includes a first prefix and third data, and the first preset value is a positive number. The integer result of the product of the third preset value and the first verification data is divided by a first precision value to obtain a third calculation result, wherein the first precision value is determined based on the data with the most decimal places in the first set of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, the third preset value is a positive number, and the first data is determined based on the third calculation result.

[0219] After acquiring the second set of data, the decompression unit can extract the first prefix and the third data from the second set of data and recover the first setting result. For example, the first setting result can be determined by concatenating the first prefix and the third data. Specifically, if the sign bit and exponent bit of the first numerical interval are "0111111111111", and the first setting result is assumed to have 1 bits after the H+S+Bth bit, combined with the first prefix being "11" and the third data being "001100001011", the first setting result can be determined to be "011111111111111100110000101111111111111...". Optionally, in some scenarios, the first prefix includes the sign bit and the exponent bit. Optionally, the sign bit and exponent bit can be set to default, so that the second set of data does not need to record the sign bit and exponent bit of the first setting result. For example, the first value range can be [1,2), so that the sign bit of the first setting result is "011111111111".

[0220] After the decompression unit determines the first setting result, it can determine the first verification data based on the first setting result. The first verification data is the difference between the first setting result and the first preset value, where the first preset value is a positive number, for example, 1 or 2. The first setting result is in binary form. The decompression unit can first convert the binary first setting result to decimal, and then subtract the first preset value to obtain the first verification data. For example, if the first setting result is "01111111111111100110000101111111111111...", its decimal value is approximately 1.797563. If the first preset value is 1, then the first verification data can be determined to be 0.797563.

[0221] The decompression unit divides the product of the third preset value and the first verification data by the first precision value to obtain the third calculation result. The third preset value is determined based on the first precision value. The explanation of the first precision value (P) can be found in the preceding content, and the explanation of the third preset value can also be found in the preceding content; it will not be repeated here. For example, the third preset value can be M*P. Taking P=1000 and M=3.9121 as an example, the third preset value is equal to 3912.1. Therefore, the calculation process of the third calculation result can be expressed as floor(3912.1*0.797563) / 1000 = 3.12.

[0222] After calculating the third operation result, the decompression unit determines the first data based on the third operation result. For example, the decompression unit can combine the sign bit stored in the second set of data to determine the sign of the first data, and finally determine the first data. For instance, if the sign bit stored in the second set of data indicates that the first data is positive, then the first data equals 3.12. Alternatively, if the sign bit stored in the second set of data indicates that the first data is negative, then the first data equals -3.12.

[0223] It is understandable that the process of the decompression unit decompressing the third data to obtain the first data can be understood as the reverse process of the above data verification process. Therefore, the process of the decompression unit decompressing data can also refer to the relevant introduction of the above data verification process.

[0224] Optionally, if the sign bit in the second set of data indicates that the fifth data included in the second set of data is uncompressed data, the decompression unit can directly extract the fifth data as decompressed data without performing the above calculation process.

[0225] Optionally, the sign bit in the second set of data indicates that the compressed data corresponding to the fifth data included in the second set of data is 0. The decompression unit can directly confirm 0 as the decompressed data of the fifth data without performing the above calculation process.

[0226] In one possible implementation, the second set of data includes a first symbol, which is a symbol in the sign bit of the second set of data. The decompression unit determines the first data based on the result of a third operation, including: if the first symbol indicates that the first data is positive, using the result of the third operation as the first data; if the first symbol indicates that the first data is negative, setting the first bit of the result of the third operation to 1 and using it as the first data.

[0227] Based on the foregoing description, we know that the result of the third operation is positive. Therefore, if the first sign indicates that the first data is positive, the result of the third operation can be directly used as the first data. If the first sign indicates that the first data is negative, the first bit (sign bit) of the result of the third operation is set to 1 (indicating a negative value) and then used as the first data.

[0228] S403: The data processing system sends the first set of data to the client.

[0229] Accordingly, the client receives the first set of data from the data processing system.

[0230] As can be seen from the above description, the decompression scheme provided in this application can accurately decompress compressed data, and the decompression process is relatively simple. Therefore, the decompression method provided in this application has high decompression performance.

[0231] Specifically, this technical solution can use data occupying fewer bits (i.e., the third and fourth data) to restore data occupying more bits (i.e., the first and second data). This ensures the correctness of decompression and saves the data storage cost of the database, because the database can store only the second set of data and not the first set of data, thus achieving the effect of storing the first set of data.

[0232] More specifically, this technical solution obtains the third calculation result by dividing the integer result of the product of the third preset value and the first verification data by the first precision value, and then obtains the first data based on the third calculation result. The first preset value, the third preset value, and the first precision value can all be pre-stored in the computing device, allowing for rapid retrieval of these values ​​during decompression of the third data, thus accelerating the calculation of the third calculation result. Therefore, the above decompression process is simple and efficient, enabling rapid decompression of compressed data, and the above solution has high decompression performance.

[0233] Moreover, this technical solution can accurately decompress the third data into the first data using the first symbol, avoiding symbol errors in the decompressed data.

[0234] This application also provides a chip system including a processor and a power supply circuit, wherein the power supply circuit supplies power to the processor, and the processor performs the aforementioned operations. Figure 2 The steps in the data processing method are executed by the data processing system. Specifically, the processor is used to execute the aforementioned steps. Figure 2 The steps in the data processing method are executed by the first acquisition unit, the compression unit, and the first output unit in the data processing system. Alternatively, the processor is used to execute the aforementioned steps. Figure 3The steps in the data processing method are executed by the data processing system. Specifically, the processor is used to execute the aforementioned steps. Figure 3 The data processing method comprises the steps executed by the second acquisition unit, decompression unit, and second output unit within the data processing system. For simplicity, these steps will not be elaborated upon here. The processor can be implemented using a GPU, or through computing devices such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.

[0235] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a computing device provided in this application. Figure 7 As shown, the computing device 400 provided in this application includes: a bus 401, a processor 402, a memory 403, and a communication interface 404. The processor 402, the memory 403, and the communication interface 404 communicate with each other via the bus 401. The computing device 400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 400.

[0236] Bus 401 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 401 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 401 may include a path for transmitting information between various components of the computing device 400 (e.g., memory 403, processor 402, communication interface 404).

[0237] Processor 402 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0238] Memory 403 may include volatile memory, such as random access memory (RAM). Processor 402 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 403 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0239] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 403, or two or more types of storage media can be configured to realize the function of memory 403. This application does not limit this.

[0240] The memory 403 stores executable program code, and the processor 402 executes the executable program code to implement the aforementioned functions respectively. Figure 1 The data processing system 200 performs the functions of the first acquisition unit 210 and the first output unit 230, thereby executing the aforementioned functions respectively. Figure 2 The data processing method comprises steps executed by the first acquisition unit, compression unit, and first output unit in the data processing system. That is, the memory 403 stores the steps for executing... Figure 2 , Figure 3 or Figure 5 Instructions for data processing methods.

[0241] Optionally, the program code in memory 403 used to implement the functions of the first acquisition unit 210 and the first output unit 230 is the same as described above. Figure 1 The relationship between the first acquisition unit 210, compression unit 220, and first output unit 230 in the data processing system 200 can also be dynamically adjusted according to business needs, so that the program code stored in the memory 403 can also achieve the aforementioned Figure 1 The data processing system 200 includes one or more units such as the first acquisition unit 210, the compression unit 220, and the first output unit 230. For example, the program code in the memory 403 used to implement the function of the first acquisition unit 210 can not only achieve the aforementioned functions. Figure 1 The first acquisition unit 210 in the data processing system 200 can also perform the aforementioned functions. Figure 1The function of the compression unit 220 in the data processing system 200.

[0242] Alternatively, the memory 403 stores executable code, and the processor 402 executes the executable program code to implement the aforementioned functions respectively. Figure 1 The data processing system 200 performs the functions of the second acquisition unit 240 and the second output unit 260, thereby executing the aforementioned functions respectively. Figure 3 The data processing method is executed by the second acquisition unit, decompression unit, and second output unit in the data processing system. That is, the memory 403 stores the steps for executing... Figure 6 Instructions for data processing methods.

[0243] Optionally, the program code in memory 403 used to implement the functions of the second acquisition unit 240 and the second output unit 260 is the same as described above. Figure 1 The relationship between the second acquisition unit 240, decompression unit 250, and second output unit 260 in the data processing system 200 can also be dynamically adjusted according to business needs, so that the program code stored in the memory 403 can also achieve the aforementioned Figure 1 The data processing system 200 includes one or more units such as the second acquisition unit 240, the decompression unit 250, and the second output unit 260. For example, the program code in the memory 403 used to implement the function of the second acquisition unit 240 can not only achieve the aforementioned functions. Figure 1 The second acquisition unit 240 in the data processing system 200 can also perform the aforementioned functions. Figure 1 The function of the decompression unit 250 in the data processing system 200.

[0244] The communication interface 404 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 400 and other computing devices or communication networks.

[0245] As one possible implementation, the computing device 400 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor performs the aforementioned operations. Figure 2 The data processing method includes the operation steps performed by the data processing system, or the aforementioned steps. Figure 3 The data processing method involves operational steps executed by the data processing system. These steps can be implemented using a processor, such as a GPU, a DPU, NPU, XPU, SoC, offload cards, accelerator cards, or AI chips.

[0246] As one possible implementation, the computing device 400 may include multiple types of processors 402, that is, the computing device 400 is a heterogeneous device. For example, the computing device 400 includes a CPU and a GPU, and the aforementioned functions can be executed by at least one of the processors 402. Figure 2 The data processing method includes the operation steps performed by the data processing system, or the aforementioned steps. Figure 3 The data processing methods involve operational steps performed by the data processing system. For the sake of brevity, these steps will not be elaborated upon here.

[0247] See Figure 8 , Figure 8 This is a schematic diagram of a computing device cluster provided in an embodiment of this application. The computing device cluster provided in this embodiment includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0248] like Figure 8 As shown, the computing device cluster includes at least one computing device 400. The memory 403 of one or more computing devices 400 in the computing device cluster may store the same memory for performing the aforementioned tasks. Figure 2 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 3 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 5 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 6 The instructions for the steps executed by the data processing system in the data processing method.

[0249] In some possible implementations, the memory 403 of one or more computing devices 400 in the computing device cluster may also respectively store memory for executing the aforementioned. Figure 2 The instructions in the data processing method that are executed by the data processing system, or those used to execute the aforementioned steps. Figure 3 The instructions in the data processing method that are executed by the data processing system, or those used to execute the aforementioned steps. Figure 5 The instructions in the data processing method that are executed by the data processing system, or those used to execute the aforementioned steps. Figure 6 The data processing method involves a subset of instructions executed by the data processing system. In other words, a combination of one or more computing devices 400 can jointly execute instructions for performing the aforementioned steps. Figure 2 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned...Figure 3 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 5 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 6 The instructions for the steps executed by the data processing system in the data processing method.

[0250] It should be noted that the memory 403 in different computing devices 400 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 1 The data processing system 200 performs some of its functions. That is, the instructions stored in the memory 403 of different computing devices 400 can implement the functions of one or more of the units, including the first acquisition unit 210, the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250, and the second output unit 260.

[0251] Optionally, the program code in the memory 403 of different computing devices 400 used to implement the functions of the first acquisition unit 210, compression unit 220, first output unit 230, second acquisition unit 240, decompression unit 250, and second output unit 260 is the same as described above. Figure 1 The relationships between the first acquisition unit 210, compression unit 220, first output unit 230, second acquisition unit 240, decompression unit 250, and second output unit 260 in the data processing system 200 can also be dynamically adjusted according to business needs, so that the program code stored in the memory 403 can also achieve the aforementioned Figure 1 The data processing system 200 includes the functions of one or more units among the first acquisition unit 210, compression unit 220, first output unit 230, second acquisition unit 240, decompression unit 250, and second output unit 260.

[0252] See Figure 9 , Figure 9 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. In some possible implementations, one or more computing devices in the computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 As shown, the two computing devices 400A and 400B are connected via a network. Specifically, they are connected to the network through the communication interface in each computing device. In this possible implementation, the memory 403 in computing device 400A stores instructions for executing the function of the first acquisition unit 210. Simultaneously, the memory 403 in computing device 400B stores instructions for executing the function of the first output unit 230.

[0253] Figure 9The connection method between the computing device clusters shown can be that, considering that the data processing method provided in this application requires a large amount of data to be compressed, the function implemented by the first acquisition unit 210 is delegated to the computing device 400A.

[0254] It should be understood that Figure 9 The functions of the computing device 400A shown can also be performed by multiple computing devices 400. Similarly, the functions of the computing device 400B can also be performed by multiple computing devices 400.

[0255] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster. The difference is that the memory 403 in one or more computing devices 400 within this computing device cluster can store the same memory for executing the aforementioned... Figure 2 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 3 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 5 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 6 The instructions for the steps executed by the data processing system in the data processing method.

[0256] In some possible implementations, the memory 403 of one or more computing devices 400 in the computing device cluster may also respectively store memory for executing the aforementioned. Figure 2 The instructions in the data processing method that are executed by the data processing system, or those used to execute the aforementioned steps. Figure 3 The instructions in the data processing method that are executed by the data processing system, or those used to execute the aforementioned steps. Figure 5 The instructions in the data processing method that are executed by the data processing system, or those used to execute the aforementioned steps. Figure 6 The data processing method involves a subset of instructions executed by the data processing system. In other words, a combination of one or more computing devices 400 can jointly execute instructions for performing the aforementioned steps. Figure 2 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 3 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 5 Instructions for steps executed by the data processing system in a data processing method, or instructions for executing the aforementioned... Figure 6 The instructions for the steps executed by the data processing system in the data processing method.

[0257] It should be noted that the memory 403 in different computing devices 400 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 1 The data processing system 200 performs some of its functions. That is, the instructions stored in the memory 403 of different computing devices 400 can implement the functions of one or more of the units, including the first acquisition unit 210, the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250, and the second output unit 260.

[0258] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the aforementioned... Figure 2 The steps in the data processing method performed by the data processing system, or the aforementioned steps. Figure 3 The steps in the data processing method performed by the data processing system, or the aforementioned steps. Figure 5 The steps in the data processing method performed by the data processing system, or the aforementioned steps. Figure 6 The steps performed by the data processing system in the data processing method.

[0259] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any usable medium that a computing device can store, or a data storage device such as a data center containing one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the aforementioned actions. Figure 2 The steps in the data processing method performed by the data processing system, or the aforementioned steps. Figure 3 The steps in the data processing method performed by the data processing system, or the aforementioned steps. Figure 5 The steps in the data processing method performed by the data processing system, or the aforementioned steps. Figure 6 The steps performed by the data processing system in the data processing method.

[0260] It should be understood that in the embodiments of this application, "when," "...when," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not time-limited, nor do they require the device to make a judgment action, nor do they imply any other limitations.

[0261] It should be understood that in the embodiments of this application, "simultaneous" does not necessarily require that they occur at the same time, minute, second, or even at the same moment. When the times of occurrence are slightly different, they can also be understood as occurring "simultaneously".

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a first set of data, which includes first data and second data, wherein both the first data and the second data are used to represent floating-point data, and both the first data and the second data occupy a first number of bits; Based on the first set of data, the first data is normalized to obtain the first target data; Perform an XOR operation on the second target data and the third target data to obtain a first operation result. The first K bits of the first operation result are 0. The second target data and the third target data both belong to a first numerical range. The second target data is the sum of the first target data and a first preset value. The third target data is the sum of the second target data and a second preset value. The first preset value and the second preset value are both positive numbers, and K≥H. The first setting result is obtained by setting bits K+2 to A in the second target data to 1, or by setting bits K+2 to A in the third target data to 0. The number of bits from the H+Sth bit to the (H+S)+B-1th bit in the first setting result is determined as the third data, where S is a positive integer, KH < B < AH, (H+S)+B-1 < A or (H+S)+B-1 = A; Output the second set of data, which includes the third and fourth data; Wherein, A is the first quantity, B is the second quantity, H is the third quantity, the third data is the data obtained by compressing the first data, the third data occupies the bits of the second quantity, based on the floating-point data represented by the third data and the first prefix belonging to the first numerical range, the third quantity is the sum of the number of bits occupied by the sign bit and exponent bit of the floating-point data storing the first numerical range, and the first prefix includes the sign bit and exponent bit of the floating-point data in the first numerical range; The fourth data is the data obtained by compressing the second data. The fourth data occupies a fourth number of bits. The sum of the fourth number and the third number is less than or equal to the first number. The floating-point data represented by the fourth data and the first prefix belongs to the first numerical range.

2. The method according to claim 1, characterized in that, S=1。 3. The method according to claim 1, characterized in that, The first data is the data with the smallest absolute value in the first group of data, and the first S bits of the last S bits of the second target data are 1.

4. The method according to any one of claims 1-3, characterized in that, The product of the first data and the first precision value is equal to the rounded result of the product of the third preset value and the first verification data. The first precision value is determined based on the data with the most decimal places in the first group of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The first verification data is the difference between the first setting result and the first preset value.

5. The method according to claim 4, characterized in that, The first set of data includes a fifth set of data, which occupies the first number of bits; If the product of the fifth data and the first precision value is not equal to the rounded result of the product of the third preset value and the second verification data, then the second set of data includes the fifth data, wherein the second verification data is obtained based on the fifth data; If the fifth data is 0, then the second group of data includes a sixth data, which occupies a fifth number of bits, and the fifth number is less than the first number.

6. The method according to any one of claims 1-3, characterized in that, The second set of data includes a first symbol, which is used to indicate whether the first data is a positive or negative number. The first symbol and the third data are used together to decompress and obtain the first data.

7. A data processing method, characterized in that, The method includes: Obtain the second set of data, which includes the third and fourth sets of data; Obtain first verification data, wherein the first verification data is the difference between a first setting result and a first preset value, the first setting result includes a first prefix and the third data, and the first preset value is a positive number; The product of the third preset value and the first verification data is rounded down and divided by the first precision value to obtain the third calculation result. The first precision value is determined based on the data with the most decimal places in the first group of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The third preset value is a positive number. The first data is determined based on the result of the third calculation. Output the first set of data, which includes the first data and the second data; wherein, The third data occupies a second number of bits, the sum of the second number and the third number is less than or equal to the first number, based on the fact that the floating-point data represented by the third data and the first prefix belongs to a first numerical range, the third number is the sum of the number of bits occupied by the sign bit and exponent bit of the floating-point data storing the first numerical range, and the first prefix includes the sign bit and exponent bit of the floating-point data in the first numerical range. The fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical range. The first data is the data obtained by decompressing the third data, and the second data is the data obtained by decompressing the fourth data. Both the first data and the second data are used to represent floating-point data, and both the first data and the second data occupy the first number of bits.

8. The method according to claim 7, characterized in that, The second set of data includes the first symbol; Determining the first data based on the third calculation result includes: When the first symbol is used to indicate that the first data is a positive number, the result of the third operation is taken as the first data; When the first symbol is used to indicate that the first data is negative, the first bit in the third operation result is set to 1 and then used as the first data.

9. A data processing system, characterized in that, include: The system comprises a first acquisition unit, a compression unit, and a first output unit. The first acquisition unit is used to acquire a first set of data, the first set of data including first data and second data, wherein the first data and the second data are both used to represent floating-point data, and the first data and the second data each occupy a first number of bits; The compression unit is configured to: normalize the first data based on the first set of data to obtain first target data; perform an XOR operation on the second target data and the third target data to obtain a first operation result where the first K bits are 0; set the K+2 to A bits of the second target data to 1 as a first setting result, or set the K+2 to A bits of the third target data to 0 as the first setting result; determine the number of bits from the H+S to (H+S)+B-1 bits of the first setting result as the third data; wherein the second target data and the third target data both belong to a first numerical range, the second target data is the sum of the first target data and a first preset value, the third target data is the sum of the second target data and a second preset value, the first preset value and the second preset value are both positive numbers, S is a positive integer, K≥H, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A; The first output unit is used to output a second set of data, the second set of data including the third data and the fourth data; Wherein, A is the first quantity, B is the second quantity, H is the third quantity, the third data is the data obtained by compressing the first data, the third data occupies the bits of the second quantity, based on the floating-point data represented by the third data and the first prefix belonging to the first numerical range, the third quantity is the sum of the number of bits occupied by the sign bit and exponent bit of the floating-point data storing the first numerical range, and the first prefix includes the sign bit and exponent bit of the floating-point data in the first numerical range; The fourth data is the data obtained by compressing the second data. The fourth data occupies a fourth number of bits. The sum of the fourth number and the third number is less than or equal to the first number. The floating-point data represented by the fourth data and the first prefix belongs to the first numerical range.

10. The system according to claim 9, characterized in that, S=1。 11. The system according to claim 9, characterized in that, The first data is the data with the smallest absolute value in the first group of data, and the first S bits of the last S bits of the second target data are 1.

12. The system according to any one of claims 9-11, characterized in that, The product of the first data and the first precision value is equal to the rounded result of the product of the third preset value and the first verification data. The first precision value is determined based on the data with the most decimal places in the first group of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The first verification data is the difference between the first setting result and the first preset value.

13. The system according to claim 12, characterized in that, The first set of data includes a fifth set of data, which occupies the first number of bits; If the product of the fifth data and the first precision value is not equal to the rounded result of the product of the third preset value and the second verification data, then the second set of data includes the fifth data, wherein the second verification data is obtained based on the fifth data; If the fifth data is 0, then the second group of data includes a sixth data, which occupies a fifth number of bits, and the fifth number is less than the first number.

14. The system according to any one of claims 9-11, characterized in that, The second set of data includes a first symbol, which is used to indicate whether the first data is a positive or negative number. The first symbol and the third data are used together to decompress and obtain the first data.

15. A data processing system, characterized in that, include: The system comprises a second acquisition unit, a decompression unit, and a second output unit. The second acquisition unit is used to acquire a second set of data, which includes a third set of data and a fourth set of data; The decompression unit is configured to: acquire first verification data; divide the integer result of the product of a third preset value and the first verification data by a first precision value to obtain a third operation result; and determine first data based on the third operation result; wherein the first verification data is the difference between a first setting result and a first preset value, the first setting result includes the first prefix and the third data, the first preset value is a positive number, the first precision value is determined based on the data with the most decimal places in the first group of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the third preset value is a positive number; The second output unit is used to output the first set of data, which includes the first data and the second data. The third data occupies a second number of bits, the sum of the second and third numbers is less than or equal to the first number, and the floating-point data represented by the third data and the first prefix belongs to a first numerical range. The third number is the sum of the number of bits occupied by the sign bit and exponent bit of the floating-point data storing the first numerical range. The first prefix includes the sign bit and exponent bit of the floating-point data in the first numerical range. The fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical range. The first data is the data obtained by decompressing the third data, and the second data is the data obtained by decompressing the fourth data. Both the first data and the second data are used to represent floating-point data, and both the first data and the second data occupy the first number of bits.

16. The system according to claim 15, characterized in that, The second set of data includes the first symbol; The decompression unit is specifically used to use the third operation result as the first data when the first symbol indicates that the first data is positive; and to set the first bit of the third operation result to 1 and use it as the first data when the first symbol indicates that the first data is negative.

17. A chip system, characterized in that, The chip system includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to perform the operation steps of the method as described in any one of claims 1 to 8.

18. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

19. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 8.

20. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 8.

Images