Data processing method and system and related equipment

Abstract

The invention provides a data processing method and system and related equipment, and can be applied to the technical field of computers. The method includes acquiring a first group of data including first data and second data, and outputting a second group of data including third data and fourth data. Wherein the first data and the second data occupy a first number of bits, the third data and the fourth data are data obtained by compressing the first data and the second data respectively, the third data occupy a second number of bits, the fourth data occupy a fourth number of bits, and the sum of the second number and the third number is smaller than or equal to the first number. The sum of the fourth number and the third number is smaller than or equal to the first number, and the third number is the number of bits occupied by sign bits and index bits of the floating point type data. Therefore, according to the method, the number of bits occupied by the compressed data can be far smaller than the number of bits occupied by the data before compression, the compression ratio of the data is reduced, and therefore the data storage cost of the database is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, system and related equipment. Background Art

[0002] A time series database is a database designed specifically for time series data. Floating-point data is one of the commonly used data types in time series databases, and some time series databases even only support the import of floating-point data.

[0003] The amount of floating-point data in a time series database is usually very large. In order to reduce the data storage cost, the time series database can compress a large amount of floating-point data and only store the compressed results. In this case, the data storage cost can be measured by the compression rate of the data. The smaller the compression rate, the lower the data storage cost; the larger the compression rate, the higher the data storage cost. However, the compression rate of general compression algorithms is high. If applied to time series databases, it cannot meet the demand for reducing data storage costs. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a data processing method, system and related equipment, which can reduce the compression rate of data and thus reduce the data storage cost of the database.

[0005] In a first aspect, a data processing method is provided, the method comprising: obtaining a first group of data including first data and second data, and outputting a second group of data including third data and fourth data. The first data and the second data are both used to represent floating-point data, and the first data and the second data both occupy a first number of bits. The third data is data obtained by compressing the first data, the third data occupies a second number of bits, the sum of the second number and the third number is less than or equal to the first number, the floating-point data represented by the third data and the first prefix belongs to the first numerical interval, the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, and the first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval. The fourth data is data obtained by compressing the second data, the fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical interval.

[0006] It is understandable that floating point data can be divided into single-precision floating point data and double-precision floating point data according to data precision. Computers store floating point data according to preset rules. Taking the standard of Institute of Electrical and Electronics Engineers (IEEE) 754 (also known as ANSI / IEEE Std 754-1985) as an example, computers use 32 bits to store single-precision floating point numbers, where the first bit is the sign bit, 0 represents a positive number, 1 represents a negative number, the second to the ninth bits are the exponent bits, and the tenth to the 32nd bits are the mantissa bits (the leftmost bit of the data is the first bit). Computers use 64 bits to store double-precision floating point numbers, where the first bit is also the sign bit, the second to the twelfth bits are the exponent bits, and the thirteenth to the 64th bits are the mantissa bits.

[0007] In the above scheme, the difference between the number of bits occupied by the first data (i.e., the first number) and the number of bits occupied by the data obtained by compressing the first data (i.e., the third data) (i.e., the second number) is greater than or equal to the fourth number, wherein the fourth number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, and therefore, the number of bits occupied by the third data is much smaller than the number of bits occupied by the first data. Similarly, the number of bits occupied by the fourth data obtained after compressing the second data is also much smaller than the number of bits occupied by the second data. It can be understood that the more data in the first group of data is compressed, the greater the proportion of compressed data contained in the second group of data, so that the total number of bits occupied by the second group of data is much smaller than the total number of bits occupied by the first group of data, and the gap between the two increases as the number of compressed data increases. Therefore, the above scheme can greatly reduce the compression rate of the data, and only the compressed data (i.e., the second group of data) is subsequently stored in the database, which can reduce the data storage cost of the database.

[0008] Optionally, in this article, "the data occupies XX number of bits" can be understood as the binary form of the data occupies XX number of bits, for example, the third data occupies the second number of bits, which can be understood as the binary form of the third data occupies the second number of bits. "The floating-point data represented by XX belongs to the first numerical interval" can be understood as the value of the floating-point data represented by XX belongs to the first numerical interval, for example, the floating-point data represented by the third data and the first prefix belong to the first numerical interval, which can be understood as the value of the floating-point data represented by the third data and the first prefix belongs to the first numerical interval.

[0009] Optionally, the data in the first group of data are all single-precision floating point numbers or all double-precision floating point numbers.

[0010] Optionally, the first number is 32, or the first number is 64.

[0011] Optionally, when the first number is 32, the third number is 9. When the first number is 64, the third number is 12.

[0012] Optionally, the first numerical interval is [1, 2), or the first numerical interval is [2, 3).

[0013] In some possible implementations, the first number is A, the second number is B, and the third number is H. Before outputting the second set of data, the method further includes: normalizing the first data based on the first set of data to obtain the first target data; performing an XOR operation on the second target data and the third target data to obtain a first operation result, and the first K bits of the first operation result are 0; setting the K+2th to Ath bits in the second target data to 1 as the first setting result, or setting the K+2th to Ath bits in the third target data to 0 as the first setting result; determining the number of bits from the H+Sth to the (H+S)+B-1th bits in the first setting result as the third data. Wherein, the second target data and the third target data both belong to the first numerical interval. The second target data is the sum of the first target data and the first preset value. The third target data is the sum of the second target data and the second preset value. The first preset value and the second preset value are both positive numbers. S is a positive integer, K≥H, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A.

[0014] In the above scheme, the first target data is obtained by normalizing the first data based on the first group of data, the sum of the first target data and the first preset value is the second target data, and the second target data belongs to the first numerical interval. The second target data is subjected to an XOR operation with the third target data belonging to the first numerical interval to obtain a first operation result in which the first K bits are 0, that is, the first K bits of the second target data are the same as those of the third target data. K≥H, which can be understood as the sign bit and exponent bit of the second target data are exactly the same as those of the third target data, and the second target data is different from the third target data at the K+1 bit. Since the second target data is smaller than the third target data (the second preset value is a positive number), the K+1th bit of the second target data is 0, and the K+1th bit of the third target data is 1. The first setting result is set to 1 from the K+2th bit to the Ath bit in the second target data, or the first setting result is set to 0 from the K+2th bit to the Ath bit in the third target data, so that the first setting result is greater than or equal to the second target data, and the first setting result is less than or equal to the third target data, that is, the value represented by the first setting result also belongs to the first numerical interval, so that the first H bits of the first setting result are the same as the first H bits of any floating point number in the first numerical interval, the K+1th bit of the first setting result is 0, and the data bits after the K+1th bit are 1, therefore, the number of bits from the H+Sth bit to the (H+S)+B-1th bit in the first setting result is determined as the third data (KH<B<AH), which can accurately represent the first data. On the other hand, the number of bits from the H+Sth bit to the (H+S)+B-1th bit in the first setting result is determined as the third data, so that the storage space occupied by the third data is much smaller than the first data, which can greatly save the storage space occupied by storing the first data (that is, it has an extremely low compression rate), thereby greatly reducing the cost of storing the first group of data. In addition, the process of determining the third data only involves normalization and XOR operation, so the above implementation scheme also has higher compression efficiency.

[0015] Optionally, the first preset value is a positive integer, for example, the first preset value is 1, 2, 3 or 4, etc.

[0016] In some possible implementations, S=1.

[0017] The above scheme provides two specific implementation methods for determining the third data. The first implementation method is based on (H+S)+B-1=A. When S=1, B=HA, that is, the third data is all the mantissa bits in the first setting result. The second implementation method is based on (H+S)+B-1<A. When S=1, B<HA, that is, the third data is part of the mantissa bits in the first setting result. It can be seen that these two implementation methods can make the number of bits occupied by the third data much smaller than the number of bits occupied by the first data, thereby greatly reducing the data compression rate. Moreover, in these two implementation methods, the first prefix is ​​composed of the sign bit and exponent bit of any floating point number in the first numerical interval.

[0018] Optionally, the second set of data includes the first prefix.

[0019] In some possible implementations, the first data is data with the smallest absolute value in the first group of data, and the first S bits of the mantissa of the second target data are 1.

[0020] In the above scheme, the first target data is the normalized result of the first data. Since the first data is the data with the smallest absolute value in the first set of data, the first target data is also the smallest in the normalized results of all the data in the first set of data. On this basis, the normalized results of all the data are added to the first preset value, and the sum of the first target data and the first preset value (i.e., the second target data) is the smallest among all the summed results, that is, all the summed results are greater than or equal to the second target data. In the case where the first S bits of the mantissa of the second target data are 1, the first S bits of the mantissa of all the summed results are also 1. Therefore, the first S bits of the mantissa of the second target data can be used as the common part of all the data in the second set of data, which is equivalent to using the 1st to (H+S)-1st bits of the second target data as the first prefix, which can further reduce the total number of bits occupied by the second set of data and further reduce the compression rate of the data.

[0021] Optionally, the first preset value is a positive integer. For example, the first preset value is 1, 2, 3 or 4.

[0022] Optionally, the data obtained by normalizing any data in the first set of data is a positive number. For example, the data obtained by normalizing any data in the first set of data is a positive number [0, 1] or [1, 2].

[0023] Optionally, S is a positive integer, for example, S is 1, 2, 3 or 4, etc.

[0024] Optionally, the data in the first set of data may be normalized in the following manner: For example, the absolute value of the data in the first set of data is taken and divided by M to obtain a normalized value, wherein M is greater than or equal to the maximum absolute value of the first set of data.

[0025] Optionally, M is equal to max+L, where max is the maximum absolute value of the first set of data and L is a positive number.

[0026] Optionally, N is related to the data precision of the first set of data, and the data precision of the first set of data is related to the maximum number of decimal places of the first set of data. For example, if the maximum number of decimal places of the first set of data is N, then L and N satisfy the following relationship: L = (1 + x) / pow (10, N), where x is a positive number, for example, x = 0.05, 0.1 or 0.2.

[0027] In some possible implementations, the product of the first data and the first precision value is equal to the rounded result of the product of the third preset value and the first verification data, wherein the first precision value is determined based on the data with the largest number of decimal places in the first group of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the first verification data is the difference between the first setting result and the first preset value.

[0028] In the above scheme, the first verification data is the difference between the first setting result and the first preset value. The first setting result is determined based on the first data, and the first setting result is used to determine the third data (that is, the data obtained by compressing the first data). Therefore, by making the product of the first data and the first precision value equal to the rounded result of the product of the third preset value and the first verification data, the third data can be restored and the first data can be obtained.

[0029] In some possible implementations, the first set of data includes fifth data, and the fifth data occupies a first number of bits. If the product of the fifth data and the first precision value is not equal to the rounded result of the product of the third preset value and the second verification data, the second set of data includes the fifth data, wherein the second verification data is obtained based on the fifth data. If the fifth data is 0, the second set of data includes sixth data, and the sixth data occupies a fifth number of bits, and the fifth number is less than the first number.

[0030] In the above scheme, by judging whether the product of the fifth data and the first precision value is equal to the rounded result of the product of the third preset value and the second verification data, it is determined whether to compress the fifth data, which can ensure the accuracy of decompression and avoid decompression errors. In addition, if the fifth data is 0, the second group of data includes the sixth data, the sixth data occupies the fifth number of bits, and the fifth number is less than the first number, which also helps to reduce the number of bits occupied by storing '0'. Optionally, the sixth data can be a special bit number combination, for example, the sixth data is "0000000000".

[0031] In some possible implementations, the second group of data includes a first symbol, where the first symbol is used to indicate whether the first data is a positive number or a negative number, and the first symbol and the third data are used together to decompress the first data.

[0032] In the above scheme, the first symbol is used to indicate whether the first data is positive or negative, which can provide more information for restoring the first data using the third data, thereby ensuring the correctness of restoring the first data. In addition, storing the symbol information of the first data using the first symbol can also save storage space and reduce data storage costs.

[0033] In a second aspect, a data processing method is provided, the method comprising: obtaining a second group of data including third data and fourth data, and outputting a first group of data including first data and second data. The third data occupies a second number of bits, the sum of the second number and the third number is less than or equal to the first number, the floating-point data represented by the third data and the first prefix belongs to the first numerical interval, the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, and the first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval. The fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical interval. The first data is the data obtained by decompressing the third data, the second data is the data obtained by compressing the fourth data, the first data and the second data are both used to represent the floating-point data, and the first data and the second data both occupy the first number of bits.

[0034] Therefore, the present technical solution can use data occupying a smaller number of bits (i.e., the third data and the fourth data) to restore data occupying a larger number of bits (i.e., the first data and the second data). This not only ensures the correctness of the decompression, but also saves the data storage cost of the database, because the database can only store the second set of data instead of the first set of data, thus achieving the effect of storing the first set of data.

[0035] In some possible implementations, before outputting the first set of data, the method further includes:

[0036] Obtain first verification data; divide the rounded result of the product of the third preset value and the first verification data by the first precision value to obtain a third operation result; determine the first data based on the third operation result. The first verification data is the difference between the first setting result and the first preset value. The first setting result includes the first prefix and the third data. The first preset value is a positive number. The first precision value is determined based on the data with the largest number of decimal places in the first group of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The third preset value is a positive number.

[0037] In the above scheme, the third operation result can be obtained by dividing the rounded result of the product of the third preset value and the first verification data by the first precision value, and then the first data can be obtained based on the third operation result. Among them, the first preset value, the third preset value and the first precision value can all be pre-stored in the computing device, so that the first preset value, the third preset value and the first precision value can be quickly obtained when decompressing the third data, thereby speeding up the calculation speed of the third operation result. Therefore, the above decompression process is simple and efficient, so that the compressed data can be quickly decompressed, and the above scheme has a high decompression performance.

[0038] In some possible implementations, the second group of data includes a first symbol. The determining the first data based on the third operation result includes: when the first symbol is used to indicate that the first data is a positive number, the third operation result is used as the first data; when the first symbol is used to indicate that the first data is a negative number, the first bit in the third operation result is set to 1 and used as the first data.

[0039] In the above solution, the third data can be accurately decompressed into the first data through the first symbol, thereby avoiding symbol errors in the decompressed data.

[0040] Optionally, the first symbol is also used to indicate that the fifth data is uncompressed data. In this case, the fifth data can be directly extracted as decompressed data.

[0041] Optionally, the first symbol is also used to indicate that the sixth data is 0, and the decompression result of the sixth data can be set to 0.

[0042] In a third aspect, a data processing system is provided, the system comprising a first acquisition unit and a first output unit. The first acquisition unit is used to acquire a first set of data, the first set of data comprising first data and second data. The first output unit is used to output a second set of data, the second set of data comprising third data and fourth data. The first data and the second data are both used to represent floating-point data. The first data and the second data both occupy a first number of bits. The third data is data compressed from the first data. The third data occupies a second number of bits. The sum of the second number and the third number is less than or equal to the first number. The floating-point data represented based on the third data and the first prefix belongs to the first numerical interval. The third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval. The first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval. The fourth data is data compressed from the second data. The fourth data occupies a fourth number of bits. The sum of the fourth number and the third number is less than or equal to the first number. The floating-point data represented based on the fourth data and the first prefix belongs to the first numerical interval.

[0043] In some possible implementations, the first number is A, the second number is B, and the third number is H. The data processing system also includes a compression unit. The compression unit is used to normalize the first data based on the first group of data before the first output unit outputs the second group of data to obtain the first target data; perform an XOR operation on the second target data and the third target data to obtain a first operation result with the first K bits being 0; set the K+2th to Ath bits in the second target data to 1 as the first setting result, or set the K+2th to Ath bits in the third target data to 0 as the first setting result; determine the number of bits from the H+Sth to the (H+S)+B-1th bits in the first setting result as the third data. The second target data and the third target data both belong to the first numerical interval. The second target data is the sum of the first target data and the first preset value. The third target data is the sum of the second target data and the second preset value. The first preset value and the second preset value are both positive numbers. S is a positive integer, K≥H, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A.

[0044] In some possible implementations, S=1.

[0045] In some possible implementations, the first data is data with the smallest absolute value in the first group of data. The first S bits of the mantissa of the second target data are 1.

[0046] In some possible implementations, the product of the first data and the first precision value is equal to the rounded product of the third preset value and the first verification data. The first precision value is determined based on the data with the largest number of decimal places in the first group of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The first verification data is the difference between the first setting result and the first preset value.

[0047] In some possible implementations, the first set of data includes fifth data. The fifth data occupies a first number of bits. If the product of the fifth data and the first precision value is not equal to the rounded product of the third preset value and the second verification data, the second set of data includes the fifth data. The second verification data is obtained based on the fifth data. If the fifth data is 0, the second set of data includes sixth data. The sixth data occupies a fifth number of bits. The fifth number is less than the first number.

[0048] In some possible implementations, the second set of data includes a first symbol. The first symbol is used to indicate whether the first data is a positive number or a negative number. The first symbol and the third data are used together to decompress the first data.

[0049] In a fourth aspect, a data processing system is provided, the system comprising a second acquisition unit and a second output unit. The second acquisition unit is used to acquire a second set of data, the second set of data comprising third data and fourth data. The second output unit is used to output a first set of data, the first set of data comprising first data and second data. The third data occupies a second number of bits. The sum of the second number and the third number is less than or equal to the first number. The floating-point data represented based on the third data and the first prefix belongs to the first numerical interval. The third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval. The first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval. The fourth data occupies a fourth number of bits. The sum of the fourth number and the third number is less than or equal to the first number. The floating-point data represented based on the fourth data and the first prefix belongs to the first numerical interval. The first data is the data obtained by decompressing the third data. The second data is the data obtained by compressing the fourth data. The first data and the second data are both used to represent the floating-point data. The first data and the second data both occupy the first number of bits.

[0050] In some possible implementations, the data processing system further includes a decompression unit. The decompression unit is used to obtain first verification data before the second output unit outputs the first group of data; divide the rounded result of the product of the third preset value and the first verification data by the first precision value to obtain a third operation result; and determine the first data based on the third operation result. The first verification data is the difference between the first setting result and the first preset value. The first setting result includes a first prefix and third data. The first preset value is a positive number. The first precision value is determined based on the data with the largest number of decimal places in the first group of data. The third preset value is determined based on the first precision value. The third preset value is different from the first precision value. The third preset value is a positive number.

[0051] In some possible implementations, the second group of data includes a first symbol. The decompression unit is specifically configured to use the third operation result as the first data when the first symbol is used to indicate that the first data is a positive number; and to use the first bit of the third operation result as 1 after it is used as the first data when the first symbol is used to indicate that the first data is a negative number.

[0052] In a fifth aspect, a chip system is provided, comprising a processor and a power supply circuit, wherein the power supply circuit is used to supply power to the processor, and the processor is used to execute any method of the first aspect, or execute any method of the second aspect.

[0053] In a sixth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory.

[0054] The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes any method of the first aspect, or executes any method of the second aspect.

[0055] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed by a computing device, causes the computing device to execute any method of the first aspect, or to execute any method of the second aspect.

[0056] In an eighth aspect, a computer-readable storage medium is provided, characterized in that it includes computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes a method as described in any one of the first aspects, or executes a method as described in any one of the second aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is an architecture diagram of a data processing system provided in an embodiment of the present application;

[0058] Figure 2It is a flowchart of a data processing method provided in an embodiment of the present application;

[0059] Figure 3 It is a flowchart of another data processing method provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of data composition of a first setting result provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of another data processing method provided in an embodiment of the present application;

[0062] Figure 6 A flowchart of another data processing method provided in an embodiment of the present application;

[0063] Figure 7 It is a structural schematic diagram of a computing device provided by the present application;

[0064] Figure 8 It is a structural diagram of a computing device cluster provided by this application;

[0065] Fig. 9 It is a structural diagram of another computing device cluster provided by this application. DETAILED DESCRIPTION

[0066] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] In order to reduce the data storage cost of a database, the present application provides a data processing system, which can compress data so that the difference in the number of bits occupied by the data before and after compression is greater than or equal to the number of bits occupied by the sign bit and the exponent bit of the floating-point data. That is, the system can make a large difference in the number of bits occupied by the data before and after compression, so the system can greatly reduce the compression rate of the data, and subsequently only store the compressed data in the database, which can reduce the data storage cost of the database.

[0068] See also Figure 1 , Figure 1 is an architecture diagram of a data processing system provided in an embodiment of the present application. Figure 1As shown, the architecture includes a client 100, a data processing system 200, and a database 300. The client 100 and the data processing system 200, as well as the data processing system 200 and the database 300, can communicate via wired or wireless means.

[0069] The client 100 is used to generate a first set of data including first data and second data, and send the first set of data to the data processing system 200. The data in the first set of data are all used to represent floating point data, and the data in the first set of data occupy a first number of bits. The first set of data includes the first data and the second data.

[0070] The data processing system 200 is used to compress the data in the first group of data, use the compressed data as the data in the second group of data, thereby obtaining the second group of data, and sending the second group of data to the database 300. Specifically, the data processing system 200 is used to compress the first data into the third data in the second group of data, and compress the second data into the fourth data in the second group of data. Among them, the third data occupies the second number of bits, the fourth data occupies the fourth number of bits, the sum of the second number and the third number is less than or equal to the first number, the sum of the fourth number and the third number is less than or equal to the first number, and the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data.

[0071] The database 300 is used to store the second set of data. The database 300 may be a time series database, a relational database, a graph database, or other database that supports the storage of floating-point data.

[0072] The data processing system 200 is also used to obtain the second set of data from the database 300, decompress the data in the second set of data, obtain the data in the first set of data, and send the first set of data to the client 100. Specifically, the data processing system 200 is used to decompress the third data into the first data and decompress the fourth data into the second data.

[0073] In some possible implementations, the client 100, the data processing system 200, and the database 300 may all be deployed on a computing device, which is an electronic device used to calculate, process, and store data, including a server, a supercomputer, a personal computer, a workstation, a mobile device, and the like.

[0074] Among them, the client 100, the data processing system 200, and the database 300 can be deployed on different computing devices, or deployed on the same computing device. When the client 100, the data processing system 200, and the database 300 are deployed on different computing devices, at least two of the client 100, the data processing system 200, and the database 300 are deployed on different computing devices in the same computing device cluster, or deployed on different computing devices in different computing clusters. Among them, the computing device cluster may include multiple of the above computing devices. The specific deployment situation can be determined according to the actual application scenario, and this application does not make specific limitations.

[0075] In a specific implementation, the client 100 may also be deployed on a terminal device, which is an electronic device used to access a computing device, including a personal computer, a smart phone, a handheld processing device, a tablet computer, a mobile notebook, an integrated handheld device, an intelligent conference device, an intelligent learning machine, and the like.

[0076] Optionally, the data processing system 200 may be software, a plug-in, or a console of a cloud platform.

[0077] When the data processing system 200 is software, the data processing system 200 may be independent software specifically used for compressing data. Users can use the data compression software by downloading and installing it. Figure 1 data processing system 200.

[0078] When the data processing system 200 is a plug-in, the data processing system 200 can be built into the application tool. For example, as a data compression plug-in of the database 300, the data in the database 300 is compressed. The user can use it by upgrading the application tool. Figure 1 A data processing system 200 in FIG.

[0079] When the data processing system 200 is a console, the data processing system 200 can access cloud services that provide data compression / decompression functions, including clients based on the World Wide Web (web), application clients, application programming interfaces (APIs), etc. Users can use cloud services by purchasing them. Figure 1 data processing system 200.

[0080] In some possible implementations, the data processing system 200 includes multiple units. Figure 1In the embodiment, the data processing system 200 includes a first acquisition unit 210, a compression unit 220, a first output unit 230, a second acquisition unit 240, a decompression unit 250 and a second output unit 260.

[0081] Among them, the first acquisition unit 210, the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250 and the second output unit 260 can be implemented by software or by hardware. Exemplarily, the implementation of the first acquisition unit 210 is described below by taking the first acquisition unit 210 as an example. Similarly, the implementation of the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250 and the second output unit 260 can refer to the implementation of the first acquisition unit 210.

[0082] The first acquisition unit 210 is taken as an example of a software functional unit, and the first acquisition unit 210 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first acquisition unit 210 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0083] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0084] The first acquisition unit 210 is taken as an example of a hardware functional unit, and the first acquisition unit 210 may include at least one computing device, such as a server, etc. Alternatively, the first acquisition unit 210 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0085] The multiple computing devices included in the first acquisition unit 210 can be distributed in the same region or in different regions. The multiple computing devices included in the first acquisition unit 210 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first acquisition unit 210 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGA GALs, DPUs, NPUs, SoCs, offload cards, and acceleration cards.

[0086] The functions of each unit in the data processing system 200 are described below.

[0087] The first acquisition unit 210 is used to receive the first set of data input by the user through the client 100, and input the first set of data into the compression unit 220. The data in the first set of data are all used to represent floating point data, and the data in the first set of data occupy a first number of bits. The first set of data includes the first data and the second data. The process can be referred to as follows Figure 2 The execution process of step S101 in the data processing method.

[0088] The compression unit 220 is used to compress the first set of data to obtain the second set of data, and send the second set of data to the first output unit 230. Specifically, the data in the first set of data is compressed, and the compressed data is used as the data in the second set of data, thereby obtaining the second set of data.

[0089] Specifically, in order to compress the first data in the first group of data into the third data in the first group of data, the compression unit 220 is specifically used to: normalize the first data based on the first group of data to obtain the first target data; take the sum of the first target data and the first preset value as the second target data, and take the sum of the second target data and the second preset value as the third target data; perform an XOR operation on the second target data and the third target data to obtain a first operation result in which the first K bits are 0, set the K+2th to Ath bits in the second target data to 1 as the first setting result, or set the K+2th to Ath bits in the third target data to 0 as the first setting result; determine the number of bits from the H+Sth to the (H+S)+B-1th bits in the first setting result as the third data. Among them, the first preset value and the second preset value are both positive numbers, A represents the number of bits occupied by the first data, B represents the number of bits occupied by the third data, H represents the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, S is a positive integer, K≥H, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A.

[0090] The above process of compressing the first set of data to obtain the second set of data, and the process of compressing the first data to obtain the third data, can be referred to as follows Figure 3 The execution process of steps S201 to S205 in the data processing method of Figure 5 The execution process of steps S301 to S305 in the data processing method.

[0091] The first output unit 230 is used to send the second set of data to the database 300. This process can be seen in the following Figure 2 The execution process of step S103 in the data processing method.

[0092] The second acquisition unit 240 is used to acquire a second set of data from the database 300 and input the second set of data into the decompression unit 250 .

[0093] The decompression unit 250 is used to decompress the second set of data to obtain the first set of data, and send the first set of data to the second output unit 260. Specifically, the data in the second set of data is decompressed, and the decompression result of the data is used as the data in the first set of data, thereby obtaining the first set of data.

[0094] Specifically, in order to decompress the third data in the second set of data into the first data in the first set of data, the decompression unit 250 is specifically used to: obtain the first verification data; divide the rounded result of the product of the third preset value and the first verification data by the first precision value to obtain the third operation result; and determine the first data based on the third operation result. The first verification data is the difference between the first setting result and the first preset value, the first precision value is determined based on the data with the largest number of decimal places in the first set of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the third preset value is a positive number.

[0095] The above process of decompressing the second set of data to obtain the first set of data, and the process of decompressing the third data to obtain the first data, can be referred to as follows Figure 6 The execution process of step S402 in the data processing method.

[0096] The second output unit 260 is used to send the first set of data to the client 100 .

[0097] It should be understood that the units in the above data processing system 200 are based on Figure 1 The data processing system 200 is described by taking the first acquisition unit 210 to the second output unit 260 as an example. In practical applications, the data processing system 200 may also include more or fewer units, which is not specifically limited in this application.

[0098] It should be understood that the architecture of the above data processing system is based on Figure 1 The client 100 and the database 300 communicating with the data processing system 200 are each described as an example. In actual applications, the number of the client 100 and the database 300 can be one or more, and this application does not specifically limit it. When the number of the client 100 is multiple, one of the clients 100 can generate a set of data to be compressed and input the set of data into the data processing system 200, and another client 100 can generate another set of data to be compressed and input the set of data into the data processing system 200. When the number of the database 300 is multiple, the data processing system 200 can store a set of compressed data in one of the databases 300, and store another set of compressed data in another database 300.

[0099] In some possible application scenarios, the data processing system 200 can also be used to compress a set of data to be compressed that has been stored in the database 300 to obtain a set of compressed data, and then use the compressed data to replace the data to be compressed in the database 300, thereby reducing the data storage cost of the database 300.

[0100] In summary, the data processing system 200 provided by the present application can make the difference between the number of bits occupied by the data before compression and the number of bits occupied by the data after compression greater than or equal to the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data, that is, the number of bits occupied by the compressed data is much smaller than the number of bits occupied by the data before compression. It can be understood that the more data in the first group of data is compressed, the greater the proportion of compressed data contained in the second group of data, so that the total number of bits occupied by the second group of data can be much smaller than the total number of bits occupied by the first group of data, and the gap between the two increases as the number of compressed data increases. Therefore, the data processing system 200 can greatly reduce the compression rate of the data, and only the compressed data (i.e., the second group of data) is subsequently stored in the database 300, which can reduce the data storage cost of the database 300.

[0101] See also Figure 2 , Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application. Figure 2 As shown, the data processing method provided in the embodiment of the present application includes:

[0102] S101: The client sends a first set of data to a data processing system.

[0103] Accordingly, the data processing system receives a first set of data from the client.

[0104] In a specific implementation, the first set of data is received by a first acquisition unit in a data processing system. The data processing system may be the aforementioned Figure 1 In the data processing system 200, the first acquisition unit may be the aforementioned Figure 1 The first acquisition unit 210 in the data processing system 200.

[0105] In practical applications, the first acquisition unit may also acquire the first set of data by reading a storage medium, or may acquire the first set of data from a database. This application does not specifically limit the method for acquiring the first set of data.

[0106] The first group of data includes a plurality of floating point data. For example, the first group of data includes first data and second data. The first data and the second data are floating point data. Specifically, the first data is 3.1, the second data is 3.2, and so on.

[0107] The application does not limit the number of data in the first set of data. For example, the first set of data is 10, 20, or 30 data.

[0108] Optionally, the amount of data in the first set of data can be preset.

[0109] Optionally, the first group of data is a group of data obtained by intercepting a data stream, and the data arrangement order in the first group of data is the same as the arrangement order in the data stream before interception.

[0110] According to the data precision, floating-point data is divided into single-precision floating-point data and double-precision floating-point data, and the first acquisition unit will store floating-point data according to preset rules. Taking the IEEE754 standard as an example, the first acquisition unit uses 32 bits to store single-precision floating-point numbers, where the first bit is the sign bit, 0 represents a positive number, 1 represents a negative number, the second to the ninth bit is the exponent bit, and the tenth to the 32nd bit is the mantissa bit (the leftmost bit of the data is the first bit). The first acquisition unit uses 64 bits to store double-precision floating-point numbers, where the first bit is also the sign bit, the second to the 12th bit is the exponent bit, and the 13th to the 64th bit is the mantissa bit. For ease of description, the leftmost bit of the data is used as the first bit of the data, that is, the sign bit is the first bit of the data.

[0111] The data in the first group of data occupy a first number of bits, for example, the first data and the second data both occupy a first number of bits. Specifically, if the first data is stored as single-precision floating-point data, the first number is 32; if the first data is stored as double-precision floating-point data, the first number is 64.

[0112] For the convenience of description, unless otherwise specified below, the data in the first set of data and the second set of data to be presented below may be understood as double-precision floating-point data.

[0113] S102: A compression unit in the data processing system obtains a second set of data based on the first set of data.

[0114] The compression unit may be the aforementioned Figure 1 The compression unit 220 in the data processing system 200.

[0115] The second group of data is data obtained by compressing the first group of data by the compression unit, and the second group of data includes multiple compressed data, such as third data and fourth data. The third data is data obtained by compressing the first data by the compression unit, and the fourth data is data obtained by compressing the second data by the compression unit.

[0116] The third data occupies a second number of bits, the sum of the second number and the third number is less than or equal to the first number, and the third number stores the sum of the bit data occupied by the sign bit and the exponent bit of the first data. For example, if the first data is single-precision floating-point data, the first number is 32, and the third number is 9. It can be calculated that the second number is less than or equal to 23, and the third data saves at least 9 bits of storage space compared to the first data. If the first data is double-precision floating-point data, the first number is 64, and the third number is 12. It can be calculated that the second number is less than or equal to 52, and the third data saves at least 9 bits of storage space compared to the first data.

[0117] The fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the third number stores the sum of the bit data occupied by the sign bit and the exponent bit of the second data. For example, if the second data is single-precision floating-point data, the first number is 32, and the third number is 9. It can be calculated that the fourth number is less than or equal to 23, and the fourth data saves at least 9 bits of storage space compared to the second data. If the second data is double-precision floating-point data, the first number is 64, and the third number is 12. It can be calculated that the fourth number is less than or equal to 52, and the fourth data saves at least 9 bits of storage space compared to the second data.

[0118] Therefore, the compression scheme provided by this application has good compression performance.

[0119] The floating-point data represented by the compression unit based on the third data and the first prefix belongs to the first numerical interval, and the floating-point data represented by the compression unit based on the fourth data and the first prefix belongs to the first numerical interval, and the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval. For how the compression unit determines the floating-point data represented based on the third data and the first prefix, or based on the fourth data and the first prefix, please refer to the following Figure 3 The introduction of the relevant contents in the description will not be described in detail here. It should be noted that the number of bits occupied by the floating-point data represented by the third data and the first prefix, or the floating-point data represented by the fourth data and the first prefix, is the same as the number of bits occupied by the data in the first group of data. Therefore, the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data in the first numerical interval is the same as the sum of the number of bits occupied by the sign bit and the exponent bit of the first data.

[0120] Optionally, the first numerical interval is [1, 2), or the first numerical interval is [2, 3).

[0121] S103: The data processing system sends a second set of data to the database.

[0122] The data processing system sends the second set of data to the database so that the database stores the second set of data. In this way, the database only stores the second set of data but not the first set of data, thus achieving the effect of storing the first set of data, thereby saving the data storage cost of the database.

[0123] Next, we will introduce in detail the above Figure 2 The process in which the compression unit compresses the first set of data to obtain the second set of data in step S102 of the data processing method.

[0124] See also Figure 3 , Figure 3 This is a flow chart of another data processing method provided in an embodiment of the present application. It should be noted that: Figure 3 It can be used as a separate embodiment or as the above Figure 2 The data processing method shown in FIG. 1 is a supplement to the data processing method shown in FIG. 1 , and this application does not limit this. Figure 3 As shown, the data processing method provided by the present application includes:

[0125] S201: The compression unit normalizes the first data in the first group of data to obtain first target data.

[0126] Normalization is a data preprocessing technique used to convert the feature values ​​of data to a specific range, usually [0,1] or [-1,1]. Its purpose is to eliminate the differences between different features due to factors such as dimension and order of magnitude, so that the data is comparable and can be analyzed and processed at the same scale. The normalization method provided in this application specifically includes the following steps:

[0127] Step 1: The compression unit takes the absolute value of the first set of data and determines the maximum absolute value of the first set of data.

[0128] For example, the first set of data includes: 3.12 -3.18 3.89 -3.91 3.27 -3.55 3.66 -3.30, then the maximum absolute value of the first set of data is 3.91.

[0129] For another example, the first set of data includes: 6.24, 7.11 -8.98 9.22, then the maximum absolute value of the first set of data is 9.22.

[0130] For ease of description, Max may be used hereinafter to represent the maximum absolute value of the first set of data, and similarly, Min may be used to represent the minimum absolute value of a set of data.

[0131] Step 2: The compression unit determines the dividend M in the normalization process.

[0132] Wherein, M is a value greater than Max, for example, M=Max+L, and L is a positive number, for example, L is 0.05, 0.1 or 0.2.

[0133] In a possible implementation, L depends on a first precision value of the first set of data, where the first precision value is determined based on data with the largest number of decimal places in the first set of data.

[0134] Taking the first set of data including: 3.12 -3.18 3.89 -3.911 3.27 -3.55 3.66 -3.30 as an example, the data bit with the largest number of decimal places in the first set of data is -3.911, and the number of decimal places of -3.911 is 3. It should be noted that the number of decimal places in this application refers to the number of valid decimal places, for example, the valid decimal places of -3.910 are 2, and the valid decimal places of 3.201 are 3. For the convenience of description, the maximum number of valid decimal places in the first set of data is represented by the symbol N below.

[0135] The first precision value P is positively correlated with the maximum number of effective decimal places N in the first set of data. For example, P=pow(10,N). If N=3, then P=1000. For another example, P=100*N. For another example, P=pow(N,5). For ease of description, P=pow(10,N) is taken as an example for related introduction.

[0136] L is negatively correlated with P. For example, L = (1 + x) / P, x is a positive number, for example, x is 0.1, 0.2 or 0.3, etc.; for example, L = (1 + x) / P 2 , for example, x is 0.1, 0.2 or 0.3, etc. For the convenience of description, L=(1+0.1) / P is taken as an example for related introduction.

[0137] M=Max+L, L and Max are both positive numbers, then M is a positive number greater than Max.

[0138] The present application also provides a specific implementation method for the compression unit to determine the maximum number N of valid decimal places in the first group of data. Taking the first data v in the first group of data as an example, and setting the initial m equal to 0, the specific calculation process is as follows:

[0139] 1. Calculate vv = v*pow(10,n)

[0140] 2. Convert vv to IEEE754 binary representation and store it in uint64 (a data type of unsigned 64-bit integer), denoted by u. For example, u = float64bits(vv).

[0141] 3. Calculate the exponential part e of u, for example, e = u>>52–1023.

[0142] 4. Remove the integer part, for example, u=u&(1<<(52-e)-1).

[0143] 5. Determine whether u is 0. If u is 0, it can be determined that N=m and the calculation ends. If u is not equal to 0, add 1 to the value of m and re-execute the above steps 1-4. Optionally, if the output result of u<8||u==(1<<(52-e)-1) is "yes", it can also be determined that N=m and the calculation ends. This is due to the precision of floating-point numbers. There may be errors by directly judging whether u is 0. For example, the compression unit may output the calculation result of 1.001*1000 as 1000.99999999999999; for another example, the compression unit may output the calculation result of 1.09*100 as 109.00000010000001.

[0144] Step 3: The compression unit normalizes the first data to obtain first target data.

[0145] In a possible implementation, the compression unit divides the absolute value of each data in the first set of data by M, and the obtained value is the normalized result of the data. Therefore, the compression unit divides the absolute value of the first data by M, and obtains the normalized result of the first data as the first target data.

[0146] Exemplarily, continuing to take the first set of data including: 3.12 -3.18 3.89 -3.911 3.27 -3.55 3.66-3.3 as an example, for the convenience of description, the normalization process, please refer to Table 1.

[0147] Table 1

[0148]

[0149] Among them, M=Max+1.1 / pow(10,N), Max=3.911, N=3, and the calculated value is M=3.9121.

[0150] The compression unit normalizes the first data based on the first set of data to obtain the first target data. Continuing to refer to Table 1, the first set of data is, for example, the original value in Table 1, and the first data is, for example, 3.12 in Table 1, then the first target data is 0.797526. Since M is a positive number greater than Max, all normalization results are values ​​between 0 and 1.

[0151] S202: The compression unit determines second target data and third target data based on the first target data.

[0152] Among them, the second target data is the sum of the first target data and the first preset value, the third target data is the sum of the second target value and the second preset value, and the first preset value and the second preset value are both positive numbers, for example, the first preset value is 1 or 2, and the second preset value is 1 or 2.

[0153] In a possible implementation manner, the following relationship exists between the second preset value F, the first precision value P and the dividend M: F=1 / (P*M).

[0154] Exemplarily, taking the first target data as 0.797526, the first preset value as 1, P=1000 and M=3.9121 as an example, the second target data is 0.797526+1=1.797526, and the third target data is 1.797526+1 / (1000*3.9121)=1.797781617187699.

[0155] S203: The compression unit performs an XOR operation on the second target data and the third target data to obtain a first operation result.

[0156] Taking the above-mentioned second target data of 1.797526 and the third target data of 1.797781617187699 as an example, when the computing device stores the second target data and the third target data, the sign bits are all 0, the exponent bits are all 1, and the first several bits of the mantissa are the same. Therefore, the computing device performs an XOR operation on the second target data and the third target data, and the first K bits of the first operation result are all 0. The size of K depends on the difference between the second target data and the third target data. The smaller the difference, the larger the K value, and the larger the difference, the smaller the K value. For example, the mantissa of 1.797526 is "11001100001010...", and the mantissa of 1.797781617187699 is "11001100001110...". It can be seen that when 1.797526 and 1.797781617187699 are stored in binary, the first 11 bits of the mantissa are the same, plus 1 sign bit and 11 exponent bits, the first 23 bits of the binary form of 1.797526 and 1.797781617187699 are the same, so K is equal to 23.

[0157] S204: The compression unit determines a first setting result based on the first operation result.

[0158] Among them, the first setting result is obtained by setting the K+2th to Ath bits in the binary form of the second target data to 1, or the first setting result is obtained by setting the K+2th to Ath bits in the binary form of the third target data to 0 as the first setting result.

[0159] Taking the binary number of the second target data as “011111111111111001100001010…” as an example, after setting the K+2th to Ath bits of the second target data to 1, “0111111111111100110000101111111…” is obtained. Taking the binary number of the third target data as “01111111111111001100001110…” as an example, after setting the K+2th to Ath bits of the third target data to 0, “01111111111111100110000110000000…” is obtained. Combined with the above description, it can be known that the value range of the first setting result is from the second target data to the third target data, that is, the second target data is less than or equal to the first setting result, and the first setting result is less than or equal to the third target data.

[0160] Taking the first numerical interval [1,2) as an example, the second target data, the third target data and the first setting result all belong to the first numerical interval.

[0161] K≥H, H is the third number, the third number is the binary form of the floating-point data in the first numerical interval, the number of bits occupied by the sign bit and the exponent bit, for example, taking double-precision floating-point numbers as an example, H is 12, combined with the above description K=23, therefore, K≥H.

[0162] S205: The compression unit determines third data based on the first setting result.

[0163] In a possible implementation, the compression unit determines the number of bits from the H+Sth bit to the (H+S)+B-1th bit in the first setting result as the third data. Wherein, S is a positive integer, for example, S=1, 2, 3 or 4. (H+S)+B-1<A or (H+S)+B-1=A, KH<B<AH, which means that the third data can be represented by at least K-H+1 bits.

[0164] Taking H=12 and K=23 as an example, the value range of B is [12, 52]. Combined with the first setting result of "0111111111111100110000101111111...", and taking S=1 and B=12, it can be obtained that the third data is the 13th to 27th bits in the first result, that is, the third data is "100110000101111". Combined with the first setting result of "0111111111111100110000101111111...", and taking S=2 and B=15, it can be obtained that the third data is the 14th to 30th bits in the first result, that is, the third data is "00110000101111111".

[0165] In another example, when the first set of data includes 6.24, 7.11, 9.89, 9.22, then N = 2, P = 100, M = 9981. Taking 6.24 as the first data, the second target data can be obtained as 0.63087, the third target data can be obtained as 0.63188, and the first setting result can be determined as 0.631835375 from them. According to the ieee754 rule, the third data can be represented as 1010000111.

[0166] It should be noted that since all bits after the Kth bit of the first setting result are either all 1 or all 0, and the sign bit and exponent bit of the first setting result are also known (both are "011111111111"), therefore, the third data determined by the above method can accurately restore the first setting result, and then decompress to obtain the first data. The decompression process will be introduced in the data processing method below and will not be elaborated here for the time being. Figure 6 The data processing method will be introduced below and will not be elaborated here for the time being.

[0167] Combined with the above description, it can be known that Figure 3 the compression scheme shown has an extremely low compression ratio, can save a large amount of storage space, and thus reduce the data storage cost.

[0168] This application also provides a specific implementation manner for the above compression unit to determine the third data. The calculation process is as follows:

[0169] 1. The compression unit calculates the second target data f1 and the third target data f2 corresponding to the first data v. For example, f1 = v / M + 1, f1 = (v + 1 / P) / Max + 1. It can be understood that the first preset value is taken as 1 here.

[0170] 2. The compression unit determines the first target data. For example, the compression unit selects a value as the first target data from before the second target data to the third target data. Exemplarily, the computing device converts f1 and f2 into binary representations. For example, the computing device obtains the binary forms of f1 and f2 through u1 = float64bits(f2) and u2 = float64bits(f2) respectively.

[0171] 3. The compression unit calculates the number of leading zeros after the exclusive OR of u1 and u2, denoted as Z = LeadingZeros64(u1^u2).

[0172] 4. Since 1 <= f1 < f2 < 2, under the IEEE754 standard: the high 12 bits of u1 and u2 must be the same, then Z >= 12, and the number of common prefixes of the fractional parts can be represented by Z - 12.

[0173] 5. Take u1, set the (Z + 1)-th bit of u1 to 1 and set all the bits after the (Z + 1)-th bit to 0, or, set all the bits after the (Z + 2)-th bit of u2 to 1, and convert it to a new floating-point number f3. Then f3 satisfies f1 <= f3 < f2, that is, f3 is the first setting result.

[0174] 6. The compression unit determines the bits from the (H + 1)-th bit to the (Z + 1)-th bit in the binary form of f3 as the third data.

[0175] Optionally, before the compression unit determines the third data, it is necessary to check the determined compressed data to determine whether it can be losslessly decompressed. If the check passes, it is used as the third data, otherwise, the data is marked as unable to be compressed. Exemplarily, the compression unit determines whether the product of the first data and the first precision is equal to the floor of the product of the third preset value and the first check data. If the result is "yes", it can be losslessly decompressed, and if the result is "no", it cannot be losslessly decompressed. Among them, the first precision value is determined based on the data with the largest number of decimal places in the first group of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the first check data is the difference between the first setting result and the first preset value. Among them, the first precision value can be the above-mentioned first precision value P, and the third preset value can be M * P. Taking P = 1000, M = 3.9121, the first preset value as 1, and the first setting result as approximately 1.797563 as an example, then, the first precision value is 1000, the third preset value is 3912.1, the first setting result is 1.797563, the product of the first data and the first precision is 3.12 * 1000 = 3120, and the floor of the product of the third preset value and the first check data is floor(3912.1 * (1.797563 - 1)) = 3120. Therefore, 3.12 can be compressed to 1.797563.

[0176] The above scheme is equivalent to v == floor(M * F * P) / P, where v is the first data, F is the first setting result, M is the dividend, and P is the first precision value.

[0177] In a possible implementation, the first group of data further includes a fifth data, and the fifth data occupies a first number of bit positions. If the product of the fifth data and the first precision value is not equal to the rounded result of the product of the third preset value and the second check data, then the second group of data includes the fifth data, where the second check data is obtained based on the fifth data. If the fifth data is 0, then the second group of data further includes a sixth data, and the sixth data occupies a fifth number of bit positions, and the fifth number is less than the first number.

[0178] If the product of the fifth data and the first precision value is not equal to the rounded result of the product of the third preset value and the second verification data, it can be understood that the fifth data cannot pass the above verification process. Therefore, it is determined that the fifth data cannot be compressed. In this case, the fifth data can be directly saved in the first group of data. Among them, the introduction of the second verification data can refer to the introduction of the first verification data above, and will not be repeated here. If the fifth data is 0, it is usually not necessary to compress it. The position of the fifth data and the fact that the fifth data is 0 can be indicated by storing special characters. A special queue can also be maintained to specifically record the data with a subscript of '0'. Among them, the special character and one of the data in the special queue can both be the sixth data, and the sixth data occupies the fifth number of bits, and the fifth number of bits is less than the first number, for example, the fifth number of bits is 1-63.

[0179] In another possible implementation, the second set of data also includes a first symbol, and the first symbol is used to indicate whether the first data is a positive number or a negative number. For example, when the first data is a positive number, the first symbol is determined to be a positive number; when the first data is a negative number, the first symbol is determined to be a negative number, so that the electronic device can correctly restore the first data during the decompression process.

[0180] Optionally, the second group of data further includes a second symbol, and the second symbol is used to indicate that the fifth data cannot be compressed, or to indicate that the fifth data is 0. In this way, it is helpful for the electronic device to correctly restore the data when decompressing the data.

[0181] Exemplarily, an extended bitmap can be used to record the subscripts of each data in a set of data, where each subscript occupies 2 bits. For example, "00" is used to indicate a compressible positive number, "01" is used to indicate a value of 0, "10" is used to indicate a compressible negative number, and "11" is used to indicate an incompressible number.

[0182] In a possible implementation, in order to further reduce the compression rate of data, the compression rate may be further reduced by extracting a common prefix.

[0183] Above Figure 3 The method shown is to take the first data as an example to exemplarily introduce the data compression process. Figure 3In the scheme shown, data compression is performed on each data in the first group of data, and a setting result corresponding to each data is generated. When the first preset value is fixed, each setting result generated belongs to the first numerical interval. Therefore, the sign bit and the exponent bit of the multiple setting results (including the first setting result) generated by the compression unit are the same, and the first S bits of the mantissa are all 1. Therefore, the first H+S bits of the first setting result can be determined as a common prefix (also called the first prefix), and since the first H bits of the multiple setting results are the sign bit and the exponent bit, and the first H bits of the multiple setting results are the same, the data of the first H bits can be determined, and therefore, the common prefix can be further limited to the H+1th bit to the H+S-1th bit of the first setting result.

[0184] See also Figure 4 , Figure 4 This is a schematic diagram of the data composition of a first setting result provided in an embodiment of the present application. Figure 4 As shown, the 1st to Hth bits are the sign bit and the exponent bit, the H+1th to H+S-1th bits are the common prefix, the H+Sth to H+S+B-1th bits are the third data, and the number of bits from the H+S+Bth to the Ath bits are all 1 or all 0. Since the sign bit, the exponent bit, and the number of bits after the H+S+Bth bit are all known, the first setting result can be restored through the common prefix and the third data.

[0185] Exemplarily, taking the first setting result as "01111111111111100110000101111111...", S=2, B=12 as an example, the sign bit and the exponent bit are the 1st to Hth bits: "011111111111", the common prefix is ​​the H+1th to the H+S-1th bits: "11", and the third data is the H+Sth to the H+S+B-1th bits: "001100001011". Since the sign bit, the exponent bit, and the bits after the H+S+B-1th bit are all known, the first setting result can be determined based on the common prefix and the third data.

[0186] The present application also provides a data processing method to determine the number of bits S occupied by a common prefix, which is exemplarily shown as follows:

[0187] 1. The compression unit determines K=Min / Max+1, where Min is the smallest absolute value in the first set of data, and Max is the largest absolute value in the first set of data.

[0188] 2. The compression unit converts K into binary. For example, the compression unit determines that the binary of K is uk through uk=float64bits(K).

[0189] 3. The compression unit removes the sign bit of uk. For example, the compression unit removes the sign bit by setting uk=uk<<1.

[0190] 4. The compression unit determines the number of common prefixes S of the first group of data = the number of uk leading 1s - 1.

[0191] Optionally, the second group of data includes the following parts: a common prefix (first prefix), a sign bit (for example, "00", "01", "10", "11") and compressed data (for example, third data, etc.). This application does not limit the order of the above parts. Optionally, the second group of data also includes data on the sign bit and exponent bit of the floating-point number in the first numerical interval. For example, when the first numerical interval is [0,1), the sign bit and exponent bit of the floating-point number in the first numerical interval are "0111111111111". Optionally, the second group of data may also include a first preset value. Of course, the data on the sign bit and exponent bit of the floating-point number in the first numerical interval and the first preset value may be set to default, so that the second group of data does not need to include such data.

[0192] In order to better understand the compression method provided by this application, this application also provides the following Figure 5 The data processing method shown.

[0193] See also Figure 5 , Figure 5 A flow chart of another data processing method provided in an embodiment of the present application. Figure 5 As shown, the data processing method provided by the present application includes:

[0194] S301: The compression unit preprocesses the first group of data.

[0195] For example, the compression unit determines the maximum absolute value Max, the minimum absolute value Min, the most significant digit N of the decimal place, etc. in the first group of data, and further determines the first precision value P=pow(10, N).

[0196] S302: The compression unit encodes the first set of data. This process can refer to the above Figure 3 The introduction of relevant contents will not be repeated here.

[0197] S303: The compression unit verifies the encoded data. This process can refer to the above Figure 3 The introduction of relevant contents will not be repeated here.

[0198] S304: The compression unit determines the first prefix. This process can refer to the above Figure 3 The introduction of relevant contents will not be repeated here.

[0199] S305: The compression unit extracts the common prefix again from the data after the first prefix is ​​extracted. Extracting the common prefix again can be understood as determining a plurality of data that do not contain the first prefix based on the first group of data, and then performing the above steps on some of the data that do not contain the first prefix. Figure 3 The operation of extracting the first prefix is ​​performed to further compress the data.

[0200] In summary, by adopting this compression scheme, the difference between the number of bits occupied by the first data (i.e., the first number) and the number of bits occupied by the data obtained by compressing the first data (i.e., the third data) (i.e., the second number) can be greater than or equal to the fourth number, wherein the fourth number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, and therefore, the number of bits occupied by the third data is much smaller than the number of bits occupied by the first data. Similarly, the number of bits occupied by the fourth data obtained after compressing the second data is also much smaller than the number of bits occupied by the second data. It can be understood that the more data in the first group of data is compressed, the greater the proportion of compressed data contained in the second group of data, so that the total number of bits occupied by the second group of data is much smaller than the total number of bits occupied by the first group of data, and the gap between the two increases as the number of compressed data increases. Therefore, the above scheme can greatly reduce the compression rate of the data, and only the compressed data (i.e., the second group of data) is subsequently stored in the database, which can reduce the data storage cost of the database.

[0201] Specifically, in the present technical solution, the first target data is obtained by normalizing the first data based on the first group of data, the sum of the first target data and the first preset value is the second target data, and the second target data belongs to the first numerical interval. The second target data is subjected to an XOR operation with the third target data belonging to the first numerical interval to obtain a first operation result in which the first K bits are 0, that is, the first K bits of the second target data are the same as those of the third target data. K≥H, which can be understood as the sign bit and exponent bit of the second target data are exactly the same as those of the third target data, and the second target data is different from the third target data at the K+1 bit. Since the second target data is smaller than the third target data (the second preset value is a positive number), the K+1th bit of the second target data is 0, and the K+1th bit of the third target data is 1. The first setting result is set to 1 from the K+2th bit to the Ath bit in the second target data, or the first setting result is set to 0 from the K+2th bit to the Ath bit in the third target data, so that the first setting result is greater than or equal to the second target data, and the first setting result is less than or equal to the third target data, that is, the value represented by the first setting result also belongs to the first numerical interval, so that the first H bits of the first setting result are the same as the first H bits of any floating point number in the first numerical interval, the K+1th bit of the first setting result is 0, and the data bits after the K+1th bit are 1, therefore, the number of bits from the H+Sth bit to the (H+S)+B-1th bit in the first setting result is determined as the third data (KH<B<AH), which can accurately represent the first data. On the other hand, the number of bits from the H+Sth bit to the (H+S)+B-1th bit in the first setting result is determined as the third data, so that the storage space occupied by the third data is much smaller than the first data, which can greatly save the storage space occupied by storing the first data (that is, it has an extremely low compression rate), thereby greatly reducing the cost of storing the first group of data. In addition, the process of determining the third data only involves normalization and XOR operation, so the above implementation scheme also has higher compression efficiency.

[0202] In addition, the technical solution provides two specific implementation methods for determining the third data. The first implementation method is based on (H+S)+B-1=A. When S=1, B=HA, that is, the third data is all the mantissa bits in the first setting result. The second implementation method is based on (H+S)+B-1<A. When S=1, B<HA, that is, the third data is part of the mantissa bits in the first setting result. It can be seen that these two implementation methods can make the number of bits occupied by the third data much smaller than the number of bits occupied by the first data, thereby greatly reducing the data compression rate. Moreover, in these two implementation methods, the first prefix is ​​composed of the sign bit and exponent bit of any floating point number in the first numerical interval.

[0203] Moreover, the technical scheme also provides a third specific implementation method for determining the third data. Specifically, the first target data is the normalized result of the first data. Since the first data is the data with the smallest absolute value in the first group of data, the first target data is also the smallest in the normalized results of all data in the first group of data. On this basis, the normalized results of all data are added to the first preset value, and the sum of the first target data and the first preset value (i.e., the second target data) is the smallest among all the summed results, that is, all the summed results are greater than or equal to the second target data. In the case where the first S bits of the mantissa of the second target data are 1, the first S bits of the mantissa of all the summed results are also 1, therefore, the first S bits of the mantissa of the second target data can be used as the common part of all the data in the second group of data, which is equivalent to using the first to (H+S)-1 bits of the second target data as the first prefix, which can further reduce the total number of bits occupied by the second group of data and further reduce the compression rate of the data.

[0204] In addition, in the present technical solution, the first verification data is the difference between the first setting result and the first preset value. The first setting result is determined based on the first data, and the first setting result is used to determine the third data (i.e., the data obtained by compressing the first data). Therefore, by making the product of the first data and the first precision value equal to the rounded result of the product of the third preset value and the first verification data, the third data can be restored and the first data can be obtained.

[0205] On the other hand, the technical solution determines whether to compress the fifth data by judging whether the product of the fifth data and the first precision value is equal to the rounded result of the product of the third preset value and the second verification data, so as to ensure the accuracy of decompression and avoid errors in decompression. In addition, if the fifth data is 0, the second group of data includes the sixth data, the sixth data occupies the fifth number of bits, and the fifth number is less than the first number, which also helps to reduce the number of bits occupied by storing '0'.

[0206] Moreover, the present technical solution uses the first symbol to indicate whether the first data is a positive number or a negative number, which can provide more information for restoring the first data using the third data, thereby ensuring the correctness of restoring the first data. In addition, storing the symbol information of the first data by the first symbol can also save storage space and reduce data storage costs.

[0207] This application also provides a decompression solution for decompressing the above Figure 3 The second set of output data.

[0208] See also Figure 6 , Figure 6 A flow chart of another data processing method provided in an embodiment of the present application. Figure 6As shown, the data processing method provided by the present application includes:

[0209] S401: The data processing system obtains a second set of data from a database.

[0210] In a specific implementation, the second set of data is received by a second acquisition unit in the data processing system. The data processing system may be the aforementioned Figure 1 In the data processing system 200, the second acquisition unit may be the aforementioned Figure 1 The second acquisition unit 240 in the data processing system 200.

[0211] In actual applications, the second acquisition unit may also receive the second set of data sent from the client, and this application does not specifically limit the method for acquiring the second set of data.

[0212] The introduction of the second set of data can refer to the above Figure 2 The content related to the second group of data in step S102 of the data processing method is introduced.

[0213] The second group of data includes a plurality of compressed data, such as third data and fourth data.

[0214] S402: A decompression unit in the data processing system obtains a first group of data based on the second group of data.

[0215] Among them, the decompression unit can be the aforementioned Figure 1 The decompression unit 250 in the data processing system 200.

[0216] The first group of data includes a plurality of floating point data. For example, the first group of data includes first data and second data. The first data and the second data are floating point data. Specifically, the first data is 3.1, the second data is 3.2, and so on.

[0217] As described above, the third data occupies the second number of bits, the sum of the second number and the third number is less than or equal to the first number, the floating-point data represented by the third data and the first prefix belongs to the first numerical interval, the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, and the first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval. The fourth data occupies the fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical interval. The first data is the data obtained by decompressing the third data, the second data is the data obtained by compressing the fourth data, the first data and the second data are both used to represent the floating-point data, and the first data and the second data both occupy the first number of bits. The introduction of the first group of data can also refer to the aforementioned S101.

[0218] In a possible implementation, before outputting the first set of data, the method further includes: obtaining first verification data, wherein the first verification data is the difference between the first setting result and the first preset value, the first setting result includes the first prefix and the third data, and the first preset value is a positive number. The rounding result of the product of the third preset value and the first verification data is divided by the first precision value to obtain a third operation result, wherein the first precision value is determined based on the data with the largest number of decimal places in the first set of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, the third preset value is a positive number, and the first data is determined based on the third operation result.

[0219] After obtaining the second set of data, the decompression unit can extract the first prefix and the third data from the second set of data, and restore the first setting result. For example, the first setting result can be determined by concatenating the first prefix and the third data. Specifically, if the sign bit and the exponent bit of the first numerical interval are "01111111111111", the bit after H+S+B of the first setting result is 1 by default. Combined with the first prefix of "11" and the third data of "001100001011", the first setting result can be determined to be "01111111111111110011000010111111111111...". Optionally, in some scenarios, the first prefix includes a sign bit and an exponent bit. Optionally, the sign bit and exponent bit can be set to default, so that the second set of data does not record the sign bit and exponent bit of the first setting result. For example, the first numerical interval can be made [1,2) and the sign bit of the first setting result is "011111111111".

[0220] After the decompression unit determines the first setting result, the first verification data can be determined based on the first setting result, which is the difference between the first setting result and the first preset value, wherein the first preset value is a positive number, for example, the first preset value is 1 or 2. The first setting result determined above is in binary form, and the decompression unit can first convert the binary first setting result into decimal, and then subtract the first preset value to obtain the first verification data. For example, the first setting result is "0111111111111110011000010111111111111...", and its decimal number is approximately 1.797563. If the first preset value is 1, it can be determined that the first verification data is 0.797563.

[0221] The decompression unit divides the rounded result of the product of the third preset value and the first verification data by the first precision value to obtain a third operation result. The third preset value is determined based on the first precision value. The introduction of the first precision value can refer to the first precision value P in the aforementioned content, and the third preset value can also refer to the introduction of the third preset value in the aforementioned content, which will not be repeated here. For example, the third preset value can be M*P. Taking P=1000 and M=3.9121 as an example, the third preset value is equal to 3912.1. Therefore, the calculation process of the third operation result can be expressed as floor(3912.1*0.797563) / 1000=3.12.

[0222] After calculating the third operation result, the decompression unit determines the first data based on the third operation result. Exemplarily, the decompression unit can determine the positive or negative of the first data in combination with the sign bit stored in the second set of data, and finally determine the first data. For example, if the sign bit stored in the second set of data indicates that the first data is positive, then the first data is equal to 3.12. For example, if the sign bit stored in the second set of data indicates that the first data is negative, then the first data is equal to -3.12.

[0223] It can be understood that the process of the decompression unit decompressing the third data to obtain the first data can be understood as the inverse process of the above-mentioned data verification process. Therefore, the process of the decompression unit decompressing data can also refer to the relevant introduction of the above-mentioned data verification process.

[0224] Optionally, if the sign bit in the second group of data indicates that the fifth data included in the second group of data is uncompressed data, the decompression unit may directly extract the fifth data as decompressed data without performing the above calculation process.

[0225] Optionally, the sign bit in the second group of data indicates that the compressed data corresponding to the fifth data included in the second group of data is 0, and the decompression unit can directly determine 0 as the decompressed data of the fifth data without performing the above calculation process.

[0226] In a possible implementation, the second group of data includes a first symbol, and the first symbol belongs to a symbol in a symbol bit of the second group of data. The decompression unit determines the first data based on the third operation result, including: when the first symbol is used to indicate that the first data is a positive number, the third operation result is used as the first data. When the first symbol is used to indicate that the first data is a negative number, the first bit in the third operation result is set to 1 and used as the first data.

[0227] Combined with the above description, it can be known that the third operation result is a positive value. Therefore, when the first symbol is used to indicate that the first data is a positive number, the third operation result can be directly used as the first data. When the first symbol is used to indicate that the first data is a negative number, the first bit (sign bit) in the third operation result is set to 1 (indicating a negative value) and then used as the first data.

[0228] S403: The data processing system sends a first set of data to the client.

[0229] Accordingly, the client receives a first set of data from the data processing system.

[0230] From the above description, it can be known that the decompression solution provided by the present application can accurately decompress the compressed data, and the decompression process is relatively simple. Therefore, the decompression method provided by the present application has high decompression performance.

[0231] Specifically, the present technical solution can use data occupying a smaller number of bits (i.e., the third data and the fourth data) to restore data occupying a larger number of bits (i.e., the first data and the second data). This not only ensures the correctness of the decompression, but also saves the data storage cost of the database, because the database can only store the second set of data instead of the first set of data, thus achieving the effect of storing the first set of data.

[0232] More specifically, the technical solution can obtain the third operation result by dividing the rounded result of the product of the third preset value and the first verification data by the first precision value, and then obtain the first data based on the third operation result. The first preset value, the third preset value and the first precision value can all be pre-stored in the computing device, so that the first preset value, the third preset value and the first precision value can be quickly obtained when decompressing the third data, thereby speeding up the calculation speed of the third operation result. Therefore, the above decompression process is simple and efficient, so that the compressed data can be quickly decompressed, and the above scheme has a high decompression performance.

[0233] Moreover, the technical solution can accurately decompress the third data into the first data through the first symbol, thereby avoiding symbol errors in the decompressed data.

[0234] The present application also provides a chip system, which includes a processor and a power supply circuit, wherein the power supply circuit is used to supply power to the processor, and the processor is used to execute the aforementioned Figure 2 Specifically, the processor is used to execute the aforementioned steps performed by the data processing system in the data processing method. Figure 2 The steps in the data processing method are performed by the first acquisition unit, the compression unit and the first output unit in the data processing system. Alternatively, the processor is used to perform the aforementioned Figure 3Specifically, the processor is used to execute the aforementioned steps performed by the data processing system in the data processing method. Figure 3 The steps of the data processing method performed by the second acquisition unit, the decompression unit and the second output unit in the data processing system. For the sake of brevity, they are not repeated here. Among them, the processor can be implemented by a GPU, and can also be implemented by a computing device such as a DPU, NPU, XPU, SoC, offload card, acceleration card, etc.

[0235] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computing device provided by this application. Figure 7 As shown, the computing device 400 provided in the present application includes: a bus 401, a processor 402, a memory 403 and a communication interface 404. The processor 402, the memory 403 and the communication interface 404 communicate with each other through the bus 401. The computing device 400 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 400.

[0236] The bus 401 may be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. Among them, the unified bus is also called a Lingqu bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus 401 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 401 may include a path for transmitting information between various components of the computing device 400 (eg, the memory 403, the processor 402, and the communication interface 404).

[0237] The processor 402 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, etc.

[0238] The memory 403 may include a volatile memory, such as a random access memory (RAM). The processor 402 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). In addition, the memory 403 may also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0239] It is worth noting that the same type of storage medium can be configured in the same computing device to implement the function of memory 403, or two or more types of storage media can be configured to implement the function of memory 403, and this application does not limit this.

[0240] The memory 403 stores executable program codes, and the processor 402 executes the executable program codes to respectively implement the aforementioned Figure 1 The functions of the first acquisition unit 210 and the first output unit 230 in the data processing system 200 are respectively executed as described above. Figure 2 The steps in the data processing method are performed by the first acquisition unit, the compression unit, and the first output unit in the data processing system. That is, the memory 403 stores the steps for performing Figure 2 , Figure 3 or Figure 5 Instructions for data processing methods.

[0241] Optionally, the program code in the memory 403 for implementing the functions of the first acquisition unit 210 and the first output unit 230 is the same as the aforementioned Figure 1 The relationship between the first acquisition unit 210, the compression unit 220, and the first output unit 230 in the data processing system 200 can also be dynamically adjusted according to business requirements, so that the program code stored in the memory 403 can also implement the aforementioned Figure 1 The functions of one or more units of the first acquisition unit 210, the compression unit 220, and the first output unit 230 in the data processing system 200. For example, the program code in the memory 403 for implementing the functions of the first acquisition unit 210 can not only implement the aforementioned Figure 1 The function of the first acquisition unit 210 in the data processing system 200 can also realize the aforementioned Figure 1The function of the compression unit 220 in the data processing system 200 is shown in FIG.

[0242] Alternatively, the memory 403 stores executable codes, and the processor 402 executes the executable program codes to respectively implement the aforementioned Figure 1 The functions of the second acquisition unit 240 and the second output unit 260 in the data processing system 200 are respectively performed as described above. Figure 3 The steps in the data processing method are performed by the second acquisition unit, the decompression unit, and the second output unit in the data processing system. That is, the memory 403 stores the steps for performing Figure 6 Instructions for data processing methods.

[0243] Optionally, the program code in the memory 403 for implementing the functions of the second acquisition unit 240 and the second output unit 260 is the same as the aforementioned Figure 1 The relationship between the second acquisition unit 240, the decompression unit 250, and the second output unit 260 in the data processing system 200 can also be dynamically adjusted according to business requirements, so that the program code stored in the memory 403 can also implement the aforementioned Figure 1 The program code in the memory 403 for implementing the function of the second acquisition unit 240 can not only implement the aforementioned Figure 1 The function of the second acquisition unit 240 in the data processing system 200 can also realize the aforementioned Figure 1 The function of the decompression unit 250 in the data processing system 200 is as follows.

[0244] The communication interface 404 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 400 and other computing devices or communication networks.

[0245] As a possible implementation, the computing device 400 may also include a chip system, the chip system includes a processor and a power supply circuit, the power supply circuit is used to supply power to the processor, and the processor is used to perform the above Figure 2 The operation steps performed by the data processing system in the data processing method of Figure 3 The operation steps performed by the data processing system in the data processing method. The processor may be implemented by a GPU, or by a computing device such as a DPU, NPU, XPU, SoC, offload card, accelerator card, or AI chip.

[0246] As a possible implementation, the computing device 400 may include multiple types of processors 402, that is, the computing device 400 is a heterogeneous device. For example, the computing device 400 includes a CPU and a GPU, and at least one of the processors 402 may perform the aforementioned Figure 2 The operation steps performed by the data processing system in the data processing method, or the execution of the aforementioned Figure 3 The operation steps performed by the data processing system in the data processing method are omitted for brevity.

[0247] See also Figure 8 , Figure 8 1 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. The computing device cluster provided in an embodiment of the present application includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0248] like Figure 8 As shown, the computing device cluster includes at least one computing device 400. The memory 403 in one or more computing devices 400 in the computing device cluster may store the same Figure 2 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 3 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 5 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 6 Instructions for the steps of a data processing method executed by a data processing system.

[0249] In some possible implementations, the memory 403 of one or more computing devices 400 in the computing device cluster may also store the memory 403 for executing the aforementioned Figure 2 Partial instructions of the steps executed by the data processing system in the data processing method, or for executing the aforementioned Figure 3 Partial instructions of the steps executed by the data processing system in the data processing method, or for executing the aforementioned Figure 5 Partial instructions of the steps executed by the data processing system in the data processing method, or for executing the aforementioned Figure 6 In other words, the combination of one or more computing devices 400 can jointly execute the instructions for executing the aforementioned Figure 2 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 3 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 5 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 6 Instructions for the steps of a data processing method executed by a data processing system.

[0250] It should be noted that the memory 403 in different computing devices 400 in the computing device cluster may store different instructions, which are respectively used to execute the aforementioned Figure 1 In other words, the instructions stored in the memory 403 in different computing devices 400 can implement the functions of one or more of the first acquisition unit 210, the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250 and the second output unit 260.

[0251] Optionally, the program codes for implementing the functions of the first acquisition unit 210, the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250, and the second output unit 260 in the memory 403 of the different computing devices 400 are the same as those in the above Figure 1 The relationship between the first acquisition unit 210, the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250, and the second output unit 260 in the data processing system 200 can also be dynamically adjusted according to business needs, so that the program code stored in the memory 403 can also implement the aforementioned Figure 1 The functions of one or more units among the first acquisition unit 210, the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250, and the second output unit 260 in the data processing system 200.

[0252] See also Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram of another computing device cluster provided in an embodiment of the present application. In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. Fig. 9 As shown, two computing devices 400A and 400B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device. In this type of possible implementation, the memory 403 in the computing device 400A stores instructions for executing the functions of the first acquisition unit 210. At the same time, the memory 403 in the computing device 400B stores instructions for executing the functions of the first output unit 230.

[0253] Fig. 9The connection mode between the computing device clusters shown may be that considering that the data processing method provided in the present application needs to obtain a large amount of data to be compressed, it is considered to hand over the functions implemented by the first obtaining unit 210 to the computing device 400A for execution.

[0254] It should be understood that Fig. 9 The functions of the computing device 400A shown in FIG. 4 may also be completed by multiple computing devices 400. Similarly, the functions of the computing device 400B may also be completed by multiple computing devices 400.

[0255] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Figure 8 and Fig. 9 The difference is that the memory 403 in one or more computing devices 400 in the computing device cluster may store the same memory 403 for executing the aforementioned Figure 2 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 3 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 5 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 6 Instructions for the steps of a data processing method executed by a data processing system.

[0256] In some possible implementations, the memory 403 of one or more computing devices 400 in the computing device cluster may also store the memory 403 for executing the aforementioned Figure 2 Partial instructions of the steps executed by the data processing system in the data processing method, or for executing the aforementioned Figure 3 Partial instructions of the steps executed by the data processing system in the data processing method, or for executing the aforementioned Figure 5 Partial instructions of the steps executed by the data processing system in the data processing method, or for executing the aforementioned Figure 6 In other words, the combination of one or more computing devices 400 can jointly execute the instructions for executing the aforementioned Figure 2 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 3 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 5 Instructions for the steps executed by a data processing system in a data processing method, or for executing the aforementioned Figure 6 Instructions for the steps of a data processing method executed by a data processing system.

[0257] It should be noted that the memory 403 in different computing devices 400 in the computing device cluster may store different instructions, which are respectively used to execute the aforementioned Figure 1 In other words, the instructions stored in the memory 403 in different computing devices 400 can implement the functions of one or more of the first acquisition unit 210, the compression unit 220, the first output unit 230, the second acquisition unit 240, the decompression unit 250 and the second output unit 260.

[0258] The present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the aforementioned Figure 2 The steps performed by the data processing system in the data processing method of Figure 3 The steps performed by the data processing system in the data processing method of Figure 5 The steps performed by the data processing system in the data processing method of Figure 6 The steps in the data processing method are performed by the data processing system.

[0259] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned Figure 2 The steps performed by the data processing system in the data processing method of Figure 3 The steps performed by the data processing system in the data processing method of Figure 5 The steps performed by the data processing system in the data processing method of Figure 6 The steps in the data processing method are performed by the data processing system.

[0260] It should be understood that in the embodiments of the present application, "when", "when" and "if" all mean that the device will make corresponding processing under certain objective circumstances, and do not limit the time, nor do they require that the device must have a judgment action when it is implemented, nor do they mean that there are other limitations.

[0261] It should be understood that in the embodiments of the present application, "simultaneous" does not necessarily strictly require that it occurs at the same time, minute, second, or even at the same moment. When the moments of occurrence are slightly different, it can also be understood as occurring "simultaneously".

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a first set of data, where the first set of data includes first data and second data; outputting a second set of data, wherein the second set of data includes third data and fourth data; The first data and the second data are both used to represent floating-point data, and the first data and the second data both occupy a first number of bits; The third data is data obtained by compressing the first data, the third data occupies a second number of bits, the sum of the second number and the third number is less than or equal to the first number, the floating-point data represented by the third data and the first prefix belongs to a first numerical interval, the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, and the first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval; The fourth data is data compressed from the second data, the fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical range.

2. The method according to claim 1, characterized in that The first quantity is A, the second quantity is B, the third quantity is H, Before outputting the second set of data, the method further includes: Normalizing the first data based on the first group of data to obtain first target data; Perform an XOR operation on the second target data and the third target data to obtain a first operation result, wherein the first K bits of the first operation result are 0, wherein the second target data and the third target data both belong to the first numerical interval, the second target data is the sum of the first target data and a first preset value, and the third target data is the sum of the second target data and a second preset value, the first preset value and the second preset value are both positive numbers, and K≥H; Setting the K+2th to Ath bits in the second target data to 1 as the first setting result, or setting the K+2th to Ath bits in the third target data to 0 as the first setting result; Determine the number of bits from the H+Sth to the (H+S)+B-1th in the first setting result as the third data, where S is a positive integer, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A.

3. The method according to claim 2, characterized in that S＝1。 4. The method according to claim 2, characterized in that: The first data is data with the smallest absolute value in the first group of data, and the first S bits of the mantissa of the second target data are 1.

5. The method according to any one of claims 2 to 4, characterized in that: The product of the first data and the first precision value is equal to the rounded result of the product of the third preset value and the first verification data, wherein the first precision value is determined based on the data with the largest number of decimal places in the first group of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the first verification data is the difference between the first setting result and the first preset value.

6. The method according to claim 5, characterized in that The first set of data includes fifth data, and the fifth data occupies the first number of bits; If the product of the fifth data and the first precision value is not equal to the rounded product of the third preset value and the second verification data, the second set of data includes the fifth data, wherein the second verification data is obtained based on the fifth data; If the fifth data is 0, the second group of data includes sixth data, the sixth data occupies a fifth number of bits, and the fifth number is smaller than the first number.

7. The method according to any one of claims 1 to 6, characterized in that: The second group of data includes a first symbol, where the first symbol is used to indicate whether the first data is a positive number or a negative number, and the first symbol and the third data are used together to decompress the first data.

8. A data processing method, characterized in that: The method comprises: Acquire a second set of data, where the second set of data includes third data and fourth data; Output a first set of data, the first set of data includes first data and second data; wherein, The third data occupies a second number of bits, the sum of the second number and the third number is less than or equal to the first number, the floating-point data represented by the third data and the first prefix belongs to a first numerical interval, the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, and the first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval; The fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical range; The first data is data obtained by decompressing the third data, the second data is data obtained by compressing the fourth data, the first data and the second data are both used to represent floating-point data, and the first data and the second data both occupy the first number of bits.

9. The method according to claim 8, characterized in that Before outputting the first set of data, the method further includes: Acquire first verification data, wherein the first verification data is a difference between a first setting result and a first preset value, the first setting result includes the first prefix and the third data, and the first preset value is a positive number; Divide the rounded result of the product of a third preset value and the first verification data by a first precision value to obtain a third operation result, wherein the first precision value is determined based on the data with the largest number of decimal places in the first group of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the third preset value is a positive number; The first data is determined based on the third operation result.

10. The method according to claim 9, characterized in that The second set of data includes a first symbol; The determining the first data based on the third operation result includes: When the first symbol is used to indicate that the first data is a positive number, taking the third operation result as the first data; In the case where the first symbol is used to indicate that the first data is a negative number, the first bit in the third operation result is set to 1 and used as the first data.

11. A data processing system, characterized in that: include: a first acquisition unit and a first output unit, The first acquisition unit is used to acquire a first set of data, where the first set of data includes first data and second data; The first output unit is used to output a second set of data, where the second set of data includes third data and fourth data; The first data and the second data are both used to represent floating-point data, and the first data and the second data both occupy a first number of bits; The third data is data obtained by compressing the first data, the third data occupies a second number of bits, the sum of the second number and the third number is less than or equal to the first number, the floating-point data represented by the third data and the first prefix belongs to a first numerical interval, the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, and the first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval; The fourth data is data compressed from the second data, the fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, and the floating-point data represented by the fourth data and the first prefix belongs to the first numerical range.

12. The system according to claim 11, characterized in that The first number is A, the second number is B, and the third number is H. The data processing system further includes a compression unit. The compression unit is configured to normalize the first data based on the first group of data to obtain first target data before the first output unit outputs the second group of data; Perform an XOR operation on the second target data and the third target data to obtain a first operation result in which the first K bits are 0; set the K+2th to Ath bits in the second target data to 1 as the first setting result, or set the K+2th to Ath bits in the third target data to 0 as the first setting result; determine the number of bits from the H+Sth to the (H+S)+B-1th bits in the first setting result as the third data; wherein the second target data and the third target data both belong to the first numerical range, the second target data is the sum of the first target data and a first preset value, the third target data is the sum of the second target data and a second preset value, the first preset value and the second preset value are both positive numbers, S is a positive integer, K≥H, KH<B<AH, (H+S)+B-1<A or (H+S)+B-1=A.

13. The system according to claim 12, characterized in that S＝1。 14. The system according to claim 12, characterized in that The first data is data with the smallest absolute value in the first group of data, and the first S bits of the mantissa of the second target data are 1.

15. The system according to any one of claims 12 to 14, characterized in that: The product of the first data and the first precision value is equal to the rounded result of the product of the third preset value and the first verification data, wherein the first precision value is determined based on the data with the largest number of decimal places in the first group of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the first verification data is the difference between the first setting result and the first preset value.

16. The system according to claim 15, characterized in that The first set of data includes fifth data, and the fifth data occupies the first number of bits; If the product of the fifth data and the first precision value is not equal to the rounded product of the third preset value and the second verification data, the second set of data includes the fifth data, wherein the second verification data is obtained based on the fifth data; If the fifth data is 0, the second group of data includes sixth data, the sixth data occupies a fifth number of bits, and the fifth number is smaller than the first number.

17. The system according to any one of claims 11 to 16, characterized in that: The second group of data includes a first symbol, where the first symbol is used to indicate whether the first data is a positive number or a negative number, and the first symbol and the third data are used together to decompress the first data.

18. A data processing system, characterized in that: include: a second acquisition unit and a second output unit, The second acquisition unit is used to acquire a second set of data, where the second set of data includes third data and fourth data; The second output unit is used to output a first set of data, where the first set of data includes first data and second data; Among them, the third data occupies a second number of bits, the sum of the second number and the third number is less than or equal to the first number, the floating-point data represented by the third data and the first prefix belongs to the first numerical interval, the third number is the sum of the number of bits occupied by the sign bit and the exponent bit of the floating-point data storing the first numerical interval, the first prefix includes the sign bit and the exponent bit of the floating-point data of the first numerical interval, the fourth data occupies a fourth number of bits, the sum of the fourth number and the third number is less than or equal to the first number, the floating-point data represented by the fourth data and the first prefix belongs to the first numerical interval, the first data is data decompressed by the third data, the second data is data compressed by the fourth data, the first data and the second data are both used to represent floating-point data, and the first data and the second data both occupy the first number of bits.

19. The system according to claim 18, characterized in that The data processing system further comprises a decompression unit, The decompression unit is used to obtain first verification data before the second output unit outputs the first group of data; divide the rounded result of the product of a third preset value and the first verification data by a first precision value to obtain a third operation result; determine the first data based on the third operation result; wherein the first verification data is the difference between a first setting result and a first preset value, the first setting result includes the first prefix and the third data, the first preset value is a positive number, the first precision value is determined based on the data with the largest number of decimal places in the first group of data, the third preset value is determined based on the first precision value, the third preset value is different from the first precision value, and the third preset value is a positive number.

20. The system according to claim 19, characterized in that The second set of data includes a first symbol; The decompression unit is specifically used to use the third operation result as the first data when the first symbol is used to indicate that the first data is a positive number; and to set the first bit in the third operation result to 1 and use it as the first data when the first symbol is used to indicate that the first data is a negative number.

21. A chip system, characterized in that: The chip system includes a processor and a power supply circuit, wherein the power supply circuit is used to supply power to the processor, and the processor is used to execute the operating steps of the method described in any one of claims 1 to 7, or execute the operating steps of the method described in any one of claims 8 to 10.

22. A computing device cluster, characterized in that: It includes at least one computing device, each computing device includes a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method as claimed in any one of claims 1 to 7, or executes the method as claimed in any one of claims 8 to 10.

23. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device executes the method according to any one of claims 1 to 7, or executes the method according to any one of claims 8 to 10.

24. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method according to any one of claims 1 to 7, or executes the method according to any one of claims 8 to 10.

Images