Historical data compression and decompression methods, programs, and devices

Abstract

The present invention provides a method, program, and apparatus for efficiently compressing historical data, which is data recording the past price movements of financial instruments, as well as a method, program, and apparatus for decompressing compressed historical data. [Solution] Various differences are calculated from the preceding and succeeding records of the historical data (however, to ensure that data close to 0 appears as frequently as possible, the difference between the opening price and the closing price of the previous data is calculated for the opening price, and the difference between the high price, low price, and closing price and the opening price of the current data is calculated for the high price and the closing price). The numerical values ​​related to price are converted to integers, and for integers that may be negative, both positive and negative numbers close to 0 are mapped to non-negative integers close to 0. By representing the non-negative integers with the fewest possible octets, the historical data can be represented with less data than before.

Description

Technical Field

[0001] The present invention relates to a method for compressing and decompressing time-series data, and more particularly, to a method for compressing and decompressing time-series data (referred to as "historical data") that records past price movements of financial products and the like.

Background Art

[0002] In applications such as considering trading strategies for financial products, historical data, which is a record of past price movements of financial products, is widely used.

[0003] Historical data is generally stored by financial product handling companies and the like, and is provided to those who wish to use it through communication or the like.

[0004] Historical data describes a timestamp, price, and other attached information in a CSV (Comma Separated Values) format or a format equivalent thereto, and is provided after being compressed by a general-purpose data compression technology.

Prior Art Documents

Non-Patent Documents

[0006] Historical data can be recorded in two formats: timeframe format (where the opening price, highest price, lowest price, and closing price for each period of time are recorded along with a timestamp) or tick format (where the timestamp and price are recorded each time there is a price change).

[0007] As of 2024, several services exist that provide historical data spanning several years to several decades. These services typically provide historical data in a general-purpose format such as CSV, compressed using general-purpose data compression technologies.

[0008] However, historical data compressed using general-purpose data compression technologies still faces the challenge of being large in volume, even after compression.

[0009] Due to the aforementioned issues, more storage space is required to store historical data, and for those providing historical data services, this leads to an unnecessary increase in server and communication load.

[0010] Furthermore, historical data compressed using general-purpose data compression technology needs to be decompressed before use, and since the decompressed historical data is extremely large in size, there is a problem in that it is difficult to keep large amounts of historical data readily available at all times.

[0011] This invention has been made in view of the above problems, and aims to reduce the storage capacity required for storing historical data and to reduce various burdens when transmitting historical data over communications by performing data compression based on the characteristics of historical data.

[0012] Furthermore, the present invention aims to enable application software and the like to directly read compressed historical data in a practical amount of time by designing it to reduce the amount of computation required for decompression. [Means for solving the problem]

[0013] The historical data compression method according to the present invention comprises at least a difference calculation step and a compaction step.

[0014] Historical data generally consists of multiple records sorted in ascending order of timestamps.

[0015] Each record in historical data has a timestamp, a price, and may also have volume and other data. In tick-based historical data, the price of each record consists of a single price or a bid price and an ask price. However, the difference between the bid and ask prices may be used instead of the bid or ask price. In hourly-based historical data, the price of each record consists of the open price, high price, low price, and close price.

[0016] In the difference calculation step, for a record in the historical data at a given time (referred to as the "current record"), the difference is calculated by referring to the record immediately preceding the current record (referred to as the "previous record"), and, if necessary, the current record itself. However, in calculating the difference, the difference is calculated for the timestamp, price, and volume separately. If there is no previous record corresponding to the first record, in this case, it is sufficient to stipulate that the timestamp, price, and volume of the previous record are all treated as specific constants.

[0017] In the case of historical data using a time frame method, the calculation of price differences in the difference calculation step refers to the previous record and the current record to calculate the difference between the opening price of the current record and the closing price of the previous record, the difference between the high price of the current record and the opening price of the current record, the difference between the opening price of the current record and the low price of the current record, and the difference between the closing price of the current record and the opening price of the current record.

[0018] In the case of tick-based historical data, the price difference calculation in the difference calculation step involves calculating the difference between the selling prices or buying prices of the current record and the previous record, and further calculating the difference between the selling price of the current record and the buying price of the current record.

[0019] In the case of tick-based historical data, the calculation of the difference in volume in the difference calculation step involves calculating the difference between the current record and the previous record in terms of buy quantities or sell quantities, and further calculating the difference between the sell quantity of the current record and the buy quantity of the current record.

[0020] Because the price fluctuations of financial instruments are highly continuous, the difference calculated in the difference calculation step is likely to be close to zero.

[0021] In the compactification step, the difference calculated in the difference calculation step is converted into a representation (called a "compact representation") that has the property of being representable with fewer bits if the absolute value is small.

[0022] A record in which each part is described in a compact representation is called a compacted record.

[0023] By representing historical data using the compacted record, compacted historical data is created.

[0024] In the compacted historical data, the compacted records are sorted in the same order as they were stored in the historical data.

[0025] Preferably, the compacted historical data can be further reversibly compressed by a general-purpose data compression method to obtain compressed compacted historical data. General-purpose data compression methods such as entropy encoding (Huffman encoding, range encoding, etc.) and dictionary encoding methods (LZ77, LZW, LZMA, Brotli, etc.) can be used. These are appropriately selected in consideration of the computational amount required for desired compression and decompression, the main memory amount, and the achieved compression ratio, etc.

Advantages of the Invention

[0026] According to the present invention, compared with the conventional format, the data amount of historical data can be significantly reduced, thereby reducing the costs associated with storage and communication.

[0027] Furthermore, since the compacted historical data can be read out faster than the conventional format, by replacing the historical data stored in the conventional format with the compacted historical data, it also contributes to the speeding up of application software that utilizes historical data.

Brief Description of the Drawings

[0028] [Figure 1] It is a diagram showing the input / output relationship and processing of historical data compression method according to an embodiment of the present invention. [Figure 2] It is a functional block diagram of a historical data compression apparatus according to an embodiment of the present invention. [Figure 3] This diagram shows the data input / output relationship and processing during the compaction step. [Figure 4] This diagram illustrates historical data (using a time frame format). [Figure 5] This diagram illustrates differential historical data obtained from historical data (timeframe format). [Figure 6] This figure illustrates integer difference historical data obtained from historical data (timeframe format). [Figure 7] This figure illustrates non-negative integer difference historical data obtained from historical data (timeframe format). [Figure 8] This is a flowchart for the process of converting an unsigned integer to a compact representation. [Figure 9] This diagram illustrates how records of non-negative integer difference historical data obtained from historical data (timeframe format) are converted into compacted records. [Figure 10] This diagram illustrates the completed form of compacted historical data obtained from historical data (timeframe format). [Figure 11] This figure shows the data input / output relationship and processing of a historical data expansion method according to an embodiment of the present invention. [Figure 12] This diagram illustrates historical data (tick-based). [Figure 13] This diagram illustrates differential historical data obtained from historical data (tick method). [Figure 14] This figure illustrates integer-scale difference historical data obtained from historical data (tick method). [Figure 15] This figure illustrates non-negative integer difference historical data obtained from historical data (tick method). [Figure 16]This diagram illustrates how records of non-negative integer difference historical data obtained from historical data (tick format) are converted into compacted records. [Figure 17] This diagram illustrates the completed form of compacted historical data obtained from historical data (tick method). [Modes for carrying out the invention]

[0029] Embodiments of the present invention, including a historical data compression method, program, and apparatus, as well as a historical data decompression method, program, and apparatus, will be described with reference to the figures. [Examples]

[0030] Figure 1 is a diagram showing the data input / output relationship and processing in a historical data compression method according to an embodiment of the present invention.

[0031] As shown in Figure 1, the historical data compression method 10 comprises a difference calculation step 11 and a compactification step 12, and preferably further comprises a general-purpose compression step 13.

[0032] Figure 2 is a functional block diagram of a historical data compression device according to an embodiment of the present invention.

[0033] As shown in Figure 2, the historical data compression device 15 is implemented by an information processing device such as a personal computer, and comprises an arithmetic unit 17, a main memory unit 18, and an auxiliary storage unit 19. The arithmetic unit 17 is implemented using, for example, a semiconductor integrated circuit called a CPU (Central Processing Unit). The main memory unit 18 is implemented using, for example, a semiconductor integrated circuit called a DRAM (Dynamic Random Access Memory). The auxiliary storage unit 19 is composed of devices such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), which are slower than DRAM but have a larger storage capacity and a lower price per unit of storage capacity.

[0034] The historical data compression method 10 is implemented, for example, as a program (referred to as the "historical data compression program") that runs on the historical data compression device 15.

[0035] The historical data compression program 16 is, for example, recorded in the auxiliary storage unit 19, loaded into the main storage unit 18 as needed, and executed by the arithmetic unit 17.

[0036] The cases where historical data is presented using a time frame method and where it is presented using a tick method will be explained separately in the following Examples 1 and 2. [Examples]

[0037] [Process for converting historical data (timeframe format) into compact historical data, and for expanding the data.] Figure 4 illustrates historical data (timeframe format). As shown in Figure 4, the historical data 50 consists of multiple records (59a, 59b, 59c, etc.). Each record has a timestamp 51, opening price 52, high price 53, low price 54, and closing price 55, and may also have volume 56, as in this example.

[0038] Here, the prices (opening price 52, high price 53, low price 54, closing price 55) may include values ​​less than an integer. However, given a specific period for a particular financial instrument, the minimum unit of price is predetermined. The minimum unit of price is often provided in advance as supplementary information to the historical data. Even if the minimum unit of price is not provided in advance as supplementary information to the historical data, it can be easily determined by scanning all records in the historical data and detecting the least significant digit of all price values. In the example in Figure 4, the minimum unit of price in this historical data is 0.001.

[0039] Therefore, note that in this example, the price can be converted to an integer by multiplying it by 1000. This fact will be used in the compactification step 12.

[0040] The process of difference calculation step 11 will be explained with reference to Figures 1, 4, and 5.

[0041] Refer to Figures 1 and 4. The difference calculation step 11 receives historical data 50 as input.

[0042] Refer to Figures 1 and 4. The difference calculation step 11 calculates the difference for each record in the historical data 50 based on the previous record. For example, if 59b is the current record, then the previous record is 59a. The difference to derive the current record 59b from the previous record 59a is calculated as follows. (Difference between the timestamps of the current record and the previous record) = (January 8, 2020, 8:02) - (January 8, 2020, 8:01) = 60 seconds (Difference between the opening price of the current record and the closing price of the previous record) = 108.447 - 108.447 = 0 (Difference between the current record's high price and its opening price) = 108.451 - 108.447 = 0.004 (Difference between the opening price and the low price of the current record) = 108.447 - 108.442 = 0.005 (Difference between the closing price and opening price of the current record) = 108.444 - 108.447 = -0.003 (Difference in volume between current record and previous record) = 1245678 - 1234567 = 11111 Perform this calculation for all records. However, for the first record, 59a, the previous record is assumed to be, for example, January 1, 1970, 00:00:00, with a closing price of 0 and a trading volume of 0.

[0043] Refer to Figures 4 and 5. As a result of this difference calculation, historical data 50 is converted into difference historical data 60. Furthermore, each part that constitutes a record in difference historical data 60 is then divided into: The difference obtained by subtracting the timestamp of the previous record from the timestamp of the current record is calculated as difference timestamp 61. The difference obtained by subtracting the closing price of the previous record from the opening price of the current record is calculated as the difference opening price, 62. The difference obtained by subtracting the opening price of the current record from the high price of the current record is calculated as the difference high price of 63. The difference obtained by subtracting the current record's low price from the current record's opening price is calculated as the difference low price of 64. The difference obtained by subtracting the opening price of the current record from the closing price of the current record is calculated as the difference closing price of 65. The difference obtained by subtracting the sales volume of the previous record from the sales volume of the current record is calculated as the difference in sales volume of 66. It is called [this]. However, the symbols are added for the purpose of matching with the diagram and are not included in the name.

[0044] The process of compaction step 12 will be explained with reference to Figures 1, 3, and 5 through 10.

[0045] Refer to Figure 3. The compactification step 12 takes the differential historical data 60 as input and outputs the compactified historical data 90.

[0046] The compactification step 12 includes an integerization step 121, a non-negative integerization step 122, and an encoding step 123.

[0047] The process of integerization step 121 will be explained with reference to Figures 3 and 5 to 6.

[0048] The integer conversion step 121 takes the difference historical data 60 as input and converts the data contained in the difference historical data 60 into integers. In this example, the conversion is performed as follows. The differential timestamp 61 is converted to an integer value in seconds, and preferably further converted according to Table 1. The difference opening price of 62, the difference high price of 63, the difference low price of 64, and the difference closing price of 65 are converted to integers by multiplying them by the smallest possible positive number that can be made into an integer when multiplied by the smallest unit of price. In this example, the smallest unit of price is 0.001, so the difference opening price of 62, the difference high price of 63, the difference low price of 64, and the difference closing price of 65 are converted to integer values ​​by multiplying them by 1000. The difference of 66 will not be converted. Perform this conversion on all records.

[0049] [Table 1]

[0050] The purpose of further converting the integer values ​​in seconds as shown in Table 1 is to replace specific numerical values ​​that frequently appear as differential time in the historical data with numerical values ​​with fewer bits. The conversions shown in Table 1 are examples, and the converted values ​​corresponding to integer values ​​in seconds of 60 or more can be modified as appropriate, within the range that achieves the objective of replacing them with numerical values ​​with fewer bits.

[0051] As a result of this conversion, the differential historical data 60 is converted into integer differential historical data 70. Furthermore, each part that makes up the record of the integer differential historical data 70 is called the integer differential timestamp 71, integer differential open price 72, integer differential high price 73, integer differential low price 74, integer differential close price 75, and integer differential volume 76, respectively (however, the signs are added for the purpose of matching with the figure and are not included in the names).

[0052] The process of the non-negative integer conversion step 122 will be explained with reference to Figures 3, 6, and 7.

[0053] The non-negative integer conversion step 122 receives the integer difference historical data 70 as input and converts the integer difference timestamp 71, integer difference opening price 72, integer difference closing price 75, and integer difference volume 76 from the integer difference historical data 70 into non-negative integers according to the following rules. If the original number (referred to as the "original number" in this paragraph) is non-negative, the original number is converted to twice its original number; if the original number is negative, the original number is converted to -2 times the original number minus 1. This rule allows us to establish a one-to-one correspondence between the original number and non-negative integers, and moreover, to establish a correspondence between integers close to zero and non-negative integers close to zero.

[0054] Furthermore, when focusing on a specific record, the high value is always greater than or equal to the opening price, and the low value is always less than or equal to the opening price. Therefore, by definition, the difference high value 63 and the difference low value 64 are always non-negative. Consequently, the integer difference high value 73 and the integer difference low value 74 cannot be negative. Therefore, in the non-negative integerization step 122, these values ​​are not converted and are used as they are, as the non-negative integer difference high value 83 and the non-negative integer difference low value 84, respectively. By ensuring that the records are sorted in ascending order of timestamps, the processing described in the previous paragraph may be omitted for integer differential timestamps 71 as well, and the value of integer differential timestamp 71 may be used directly as the non-negative integer differential timestamp 81.

[0055] The following is an example of the processing when the non-negative integerization step 122 is performed on record 79b of the integerized difference historical data 70. Since the integer difference timestamp 71 of record 79b is 0, the value of the non-negative integer difference timestamp 81 is 0 × 2 = 0. Since the integer difference starting value 72 for record 79b is 0, the value of the non-negative integer difference starting value 82 is 0 × 2 = 0. The integer difference high value 73 and the integer difference low value 74 of record 79b are 4 and 5, respectively. Since these values ​​are not changed in the non-negative integerization step 122, the non-negative integer difference high value 83 and the non-negative integer difference low value 84 become 4 and 5, respectively. Since the integer difference closing price 75 for record 79b is -3, the non-negative integer difference closing price 85 is -3 × (-2) - 1 = 5. Since the integer difference volume 76 of record 79b is 11111, the value of the non-negative integer difference closing price 85 is 11111 × 2 = 22222. This converts record 79b to record 89b.

[0056] Perform this conversion on all records.

[0057] As a result of this conversion, the integer difference historical data 70 is converted into non-negative integer difference historical data 80. Each part that makes up the record of the non-negative integer difference historical data 80 is called the non-negative integer difference timestamp 81, the non-negative integer difference opening price 82, the non-negative integer difference high price 83, the non-negative integer difference low price 84, the non-negative integer difference closing price 85, and the non-negative integer difference volume 86, respectively (however, the signs are added for the purpose of matching with the figure and are not included in the names).

[0058] The processing of the encoding step 123 will be explained with reference to Figures 3 and 7 through 10.

[0059] The encoding step 123 converts the non-negative integerized difference historical data 80 into compactified historical data 90.

[0060] The encoding step 123, for each record of the non-negative integer difference historical data 80, sequentially converts the values ​​of the non-negative integer difference timestamp 81, the non-negative integer difference opening price 82, the non-negative integer difference high price 83, the non-negative integer difference low price 84, the non-negative integer difference closing price 85, and the non-negative integer difference volume 86 into compact representations according to the procedure 125 shown in Figure 8, and arranges them in order.

[0061] Referring to Figure 8, the procedure for converting the non-negative integer 22222 (binary representation: 101011011001110, hexadecimal representation: 0x56CE) to a compact representation according to procedure 125 is shown.

[0062] In step 125a, set v to 22222 and r to an empty octet array.

[0063] In step 125b, the value of v is not less than or equal to 0x7F (decimal representation: 127), so we proceed to step 125c, which is on the FALSE side.

[0064] In step 125c, the bitwise AND of v and 0x7F (i.e., the lower 7 bits of v) is taken, and the bitwise OR of this with 0x80 is taken and the result is added to the end of r. The lower 7 bits of v are 1001110 in binary representation, and the bitwise OR of this with 0x80, i.e., 0xCE (binary representation: 11001110), is added to the end of r. r becomes an array {0xCE} with 1 element. Next, v is right-shifted by 7 bits and changes to 10101101 in binary representation. Return to step 125b.

[0065] In step 125b, the value of v is not less than or equal to 0x7F, so we proceed to step 125c, which is on the FALSE side.

[0066] In step 125c, the bitwise AND of v and 0x7F (i.e., the lower 7 bits of v) is taken, and the bitwise OR of this with 0x80 is taken and the result is added to the end of r. The lower 7 bits of v have changed to 0101101 in binary representation, and the bitwise OR of this with 0x80, i.e., 0xAD (binary representation 10101101), is added to the end of r. r becomes an array of 2 elements {0xCE, 0xAD}. Next, v is right-shifted by 7 bits and changed to 1 in binary representation. Return to step 125b.

[0067] In step 125b, the value of v is now less than or equal to 0x7F, so we proceed to step 125d, which is on the TRUE side.

[0068] In step 125d, the value of v, 1 (hexadecimal representation: 0x01), is added to the end of r. r becomes an array of 3 elements, {0xCE,0xAD,0x01}, and we obtain the compact representation of 22222, {0xCE,0xAD,0x01}.

[0069] According to procedure 125, record 89b of the non-negative integer difference historical data 80 is transformed as follows: Referring to Figure 9, the non-negative integer difference timestamp 81 and the non-negative integer difference start value 82 of 89b are both 0, so they are both converted to an octet sequence of length {0x00}. The non-negative integer difference high value 83 of 89b is 4, so this is converted to an octet sequence of length {0x04}. The non-negative integer difference low price 84 and the non-negative integer difference closing price 85 of 89b are both 5, so they are both converted to an octet sequence of length {0x05}. The non-negative integer output of 89b, 86, which is 22222, is converted to {0xCE, 0xAD, 0x01} as illustrated in the previous paragraph. By arranging these octet sequences in order, record 89b is transformed into the compacted record 99b, {0x00,0x00,0x04,0x05,0x05,0xCE,0xAD,0x01}.

[0070] Refer to Figure 10. The encoding step 123 outputs compacted historical data 90, which consists of a value 98 indicating the smallest unit of price in the historical data, followed by the compact representations (99a, 99b, 99c, etc.) corresponding to each record, in order. In this example, the value 98 indicating the smallest unit of price in the historical data is the maximum number of decimal places in the price. In this example, the smallest unit of price in the historical data 50 is 0.001, so the value 98 indicating the smallest unit of price in the historical data is 3, which means the third decimal place.

[0071] Referring to Figure 1, the general compression step 13 will be described.

[0072] The historical data compression method 10 preferably further comprises a general-purpose compression step 13. The general-purpose compression step 13 receives the compactified historical data 90 as input, performs compression processing using a general-purpose data compression scheme, and outputs the compressed compactified historical data 95.

[0073] General-purpose data compression methods include, for example, GZIP, LZMA2, and Brotli, and are selected appropriately based on factors such as computational cost and compression ratio.

[0074] Although the compacted historical data 90 is created by removing redundancy from the historical data 50, it still contains redundancy, such as a large number of integer values ​​close to zero. Therefore, the amount of data can be further reduced using a general-purpose data compression method.

[0075] Furthermore, since the compressed historical data 90 has had redundancy based on the characteristics of the historical data removed from the historical data 50 in advance, the compressed compressed historical data 95 will have a smaller data size compared to when the historical data 50 is compressed using a general-purpose data compression method.

[0076] Furthermore, the difference calculation step 11 and the compactification step 12 in the present invention consist of simple arithmetic operations, bitwise operations, and substitutions, making them significantly lighter than general-purpose data compression methods that require complex calculations such as pattern matching and range coding. Moreover, since the compactified historical data 90 is approximately one-third or less of the historical data in size, even when used in conjunction with a general-purpose data compression method, the amount of data processed by the computationally intensive general-purpose data compression method can be reduced to about one-third or less compared to simply applying the general-purpose data compression method. Therefore, the present invention is also advantageous in terms of computational complexity.

[0077] Through the above series of processes, the historical data 50 is converted into a shorter representation, either compacted historical data 90 or compressed compacted historical data 95.

[0078] In this example, for ease of understanding, the differential historical data 60, integerized differential historical data 70, non-negative integerized differential historical data 80, and compactified historical data 90 are completed before proceeding to the next step. However, in actual implementation, the order of processing can be changed to the extent that it does not affect the result, such as repeating all steps for each record in order from the first record. A historical data compression method and program implemented in this way are also included within the technical scope of the present invention.

[0079] Referring to Figure 11, a historical data decompression method 20 for recovering historical data 50 from compressed compactified historical data 95 or compactified historical data 90 will be described.

[0080] As shown in Figure 11, the historical data expansion method 20 comprises a compactification and restoration step 22 and a differential restoration step 21, and preferably further comprises a general-purpose expansion step 23 prior to the compactification and restoration step 22.

[0081] The historical data decompression method 20 is implemented, for example, as a program (referred to as the "historical data decompression program") that runs on an information processing device such as a personal computer. The historical data decompression program may run on the same information processing device as the historical data compression device 15, or it may run on a different information processing device.

[0082] Referring to Figure 11, the process of general deployment step 23 will be explained.

[0083] If the historical data decompression method 20 receives compressed compacted historical data 95 as input, the general decompression step 23 receives the compressed compacted historical data 95 as input and applies a decompression process corresponding to the general data compression method applied in the general compression step 13 when creating the compressed compacted historical data 95, thereby obtaining compacted historical data 90. If the historical data decompression method 20 receives compacted historical data 90 as input, the processing in this paragraph is unnecessary and is not executed.

[0084] The process of compactification and restoration step 22 will be described with reference to Figures 11, 10, 9, 7, and 3.

[0085] The compactification and reconstruction step 22 receives the compactified historical data 90 as input. From the compactified historical data 90, the compactification and reconstruction step 22 reads the value 98 that represents the smallest unit of price in the historical data and determines the value to multiply by when converting the integerized price to the true price. In this example, the value 98 that represents the smallest unit of price in the historical data is written as 3, so the value to multiply by when converting the integerized price to the true price is 10 to the power of (-3), or 0.001.

[0086] The compactification and restoration step 22 reads records from the starting position of the first record contained in the compactified historical data 90. An empty queue (first-in, first-out list) q of unsigned integer values ​​is prepared.

[0087] The initial value of the reference bit n is set to 0, and the initial value of the read value v is also set to 0.

[0088] Read one octet of data. Perform a bitwise logical AND operation on the read octet and 0x7F, then left-shift it by n bits, and perform a bitwise logical OR operation with v to obtain the new value of v. If the most significant bit of the read octet is 0, proceed to the operation in the next paragraph. If the most significant bit of the read octet is 1, add 7 to the reference bit n and repeat the process in this paragraph.

[0089] Add v to queue q and return to the processing in the previous paragraph unless the end of the compactified historical data 90 has not been reached. If the end of the compactified historical data 90 has been reached, proceed to the processing in the next paragraph.

[0090] By taking six elements from the queue q obtained in this way and assigning them sequentially to the non-negative integer difference timestamp 81, non-negative integer difference open price 82, non-negative integer difference high price 83, non-negative integer difference low price 84, non-negative integer difference close price 85, and non-negative integer difference volume 86, one record of the non-negative integer difference historical data 80 can be reconstructed. By continuing this until the queue q is empty, the entire non-negative integer difference historical data 80 can be reconstructed.

[0091] Refer to Figures 11, 7, and 6. For all records in the non-negative integerized difference historical data 80, the values ​​of the non-negative integerized difference timestamp 81, non-negative integerized difference start value 82, non-negative integerized difference end value 85, and non-negative integerized difference volume 86 are examined. If the least significant bit of the value is 0, the value is divided by 2. If the least significant bit of the value is 1, 1 is added to the value and then divided by -2. By replacing the value with the value obtained in this way, the entire integerized difference historical data 70 can be restored from the non-negative integerized difference historical data 80. However, if the integerized difference timestamp 71 is not converted in the non-negative integerization step 122, the value of the non-negative integerized difference timestamp 81 is not converted, regardless of the above.

[0092] Refer to Figures 11, 6, and 5. For all records in the integer-difference historical data 70, the integer-difference opening price 72, integer-difference high price 73, integer-difference low price 74, and integer-difference closing price 75 are each multiplied by the value that should be multiplied when converting the integer-difference prices to true prices. Furthermore, the value of the integer-difference timestamp 71 is treated as the converted value in Table 1 and converted to an integer value in seconds according to Table 1. This makes it possible to reconstruct the entire difference historical data 60 from the integer-difference historical data 70.

[0093] The process of the differential restoration step 21 will be explained with reference to Figures 11, 5, and 4.

[0094] The differential restoration step 21 applies the following operations to all records in the differential historical data 60, starting from the first record. However, in this paragraph, the previous record that has been restored to the historical data 50 is denoted as R. Add the current record's differential timestamp 61 to R's timestamp 51 to obtain the current record's timestamp 51. Add the closing price of R (55) to the difference opening price of the current record (62) to obtain the opening price of the current record (52). The current record's high price of 53 is obtained by adding the current record's difference high price of 63 to the current record's opening price of 52. Subtracting the current record's low price of 64 from the current record's opening price of 53 gives the current record's low price of 54. The closing price of the current record, 55, is obtained by adding the closing price of the current record (65) to the opening price of the current record (54). Add the difference in output volume of the current record (66) to the output volume of R (56) to obtain the output volume of the current record (56).

[0095] Through the above operations, the differential historical data 60 is restored to the historical data 50.

[0096] In this example, for the sake of ease of understanding, the compactified historical data 90, the non-negative integerized difference historical data 80, the integerized difference historical data 70, and the difference historical data 60 are each completed before proceeding to the next step. However, in actual implementation, the order of processing can be changed as long as it does not affect the result, such as repeating all steps for each record in order from the first record. Such a historical data expansion method is also included within the technical scope of the present invention. [Examples]

[0097] [Process for converting historical data (tick format) into compact historical data, and the process of expanding the data] Figure 12 illustrates historical data (tick format). As shown in Figure 12, the historical data 50t consists of multiple records (59ta, 59tb, 59tc, etc.). Each record has a timestamp 51t, selling price 52t, buying price 53t, selling quantity 54t, buying quantity 55t, and may also have other supplementary information. Each record may also have a single transaction volume instead of the selling quantity 54t and buying quantity 55t in this example. In this case, by replacing the selling quantity 54t below with a single transaction volume and omitting all processing related to the buying quantity 55t, compacted historical data can be generated in the same way as in this example.

[0098] Here, the prices of financial instruments (selling price 52t, buying price 53t) may include values ​​less than integers. However, given a specific period for a particular financial instrument, the minimum unit of price is predetermined. The minimum unit of price is often provided in advance as supplementary information to the historical data. Even if the minimum unit of price is not provided in advance as supplementary information to the historical data, it can be easily determined by scanning all records in the historical data and detecting the least significant digit of all price values. In this example, the minimum unit of price in this historical data is 0.001.

[0099] Therefore, note that in this example, the price can be converted to an integer by multiplying it by 1000. This fact will be used in the compactification step 12.

[0100] The sales quantity of 54t and the purchase quantity of 55t would have many digits if expressed as actual numbers, so they are sometimes given as numbers obtained by dividing the actual numbers by a number such as 1 million. In this case, the sales quantity of 54t and the purchase quantity of 55t may include numbers that are less than integers. In this example, the minimum trading unit is 10,000, and the sales quantity of 54t and the purchase quantity of 55t are given as numbers obtained by dividing the actual numbers by 1 million, and therefore, the numbers may have up to two decimal places.

[0101] Therefore, note that in this example, the values ​​for the sales quantity of 54t and the purchase quantity of 55t can be converted to integers by multiplying them by 100. This fact will be used in the compactification step 12.

[0102] The process of the difference calculation step 11 will be explained with reference to Figures 1, 12, and 13.

[0103] Step 11 of the difference calculation receives historical data 50t as input.

[0104] The difference calculation step 11 calculates the difference for each record in the historical data 50t based on the previous record. For example, if 59tb is the current record, then the previous record is 59ta. The difference to derive 59tb from 59ta is calculated as follows. (Difference between the timestamps of the current record and the previous record) = (January 8, 2020 8:01.391) - (January 8, 2020 8:01.024) = 367 milliseconds (Difference between the selling price of the current record and the selling price of the previous record) = 149.931 - 149.941 = -0.010 (Difference between the current record's purchase price and current record's selling price) = 150.015 - 149.931 = 0.084 (Difference between the sales quantity of the current record and the sales quantity of the previous record) = 0.18 - 1.98 = -1.80 (Difference between the number of current records purchased and the number of current records sold) = 0.90 - 0.18 = 0.72 Perform this calculation for all records. However, for the first record, 59ta, the previous record is assumed to be, for example, January 1, 1970, 00:00:00, with a selling price of 0 and a selling quantity of 0.

[0105] As a result of this difference calculation, the historical data 50t is converted into differential historical data 60t. Furthermore, each part that makes up the records of the differential historical data 60t is then divided into its respective parts. The difference obtained by subtracting the timestamp of the previous record from the timestamp of the current record is calculated as the difference timestamp 61t. The difference obtained by subtracting the selling price of the previous record from the selling price of the current record is calculated as the difference selling price of 62t. The difference obtained by subtracting the selling price of the current record from the purchase price of the current record is calculated as the difference purchase price of 63t. The difference between the sales volume of the current record and the sales volume of the previous record is calculated as the difference in sales volume: 64 tons. The difference obtained by subtracting the current record's sales quantity from the current record's purchase quantity is calculated as a difference of 65 tons. It is called [this]. However, the symbols are added for the purpose of matching with the diagram and are not included in the name.

[0106] The process of compaction step 12 will be explained with reference to Figures 1, 3, and 13 to 17.

[0107] Refer to Figure 3. Compaction step 12 takes the differential historical data 60t as input and outputs the compactified historical data 90t.

[0108] The process of integerization step 121 will be explained with reference to Figures 3, 13, and 14.

[0109] The integerization step 121 takes the differential historical data 60t as input and converts all timestamps and numerical values ​​contained in the differential historical data 60t into integers. In this example, the conversion is performed as follows: The differential timestamp 61t is converted to an integer value in milliseconds. The difference in selling price of 62t and the difference in buying price of 63t are converted to integers by multiplying them by the smallest possible positive number that can be converted to an integer when multiplied by the smallest unit of price. In this example, the smallest unit of price is 0.001, so the difference in selling price of 62t and the difference in buying price of 63t are converted to integer values ​​by multiplying them by 1000. As mentioned earlier, the difference in sales quantity of 64t and the difference in purchase quantity of 65t are converted to integer values ​​by multiplying them by 100.

[0110] As a result of this conversion, the differential historical data 60t is converted into integer differential historical data 70t. Furthermore, each part that makes up the record of the integer differential historical data 70t is called the integer differential timestamp 71t, the integer differential selling price 72t, the integer differential buying price 73t, the integer differential selling quantity 74t, and the integer differential buying quantity 75t, respectively (however, the signs are added for the purpose of matching with the figure and are not included in the names).

[0111] The process of the non-negative integer conversion step 122 will be explained with reference to Figures 3, 14, and 15.

[0112] The non-negative integerization step 122 takes the integerized difference historical data 70t as input and converts the numerical values ​​contained in each record of the integerized difference historical data 70t to non-negative integers according to the following rules. If the original number (referred to as the "original number" in this paragraph) is non-negative, the original number is converted to twice its original number; if the original number is negative, the original number is converted to -2 times the original number minus 1. This rule allows us to establish a one-to-one correspondence between the original number and non-negative integers, and moreover, to establish a correspondence between integers close to zero and non-negative integers close to zero. However, with respect to timestamp 71t, by separately guaranteeing that the records are sorted in ascending order of timestamps, the timestamp 71t can be treated as a non-negative integer, and the processing in this paragraph can be omitted.

[0113] Perform this conversion on all records.

[0114] As a result of this conversion, the integer difference historical data 70t is converted into non-negative integer difference historical data 80t. Each part that makes up the record of the non-negative integer difference historical data 80t is called the non-negative integer difference timestamp 81t, the non-negative integer difference selling price 82t, the non-negative integer difference buying price 83t, the non-negative integer difference selling quantity 84t, and the non-negative integer difference buying quantity 85t, respectively (however, the signs are added for the purpose of matching with the figure and are not included in the names).

[0115] The processing of the encoding step 123 will be described with reference to Figures 3, 8, and 15 to 17.

[0116] Encoding step 123 converts the non-negative integerized difference historical data 80t into compactified historical data 90t.

[0117] The encoding step 123, for each record of the non-negative integer difference historical data 80t, sequentially converts the values ​​of the non-negative integer difference timestamp 81t, the non-negative integer difference selling price 82t, the non-negative integer difference buying price 83t, the non-negative integer difference selling quantity 84t, and the non-negative integer difference buying quantity 85t into octet columns according to the procedure 125 shown in Figure 8, and arranges them in order. The details of procedure 125 are the same as for historical data (time frame method), so they will not be repeated.

[0118] According to the procedure in the previous paragraph, for example, record 89tb is converted into a compacted record 99tb, {0xDE,0x05,0x13,0xA8,0x01,0xE7,0x20,0x90,0x01}.

[0119] Refer to Figure 17. The encoding step 123 outputs compacted historical data 90t, which consists of a value 98 representing the smallest unit of price in the historical data, followed by the compact representations (99ta, 99tb, 99tc, etc.) corresponding to each record, in sequence. In this example, the value 98 representing the smallest unit of price in the historical data is the maximum number of decimal places in the price. In this example, the smallest unit of price in the historical data 50t is 0.001, so the value 98 representing the smallest unit of price in the historical data is 3, which represents the third decimal place.

[0120] Refer to Figure 1. Preferably, the historical data compression method 10 further comprises a general-purpose compression step 13. The general-purpose compression step 13 receives compacted historical data 90t as input, performs compression processing using a general-purpose data compression scheme, and outputs compressed compacted historical data 95t.

[0121] Although the compacted historical data 90t is created by removing redundancy from the historical data 50t, it still retains redundancy. Therefore, the amount of data can be further reduced using a general-purpose data compression method.

[0122] Furthermore, since the compacted historical data 90t has had redundancy removed from the historical data 50t based on the characteristics of the historical data, the compressed compacted historical data 95t will have a smaller data size compared to compressing the historical data 50t using a general-purpose data compression method.

[0123] Through the above series of processes, the historical data 50t is converted into a shorter representation, compacted historical data 90t, or compressed compacted historical data 95t.

[0124] In this example, for ease of understanding, the differential historical data 60t, integerized differential historical data 70t, non-negative integerized differential historical data 80t, and compactified historical data 90t are completed before proceeding to the next step. However, in actual implementation, the order of processing can be changed to the extent that it does not affect the result, such as repeating all steps for each record in order from the first record. A historical data compression method implemented in this way is also included within the technical scope of the present invention.

[0125] The procedure for restoring the compactified historical data 90t and the compressed compactified historical data 95t to the historical data 50t is the reverse of the procedure for converting the historical data 50t to the compactified historical data 90t and the compressed compactified historical data 95t. The details of the procedure are the same as those for restoring the compactified historical data 90 and the compressed compactified historical data 95 to the historical data 50, except for the difference in data names, so the explanation will not be repeated.

[0126] [Modified examples of the embodiment] The main parts of the present invention have been described above, but modifications can also be made as shown below.

[0127] Preferably, if the number of records in a series of historical data is large (for example, exceeding tens of thousands), it is advisable to divide the series of historical data into periods that are not too long (for example, every month) and create a compacted historical data 90 or 90t for each period. This means that when you want to obtain a historical data record corresponding to a specific time, you only need to scan the records in the compacted historical data 90 or 90t that contain that record from the beginning, reducing the amount of computation required to obtain the historical data record corresponding to a specific time compared to when all records are combined into a single compacted historical data 90 or 90t.

[0128] Because the compacted historical data 90 and 90t utilize differences and have a variable data length per record, reading a record located in the middle of the data requires sequentially reading records from the beginning of the compacted historical data until the desired record is reached. Consequently, the compacted historical data inherently has the problem of not allowing random access to specific records, but this problem is mitigated by the modification described in the previous paragraph.

[0129] [Examples of the effects of the present invention] To confirm the effectiveness of the data volume reduction method according to the present invention, we will illustrate the results of applying the present invention to actual historical data.

[0130] However, the amount of data that can be reduced by the present invention depends on the nature of the original historical data 50 or 50t. More specifically, the smaller the variation in numerical values ​​between adjacent records in the historical data, and the more the historical data is described in a relatively redundant representation format, the greater the data reduction effect of the present invention. Therefore, it should be noted that the examples are merely illustrative, and the same results may not be obtained with any historical data.

[0131] In the first example of results according to the present invention, historical data in an hourly format is used, recording the price fluctuations and trading volume of the US dollar against the Japanese yen every minute from January 1, 2010 to December 31, 2019.

[0132] In a second example of the results according to the present invention, tick-based historical data is used, recording the fluctuations in the price of US dollars against Japanese yen and the trading volume from August 1, 2024 to August 31, 2024.

[0133] The amount of data when the historical data for the first example and the second example were saved in the following formats was measured, and the results are shown in Tables 2 and 3 below. However, since tick-based historical data cannot be saved in HST format, the evaluations corresponding to (a) through (ne) below are not performed in the second example. (a) Compacted historical data (i) Compressed, compact historical data (general-purpose data compression method: GZIP) (c) Compressed, compacted historical data (general-purpose data compression method: LZMA2) (e) Compressed, compacted historical data (general-purpose data compression method: Brotli) (S) CSV format (uncompressed) (C) CSV format (general-purpose data compression method: GZIP) (S) CSV format (General-purpose data compression method: LZMA2) (C) CSV format (general-purpose data compression method: Brotli) (N) HST format (MetaQuotes®' standard historical data storage format) (uncompressed) (ii) HST format (general-purpose data compression method: GZIP) (Nu) HST format (General-purpose data compression method: LZMA2) (Ne) HST format (General-purpose data compression method: Brotli) However, the software used for GZIP and LZMA2 compression was 7-zip (https: / / 7-zip.org / ) version 24.08, with compression parameters of 7z.exe -tgzip -mx9 for GZIP and lzma.exe e for LZMA2. The software used for Brotli compression was Brotli (https: / / github.com / google / brotli / releases) version 1.10, with no compression parameters specified.

[0134] [Table 2]

[0135] [Table 3]

[0136] According to Tables 2 and 3, the compacted historical data 90 and 90t (column (a)) have a smaller data size than the data obtained when widely used formats such as CSV and HST are compressed with GZIP (columns (c) and (d)). Furthermore, regardless of whether GZIP, LZMA2, or Brotli is used as the general-purpose data compression method, the compacted historical data 95 and 95t can represent the historical data with approximately 42-57% of the data size of CSV and HST formats using the same general-purpose data compression method. Therefore, it can be said that the compacted historical data 90 and 90t in the present invention effectively reduce the redundancy of the historical data 50 and 50t.

[0137] Table 4 below shows the time required to read the historical data 50 from the data saved in the example of the first result. However, since the actual required time varies depending on the hardware used to perform the historical data expansion method 20 and other miscellaneous factors, in Table 4, the required time measured in the same environment is expressed as a ratio instead of its absolute value.

[0138] [Table 4]

[0139] Comparing columns (a) and (n) in Tables 2 and 4, respectively, the file containing the compacted historical data 90 is 2.3 times faster to read than the HST file, while its data size is 85% smaller. Therefore, applications using HST files can simultaneously save disk space and speed up operation by replacing the HST file with the file containing the compacted historical data 90.

[0140] Comparing columns (i) and (iii) in Tables 2 and 4, respectively, files containing GZIP-compressed compressed historical data 95 are approximately 24% slower to read than HST files, but the data size is 90% smaller. When saving space on auxiliary storage is important, GZIP-compressed compressed historical data 95 can be an alternative to HST files.

[0141] Comparing columns (E) and (N) in Tables 2 and 4, respectively, files containing the compressed, compacted historical data 95 compressed by Brotli have the same read speed as HST files, but with a data size 92% smaller. However, because Brotli requires a very large amount of computation for compression, it is not suitable for applications that frequently update or append to historical data. Therefore, the compressed, compacted historical data 95 compressed by Brotli is suitable for situations where a file created once will be accessed by an unspecified number of people, and more specifically, it is suitable for services that distribute historical data.

[0142] The values ​​in columns (C) through (S) and (D) through (N) of Table 4 are all large, indicating that the read speed is slower compared to the compacted historical data 90. Since it is disadvantageous in terms of both data volume and read speed compared to the compacted historical data 90, there is little reason to actively choose to compress the CSV and HST formats using general-purpose data compression methods.

[0143] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed herein are also included within the technical scope of the present invention. [Explanation of Symbols]

[0144] 10. Historical Data Compression Methods 11 Difference Calculation Steps 12. Compaction Steps 13 General Compression Step 15 Historical Data Compressor 16 Historical Data Compression Program 17 Arithmetic section 18 Main memory 19 Auxiliary storage 20. Historical Data Expansion Methods 21. Differential Restoration Steps 22 Compaction and Restoration Steps 23 General Deployment Steps 50, 50t Historical Data 51, 51t timestamp 52 Opening price 52t selling price 53 High 53t Purchase price 54 Low 54t Sales Quantity 55 Closing price 55t purchase quantity 56 Volume 59a, 59b, 59c, 59ta, 59tb, 59tc Historical data records 60, 60t differential historical data 61, 61t differential timestamp 62 Difference Opening Price 62t ​​difference selling price 63 Difference High 63t difference purchase price 64 Differential Low 64t differential sales quantity 65 Difference closing price 65t differential purchase quantity 66 Differential volume 70, 70t Integerized Difference Historical Data 71, 71t integerized differential timestamp 72 Integerized difference initial value 72t integer difference selling price 73 Integerized Difference High Value 73t Integerized difference purchase price 74 Integerized difference low value 74t Integer difference sales quantity 75 Integerized difference closing price 75t Integer differential purchase quantity 76 Integer differential volume 79b Records of integer-converted difference historical data 80, 80t Integerized Difference Historical Data 81, 81t Non-negative integer difference timestamp 82 Non-negative integer difference initial value 82t Non-negative integer difference selling price 83 Non-negative integer difference high value 83t Non-negative integer difference purchase price 84 Non-negative integer difference low value 84t Non-negative integer differential sales quantity 85 Non-negative integer difference closing price 85t Non-negative integer differential purchase quantity 86 Non-negative integer differential trading volume 89b, 89tb Non-negative integerized difference historical data records 90, 90t Compaction Historical Data 95, 95t Compressed and Compacted Historical Data 98. A value indicating the smallest unit of price in historical data. 99a, 99b, 99c, 99ta, 99tb, 99tc: Records of compacted historical data. 121 Integerization Step 122 Non-negative integer step 123 Encoding Steps 125 Flowchart for converting non-negative integer values ​​to compact representation 125a, 125b, 125c, 125d: Each step in the process of converting a non-negative integer value to a compact representation.

Claims

1. The difference calculation step to calculate the difference, A compactification step involves converting the aforementioned difference into a compact representation, The difference calculation step includes, of the portion of the current record, The difference between the closing price of the previous record and the opening price of the current record, and The difference between the opening price and the high price of the current record, and The difference between the opening price and the low price of the current record, and The difference between the opening price and closing price of the current record, and Calculate, or The difference between the buy prices of the previous record and the current record, or between the sell prices, and The difference between the selling price of the current record and the buying price of the current record, Calculate, or The difference between the buy prices of the previous record and the current record, or between the sell prices, and The difference between the selling price of the current record and the buying price of the current record, The difference between the sales quantity and the purchase quantity of the previous record and the current record, The difference between the number of records sold and the number of records purchased, and Calculate, The aforementioned miniaturization step is, Steps to convert signed integers or non-negative integers to compact representations. A method for compressing historical data, characterized by comprising the following:

2. A historical data compression method according to claim 1, comprising calculating the difference between the corresponding portions of the previous record and the current record for at least one portion of the current record that is not described in claim 1.

3. The historical data compression method according to claim 2, characterized in that a specific value among the integer difference timestamp or the non-negative integer timestamp value is further converted using a conversion table.

4. A program that causes an information processing device to execute the historical data compression method described in any one of claims 1 to 3.

5. A historical data compression device, characterized in that it performs the historical data compression method described in any one of claims 1 to 3, which is an information processing device.

6. A historical data decompression method that restores compacted historical data, or compressed compacted historical data, to the original historical data by performing the reverse of the transformation performed by the historical data compression method according to any one of claims 1 to 3.

Images