Method and apparatus for losslessly compressing multiple sequentially input floating-point data
Patent Information
- Application Number
- KR1020260048041
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2026-01-19
- Filing Date
- 2026-03-17
- Publication Date
- 2026-09-04
- Estimated Expiration
- 2046-03-17
Smart Images

Figure 112026032445033-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a technology for compressing data. More specifically, the present invention relates to a technology for compressing a plurality of sequentially input floating-point data without loss. Background Technology
[0002] Recently, there has been an increasing need to store and transmit large amounts of floating-point data generated continuously in chronological order in fields such as the Internet of Things (IoT), financial time series processing, server monitoring systems, and scientific computing. This data generally follows the IEEE 754 floating-point representation, and lossless compression technology is required to reduce storage space while maintaining precision. Conventionally, file-unit compression techniques or general-purpose compression algorithms have been used, but these methods have limitations in that they are not suitable for real-time processing in streaming environments or fail to sufficiently reflect the bit structural characteristics of floating-point data.
[0003] However, conventional technology has a problem in that the total number of encoding bits does not necessarily match the offset field, length field, and entire valid bit interval required by the actual encoding format. In other words, the total number of stored bits may increase due to additional bits required to represent reference position or length information, and in environments where data distribution changes, the efficiency of searching for reference candidates may be degraded due to a fixed index structure. Therefore, there is a need for a compression technology that selects reference data by considering the entire encoding format of floating-point data to minimize the actual expected number of encoding bits, and adaptively adjusts the index structure according to changes in the statistical characteristics of the data. The problem to be solved
[0004] The technical problem to be solved through some embodiments of the present invention is to provide a method and apparatus for losslessly compressing a plurality of sequentially input floating-point data, which can improve compression efficiency under the same data conditions by more accurately minimizing the total number of bits of the compressed data actually stored compared to conventional methods.
[0005] The technical problem to be solved through some embodiments of the present invention is to provide a method and apparatus for losslessly compressing multiple sequentially input floating-point data, which can actively respond to changes in data characteristics to stably maintain reference candidate search efficiency even in a streaming environment operated for a long time, and secure consistent compression performance even for data biased in a specific distribution.
[0006] The technical problem to be solved through some embodiments of the present invention is to provide a method and apparatus for losslessly compressing multiple sequentially input floating-point data that can be flexibly applied to various computational environments, thereby improving processing speed in multi-core processor or vector computational environments while maintaining or improving compression efficiency.
[0007] The technical problem to be solved through some embodiments of the present invention is to provide a method and apparatus for losslessly compressing multiple sequentially input floating-point data, which simplifies the implementation of the decompression process and enables stable lossless restoration while reducing the possibility of errors by clearly identifying bit boundaries according to the same structure even during decoding.
[0008] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem
[0009] A method for losslessly compressing a plurality of sequentially input floating-point data according to a few embodiments of the present invention to solve the above technical problem comprises: a step of maintaining in memory a set of reference values consisting of K floating-point data (where K is an integer greater than or equal to 2) stored according to the input order; a step of generating an index structure including a plurality of buckets and a plurality of slots included in each bucket to manage the set of reference values; a step of generating an index key from the binary representation of the input n-th floating-point data and selecting a plurality of reference candidates from the set of reference values based on the index key; a step of performing an exclusive OR (XOR) operation with the n-th floating-point data for each of the plurality of reference candidates and calculating an estimated number of encoding bits based on the result of the XOR operation, the number of bits required for an offset field indicating the position of the reference candidate, (b) the number of bits required for a length field indicating the length of the zero-bit interval, and (c) the number of bits required for the effective bit interval of the result; and a step of selecting one reference candidate such that the estimated number of encoding bits is minimized. The method may include the step of generating compressed data by variable-length encoding the result of an XOR operation between the selected reference candidate and the nth floating-point data together with the offset field and the length field.
[0010] In some embodiments, the method may further include the step of changing at least one of the number of buckets, the number of slots, or the index key generation rule of the index structure based on the exponential distribution, sign distribution, or bucket collision rate of the input floating-point data.
[0011] In some embodiments, the index structure may be reconfigured when a change in data distribution exceeds a preset threshold.
[0012] In some embodiments, the reorganization of the index structure may be performed by changing the bucket mapping rules while maintaining the existing set of reference values.
[0013] In some embodiments, the method may further include the step of calculating the expected number of encoding bits for a plurality of reference candidates in parallel.
[0014] In some embodiments, the method may further include the step of calculating an expected number of encoding bits for a composite reference value generated by combining two or more different reference candidates, and encoding using the composite reference value when the number of encoding bits is reduced compared to a single reference candidate.
[0015] A device for losslessly compressing a plurality of sequentially input floating-point data according to some embodiments of the present invention includes a memory that stores one or more instructions; and at least one processor, wherein the at least one processor executes one or more instructions, thereby the device maintains in memory a set of reference values consisting of K floating-point data (where K is an integer greater than or equal to 2) stored in the order of input, and to manage the set of reference values, generates an index structure including a plurality of buckets and a plurality of slots included in each bucket, generates an index key from the binary representation of the input n-th floating-point data, selects a plurality of reference candidates from the set of reference values based on the index key, performs an exclusive OR (XOR) operation with the n-th floating-point data for each of the plurality of reference candidates, calculates an estimated number of encoding bits based on the result of the XOR operation, including (a) the number of bits required for an offset field indicating the position of the reference candidate, (b) the number of bits required for a length field indicating the length of the zero-bit interval, and (c) the number of bits required for the effective bit interval of the result, selects one reference candidate such that the estimated number of encoding bits is minimized, and the result of the XOR operation between the selected reference candidate and the n-th floating-point data is used with the offset field and the length field. Compressed data can be generated by encoding variable lengths together.
[0016] In some embodiments, the device may change at least one of the number of buckets, the number of slots, or the index key generation rule of an index structure based on the exponential distribution, sign distribution, or bucket collision rate of input floating-point data by the at least one processor executing the one or more instructions.
[0017] In some embodiments, the index structure may be reconfigured when a change in data distribution exceeds a preset threshold.
[0018] In some embodiments, the device may further calculate the expected number of encoding bits for a composite reference value generated by combining two or more different reference candidates by executing one or more instructions, and may encode using the composite reference value when the number of encoding bits is reduced compared to a single reference candidate. Effects of the invention
[0019] A method and apparatus for losslessly compressing a plurality of sequentially input floating-point data according to some embodiments of the present invention can improve compression efficiency under the same data conditions by more accurately minimizing the total number of bits of the compressed data actually stored compared to conventional methods.
[0020] A method and apparatus for losslessly compressing a plurality of sequentially input floating-point data according to some embodiments of the present invention can actively respond to changes in data characteristics, thereby stably maintaining reference candidate search efficiency even in a streaming environment operated for a long time, and can secure consistent compression performance even for data biased in a specific distribution.
[0021] A method and apparatus for losslessly compressing a plurality of sequentially input floating-point data according to some embodiments of the present invention can be flexibly applied to various computational environments, thereby maintaining or improving compression efficiency while improving processing speed in multi-core processor or vector computational environments.
[0022] A method and apparatus for losslessly compressing a plurality of sequentially input floating-point data according to some embodiments of the present invention can clearly identify bit boundaries according to the same structure even during decoding, thereby simplifying the implementation of the decompression process and enabling stable lossless restoration while reducing the possibility of errors.
[0023] The effects according to some embodiments of the present invention are not limited to those exemplified above, and a wider variety of effects are included in the present invention. Brief explanation of the drawing
[0024] FIG. 1 illustrates a method for losslessly compressing a plurality of sequentially input floating-point data according to an embodiment of the present invention. FIG. 2 is a diagram showing an index structure for managing a set of reference values according to an embodiment of the present invention. FIG. 3 is a diagram showing a structure for calculating the number of encoding bits according to an embodiment of the present invention. FIG. 4 is a diagram showing a structure for comparing the expected number of encoding bits for a plurality of reference candidates according to an embodiment of the present invention. FIG. 5 is a diagram showing a compressed data format according to variable length encoding according to an embodiment of the present invention. FIG. 6 is a diagram illustrating the adaptive reconstruction process of an index structure according to embodiments of the present invention. FIG. 7 is a block diagram illustrating the configuration of a device according to embodiments of the present invention. Specific details for implementing the invention
[0025] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, various modifications may be made to the embodiments, and thus the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, and substitutions to the embodiments are included within the scope of the rights.
[0026] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed forms, and the scope of this specification includes modifications, equivalents, or substitutions that fall within the technical concept.
[0027] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.
[0028] Furthermore, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly and specifically defined otherwise. In certain cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section. Therefore, terms used in this invention must be defined not merely by their names, but based on their meanings and the overall content of the invention.
[0029] When it is mentioned that a component is connected to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.
[0030] The terms used in the embodiments are for illustrative purposes only and should not be interpreted as intended to be limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0031] Throughout this specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, singular forms used in this specification include plural forms unless specifically stated otherwise in the text. Additionally, at least one of the expressions a, b, and c described throughout this specification may encompass 'a alone', 'b alone', 'c alone', 'a and b', 'a and c', 'b and c', or 'a, b, and c all'.
[0032] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the embodiments pertain. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0033] Additionally, terms such as …part, …module, etc. as described in this specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software. Furthermore, embodiments of the present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, embodiments of the present invention may employ integrated circuit configurations such as memory, processing, logic, look-up tables, etc., which can execute various functions under the control of one or more microprocessors or other control devices.
[0034] Each block of the process flow diagrams attached to this specification and combinations of the flow diagrams may be executed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a computer for special purposes, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means for performing the functions described in the flow diagram block(s).
[0035] These computer program instructions may be stored in computer-available or computer-readable memory that can be directed toward a computer or other programmable data processing equipment to implement a function in a specific way, and the instructions stored in said computer-available or computer-readable memory may also produce a manufactured item containing instruction means that performs the function described in the flowchart block(s).
[0036] Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).
[0037] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Furthermore, in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.
[0038] In addition, when describing with reference to the attached drawings, identical components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the embodiments, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the embodiments, such detailed description is omitted.
[0039] In one embodiment, the floating-point data compression method and / or device according to the present invention may be applied to compress a plurality of floating-point data that are input continuously in chronological order without loss. Floating-point data may be represented, for example, in single-precision (32-bit) or double-precision (64-bit) formats such as IEEE 754, but is not limited thereto, and may be applied to any floating-point representation format including a sign bit, an exponent part, and a mantissa part. Streaming may refer to a form in which input data arrives continuously, and the input data may be generated from various sources such as sensor data, monitoring indicators, financial time series values, scientific calculation results, etc. In one embodiment, the compression process may be performed in the order in which the input data arrives, and the compression result may be recorded in a storage device, memory, network transmission buffer, or file, etc.
[0040] In one embodiment, the present invention can perform compression by directly utilizing the binary representation of input floating-point data. For example, the nth input floating-point data can be treated as a binary representation (bit sequence), and a reference value set consisting of K floating-point data (where K is an integer greater than or equal to 2) stored according to the input order can be maintained in memory. The reference value set can be implemented as a sliding window, and the sliding window can be updated by removing the oldest data and adding new data whenever new data is input. Additionally, each floating-point data stored in the reference value set may have position information according to the input order, and this position information can be used to specify the reference value during the offset calculation and encoding process described later.
[0041] In one embodiment, an index structure comprising a plurality of buckets and a plurality of slots included in each bucket may be created to efficiently manage a set of reference values and select reference candidates. The index structure may be configured to allow for rapid retrieval of some of the data stored in the set of reference values by an index key, and the buckets may be selected based on the index key value (or a part thereof), and the slots included in the selected buckets may form a set of reference candidates. In one embodiment, each slot may store the location of data within the set of reference values, or identification information corresponding to the data within the set of reference values (e.g., window index), and if the number of slots in a bucket is limited, the slots may be updated according to policies such as First-In, First-Out (FIFO) or Longest-Use (LRU). Additionally, the index key may be generated based on a combination of bits in the binary representation of the input data, and may be generated in various ways, such as a method based on a portion of trailing bits, a method based on a portion of upper bits, or a method combining these methods.
[0042] In one embodiment, when a reference candidate is selected, a bitwise operation between the input data and the reference candidate may be performed. For example, the bitwise operation may include an exclusive OR (XOR) operation, and the result of the XOR operation between the binary representation of the input data and the binary representation of the reference candidate may be produced. The result of the XOR operation may include a sequence of zero bits, and the sequence of zero bits may be divided, for example, into a leading zero bit sequence and / or a trailing zero bit sequence. Additionally, the valid bit sequence to be actually stored in the result of the XOR operation (e.g., the sequence excluding the leading and trailing zero bit sequences) may be identified, and the length (number of bits) of the valid bit sequence may be calculated.
[0043] In one embodiment, the present invention may use an estimated number of encoding bits as a criterion for selecting a reference candidate. The estimated number of encoding bits may be calculated by including at least (a) the number of bits required for an offset field to indicate the position of a reference candidate, (b) the number of bits required for a length field to indicate the length of a zero-bit interval, and (c) the number of bits required to represent the valid bit interval of the XOR operation result. For example, the number of bits required for the offset field may be determined according to the size (K) of the set of reference values or the method of representing the position of the reference candidate, the number of bits required for the length field may be determined according to the method of representing the leading zero-bit length, the trailing zero-bit length, or a combination thereof, and the number of bits required for the valid bit interval may be determined according to the length of the bit sequence stored in the XOR result. Accordingly, even when having the same zero-bit interval characteristics of the XOR result, a reference candidate that more directly minimizes the total encoding length can be selected by considering the representation costs of the offset and length fields together.
[0044] In one embodiment, offset information and length information for a selected reference candidate, along with the valid bit interval of the XOR result, can be generated as compressed data using a variable-length encoding method. For example, the compressed data may be structured to include an offset field, a length field, and a valid bit interval, and each field may be encoded with a different number of bits depending on the characteristics of the input data. Furthermore, the calculation of the expected number of encoding bits for each of a plurality of reference candidates can be performed in parallel, and an extension is also possible in which two or more reference candidates are combined to form a composite reference value, and then the expected number of encoding bits for it is further calculated. Moreover, adaptive reconstruction may be performed to change the number of buckets, the number of slots, or the index key generation rules of the index structure based on statistical characteristics such as the exponential distribution, sign distribution, or bucket collision rate of the input data.
[0045] Below, under these premises, the specific configuration of a method and apparatus for losslessly compressing a plurality of sequentially input floating-point data according to one embodiment of the present invention will be described sequentially with reference to the attached drawings.
[0046] FIG. 1 illustrates a method (100) for losslessly compressing a plurality of floating-point data input sequentially according to an embodiment of the present invention. Referring to FIG. 1, a method (100) for losslessly compressing multiple floating-point data input sequentially comprises: a step (S110) of maintaining in memory a set of reference values consisting of K floating-point data (K is an integer greater than or equal to 2) stored according to the input order; a step (S120) of creating an index structure including multiple buckets and multiple slots included in each bucket to manage the set of reference values; a step (S130) of generating an index key from the binary representation of the n-th floating-point data input and selecting multiple reference candidates from the set of reference values based on the index key; a step (S140) of performing an exclusive OR (XOR) operation with the n-th floating-point data for each of the multiple reference candidates and calculating an estimated number of encoding bits based on the result of the XOR operation, including (a) the number of bits required for an offset field indicating the position of the reference candidate, (b) the number of bits required for a length field indicating the length of the zero-bit interval, and (c) the number of bits required for the effective bit interval of the result; and selecting one reference candidate such that the estimated number of encoding bits is minimized. It may include a step (S150) and a step (S160) of generating compressed data by variable-length encoding the result of an XOR operation between a selected reference candidate and the nth floating-point data together with an offset field and a length field.
[0047] In one embodiment, at step S110, the device may maintain in memory a reference value set consisting of K floating-point data (K is an integer greater than or equal to 2) stored according to the input order. The reference value set may be implemented as a sliding window, and the data included in the window may be updated whenever new floating-point data is input. For example, the reference value set may be maintained to include the K most recently stored data according to the input order, and when the window is updated, the oldest data may be removed and new data added. Additionally, each floating-point data included in the reference value set may be assigned an index (e.g., 0 to K1) to identify its position within the window, and the index may be represented within the compressed data by an offset field described later. Additionally, the reference value set may be stored as is, in the binary representation of the floating-point data (e.g., a 32-bit or 64-bit bit sequence), and the original bit sequence may be maintained for lossless restoration.
[0048] In one embodiment, at step S120, the device may create an index structure comprising a plurality of buckets and a plurality of slots included in each bucket to manage a set of reference values. The index structure may be configured to include, for example, N buckets and M slots per bucket (N and M are integers greater than or equal to 1), and the buckets may be selected by an index key. Each slot may store the location of data within the set of reference values (window index) or a corresponding identifier, thereby allowing the slot itself to specify a reference candidate without storing the entire bit sequence of floating-point data. Additionally, when a bucket is selected, the reference values specified by the slots included in that bucket may be extracted as reference candidates. Furthermore, if the slots within a bucket become saturated, the slot may be replaced according to an update policy such as FIFO or LRU, and as a result, changes in the set of reference values may be reflected in the index structure. Additionally, the index structure may be implemented in a contiguous memory layout or a cache-friendly layout considering memory access efficiency, and the reference candidate search time may be reduced.
[0049] In one embodiment, at step S130, an index key is generated from the binary representation of the input n-th floating-point data, and a plurality of reference candidates can be selected from a set of reference values based on the index key. The index key can be generated based on a combination of some bits of the bit sequence of the input data, for example, by combining some bits of the trailing bit, some bits of the upper bit, or some bits of the exponent / mantissa part. Additionally, the index key can be converted into a bucket index through a hash function, masking, shift operation, etc., and one bucket can be selected by the converted bucket index. Reference values pointed to by slots included in the selected bucket can be selected as reference candidates, and the number of reference candidates can be limited by M or the number of valid slots stored in the bucket. Additionally, when selecting reference candidates, the diversity of the candidate set and the search cost can be adjusted according to the size of the set of reference values (K), the number of buckets (N), the number of slots (M), the key generation method, etc.
[0050] In one embodiment, at step S140, an exclusive OR (XOR) operation with the n-th floating-point data is performed for each of the plurality of reference candidates, and the expected number of encoding bits can be calculated based on the result. For example, if the bit sequence of the n-th floating-point data is X and the bit sequence of the reference candidate is R, the result of the exclusive OR (XOR) operation between X and R, V, can be calculated. The XOR operation can be performed by comparing bits at the same position in the two bit sequences, outputting 1 if they are different and 0 if they are the same.
[0051] For the result of the XOR operation V, the length of the leading zero bit and the length of the trailing zero bit can be calculated, and the valid bit interval excluding the leading and trailing zero bit intervals can be determined. Additionally, the number of bits required for the valid bit interval can be calculated as the length of the valid bit interval or the number of bits required to store the bit sequence itself included in the valid bit interval.
[0052] Additionally, in step S150, the estimated number of encoding bits calculated for a plurality of reference candidates can be compared, and one reference candidate can be selected such that the value is minimized. The result of the XOR operation between the selected reference candidate and the n-th floating-point data can be variable-length encoded along with an offset field and a length field, and as a result, compressed data can be generated.
[0053] In addition, during decoding, the original floating-point data can be restored by performing an XOR operation again with the selected reference candidate and the stored XOR operation result. That is, the original bit sequence can be restored losslessly by re-performing an operation in the same manner as the XOR operation performed during compression.
[0054] In one embodiment, the number of bits required for the offset field calculated in step S140 may be determined according to the size (K) of the reference value set and the offset representation method. For example, if the offset is represented as an integer in the range of 0 to K1, a minimum number of bits sufficient to represent the range may be used. That is, the offset field may be set with a number of bits capable of distinguishing all possible positions corresponding to the size of the reference value set.
[0055] Additionally, the number of bits required for the length field can be set to a number of bits sufficient to represent the length of the zero bit interval. For example, in the case of 32-bit floating-point data, since the zero bit length can range from 0 to 31, a number of bits capable of representing the entire range can be allocated to the length field. Also, in the case of 64-bit floating-point data, since the zero bit length can range from 0 to 63, a number of bits sufficient to represent the range can be allocated to the length field. In this way, the length field can be appropriately set according to the maximum representable bit length.
[0056] In addition, the number of bits required for the effective bit interval can be determined based on the length of the actual bit sequence to be stored, excluding leading and trailing zero bits from the result of the XOR operation. If necessary, the effective bit interval can be configured to include padding bits considering byte-unit or word-unit alignment, and the actual number of bits stored can be adjusted accordingly.
[0057] In one embodiment, at step S150, the estimated number of encoding bits calculated for a plurality of reference candidates is compared, and one reference candidate is selected such that the value is minimized. For example, if the estimated number of encoding bits for reference candidate i is Bi, then i*, which minimizes Bi, can be selected. Additionally, if the minimum value is calculated identically for multiple candidates, one candidate can be determined by using additional criteria such as the candidate's position within the window, whether the candidate has been recently used, the candidate's priority within the bucket, or the length of the zero-bit interval of the XOR result. Furthermore, by selecting a reference candidate based on such a minimum estimated number of bits criterion, it is possible to avoid cases where the total encoding length increases due to high representation costs of the offset or length field even if the zero-bit interval is long, and the total number of bits of the compressed block actually stored can be directly minimized.
[0058] In one embodiment, at step S160, compressed data can be generated by variable-length encoding the result of an XOR operation between a selected reference candidate and the n-th floating-point data, along with an offset field and a length field. For example, the compressed data may consist of a bit sequence including an offset field (location of the selected reference candidate), a length field (leading zero length and / or trailing zero length, or effective bit length), and an effective bit interval (XOR result bit sequence to be stored). Additionally, the offset field may be encoded to indicate the index where the reference candidate is located within the window, and the length field may be encoded to provide zero bit interval information for restoring the effective bit interval. Additionally, the effective bit interval may be encoded to include a bit sequence obtained by removing leading and trailing zero bits from the XOR result, and upon decoding, zero bits are restored at a location specified by the length field, and the restored XOR result is XORed with the reference candidate to restore the original input data without loss. In addition, variable-length encoding can be performed to adjust the number of stored bits according to the length of the valid bit interval, and decoding efficiency can be improved by applying different codeword or header structures for specific length intervals. Furthermore, the generated compressed data can be stored sequentially in a stream buffer, written to a storage medium in block units, or transmitted over a network.
[0059] In one embodiment, the processing (100) of FIG. 1 may be repeatedly applied to each item of input floating-point data, and the reference value set and index structure may be continuously updated during the iteration process. For example, after compression for the nth data is completed, the bit sequence of the corresponding data may be added to the reference value set, and the corresponding bucket / slot of the index structure may be updated. Additionally, if data is removed from the reference value set, the reference to that data in the index structure may be updated or invalidated. Furthermore, the calculation of the expected number of encoding bits for multiple reference candidates may be performed in parallel, and may be extended to a method of combining two or more reference candidates to form a composite reference value and then calculating the expected number of encoding bits therefor.
[0060] FIG. 2 is a diagram showing an index structure (200) for managing a set of reference values according to one embodiment of the present invention.
[0061] Referring to FIG. 2, in one embodiment of the present invention, an index structure (200) can be used to quickly select a reference candidate for evaluating compression efficiency with input data among a plurality of floating-point data stored in a reference value set (220). The index structure (200) can reduce the amount of computation required for XOR operations and calculation of the expected number of encoding bits compared to the case where all items of the reference value set (220) are searched by narrowing down some items of the reference value set (220) to candidates based on an index key generated from input floating-point data.
[0062] In one embodiment, the reference value set (220) may be implemented as a sliding window consisting of K floating-point data stored according to the input order. The reference value set (220) may be maintained as a contiguous area in memory, a ring buffer, or an array structure, and each floating-point data may have an identifier (e.g., an index in the range of 0 to K1) indicating a location within the window. Additionally, the reference value set (220) may be configured to store the original binary representation of the floating-point data as is, so that the original bit sequence can be restored without loss upon decompression. Additionally, the reference value set (220) may be updated whenever new data is input, and information regarding items removed and items added upon update may be reflected in the index structure (200).
[0063] In one embodiment, the index structure (200) may include a plurality of buckets (210), and each bucket (210) may include a plurality of slots. A bucket (210) is a unit that can be selected according to an index key value, and the number of buckets (N) may be set according to the scale of data processed by the system, the size (K) of the reference value set (220), the required processing delay, or the allowable memory usage. Additionally, the number of slots (M) included in each bucket (210) may correspond to the number of reference candidates that can be maintained simultaneously in one bucket, and may be set by considering the balance between the diversity of the candidate set and the search cost. Furthermore, the buckets (210) and slots may be implemented in an array form and arranged continuously within the index structure (200), and configured to reduce cache misses by sequentially accessing slots after selecting a bucket.
[0064] In one embodiment, the device may generate an index key based on a combination of some bits in the binary representation of the input nth floating-point data (230). For example, the device may generate an index key by extracting some trailing bits of the input data, some exponent part, some mantissa part, or a combination of two or more of these, and then applying a masking or shift operation. Additionally, the device may perform a hash function, a modular operation, or a range mapping operation to convert the index key into a bucket index. Furthermore, the index key generation rule may be changed according to the characteristics of the data distribution, for example, the key may be configured by increasing the weight of the exponent part when a specific exponent range is dominant, or by increasing the weight of the mantissa part when the variation of the mantissa part is large.
[0065] In one embodiment, when an index key or bucket index generated by the device is provided to the index structure (200), the index structure (200) can select a bucket (210) corresponding to the index key. A plurality of slots included in the selected bucket (210) may store the location of data within a reference value set (220) or a corresponding identifier, and items of the reference value set (220) specified by the location or identifier may be identified as reference candidates. Additionally, instead of directly storing the actual floating-point data value within the reference value set (220), each slot may store a pointer, offset, index, or handle pointing to the reference value set (220), thereby reducing the memory usage of the index structure (200). Additionally, metadata including the last usage time, insertion time, or frequency information of the reference candidate may be additionally stored in the slot, and such metadata may be used for slot updates or candidate selection strategies.
[0066] In one embodiment, when a reference value set (220) is updated and new floating-point data is added, the device may generate an index key for the added data, and location information of the data may be recorded in one of the slots of the bucket (210) corresponding to the generated index key. Additionally, if all slots of the bucket (210) are in use, the index structure (200) may remove or update existing slots according to a slot replacement policy. For example, the index structure (200) may update slots according to a first-in, first-out method that replaces the oldest inserted slot, a recent use-first method that replaces the slot that has not been used for the longest time, or a frequency-based method that replaces slots with low usage frequency of reference candidates. Additionally, if data is removed from the reference value set (220), the index structure (200) may invalidate the slot corresponding to the removed data or update the slot to the location of another valid reference value.
[0067] In one embodiment, the index structure (200) may provide reference candidates specified by the slots of the selected bucket (210) for XOR operations and the calculation of the expected number of encoding bits in subsequent steps. For example, location information of the reference candidates extracted from the selected bucket (210) may be used to load actual candidate bit sequences through access to the reference value set (220), and subsequently, the result of the XOR operation between each candidate and the input data may be calculated. Additionally, since the number of reference candidates may be limited by the number of slots (M), the number of iterative operations performed in the step of calculating the expected number of encoding bits is limited, thereby reducing processing delay. Furthermore, as the selection of the bucket (210) is designed to reflect the bit characteristics of the input data, the probability that data having similar bit patterns will be placed in the same or similar buckets may increase, and as a result, the probability that a reference candidate providing high compression efficiency will be included in the candidate set may increase.
[0068] In one embodiment, the combined structure of the index structure (200) and the reference value set (220) can be implemented in various variations. For example, the index structure (200) may maintain multiple bucket sets corresponding to multiple key generation rules in parallel, and after extracting candidates from each different bucket set, may form an integrated candidate set. Additionally, the index structure (200) may preferentially select only some of the slots within the same bucket, or assign priority to candidates based on metadata stored in the slots. Furthermore, the index structure (200) may be reconfigured to change the number of buckets or slots according to changes in data distribution or collision rates, and the original data within the reference value set (220) may be maintained even during reconfiguration.
[0069] FIG. 3 is a diagram showing a structure for calculating the number of encoding bits according to an embodiment of the present invention. In one embodiment, a method for losslessly compressing a plurality of sequentially input floating-point data may include a step of calculating the expected number of encoding bits for a plurality of reference candidates in parallel.
[0070] Referring to FIG. 3, in one embodiment of the present invention, the encoding bit count calculation structure calculates the number of bits required for the offset field (310), the number of bits required for the length field (320), and the number of bits required for the effective bit interval (330) based on the XOR operation result (300), and can calculate the expected encoding bit count (340) using the calculated values. The expected encoding bit count (340) can be calculated for each of a plurality of reference candidates and can be used as an evaluation value to select the reference candidate that has the minimum number of bits by comparing the expected encoding bit count (340) for each reference candidate.
[0071] In one embodiment, the XOR operation result (300) can be obtained by performing an exclusive OR (XOR) operation between the binary representation of the input n-th floating-point data and the binary representation of a reference candidate. The XOR operation result (300) can be represented as a bit sequence in which bits at the same position of the two bit sequences appear as 1 when they are different and 0 when they are the same, and a long continuous zero bit interval can be formed when the input data and the reference candidate have similar bit patterns. Additionally, the XOR operation result (300) may include a leading zero bit interval and a trailing zero bit interval, and the area excluding the leading zero bit interval and the trailing zero bit interval may be determined as the valid bit interval.
[0072] In one embodiment, the offset field bit count calculation (310) can determine the number of bits to be allocated to the offset field to represent the location of a reference candidate. For example, the location of the reference candidate may be represented by an integer range limited by the size (K) of the reference value set, and the offset field may be set to a minimum number of bits that can distinguish all values in the integer range from one another. Additionally, the offset may be represented by an absolute index within the reference value set or by a relative offset based on the location of the input data; if a relative offset is used, the number of bits required for the offset field can be reduced by limiting the representation range of the relative offset. Additionally, if the reference candidate is designated as a combination of buckets and slots, the offset field may be composed of a combination of a bucket identification value and a slot identification value, and the offset field bit count calculation (310) may calculate the number of bits based on the combination method.
[0073] In one embodiment, the length field bit count calculation (320) can calculate the number of bits of a length field to represent the zero-bit interval length included in the XOR operation result (300) so that it can be restored. For example, the length field may be configured to represent a leading zero-bit length and a trailing zero-bit length, respectively, and in this case, the length field may be encoded to include a value corresponding to the leading zero length and a value corresponding to the trailing zero length. Additionally, the length field may be configured to represent a leading zero-bit length and a valid bit interval length, respectively, or may be configured to represent the leading zero-bit length and the end position of the valid bit interval. Furthermore, the expression range of the length value may be determined according to the bit length of the floating-point data, and for example, the length field may be set to represent a range of 0 to 31 in 32-bit data and a range of 0 to 63 in 64-bit data. Additionally, the length field may be encoded with a fixed length, or may be encoded with a variable length code to represent frequently occurring length values with fewer bits, and the length field bit count calculation (320) can calculate the number of bits required according to the encoding method of the length field.
[0074] In one embodiment, the effective bit interval length calculation (330) can calculate the length of the effective bit interval to be actually stored from the XOR operation result (300). For example, the length of the effective bit interval can be determined as the length excluding the leading zero bit length and the trailing zero bit length from the total bit length, and the length can correspond to the length of the bit sequence to be stored in the XOR operation result (300). Additionally, if the XOR operation result (300) is all zero, the length of the effective bit interval can be determined as zero, and in this case, an encoding mode that does not store the effective bit interval can be applied. Additionally, the effective bit interval may be stored in bit units, or may be stored to include padding bits by applying byte-unit or word-unit alignment, and the effective bit interval length calculation (330) can be calculated by adjusting the actual storage length according to the alignment rule.
[0075] In one embodiment, the calculation of the estimated number of encoding bits (340) can calculate the total number of bits to be used to encode the XOR operation result (300) by reflecting the number of offset field bits (310), the number of length field bits (320), and the number of bits required for the valid bit interval (330). For example, the estimated number of encoding bits can be calculated by summing the number of bits required for the offset field, the number of bits required for the length field, and the number of bits for storing the valid bit interval. Additionally, if a flag bit indicating the existence of a field, an identification bit indicating the encoding mode, or a header bit for exception handling is added during encoding, the estimated number of encoding bits can be calculated to include the additional bits. Furthermore, the estimated number of encoding bits can be adjusted to reflect cases where some of the length field or the valid bit interval is omitted according to the pattern of the XOR operation result (300).
[0076] In one embodiment, the calculation of the expected number of encoding bits for a plurality of reference candidates can be performed in parallel. For example, the calculation of the XOR operation result (300) for different reference candidates and the calculation of the leading zero bit length, the trailing zero bit length, and the effective bit interval length can be performed simultaneously by a plurality of threads, a plurality of cores, or vector operations. Additionally, even when performed in parallel, the number of offset field bits (310), the number of length field bits (320), the effective bit interval length (330), and the expected number of encoding bits (340) can be recorded separately for each reference candidate, and in a subsequent step, the candidate having the minimum value can be selected by comparing the expected number of encoding bits (340) for each candidate.
[0077] FIG. 4 is a diagram showing a structure for comparing the expected number of encoding bits for a plurality of reference candidates according to an embodiment of the present invention. In one embodiment, a method for losslessly compressing a plurality of floating-point data input sequentially may include the step of additionally calculating the expected number of encoding bits for a composite reference value generated by combining two or more different reference candidates, and encoding using the composite reference value when the number of encoding bits decreases compared to a single reference candidate.
[0078] Referring to FIG. 4, in one embodiment of the present invention, the results of calculating the expected number of encoding bits for each of the plurality of reference candidates (400, 410, 420) are collected, and after the minimum value among the results is determined (430), the reference candidate corresponding to the minimum value can be determined as the selected reference candidate (440). The above structure can be applied when there are multiple selectable reference candidates for the nth floating-point data input, and after the expected number of encoding bits is calculated for each reference candidate in the manner described in FIG. 3, the candidate with the smallest value can be selected, thereby minimizing the total number of bits of the compressed data actually generated.
[0079] In one embodiment, the first reference candidate (400), the second reference candidate (410), and the third reference candidate (420) may be extracted from reference values specified by the slots of the bucket selected by the index structure described in FIG. 2. The number of reference candidates may be limited by the number of slots in the bucket, for example, two or more, three or more, or more reference candidates may be provided for one input data. Additionally, the reference candidates may be floating-point data having different positions within the set of reference values, and may be configured to include candidates that are likely to have a bit pattern similar to the input data first.
[0080] In one embodiment, the result of calculating the expected number of encoding bits for each reference candidate (400, 410, 420) can be calculated based on the result of an XOR operation between the input data and the corresponding reference candidate. For example, for each candidate, the number of bits required for the offset field, the number of bits required for the length field, and the number of bits required for the valid bit interval can be calculated, and the expected number of encoding bits per candidate can be determined by summing the calculated values. Additionally, the result of calculating the expected number of encoding bits per candidate can be stored in a temporary buffer, register, or array in memory, and the values per candidate can be compared in the minimum value determination step.
[0081] In one embodiment, the minimum value determination (430) can identify a candidate having a minimum value by comparing the expected number of encoding bits corresponding to a plurality of reference candidates. For example, if the expected number of encoding bits per candidate is expressed as an integer value, the minimum value candidate can be determined by repeatedly performing comparison operations to update the minimum value. Additionally, the comparison process can be performed in proportion to the number of candidates, and if the number of candidates is limited, the time required for comparison can be maintained predictably. Furthermore, if the calculation of the expected number of encoding bits per candidate is performed in parallel, the minimum value determination (430) can be performed at the time when the parallel calculation results are collected, and the determination time can be reduced by processing the calculation results per candidate with a single vector comparison operation.
[0082] In one embodiment, the reference candidate (440) selected according to the result of the minimum value determination (430) may be specified by an offset field in a subsequent encoding process. For example, the position within the window, the bucket / slot combination, or the reference candidate identifier of the selected reference candidate (440) may be converted into offset information, and the offset information may be included in the compressed data. Additionally, a valid bit interval may be extracted from the result of the XOR operation between the selected reference candidate (440) and the input data, and a length field may be encoded to enable the restoration of the valid bit interval. By using the selected reference candidate (440) in this way, a candidate that minimizes the total encoding length, including the representation cost of the offset and length fields, can be selected even if the length of the zero bit interval itself is not long.
[0083] In one embodiment, if there are multiple candidates with the same minimum value for the expected number of encoding bits, additional tie-breaking criteria may be applied. For example, tie-breaking criteria may be set to prioritize the candidate with the shorter valid bit interval length of the XOR operation result, based on the candidate's recent usage history, position within the window, priority within the bucket, or the candidate's recent usage history. Additionally, tie-breaking criteria may be designed considering decoding simplicity, consistency with the candidate update policy, or cache locality.
[0084] In one embodiment, the present invention may additionally calculate the expected number of encoding bits for a composite reference value generated by combining two or more different reference candidates, in addition to a single reference candidate. For example, a composite reference value may be generated by combining a first reference candidate (400) and a second reference candidate (410), and the composite reference value may be generated by applying a predefined bit-unit combination operation to the bit sequences of two or more reference candidates. The bit-unit combination operation may include an operation to select one of the bit sequences of two reference candidates, an exclusive OR (XOR) operation on the bit sequences of two reference candidates, an AND or OR operation on the bit sequences of two reference candidates, or an operation to convert the weighted combination result of the two reference candidates into a binary representation. Additionally, the composite reference value may be generated by selecting a reference candidate to match the exponent or sign bit of the input data and then combining the mantissa.
[0085] In one embodiment, the estimated number of encoding bits for a composite reference value can be calculated using the same procedure as for calculating the estimated number of encoding bits for a single reference candidate. For example, the result of an XOR operation between the composite reference value and the input data can be calculated, offset information can be extended to include identifiers of the reference candidates constituting the composite reference value, and the number of bits required for the length field and the valid bit interval can be calculated. Additionally, if additional identification bits or flag bits are required to specify the composite reference value, the number of additional bits can be included in the estimated number of encoding bits. Furthermore, a composite reference value may be selected if the application of the composite reference value sufficiently reduces the valid bit interval, thereby reducing the total number of encoding bits even with additional overhead included.
[0086] In one embodiment, when the expected number of encoding bits of the composite reference value is reduced compared to a single reference candidate, a step of encoding input data using the composite reference value may be performed. For example, if the selected reference candidate (440) is determined to be a composite reference value rather than a single reference candidate, the offset field may include information indicating a plurality of reference candidates constituting the composite reference value, and the length field and valid bit interval may be encoded based on the result of an XOR operation on the composite reference value. Additionally, even when using the composite reference value, the same combination rule may be applied during the decoding process to reconstruct the composite reference value, and the original input data may be restored losslessly using the result of an XOR operation with the reconstructed composite reference value.
[0087] FIG. 5 is a diagram showing a compressed data format according to variable length encoding according to an embodiment of the present invention.
[0088] Referring to FIG. 5, in one embodiment of the present invention, a compressed data format according to variable length encoding may be configured to include an offset field (510), a length field (520), and an effective bit interval (530) within a compressed data block (500). A compressed data block (500) may be generated for each floating-point data input sequentially, and a plurality of generated compressed data blocks (500) may be connected in chronological order to form a single compressed stream. Additionally, the compressed data block (500) may be written to a storage device, memory, or network transmission buffer in the form of a bit sequence or a byte sequence.
[0089] In one embodiment, the offset field (510) may indicate the location of a reference candidate selected to encode the input nth floating-point data. For example, the offset field (510) may represent the window index occupied by the selected reference candidate within a set of reference values, and the window index may be a value in the range of 0 to K1. Additionally, the offset field (510) may represent an absolute position or a relative position relative to the current input position, and when using a relative position, it may be designed to represent frequently occurring proximity references with fewer bits. Additionally, if the index structure specifies a reference candidate as a combination of buckets and slots, the offset field (510) may be configured in a combined form of a bucket identification value and a slot identification value, thereby specifying the reference candidate as a fixed length or a variable length. Additionally, the offset field (510) may be extended to include additional identification information to distinguish whether the reference candidate is a single reference or a composite reference.
[0090] In one embodiment, the length field (520) may indicate the length of the zero bit interval or the length of the valid bit interval so that the valid bit interval (530) can be restored from the result of the XOR operation. For example, the length field (520) may be configured to indicate the length of the leading zero bit and the length of the trailing zero bit, respectively, or may be configured to indicate the length of the leading zero bit and the length of the valid bit interval. Additionally, the length field (520) may be configured with values indicating the start and end positions of the valid bit interval, in which case the result of the XOR operation can be restored by inserting a zero bit at the position specified by the length field (520) during decoding. Additionally, the length field (520) may be encoded as a fixed length, or may be encoded as a variable length code to represent frequently occurring length values with fewer bits, and the average number of stored bits may be reduced according to the distribution of the length values. Additionally, the length field (520) may include a special value or a mode value to indicate the absence of the valid bit interval (530), and when the result of the XOR operation is all 0, the repeat value can be expressed using only the length field (520).
[0091] In one embodiment, the valid bit interval (530) may include the actual bit sequence to be stored, excluding the leading and trailing zero bit intervals from the XOR operation result. For example, the valid bit interval (530) may include the bit sequence remaining after removing the consecutive zero intervals of the XOR operation result, and the entire XOR operation result may be reconstructed as the leading / trailing zero lengths specified by the length field (520) are restored during decoding. Additionally, the valid bit interval (530) may be stored in bit units, or may be stored by applying byte or word unit alignment, and may include padding bits when alignment is applied. Furthermore, if the valid bit interval (530) is very short, it may be configured to briefly represent the valid bit interval (530) using a predefined codeword for a specific length interval.
[0092] In one embodiment, the compressed data block (500) may be configured to have an offset field (510), a length field (520), and a valid bit section (530) in that order, but is not limited thereto, and the order of the fields may be changed depending on ease of decoding or alignment conditions. For example, the length field (520) may be placed first to predetermine the length of the valid bit section (530) and then the valid bit section (530) may be read, or the offset field (510) and the length field (520) may be combined to form a header area and then the valid bit section (530) may be formed as a payload area. Additionally, the offset field (510) and the length field (520) may be byte-aligned to easily identify field boundaries, or conversely, they may be arranged consecutively in bit units to increase storage efficiency.
[0093] In one embodiment, the compressed data format of FIG. 5 can be directly linked to the calculation of the expected number of encoding bits of FIG. 3. For example, the number of bits required for the offset field (510), the number of bits required for the length field (520), and the number of bits required for the effective bit interval (530) can be summed to calculate the expected number of encoding bits for each reference candidate, and the reference candidate with the minimum result can be selected. Additionally, if the length field (520) is encoded with a variable length code, the codeword length corresponding to the length value can be reflected in the calculation of the expected number of encoding bits for each reference candidate. Also, if there is a mode in which the effective bit interval (530) is omitted, the minimum number of header bits required in that mode can be reflected in the calculation of the expected number of encoding bits for each reference candidate.
[0094] In one embodiment, during decoding, an offset field (510) can be parsed from a compressed data block (500) to identify a reference candidate in a set of reference values, a length field (520) can be parsed to determine the length of a leading / successful zero or a valid bit interval, and a valid bit interval (530) can be loaded to reconstruct the XOR operation result. Additionally, the binary representation of the original floating-point data can be restored by performing an exclusive OR (XOR) operation between the reconstructed XOR operation result and the bit sequence of the identified reference candidate. Additionally, the restored binary representation can be interpreted in a floating-point format and output as the original value, and lossless restoration can be guaranteed.
[0095] FIG. 6 is a diagram illustrating the adaptive reconstruction process of an index structure according to embodiments of the present invention.
[0096] Referring to FIG. 6, the adaptive reconstruction process of an index structure according to one embodiment of the present invention may be performed in the flow of data distribution analysis (601), threshold determination (602), index structure maintenance (603), index structure reconstruction (604), and index key generation rule change (605). For example, for floating-point data input sequentially, data distribution analysis (601) may be performed at a certain number of times or at a certain time interval, and based on the analysis results, whether the data distribution change satisfies a preset criterion may be determined in the threshold determination (602) step.
[0097] Additionally, if the result of the threshold determination (602) shows that the change in data distribution does not exceed the threshold, a step (603) of maintaining the existing index structure as is may be performed, in which case the number of buckets, the number of slots, and the index key generation rules may remain unchanged. On the other hand, if the result of the threshold determination (602) shows that the change in data distribution exceeds the threshold, an index structure reconstruction step (604) may be performed, in which case the number of buckets or slots may be adjusted or the bucket mapping method may be reset.
[0098] Additionally, an index key generation rule change step (605) may be performed in conjunction with the index structure reconstruction step (604), and in this step, a mask value, shift value, or hash parameter for selecting a specific bit range of the input data may be changed. Accordingly, the index structure can be dynamically adjusted to actively respond to changes in the statistical characteristics of the input data, and reference candidate selection can continue to be performed in the subsequent compression step while maintaining the linkage relationship with the set of reference values even after reconstruction.
[0099] Additionally, data distribution analysis (601), threshold determination (602), and reconstruction-related steps (604, 605) can be performed in parallel with compression processing, and whether to reconstruct can be repeatedly determined whenever the analysis result is updated. Accordingly, the adaptive reconstruction flow illustrated in FIG. 6 can operate as a control loop structure that is repeatedly applied throughout the compression process, rather than a static initial setting.
[0100] In one embodiment, a method for losslessly compressing a plurality of sequentially input floating-point data may include a step of changing at least one of the number of buckets, the number of slots, or the index key generation rule of an index structure based on the exponential distribution, sign distribution, or bucket collision rate of the input floating-point data.
[0101] In one embodiment, the step of changing at least one of the number of buckets, the number of slots, or the index key generation rule of the index structure based on the exponential distribution, sign distribution, or bucket collision rate of the input floating-point data may be performed to stably maintain the efficiency of reference candidate selection and compression performance in response to data characteristics that change over time in a streaming environment. For example, even if floating-point data is generated from the same sensor or the same service indicator, in a specific time interval, the range of values may narrow, causing the exponent to be skewed to a limited number of values, while in another time interval, due to events or outliers, the range of values may widen, causing the exponent to spread to various values. Additionally, in a specific time interval, the frequency of occurrence of positive or negative numbers may be skewed, causing the sign bit distribution to be skewed, while in another time interval, the sign bits may appear balanced. Such changes in data distribution can cause a bias in the buckets selected by the index key, and as a result, a situation may occur where slots in some buckets are excessively updated or collisions occur frequently, resulting in insufficient maintenance of valid reference candidates.
[0102] In one embodiment, modifications based on the exponential distribution may be performed by calculating a histogram or frequency of occurrence for the exponent values of the input data (e.g., values represented by exponent bits in the IEEE 754 format). For example, an exponential distribution vector may be generated by accumulating the number of occurrences of each exponent value over a preset observation interval length (L input data or a preset time interval), and the entropy of the exponential distribution vector, the ratio of the mode, or the cumulative ratio occupied by the top few exponent values may be calculated. Additionally, if the mode exponent value accounts for more than a certain proportion of the total, the index key generation rule may be modified to reflect some bits of the exponent in the index key more, and if the exponent values appear dispersed, the index key generation rule may be modified to reduce the weight of the exponent and increase the weight of some bits of the mantissa or the trailing bit. Furthermore, if the exponential distribution is concentrated in a specific interval, the number of buckets may be increased to further subdivide the concentrated key space, and if the exponential distribution is widely dispersed, the number of buckets may be decreased to mitigate candidate sparsity caused by excessive dispersion.
[0103] In one embodiment, changes based on the code distribution can be performed by calculating the frequency of occurrence of code bits in the input data. For example, the ratio of code bits corresponding to positive numbers and code bits corresponding to negative numbers during an observation period can be calculated, and if one code occupies more than a certain proportion, the code bit can be excluded from the index key or reflected with a low weight to mitigate bucket bias. Conversely, if the code bits appear relatively evenly, the code bits can be included as part of the index key to increase the resolution of candidate selection. Additionally, if the code distribution changes periodically in a specific pattern, the difference in the ratio of code bits between the recent observation period and the previous observation period can be calculated, and if the difference exceeds a specific criterion, a change in the index key generation rule can be triggered.
[0104] In one embodiment, changes based on the bucket collision rate can be performed by measuring the frequency of slot updates occurring for selected buckets, the frequency of reference candidate replacements within the same bucket, or the degree to which input data is concentrated in a specific few buckets. For example, the bucket access distribution can be calculated by counting the bucket indices selected by each input data during the observation period, and if the cumulative proportion occupied by the top few buckets in the access distribution is high, it can be determined that bucket bias has occurred. Additionally, the number of collision events can be calculated by accumulating the number of times slot replacements occur when a new item is inserted while all slots within a bucket are filled, and if the number of collision events exceeds a certain ratio relative to the total number of inputs, it can be determined that the bucket collision rate is high. Furthermore, in the case of a high collision rate, the number of buckets can be increased to disperse the key space, the number of slots can be increased to increase the diversity of candidates maintained in the same bucket, or the index key generation rules can be changed to redistribute the bucket mapping. Conversely, if the collision rate is low and most buckets are empty, resulting in sparse candidates, the number of buckets or slots can be reduced, while making adjustments to maintain the minimum diversity required for candidate selection.
[0105] In one embodiment, changes to the number of buckets (N) and slots (M) may be performed based on a preset target value for the number of candidates or a target value for processing delay. For example, the number of slots per bucket (M) may be adjusted to maintain the number of reference candidates evaluated for a single input data within a certain range, and the number of buckets (N) may be adjusted to satisfy cache hit rate or memory usage constraints. Additionally, the number of buckets and slots may be changed in increments of integers, and the scope of change may be limited or a minimum retention time may be set to prevent frequent abrupt reconfigurations. Furthermore, when changing the number of buckets or slots, instead of immediately discarding and regenerating the existing index structure, the changes may be applied gradually by sequentially expanding or shrinking starting from some buckets.
[0106] In one embodiment, changes to the index key generation rule may be performed by changing mask values, shift values, or parameters of a hash function that select specific bit ranges of the input data. For example, a key may be constructed by combining the upper bits of the exponent and the lower bits of the mantissa, and the selection weight of the mantissa bits may be increased when the variability of the mantissa increases in the observation interval, and the selection weight of the exponent bits may be increased when the variability of the exponent increases. Additionally, when using hash-based bucket mapping, the bucket mapping may be redistributed by changing the hash seed, polynomial coefficients, or table. Furthermore, to ensure that the decoding process is not affected even when the index key generation rule is changed, the index key may be used only as internal management information for candidate selection, and the offset field in the compressed data format may be maintained to indicate the actual location of the reference candidate.
[0107] In one embodiment, the modification step may be performed for every input data, at regular intervals, or only when the results of the data distribution analysis satisfy specific conditions. For example, for every L input data, the exponential distribution, sign distribution, and bucket collision rate may be updated, and based on the updated values, it may be determined whether to modify the number of buckets, the number of slots, or the index key generation rule. Additionally, the determination of whether to perform the modification may be performed in a background thread to reduce processing delay, and the cumulative counter and histogram for analysis may be updated using atomic operations or a lock-free structure. Furthermore, to ensure that compression processing is not interrupted even when the modification is performed, a method may be applied in which the existing index structure and the modified index structure are maintained in parallel for a certain period before switching.
[0108] In one embodiment, the index structure may be reconfigured when a change in data distribution exceeds a preset threshold.
[0109] In one embodiment, the index structure may be reconfigured when the change in data distribution exceeds a preset threshold. For example, the change in data distribution may be defined as a value indicating how much the exponential distribution, sign distribution, or bucket access distribution calculated in the data distribution analysis (601) has changed compared to the distribution of the previous observation interval. Additionally, the change in data distribution may be calculated as the difference in distribution vectors calculated per observation interval, the amount of change in specific statistics, or the amount of change in collision-related indicators, and if the amount of change exceeds a preset threshold, it may be determined in the threshold judgment (602) that reconfiguration is necessary.
[0110] In one embodiment, the change in exponential distribution can be evaluated using the frequency distribution of the exponent values of the floating-point data input during the observation period. For example, an exponential histogram can be generated by accumulating counters for each exponent value, and the amount of change in exponential distribution can be calculated by determining the difference between the exponential histogram of the current observation period and the exponential histogram of the previous observation period. Additionally, the amount of change in exponential distribution can be calculated as a change in the ratio of the mode exponent value, a change in the cumulative ratio of top exponent values, a change in the entropy of the exponent value distribution, or a change in the variance of the exponent value distribution. Furthermore, if a specific exponent value or a specific exponent range is biased to occupy more than a certain proportion of the total input, or if the frequency of the mode exponent value changing increases beyond a certain threshold, it may be determined that the change in exponential distribution has exceeded a threshold.
[0111] In one embodiment, a change in the code distribution can be evaluated based on the frequency of occurrence of the code bits of the input floating-point data during the observation period. For example, the ratio of code bits corresponding to positive numbers and code bits corresponding to negative numbers can be calculated, and if the ratio fluctuates by more than a certain level compared to the previous observation period, it can be determined that a change in the code distribution has occurred. Additionally, a change in the code distribution can be evaluated based on the absolute amount of change in the code bit ratio, the rate of fluctuation, or the occurrence of periodic bias, and can be used as a reconstruction trigger when a transition is detected in which the opposite code becomes dominant while a specific code was continuously dominant.
[0112] In one embodiment, the bucket collision rate or changes in bucket bias can be evaluated using actual operational statistics of the index structure. For example, a bucket access distribution can be generated by accumulating the occurrence count of bucket indices selected by each input data during an observation period, and if the cumulative proportion occupied by the top minority buckets in the distribution increases above a certain threshold, it can be determined that bucket bias has intensified. Additionally, collision-related indicators can be calculated by accumulating the number of slot replacements that occurred during the observation period, the number of times a slot was overwritten in the same bucket, or the frequency at which the number of valid candidates per bucket fell below a minimum target value, and if the indicators exceed a preset threshold, it can be determined that reconstruction is required. Furthermore, the collision rate can be defined as the ratio of slot replacement occurrences to the total number of inputs, and can also be used as a reconstruction trigger if a period in which collision events occur continuously persists for a certain length or longer.
[0113] In one embodiment, the threshold may be set as a single criterion or as a combination of multiple criteria. For example, reconstruction may be set to be performed only when the amount of change in the exponential distribution exceeds a first threshold and simultaneously the bucket collision rate exceeds a second threshold, thereby preventing excessive reconstruction caused by temporary fluctuations. Additionally, the threshold may be adjusted in conjunction with changes in compression performance, for example, by setting reconstruction to be performed when the average expected number of encoding bits increases above a certain level or when a trend of increasing average effective bit interval length is detected. Furthermore, the threshold judgment (602) may be configured to include a hysteresis condition, and reconstruction may be suppressed by applying a higher threshold or deferring reconstruction judgment for a certain period immediately after reconstruction.
[0114] In one embodiment, the calculation of the data distribution change amount may be performed based on an accumulation counter to reduce processing delay. For example, the exponent histogram, the sign bit counter, and the bucket access counter may be updated simultaneously during the input data processing process, and the distribution change amount may be calculated using the accumulated value at the end of the observation period. Additionally, the calculation of the distribution change amount may be performed in a thread separate from the compression path, and only the calculation result may be transmitted to the threshold determination (602) to determine whether to reconstruct. Furthermore, while the determination of whether to reconstruct is being made, the index structure may continue to perform compression in an index structure maintenance state (603), and the index structure reconstruction (604) and index key generation rule change (605) may be applied only after the reconstruction is determined.
[0115] In one embodiment, the reorganization of the index structure can be performed by changing the bucket mapping rule while maintaining the existing set of reference values.
[0116] In one embodiment, the reconfiguration of the index structure can be performed by changing the bucket mapping rules while maintaining the existing set of reference values. For example, at the time of reconfiguration, K floating-point data stored in the set of reference values may be retained as they are, and the method of selecting reference candidates in the index structure may be configured to change only without recollecting or restoring the data. Accordingly, the recoverability of the compressed stream is not degraded during the reconfiguration process, and the policy for maintaining the set of reference values required for the decoding process can also be maintained without change.
[0117] In one embodiment, changes to the bucket mapping rules may be combined with changes to the index key generation rules themselves, or the index key generation rules may be maintained while only the rules for deriving bucket indices from the index keys are changed. For example, even if the same index key is generated, the bucket mapping rules may be changed so that different buckets are selected by changing the mask value, shift value, or reference value of the modular operation applied to the index key. Additionally, when using hash-based bucket mapping, the key space may be redistributed into the bucket space by changing the seed value, table value, or bit range included in the hash calculation of the hash function. Furthermore, if the number of buckets changes, the bucket mapping rules may be reset to distribute the key space evenly or semi-equally based on the changed number of buckets.
[0118] In one embodiment, when the bucket mapping rule is changed, a reassignment can be performed on the reconstructed index structure to determine which bucket each item of the reference value set belongs to. For example, an index key can be recalculated for the binary representation of each floating-point data included in the reference value set, and a bucket index can be calculated by applying the changed bucket mapping rule to the recalculated index key. Additionally, location information of the floating-point data can be recorded in one of the slots of the bucket corresponding to the calculated bucket index, and this process can be repeated for all K items of the reference value set. Furthermore, since the reassignment can be performed by rearranging only the location information without changing the reference value set, the amount of data duplication can be reduced during the reconstruction process.
[0119] In one embodiment, the loading method of slots during the reconstruction process may be designed to mitigate bucket collision rates and maintain candidate diversity. For example, if the number of items mapped to the same bucket is greater than the number of slots, a selection policy may be applied to the slots to prioritize loading recently entered items, prioritize recently used items, or prioritize loading items likely to have high similarity to the input data. Additionally, if slot replacement is required, at least one of a first-in, first-out method, a recent usage priority method, or a frequency-based method may be applied, and the system may be configured to prioritize a specific policy for a certain period immediately after reconstruction to ensure that the buckets stabilize quickly. Furthermore, if slots are not filled evenly by bucket, a method of logically expanding the number of slots in a specific bucket, or a method of merging some buckets to mitigate candidate sparsity may be applied.
[0120] In one embodiment, changes to the bucket mapping rules may be performed to improve the quality of reference candidate selection. For example, if skewness in the exponent part is detected in the data distribution analysis, the bucket mapping rules may be adjusted so that some bits of the exponent part are better distributed to different buckets. Additionally, if increased volatility is detected in a specific bit interval of the mantissa, the bucket mapping rules may be adjusted so that the bit interval contributes more significantly to bucket selection. Furthermore, if skewness in the sign bit is detected, the bucket mapping rules may be adjusted to reduce the influence of the sign bit on bucket selection, thereby mitigating concentration in a specific bucket. Additionally, if the bucket access distribution is not uniform, the bucket mapping rules may be adjusted to split the key area mapped to the bucket with high access frequency and relocate it to different buckets.
[0121] In one embodiment, reconstruction may be performed gradually so as not to interrupt the compression process. For example, the existing index structure and the reconstructed index structure may be maintained simultaneously, and the two index structures may be configured to be used in parallel for a certain period. Additionally, when selecting reference candidates for input data during the parallel period, candidates from the existing index structure and candidates from the reconstructed index structure may be combined to form an integrated candidate set, and the minimum value candidate may be selected by calculating the expected number of encoding bits for the integrated candidate set. Furthermore, when the parallel period ends, the existing index structure may be discarded and switched to the reconstructed index structure, and the transition point may be set to the point where the collision rate stabilizes or the bucket access distribution converges to a target range.
[0122] In one embodiment, the timing and scope of reorganization can be adjusted considering processing delays and memory usage. For example, instead of performing reassignment on all K items of a reference value set at once, the buckets can be divided into multiple groups and reassignment can be performed sequentially on a group basis. Additionally, reorganization can be scheduled to be performed during periods with low input data processing frequency, and can be configured to allow candidate selection through a path where the existing index structure is maintained even during reorganization. Furthermore, if the cost of reorganization is predicted to exceed a certain threshold, adjustments can be made to limit the scope of changes to bucket mapping rules or to defer changes to the number of buckets and slots to the next observation period.
[0123] In one embodiment, the reconstruction of the bucket mapping rule change method can be performed independently of the compressed data format. For example, since the actual location information of the reference candidate can be recorded in the offset field, the compressed data itself may not be affected even if the bucket mapping rule is changed. In addition, during decoding, since the reference candidate can be specified from the set of reference values according to the offset field included in the compressed data, lossless restoration may be possible without reproducing the bucket mapping rule at the time of encoding at the time of decoding.
[0124] FIG. 7 is a block diagram illustrating the configuration of a device according to embodiments of the present invention.
[0125] In one embodiment, the device (2000) may include an input / output interface (2010), a memory (2020), a processor (2030), and a communication interface (2040). However, the present invention is not limited thereto. The device (2000) may be configured with some of the components shown in FIG. 7 omitted, or may be configured to include other components in addition to the components shown in FIG. 7.
[0126] In one embodiment, the input / output interface (2010), memory (2020), processor (2030), and communication interface (2040) may each be physically / electrically connected to each other.
[0127] In one embodiment, the device (2000) may be connected to various types of external devices through an input / output interface (2010). In one embodiment, the input / output interface (2010) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module (SIM), an audio I / O (Input / Output) port, or a video I / O (Input / Output) port. In one embodiment, the input / output interface (2010) may include a USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), or DVI (Digital Visual Interface), etc.
[0128] In one embodiment, the memory (2020) may store data used in the device (2000). In one embodiment, the memory (2020) may store instructions, programs, or modules for the operation of the processor (2030).
[0129] In one embodiment, the memory (2020) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD type (Solid State Disk type), an SSD type (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, and an optical disk.
[0130] In one embodiment, the processor (2030) may include a general-purpose processor such as a CPU (Central Processing Unit), AP (Application Processor), DSP (Digital Signal Processor), etc., or a neural network processing processor such as an NPU (Neural Processing Unit). In one embodiment, the processor (2030) may be divided by one or more processors to perform operations. In one embodiment, the processor (2030) may control the operation of the device (2000). The processor (2030) may control the operation of the device (2000) according to instructions stored in memory (2020).
[0131] In one embodiment, the processor (2030) may be implemented as a memory that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of components within the device (2000) of the present invention, and a processor that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may each be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.
[0132] In addition, the processor may control one or a combination of the components described above in order to implement various embodiments according to the present invention through the device (2000).
[0133] In one embodiment, the communication interface (2040) may include one or more components that enable communication between the device (2000) and an external server or external electronic device. In one embodiment, the communication interface (2040) may include at least one of a wired communication module or a wireless communication module.
[0134] Wired communication modules may include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).
[0135] Wireless communication modules include WiBro (Wireless broadband), GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, and Bluetooth (Bluetooth). TM It may include a wireless communication module that supports a wireless communication method including at least one of ) or Wi-Fi (Wireless-Fidelity).
[0136] In one embodiment, the processor (2030) can obtain information from an external electronic device or server or provide information through a communication interface (2040).
[0137] Using the embodiments of the present invention described above, those skilled in the art will be able to easily make various changes and modifications within the scope of the essential characteristics of the present invention. The content of each claim of the patent claims may be combined with other claims that are not related by reference to the extent that it can be understood through this specification.
[0138] The effects according to the technical concept of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description in the specification.
[0139] The technical concept of the present invention, as described so far with reference to FIGS. 1 to 7, can be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). The computer program recorded on the computer-readable recording medium may be transmitted to another computing device via a network such as the Internet and installed on the other computing device, thereby being used on the other computing device.
[0140] Although it has been described above that all components constituting the embodiments of the present invention are combined as one or operate in combination, the technical concept of the present invention is not necessarily limited to such embodiments. That is, within the scope of the objectives of the present invention, all components may be selectively combined in one or more ways to operate.
[0141] Although operations are depicted in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various configurations in the embodiments described above should not be understood as a necessary separation, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.
[0142] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the technical concept defined by the present invention. Explanation of the symbols
[0143] 2000: Device 2010: Input / Output Interface 2020: Memory 2030: Processor 2040: Communication Interface
Claims
Claim 1 A method for losslessly compressing multiple floating-point data input sequentially comprises: a step of maintaining in memory a set of reference values consisting of K floating-point data (where K is an integer greater than or equal to 2) stored according to the input order; a step of generating an index structure including a plurality of buckets and a plurality of slots included in each bucket to manage the set of reference values; a step of generating an index key from the binary representation of the input n-th floating-point data and selecting a plurality of reference candidates from the set of reference values based on the index key; a step of performing an exclusive OR (XOR) operation with the n-th floating-point data for each of the plurality of reference candidates and calculating an estimated number of encoding bits based on the result of the XOR operation, the number of bits required for an offset field indicating the position of the reference candidate, (b) the number of bits required for a length field indicating the length of the zero-bit interval, and (c) the number of bits required for the effective bit interval of the result; and a step of selecting one reference candidate such that the estimated number of encoding bits is minimized. A method comprising the step of generating compressed data by variable-length encoding the result of an XOR operation between a selected reference candidate and the nth floating-point data together with the offset field and length field. Claim 2 A method according to claim 1, wherein at least one of the number of buckets, the number of slots, or the index key generation rule of an index structure is changed based on the exponential distribution, the sign distribution, or the bucket collision rate of input floating-point data, wherein the sign distribution is the ratio of the frequency in which the sign bit of the input floating-point data appears corresponding to positive and negative numbers, and the bucket collision rate is the ratio of slot replacement occurrence relative to the total number of inputs, wherein slot replacement occurs due to the insertion of a new item when all slots within the buckets of the index structure are filled. Claim 3 A method according to claim 1, wherein the index structure is reconstructed when a change in data distribution exceeds a preset threshold. Claim 4 A method according to paragraph 3, wherein the reorganization of the index structure is performed by changing the bucket mapping rules while maintaining the existing set of reference values. Claim 5 A method according to claim 1, further comprising the step of calculating the expected number of encoding bits for a plurality of reference candidates in parallel. Claim 6 A method according to claim 1, further comprising the step of additionally calculating the expected number of encoding bits for a composite reference value generated by combining two or more different reference candidates, and generating the compressed data by variable-length encoding the result of an XOR operation between the composite reference value and the nth floating-point data, together with an offset field and a length field indicating the reference candidates constituting the composite reference value, when the number of encoding bits decreases compared to a single reference candidate. Claim 7 A device for losslessly compressing multiple floating-point data input sequentially, comprising: a memory for storing one or more instructions; and at least one processor, wherein the at least one processor executes one or more instructions, thereby the device maintains in memory a set of reference values consisting of K floating-point data (where K is an integer greater than or equal to 2) stored in the order of input, and to manage the set of reference values, an index structure comprising a plurality of buckets and a plurality of slots included in each bucket, generates an index key from the binary representation of the input n-th floating-point data, selects a plurality of reference candidates from the set of reference values based on the index key, performs an exclusive OR (XOR) operation with the n-th floating-point data for each of the plurality of reference candidates, calculates an estimated number of encoding bits based on the result of the XOR operation, including (a) the number of bits required for an offset field indicating the position of the reference candidate, (b) the number of bits required for a length field indicating the length of the zero-bit interval, and (c) the number of bits required for the effective bit interval of the result, selects one reference candidate such that the estimated number of encoding bits is minimized, and the result of the XOR operation between the selected reference candidate and the n-th floating-point data is the offset field and A device that generates compressed data by variable-length encoding with a length field. Claim 8 In claim 7, the device changes at least one of the number of buckets, the number of slots, or the index key generation rule of an index structure based on the exponential distribution, the sign distribution, or the bucket collision rate of the input floating-point data, wherein the sign distribution is the ratio of the frequency at which the sign bit of the input floating-point data appears corresponding to positive and negative numbers, and the bucket collision rate is the ratio of slot replacement occurrence relative to the total number of inputs, wherein slot replacement occurs due to the insertion of a new item when all slots within the buckets of the index structure are filled. Claim 9 In claim 7, the device wherein the index structure is reconfigured when a change in data distribution exceeds a preset threshold. Claim 10 In claim 7, the device further calculates the expected number of encoding bits for a composite reference value generated by combining two or more different reference candidates by executing one or more instructions, and generates the compressed data by variable-length encoding the result of an XOR operation between the composite reference value and the nth floating-point data with an offset field and a length field indicating the reference candidates constituting the composite reference value when the number of encoding bits is reduced compared to a single reference candidate.
Citation Information
Patent Citations
Compression and decompression of numerical data
KR1020110124720A
Block floating point compression of signal data
KR1020120098651A
Parallel apparatus for high-speed, highly compressed lz77 tokenization and huffman encoding for deflate compression
KR1020140113604A
Parallel entropy coding
KR1020230140460A