Method for optimizing data transmission through the use of dynamic and adaptive slicing and reduction of serialized data and system for such optimization - Patents.com

Abstract

A system and method are provided for dynamically and adaptively slicing, reducing, and bit-packing a streamed serialized data sequence. The system includes: receiving a streamed serialized data sequence; iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values; determining, for a given iteration, whether a corresponding given sliced ​​value among the plurality of sliced ​​values ​​exists in an index; if the given sliced ​​value does not exist, adding the given sliced ​​value by bit-packing an unused ID associated with the given sliced ​​value using a first number of bits based on a current index length and encoding the sliced ​​data value using a second number of bits, thereby obtaining a given bit-packed sliced ​​value; and aggregating the given bit-packed sliced ​​values ​​to obtain a bit-packed data stream for transmission.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 264,005, filed November 12, 2021.

[0002] The present invention relates to data processing and data transmission, and more specifically to a method and system for optimizing data transmission through the use of dynamic and adaptive slicing and compression of serialized data, which can be used, for example, in the context of cloud computing networks. [Background technology]

[0003] Cloud computing platforms provide web services and application programming interfaces (APIs) to individuals, businesses and governments on a pay-per-use basis. The sheer volume of data transmitted through such platforms and the growing use and widespread adoption of such systems make it difficult for clients to forecast and budget for traffic, but such systems may be subject to security issues and may be vulnerable to cyber attacks (e.g., denial of service (DoS) attacks).

[0004] The only solution available to transform such systems into more efficient ones is a complete redesign of the platform. However, such a redesign is a challenging task as it requires specialized knowledge and runs the risk of introducing errors and defects despite interoperability and support issues. Moreover, this restructuring requires converting existing technologies and paradigms natively supported by current applications that are self-describing and human-understandable (e.g., JavaScript Object Notation (JSON), Extensible Markup Language (XML), HyperText Markup Language (HTML)) into binary formats that are not human-understandable.

[0005] In addition to the monetary cost, the large amount of data exchanged has a direct impact on the amount of electricity required by data centers due to the large size of the required storage, data being fragmented and distributed across processing units, and the ever-increasing number of processing units. The impact on electricity consumption has a non-negligible impact on the environment, especially in regions where electricity is generated using fossil fuels.

[0006] For example, the article "How to stop data centres from gobbling up the world's electricity" by Jones, Nicola in Nature Vol. 561, No. 7722 (2018): 163-166. doi:10.1038 / d41586-018-06610-y estimates that data centre electricity consumption is likely to increase by about 15 times by 2030, reaching 8% of the expected global demand, and although solutions such as hyperscale centres and innovative cooling techniques enable reductions in power consumption, the physical barriers that limit the functionality of transistors will be reached in just a few decades, limiting the improvements that can be made in further reducing consumption. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Jones, Nicola, in Nature, Vol. 561, No. 7722 (2018): 163-166. doi:10.1038 / d41586-018-06610-y: "How to stop data centres from gobbling up the world's electricity." Summary of the Invention [Problem to be solved by the invention]

[0008] It is an object of the present invention to ameliorate at least some of the disadvantages present in the prior art. One or more embodiments of the present invention may provide approaches and / or methods that achieve the goals and objectives of the present invention and / or extend the scope of these approaches and / or methods. [Means for solving the problem]

[0009] One or more embodiments of the present invention were developed based on developers' recognition that elements within serialized data structures of outbound network traffic (egress) can be used to optimize data transmission. For example, Representational State Transfer (REST) ​​web application programming interfaces (APIs) that provide microservices use certain types of data objects with similar structures and store limited text values. Developers understand that such object types, structures, and values ​​can be represented in the form of an index.

[0010] One or more embodiments of the present invention enable a reduction in the size of digital objects transmitted without loss of information, thereby reducing bandwidth and power consumption and therefore the costs associated therewith.

[0011] The present invention does not require a complete re-architecting or re-design of the cloud system (as opposed to what current solutions propose) since it applies only to the transmission channel in a non-transient manner (e.g., during the current session), and as a result, the above-mentioned drawbacks can be minimized. Furthermore, one or more embodiments of the present invention can be adapted for use in or between components such as computing devices, routers, NVMe disk drivers, IoT devices, Docker or Kubernetes containers, mobile app services, NoSQL database engines, cloud services, etc.

[0012] Accordingly, one or more embodiments of the present invention are directed to a method for optimizing data transmission through the use of dynamic, adaptive slicing and reduction of serialized data and a system for such optimization.

[0013] One or more embodiments of the present invention use a mutable polymorphic data reduction matrix to adaptively and dynamically slice and reduce the streamed serialized data sequence, bit pack it, and transmit it in an output bit-packed data stream. The serialized data sequence can also be revisited by the same or another device using a reverse process on the bit-packed data stream, without requiring storage and maintenance of large indexes or dictionaries after the end of a session (i.e., the end or interruption of the data stream).

[0014] One or more embodiments of the data reduction and compression method and system of the present invention may be viewed as a method and system for performing transcoding of a serialized data stream.

[0015] The present invention can operate on any type of serialized data format, such as, but not limited to, Javascript Object Notation (JSON), XML, YAML Aint Markup Language (YAML), Protocol Buffers / Flat Buffers, BSON, AVRO, MessagePack, Multipurpose Internet Mail Extensions (MIME), as well as non-compiled code, such as JavaScript, Python, Perl, VBScript, bash, Powershell, Groovy, Lua, R, PostScript, PHP, AppleScript, etc.

[0016] The present invention is not limited to any particular physical transmission channel and can be adapted to future data transmission networks that use higher bandwidths.

[0017] One or more embodiments of the present invention use bit arrays (also known as bitmaps, bit sets, bit strings or bit vectors) as an array data structure for compactly storing bits. One or more embodiments of the present invention slice the serialized data according to the serialized data standard and can reduce and pack the data using various numbers of bits that are not aligned to bytes (i.e., 8 bits) without passing the length of these bits, resulting in a reduction in overall size.

[0018] One or more embodiments of the present invention may use further data compression techniques (also known as bit rate reduction techniques) to encode information using fewer bits than the original representation. - Non-limiting examples of entropy-based lossless compression techniques include arithmetic codes, asymmetric number systems, Golomb codes, Huffman codes (adaptive Huffman codes, canonical Huffman codes, modified Huffman codes), range codes, Shannon codes, Shannon-Fano codes, Shannon-Fano-Elias codes, Tunstall, unary codes and universal codes (exponential-Golomb codes, Fibonacci codes, Gamma codes, Levenshtein). Dictionary-based lossless compression techniques. Non-limiting examples include byte-encoded and Lempel-Ziv (842, Brotli, Deflate, LZ, LZFSE, LZJB, LZMA, LZO, LZRW, LZS, LZSS, LZW, LZWL, LZX, Snappy and Zstandard) - Non-limiting examples of other types of lossless compression techniques include BWT, CTW, Delta, DMC, DPCM, LDCT, MTF, PAQ, ppm, RLE.

[0019] In accordance with one broad aspect of the invention, there is provided a method, executed by at least one processor, for dynamically and adaptively reducing a streamed serialized data sequence to obtain a bit-packed data stream, the method including: receiving a streamed serialized data sequence including a plurality of serialized data objects, each having a serialized object data type; and iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​based at least on the serialized object data type, the iterative applying including initializing an index; determining, for a given iteration, whether a corresponding given sliced ​​value among the plurality of sliced ​​values ​​is present in the index; and, if the given sliced ​​value is not present in the index, adding the given sliced ​​value to the index; Applying in an iterative manner, the adding includes performing the adding including bit-packing an unused ID associated with the given sliced ​​value using a first number of bits based on a current index length and encoding the sliced ​​data value using a second number of bits, thereby obtaining the given bit-packed sliced ​​value; aggregating a plurality of bit-packed sliced ​​values ​​including the given bit-packed sliced ​​value to obtain a bit-packed data stream; and transmitting the bit-packed data stream including an indicator of the index and the given sliced ​​value.

[0020] In one or more embodiments of the method, before aggregating the multiple bit-packed sliced ​​values ​​to obtain the bit data packed data sequence, the method further includes referencing the given sliced ​​value if the given sliced ​​value is present in the index, where the referencing includes bit-packing the given sliced ​​value using a third number of bits representing a reference ID to obtain the given bit-packed sliced ​​value.

[0021] In one or more embodiments of the method, iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes obtaining a given sliced ​​object, determining an individual index context based on one of a position and a key associated with the given sliced ​​object, and passing the one of the position and the key as the sliced ​​value.

[0022] In one or more embodiments of the method, passing the key as a sliced ​​value includes passing a sequence of characters representing the key as a sliced ​​value.

[0023] In one or more embodiments of the method, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes determining a distinct current index context based at least on a format of the streamed serialized data sequence.

[0024] In one or more embodiments of the method, the method further includes defaulting the ID to 0 if the index is empty for the respective current index context.

[0025] In one or more embodiments of the method, iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes iteratively applying the reduction matrix to the sliced ​​object to obtain another given sliced ​​value associated with a distinct current index context.

[0026] In one or more embodiments of the method, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​further includes obtaining a given sliced ​​control that is not associated with a value in the streamed serialized data sequence.

[0027] In one or more embodiments of this method, a given sliced ​​control is not indexed, but is associated with an individual ID.

[0028] In one or more embodiments of the method, a given sliced ​​control is processed using an overflow method.

[0029] In one or more embodiments of the method, a given sliced ​​control indicates a delimitation of a portion of a streamed serialized data sequence.

[0030] In one or more embodiments of the method, the reference ID includes controlling metadata, the controlling metadata including one of the single controlling metadata and an offset to the last index ID.

[0031] In one or more embodiments of the method, the method further includes encoding the handling operation metadata based on a predefined table, the predefined table including, for each distinct data type and each distinct control, a distinct index offset to the last ID.

[0032] In one or more embodiments of the method, a given table includes a separate control for an empty index for each distinct data type and each distinct control.

[0033] In one or more embodiments of the method, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes obtaining a given sliced ​​array and iteratively applying the reduction matrix to the streamed serialized data sequence to obtain further sliced ​​values.

[0034] In one or more embodiments of the method, the method further includes, after obtaining the given sliced ​​array, determining an individual index context based on the given sliced ​​array.

[0035] In one or more embodiments of the method, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes iteratively applying the reduction matrix to the sliced ​​array to obtain further given sliced ​​values.

[0036] In one or more embodiments of the method, the index is represented using one of a HashMap, a Vector, and an Array.

[0037] In one or more embodiments of the method, the streamed serialized data sequence includes a plurality of serialized data blocks.

[0038] In one or more embodiments of the method, the reduction matrix includes a set of conditions associated with each of the data types in the streamed serialized data sequence to generate each of the sliced ​​values, sliced ​​arrays, and sliced ​​controls.

[0039] In one or more embodiments of the method, the reduced matrix is ​​stored on a non-transitory storage medium operatively connected to the processor.

[0040] In one or more embodiments of the method, aggregating the multiple bit-packed sliced ​​values ​​to obtain the bit-packed data stream includes aggregating a given packed sliced ​​value with another set of packed sliced ​​values ​​to obtain the multiple bit-packed sliced ​​values.

[0041] In one or more embodiments of the method, a given sliced ​​control includes one of an object delimiter, an array delimiter, and a string delimiter.

[0042] In one or more embodiments of the method, the individual index context includes one of a default index context, a sliced ​​array context, and a key index context.

[0043] In one or more embodiments of the method, the streamed serialized data sequence includes a JavaScript Object Notation (JSON) data sequence.

[0044] In one or more embodiments of the method, a given sliced ​​value is associated with metadata.

[0045] In one or more embodiments of the method, the reduction matrix is ​​implemented as a mutable reduction matrix.

[0046] In accordance with one broad aspect of the invention, a system is provided for dynamically and adaptively reducing a serialized data sequence to obtain a bit-packed data stream, the system comprising at least one processor and a non-transitory storage medium comprising computer-readable instructions operatively connected to the at least one processor, the at least one processor executing the computer-readable instructions to receive a streamed serialized data sequence including a plurality of serialized data objects having a serialized object data type, and iteratively apply a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​based at least on the serialized object data type, the iteratively applying including initializing an index, determining, for a given iteration, whether a corresponding given sliced ​​value among the plurality of sliced ​​values ​​exists in the index, and applying a reduction matrix to the streamed serialized data sequence in an iterative manner to obtain a plurality of sliced ​​values ​​based at least on the serialized object data type, the iteratively applying including initializing an index, determining, for a given iteration, whether a corresponding given sliced ​​value among the plurality of sliced ​​values ​​exists in the index, applying in an iterative manner including: if the sliced ​​value is not present in the index, adding the given sliced ​​value to the index, where the adding includes bit-packing an unused ID associated with the given sliced ​​value using a first number of bits based on the current index length and encoding the sliced ​​data value using a second number of bits, thereby obtaining the given bit-packed sliced ​​value; aggregating a plurality of bit-packed sliced ​​values ​​including the given bit-packed sliced ​​value to obtain a bit-packed data stream; and configured to transmit the bit-packed data stream including the index indicator and the given sliced ​​value.

[0047] In one or more embodiments of the system, the at least one processor is further configured to reference the given sliced ​​value if it is present in the index before aggregating the multiple bit-packed sliced ​​values ​​to obtain the bit data packed data sequence, where the referencing includes bit-packing the given sliced ​​value using a third number of bits representing a reference ID to obtain the given bit-packed sliced ​​value.

[0048] In one or more embodiments of the system, iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes obtaining a given sliced ​​object, determining an individual index context based on one of a key and a position associated with the given sliced ​​object, and passing one of the key and the position as the sliced ​​value.

[0049] In one or more embodiments of the system, passing the key as a sliced ​​value includes passing a sequence of characters that represents the key as the sliced ​​value.

[0050] In one or more embodiments of the system, iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes determining a distinct current index context based at least on a format of the streamed serialized data sequence.

[0051] In one or more embodiments of the system, the at least one processor is further configured to default the ID to 0 if the index is empty for the respective current index context.

[0052] In one or more embodiments of the system, iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes iteratively applying the reduction matrix to the sliced ​​object to obtain another given sliced ​​value associated with a distinct current index context.

[0053] In one or more embodiments of the system, applying the reduction matrix in an iterative manner to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​further includes a given sliced ​​control that is not associated with a value in the streamed serialized data sequence.

[0054] In one or more embodiments of the present system, a given sliced ​​control is not indexed, but is associated with an individual ID.

[0055] In one or more embodiments of the present system, a given sliced ​​control is processed using an overflow method.

[0056] In one or more embodiments of the present system, a given sliced ​​control indicates a delimitation of a portion of a streamed serialized data sequence.

[0057] In one or more embodiments of the system, the reference ID includes controlling metadata, the controlling metadata including one of the single controlling metadata and an offset to the last index ID.

[0058] In one or more embodiments of the system, the at least one processor is further configured to encode the handling operation metadata based on a predetermined table, the predetermined table including, for each distinct data type and each distinct control, a distinct index offset to the last ID.

[0059] In one or more embodiments of the present system, a given table includes a separate control for an empty index for each distinct data type and each distinct control.

[0060] In one or more embodiments of the system, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes obtaining a given sliced ​​array and iteratively applying the reduction matrix to the streamed serialized data sequence to obtain further sliced ​​values.

[0061] In one or more embodiments of the system, the at least one processor is further configured, after obtaining the given sliced ​​array, to determine an individual index context based on the given sliced ​​array.

[0062] In one or more embodiments of the system, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​includes iteratively applying the reduction matrix to the sliced ​​array to obtain further given sliced ​​values.

[0063] In one or more embodiments of the system, the index is represented using one of a HashMap, a Vector, and an Array.

[0064] In one or more embodiments of the system, the streamed serialized data sequence includes a number of serialized data blocks.

[0065] In one or more embodiments of the system, the reduction matrix includes a set of conditions associated with each of the data types in the streamed serialized data sequence to generate each of the sliced ​​values, sliced ​​arrays, and sliced ​​controls.

[0066] In one or more embodiments of the system, the reduced matrix is ​​stored on a non-transitory storage medium operatively connected to the processor.

[0067] In one or more embodiments of the system, aggregating the plurality of bit-packed sliced ​​values ​​to obtain the bit-packed data stream may include aggregating a given packed sliced ​​value with another set of packed sliced ​​values ​​to obtain the plurality of bit-packed sliced ​​values.

[0068] In one or more embodiments of the present system, a given sliced ​​control includes one of an object delimiter, an array delimiter, and a string delimiter.

[0069] In one or more embodiments of the system, the individual index context includes one of a default index context, a sliced ​​array context, and a key index context.

[0070] In one or more embodiments of the system, the streamed serialized data sequence includes a JavaScript Object Notation (JSON) data sequence.

[0071] In one or more embodiments of the present system, a given sliced ​​value is associated with metadata.

[0072] In one or more embodiments of the present system, the reduction matrix is ​​implemented as a mutable reduction matrix.

[0073] Terms and Definitions In the context of this specification, a "server" is a computer program running on suitable hardware that is capable of receiving requests (e.g., requests from an electronic device) over a network (e.g., a communications network) and executing or directing the requests to be executed. The hardware may be a physical computer or a physical computer system, but neither is required in the context of the present invention. In the above context, the use of the term "server" is not intended to imply that all tasks (e.g., received instructions or requests) or any particular task are received, executed, or directed to be executed by the same server (i.e., the same software and / or hardware). The use of the term "server" is intended to imply that any number of software elements or hardware devices may be involved in receiving / transmitting, executing, or directing to be executed any task or request, or the results of any task or request. All of the aforementioned software and hardware may be a server or multiple servers, both of which are included in the terms "at least one server" and "server".

[0074] In the context of this specification, an "electronic device" is any computing device or computer hardware capable of executing software appropriate for the relevant task at hand. Thus, some (non-limiting) examples of electronic devices include general-purpose personal computers (desktop computers, laptops, netbooks, etc.), mobile computing devices, smartphones and tablets, Internet of Things (IoT) devices (e.g., smart speakers, smart appliances, smart home devices, sensors, etc.), and network equipment such as routers, switches and gateways. It should be noted that an electronic device in the above context is not precluded from acting as a server to other electronic devices. The use of the expression "electronic device" does not preclude multiple electronic devices from being used in receiving / transmitting, executing or causing to be executed any task or request, or the result of any task or request, or the steps of any method described herein. In the context of this specification, a "client device" refers to any of a variety of end-user client electronic devices associated with a user, such as personal computers, tablets and smartphones.

[0075] For purposes of this specification, the term "computer-readable recording medium" (also referred to as "recording media" and "storage") is intended to include non-transitory media of any nature and type, including but not limited to RAM, ROM, disks (CD-ROM, DVD, floppy disk, hard drive, etc.), USB keys, solid state drives, tape drives, etc. Multiple components, including two or more media components of the same type and / or two or more media components of different types, may also be combined to form a computer information storage medium.

[0076] For purposes of this specification, a "database" is any structured collection of data, regardless of the database's specific structure, database management software, or computer hardware on which the data is stored, implemented, or made available for use. A database may reside on the same hardware as the processes that store or utilize the information stored in the database, or it may reside on separate hardware, such as a dedicated server or multiple servers.

[0077] For purposes of this specification, the term "information" includes information of any nature or type that can be stored in a database, including, but not limited to, audiovisual works (images, films, recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc.), documents, spreadsheets, lists of words, etc.

[0078] For the purposes of this specification, unless expressly specified otherwise, an "indicator" of an information element may be the information element itself, or may be a pointer, reference, link, or other indirect mechanism that allows the recipient of the indicator to locate a network, memory, database, or other computer-readable media location from which the information element may be obtained. For example, an indicator of a document may include the document itself (i.e., the document's contents), or may be a unique document descriptor that identifies a file on a particular file system, or may be some other means that directs the recipient of the indicator to a network location, memory address, database table, or other location where the file can be accessed. As those skilled in the art will recognize, the precision required for such an indicator depends on the degree of prior understanding of the interpretation that will be given to the information exchanged between the sender and recipient of the indicator. For example, if, prior to communication between the sender and recipient, an indicator of an information element is understood to take the form of a database key for an entry in a particular table of a given database that stores the information element, then all that is required to effectively convey the information element to the recipient is the transmission of the database key, even if the information element itself is not transmitted between the sender and recipient of the indicator.

[0079] For purposes of this specification, the expression "communications network" is intended to include, but is not limited to, telecommunications networks such as computer networks, the Internet, telephone networks, telex networks, TCP / IP data networks (e.g., WAN networks, LAN networks), and virtual networks (e.g., VLANs, VNETs, ​​VXLANs, CNIs, etc.). The term "communications network" includes wired networks or direct wired connections (e.g., twisted pair wires, coaxial cable, fiber optics, etc.), as well as wireless media such as acoustic, radio frequency (RF), optical wireless communication (OMC) (e.g., visible light, infrared light, ultraviolet light) and other wireless media, and combinations of any of the foregoing.

[0080] In the context of this specification, the terms "first", "second", "third", etc. are used as adjectives only to enable distinction between the nouns they modify, and not to describe any particular relationship between said nouns. Thus, for example, it should be understood that the use of the terms "first server" and "third server" is not intended to suggest any particular (for example) order, type, chronological order, hierarchy or ranking of or between servers, nor is the use of these terms (in and of itself) intended to suggest that there must necessarily be any "second server" in any given situation. Moreover, as discussed in other contexts herein, reference to a "first" element and a "second" element does not exclude that the two elements are the same actual real-world element. Thus, for example, in some cases, the "first" server and the "second" server may be the same software and / or hardware. In other cases, the "first" server and the "second" server may be different software and / or hardware.

[0081] Each implementation of the present invention will have at least one, but not necessarily all, of the above-mentioned objects and / or aspects. It should be understood that some aspects of the present invention that arise in an attempt to achieve the above-mentioned objects may not meet the above-mentioned objects and / or may meet other objects not specifically set forth herein.

[0082] Further and / or alternative features, aspects, and advantages of implementations of the invention will become apparent from the following description, the accompanying drawings, and the appended claims.

[0083] For a better understanding of the present invention, as well as other aspects and further features thereof, reference is made to the following descriptions which should be taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0084] [Figure 1]FIG. 1 illustrates a schematic diagram of an electronic device in accordance with one or more non-limiting embodiments of the present invention. [Diagram 2] FIG. 1 illustrates a schematic diagram of a communication system in accordance with one or more non-limiting embodiments of the present invention. [Diagram 3] FIG. 3 illustrates a schematic diagram of a non-limiting example of a cloud infrastructure architecture implemented within the communication system of FIG. [Figure 4] FIG. 2 illustrates a schematic diagram of a serialized data source and a data reduction and packing procedure in accordance with one or more non-limiting embodiments of the present invention. [Diagram 5] FIG. 1 illustrates a schematic diagram of how values ​​not present in the index are added to the index (top) and how values ​​present in the index are referenced (bottom), in accordance with one or more non-limiting embodiments of the present invention. [Figure 6] FIG. 11 illustrates an example of an overflow offset in accordance with one or more non-limiting embodiments of the present invention. [Figure 7] FIG. 11 illustrates an example of an overflow offset in accordance with one or more non-limiting embodiments of the present invention. [Figure 8] FIG. 1 illustrates a non-limiting example of a JSON object and the structure of a JSON object. [Figure 9] FIG. 2 illustrates a finite state diagram of a default data reduction and packing procedure in accordance with one or more non-limiting embodiments of the present invention. [Figure 10] FIG. 10 illustrates a finite state diagram of the data reduction and packing procedure of FIG. 9 in a modified state in accordance with one or more non-limiting embodiments of the present invention. [Figure 11] FIG. 2 illustrates a flowchart of a method for dynamic, adaptive slicing and reduction of serialized data in accordance with one or more non-limiting embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0085] The language regarding the examples and conditions described herein is primarily intended to aid the reader in understanding the principles of the present invention, and is not intended to limit the scope of the present invention to such specifically described examples and conditions. It will be appreciated that those skilled in the art will be able to devise various configurations that, although not expressly described or presented herein, embody the principles of the present invention and are within the spirit and scope of the present invention.

[0086] Additionally, to aid in understanding, the following description may describe relatively simplified implementations of the invention, as those skilled in the art will appreciate that various implementations of the invention may be more complex.

[0087] In some cases, what are believed to be useful modifications of the invention are described. This is done merely as an aid to understanding, and again, does not define the scope of the invention or delimit it. These modifications are not an exhaustive list, and one of ordinary skill in the art may make other modifications while remaining within the scope of the invention. Moreover, if a modification is not described, it should not be construed that the modification is not possible and / or that what is described is the only way to implement the relevant element of the invention.

[0088] Moreover, all statements herein describing principles, aspects, and implementations of the invention, as well as specific examples of the invention, are intended to encompass both structural and functional equivalents thereof, whether currently known or developed in the future. Thus, for example, those skilled in the art will appreciate that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like, may be substantially represented in a computer-readable medium and thus represent various processes executed by a computer or processor, whether or not the computer or processor is explicitly shown.

[0089] The functionality of the various elements illustrated in the figures, including any functional blocks labeled "processor" or "graphical processing unit," may be provided through the use of dedicated hardware and hardware capable of executing software in conjunction with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. In one or more non-limiting embodiments of the present invention, the processor may be a general-purpose processor, such as a central processing unit (CPU), and / or a processor dedicated to a specific purpose, such as a graphical processing unit (GPU) and / or a data processing unit (DPU). Furthermore, explicit use of the terms "processor" or "controller" should not be construed to refer only to hardware capable of executing software, but may implicitly include, without limitation, digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage, such as non-volatile random access memory (NVRAM). Other hardware, conventional and / or custom, may also be included.

[0090] A software module that is implied to be software, or simply a module, may be expressed herein as any combination of flowchart elements or other elements indicating the performance of process steps and / or textual descriptions. Such modules may be executed by hardware that is explicitly or implicitly shown.

[0091] With the above basics in mind, some non-limiting examples will now be considered to illustrate various implementations of aspects of the present invention.

[0092] Electronic Devices Referring to FIG. 1, an electronic device 100 suitable for use with some implementations of the present invention is shown, which comprises various hardware components, including one or more single-core or multi-core processors collectively represented by processor 110, a graphical processing unit (GPU) 111, a solid-state drive 120, random access memory 130, a display interface 140 and an input / output interface 150.

[0093] Communications between the various components of electronic device 100 may be enabled by one or more internal and / or external buses 160 (e.g., PCI bus, Universal Serial Bus, IEEE 1394 "Firewire" bus, SCSI bus, Serial ATA bus, etc.) by which the various hardware components are electronically coupled.

[0094] The I / O interface 150 may be coupled to a touch screen 190 and / or one or more internal and / or external buses 160. The touch screen 190 may be part of a display. In one or more embodiments, the touch screen 190 is a display. In the same manner, the touch screen 190 may also be referred to as a screen 190. In the embodiment shown in FIG. 1, the touch screen 190 includes touch hardware 194 (e.g., pressure-sensitive cells embedded in the display layer that allow detection of a user's physical interaction with the display) and a touch I / O controller 192 that allows communication with the display interface 140, and / or one or more internal and / or external buses 160. In one or more embodiments, the I / O interface 150 may be connected to a keyboard (not shown), a mouse (not shown), or a trackpad (not shown) in addition to or simply as a software module in place of the touch screen 190 that allows a user to interact with the electronic device 100. It will be appreciated that some embodiments of the present invention may be implemented without a user interface component such as the touch screen 190.

[0095] According to implementations of the present invention, the solid-state drive 120 stores program instructions stored in the random access memory 130 and suitable for execution by the processor 110 and / or the GPU 111 to dynamically slice, reduce, and bit pack the serialized data stream to obtain packed data. For example, the program instructions may be part of a library or an application. It is contemplated that embodiments of the present invention may be implemented to transmit data between components of the electronic device 100, for example, the NVMe driver may be modified to send a compressed log to the processor 110 via the internal and / or external bus 160.

[0096] As can be appreciated by one skilled in the art, electronic device 100 may be implemented as a server, a desktop computer, a laptop, a tablet, a smartphone, a personal digital assistant, or any device that may be configured to practice the present invention.

[0097] system Referring to FIG. 2, a schematic diagram of a system 200 suitable for implementing one or more non-limiting embodiments of the present invention is shown. It should be clearly understood that the depicted system 200 is merely an exemplary implementation of the present invention. Thus, the following description of the system 200 is intended to be merely a description of an example to illustrate the present invention. This description is not intended to define the scope of the present invention or to delimit the present invention. In some cases, examples of modifications to the system 200 that may be useful are also described below. This is done merely to aid in understanding, and again, does not define the scope of the present invention or to delimit the present invention. These modifications are not an exhaustive list, and other modifications are likely possible, as will be appreciated by those skilled in the art. Moreover, if this has not been done (i.e., no modifications are described), it should not be construed that the modifications are not possible and / or that the described is the only way to implement the relevant elements of the present invention. As will be appreciated by those skilled in the art, this is likely not the case. It should be further understood that system 200 may, in certain cases, provide simple implementations of the invention, and, where such is the case, these implementations are provided in the above manner as an aid to understanding. As will be appreciated by those skilled in the art, various implementations of the invention may be more complex.

[0098] Among other things, the system 200 comprises a number of client devices 210 , a server 220 and a database 230 communicatively coupled via a communications network 250 .

[0099] Client Device The system 200 includes a number of client devices 210 .

[0100] In one or more embodiments, one or more of the client devices 210 may be associated with a user (not shown). As such, the client devices are sometimes referred to as "electronic devices," "end-user devices," or "client electronic devices." In one or more other embodiments, one or more of the client devices 210 may be servers (e.g., cloud servers) that provide functionality and resources to other client devices and may be associated with a respective operator.

[0101] Each of the multiple client devices 210 includes one or more components of the electronic device 100, such as one or more single-core or multi-core processors collectively represented by processor 110, a graphical processing unit (GPU) 111, a solid-state drive 120, random access memory 130, a display interface 140, and an input / output interface 150.

[0102] It will be appreciated that each of the client devices 210 may be implemented in different manners, and a given one of the client devices 210 may be implemented as a server, a desktop computer, a laptop, a tablet, or a smartphone, etc. In one or more other embodiments, one or more of the client devices 210 may be implemented as a smart device, such as, but not limited to, a smartphone, a smart car, a smart thermometer, a smart doorbell, a smart lock, a smart refrigerator, a phablet and tablet, a smart watch, a smart band, a smart keychain, and smart glasses.

[0103] It should be clearly understood that the implementation with multiple client devices 210 is provided for illustrative purposes only. Although only four client devices are shown (all shown in FIG. 2), it is envisioned that any number of client devices 210 may be connected to the server 220 of the system 200. It is further envisioned that in some implementations, the number of client devices 210 included in the system 200 may be in the tens or hundreds of thousands. Thus, those skilled in the art will readily appreciate other specific implementation details regarding these elements. Thus, the examples provided herein above are not meant to limit the scope of the present invention.

[0104] In one or more embodiments, a given one of the client devices 210 is specifically configured to (i) connect to the server 220, (ii) transmit a request for data to the server 220, (iii) receive packed data from the server 220, and (iv) unpack the packed data to obtain the requested data.

[0105] In one or more embodiments, a given one of the client devices 210 is configured to unpack and decode the serialized data using the methods and techniques described later in this specification.

[0106] As one non-limiting example, the serialized data requested from the server 220 may be in JSON format. As another non-limiting example, the data requested from the server 220 may be in XML format.

[0107] server The server 220 is configured, among other things, to (i) receive a request for data, (ii) obtain the requested data based on the request, (iii) serialize the requested data to obtain serialized data, (iv) dynamically slice, compress, and bit-pack the requested serialized data to obtain bit-packed data, and (v) provide the bit-packed serialized data for transmission to a given one of the plurality of client devices 210.

[0108] In one or more embodiments, server 220 performs the above process without receiving a request, i.e., server 220 receives serialized data, dynamically slices, compresses, and packs the serialized data to obtain packed data, and transmits the packed data to another computing device.

[0109] In one or more embodiments, the server 220 implements a data reduction and packing service 320 (FIG. 3) that performs a data reduction and packing procedure 400 (FIG. 4) to provide at least some of the above functionality.

[0110] The server 220 is configured to provide an application programming interface (API).

[0111] In one or more embodiments, the server 220 is configured to retrieve the serialized data from the database 230 in response to a request from a given one of the plurality of client devices 210. In one or more other embodiments, the server 220 is configured to retrieve the data from another electronic device (not shown) coupled to the communications network 240. As one non-limiting example, the requested data may be obtained from an Internet of Things (IoT) connected device, which may stream the data to the server 220.

[0112] In one or more embodiments, the functionality of the server 220 may be implemented by an IoT device.

[0113] How server 220 is configured to do this is described in more detail later in this specification.

[0114] It will be appreciated that the server 220 may be implemented as a conventional computer server and may include at least some of the features of the electronic device 100 depicted in FIG. 1. In one non-limiting example of one or more embodiments of the present invention, the server 220 is implemented as a server running an operating system (OS), such as Windows Server® or LINUX®. Of course, the server 220 may be implemented in any suitable hardware and / or software and / or firmware or combination thereof. In the disclosed non-limiting embodiment of the present invention, the server 220 is a single server. In one or more alternative non-limiting embodiments of the present invention, the functionality of the server 220 may be distributed and implemented across multiple servers (not shown).

[0115] Implementation of the server 220 is well known to those skilled in the art. However, the server 220 comprises a communication interface (not shown) configured to communicate with various entities (e.g., a database 230 and multiple client devices 210, as well as other devices potentially coupled to the communication network 240) via a communication network 240. The server 220 further comprises at least one computer processor (e.g., the processor 110 of the electronic device 100) operably coupled to the communication interface and structured and configured to execute various processes described herein.

[0116] In one or more embodiments, the server 220 is configured to provide cloud services to the client devices 210, which may include processing servers, storage, databases, networking, and software. Users of the client devices 210 or entities associated with the client devices 210 can use the cloud services provided by the server 220 to generate applications and services, store data, back up and restore data, host websites and blogs, stream audio and video, and deliver software on demand to analyze data for patterns and make predictions. A non-limiting example of a cloud architecture is described later in this specification with reference to FIG. 3.

[0117] Database Database 230 is communicatively coupled to server 220 and client device 210 via communication network 240. However, in one or more alternative implementations, database 230 may be directly coupled to server 220 without departing from the teachings of the present invention. Although database 230 is generally illustrated herein as a single entity, it will be appreciated that database 230 may also be configured in a distributed manner, e.g., database 230 may have different components, each configured for a particular type of retrieval from or storage to database 230.

[0118] Database 230 may be any structured collection of data, regardless of the particular structure of database 230 or the computer hardware on which the data is stored, implemented, or made available for use. Database 230 may reside on the same hardware as the processes that store information in database 230 or that utilize the information stored in database 230, or may reside on separate hardware, such as server 220. Database 230 may receive data from server 220 for storage of the data and may provide stored data to server 220 for use.

[0119] In one or more embodiments of the invention, database 230 is configured, among other things, to store (i) serializable data and (ii) configuration parameters of the data reduction service.

[0120] In one or more other embodiments, the database 230 may be optional.

[0121] Communication Network In one or more embodiments of the present invention, the communication network 240 is the Internet. In one or more alternative non-limiting embodiments, the communication network 240 may be implemented as any suitable local area network (LAN), wide area network (WAN), virtual local area network (VLAN) (e.g., VNET, CNI, VXLAN, etc.), private communication network, etc. The implementation of the communication network 240 is for illustrative purposes only, and while a single communication network 240 is shown, it will be understood that multiple communication networks between components of the system 200 are envisioned to be available. How the communication links 245 (not separately numbered) between the client device 210, the server 220, the database 230, and / or other electronic devices (not shown) and the communication network 240 are implemented depends, among other things, on how each electronic device is implemented.

[0122] The communications network 240 may be used to transmit data packets between the server 220 and the plurality of client devices 210. For example, the communications network 240 may be used to transmit a request from a given one of the plurality of client devices 210 to the server 220. In another example, the communications network 240 may be used to transmit data from the server 220 to another given one of the plurality of client devices 210.

[0123] It should be understood that system 200 is provided only as one non-limiting example of a computing environment in which the present invention may be practiced, and that embodiments of the present invention may be implemented between two computing devices connected to each other (e.g., a server 220 and a single client device 210), or may be implemented by a single computing device (e.g., between components within a single housing having the necessary processing power).

[0124] Cloud Infrastructure Architecture Referring to FIG. 3, a schematic diagram of a non-limiting example of a cloud infrastructure architecture 300 is shown.

[0125] Cloud infrastructure architecture 300 is implemented and executed within at least a portion of system 200. It will be understood that cloud infrastructure architecture 300 is provided only as one non-limiting example of an environment in which one or more embodiments of the present invention may be practiced, and thus, at least some embodiments of the present invention may not be practiced within a cloud computing environment.

[0126] Cloud infrastructure architecture 300 includes a public cloud service 310 connected to the Internet 340 via a gateway and firewall 345. It will be appreciated that the Internet 340 may be one implementation of communication network 240 of FIG. 2, and that public cloud service 310 may be implemented at least in part by server 220 of FIG.

[0127] The public cloud service 310 is managed by a cloud provider and provides cloud resources over the Internet 340. As a non-limiting example, the public cloud service 310 can provide cloud services and resources to multiple client devices 210 of FIG.

[0128] The public cloud service 310 includes a virtual private network (VPN) 315. It will be understood that the VPN 315 is another communication network provided for illustrative purposes only, and that the public cloud service 310 may include other types of networks, as detailed herein above. The VPN 315 uses encryption to create a private network within the public cloud service 310 over the Internet 340.

[0129] The public cloud service 310 runs a data reduction packing service 320. The data reduction packing service 320 includes a data reduction service instance 325 and a bit compressor plug-in instance 335.

[0130] The purpose of the data reduction packing service 320 is to transparently process serialized data transmitted from the public cloud service 310 (e.g., implemented using the server 220) over the Internet 340 so as to reduce bandwidth consumption and costs for clients of the public cloud service 310 (e.g., multiple client devices 210) by reducing the size of the serialized data, without losing information contained in the original data. More precisely, the data reduction packing service 320 processes redundancy in serialized data structures in a transitory manner by using a combination of dynamic and adaptive differential slicing and reduction techniques by exploiting different types and / or formats in the serialized data, by organizing and grouping redundant data structures contained in the serialized data to enable achieving compression ratios that may not be possible or may be difficult to achieve using conventional monolithic compression techniques and / or monolithic binary data reduction techniques. If necessary, structural and functional elements of the serialized data sequence may be further compressed using conventional known compression techniques.

[0131] The data reduction service instance 325 is in communication with an API manager 385 that is connected to a number of APIs 387 and a bit compressor plug-in instance 335. The bit compressor plug-in instance 335 triggers injection into the data reduction service instance 325. The API manager 385 provides the functionality of the data reduction packing service 320 through a number of APIs 387 that may be accessible to at least a portion of the clients using the public cloud service 310.

[0132] In one or more embodiments, the public cloud service 310 further comprises a public marketplace 350 that provides a certified data reduction packing service 352 and a certified bit compressor plug-in 354 for each client device (e.g., client devices 210 of FIG. 2). The certified data reduction packing service 352 is instantiated in a data reduction service instance 325, and the certified bit compressor plug-in 354 is instantiated in a bit compressor plug-in instance 335. It will be appreciated that the public marketplace 350 may be optional in some embodiments of the present invention.

[0133] The public cloud service 310 further comprises a third-party compression server 360 connected to a compression algorithm database 365. Third-party developers 370 (e.g., individuals, companies, academic researchers) can connect to the third-party compression server 360 via the Internet 340 to provide compression algorithms. The compression algorithms may be in a domain-specific language and may be stored in the compression algorithm database 365.

[0134] The third party compression server 360 uses one or more compression algorithms stored in a compression algorithm database 365 to trigger the injection into the bit compressor plug-in instance 335 .

[0135] In the context of the present invention, the third-party compression server 360 can also optionally use a domain-specific language from the compression algorithm database 365 to define additional or alternative compression techniques. The third-party developer 370 can easily define the algorithms and focus on the compression technique without having to deal with the complexities of the low-level bit-packing techniques performed by the bit compressor plug-in instance 335. The use of a domain-specific language (DSL) decouples higher-level logic from the low-level implementation of the bit compressor plug-in instance 335. This can, for example, facilitate performance improvements or allow platform-specific development at a lower layer without the direct involvement of the third-party developer 370 while maintaining compatibility. The injection can be done at run-time, as it can be easily performed by a virtual machine or just-in-time compiled.

[0136] One non-limiting example of a DSL that can be used to avoid passing the first 255 redundant characters in the prefix of a URL value is provided below. JPEG2024546443000002.jpg106166

[0137] Referring now to FIG. 4, a data reduction and packing procedure 400 implemented by the data reduction and packing service 320 in accordance with one or more non-limiting embodiments of the present invention will be described.

[0138] Data Reduction and Packing Procedures The data reduction and packing procedure 400 may be performed by one or more processors, such as processor 110 and / or GPU 111 of server 220. As a non-limiting example with respect to Figure 3, the data reduction and packing procedure 400 may be performed as part of data reduction packing service 320 by data reduction service instance 325 and bit compressor plug-in instance 335.

[0139] In one or more alternative embodiments, the data reduction and packing procedure 400 may be performed locally by a processor within a single computing device.

[0140] The data reduction and packing procedure 400 is performed for every communication session (i.e., two-way information exchange between communicating devices). If a current session is terminated or interrupted, the data is not stored in a non-transient manner and the data reduction and packing procedure 400 may be re-performed. It will be appreciated that multiple data reduction and packing procedures 400 may also be performed in parallel and independently (i.e., without sharing index information) by a processor on different serialized data sequences.

[0141] In one or more embodiments, the data reduction and packing procedure 400 is specifically configured to: (i) interface with a serialized data source 410; (ii) receive a serialized data sequence from the serialized data source 410 in the form of a serialized data stream 415; (iii) dynamically and recursively slice and reduce the serialized data stream 415 by use of a reduction matrix 440 to obtain a set of sliced ​​data portions, each of the sliced ​​data portions having an individual custom structure based on a serialized data type, the custom structure including an index and / or at least one reference value and a respective set of values ​​associated with the individual index; (iv) bit pack at least a portion of the set of sliced ​​portions to obtain bit-packed serialized data; and (v) provide the packed serialized data for transmission.

[0142] The data reduction and packing procedure 400 processes data blocks in the serialized data stream 415 on the fly as the data blocks become available for processing over time, and does not require prior knowledge of the contents of serialized data blocks that have not yet been processed or received in the serialized data stream.

[0143] In one or more embodiments, the packed serialized data stream is transmitted to another processor that performs the reverse of the data reduction and packing procedure 400 (i.e., a data unpacking and reconstruction procedure (not shown)) to unpack and decode the packed serialized data to obtain the source serialized data stream 415.

[0144] Serialized Data Source The data reduction and packing procedure 400 receives a serialized data stream 415 from a serialized data source 410 that interfaces with the data reduction and packing procedure 400. It will be appreciated that the nature of the serialized data source 410 is not limited and may include one or more computing devices or components, computer readable recording media, databases, and the like.

[0145] 2, and the serialized data stream 415 may be obtained by the server 220 from the database 230 upon receiving a request from a given one of the plurality of client devices 210. In one or more other embodiments, the serialized data source 410 may be one of the plurality of client devices 210, and the serialized data stream 415 may include data requested by another one of the plurality of client devices 210, and the serialized data stream 415 is processed by the data reduction and packing procedure 400 performed by the server 220 and transmitted back to one of the plurality of client devices 210 with the requested data in the form of packed serialized data.

[0146] In one or more embodiments, serialized data stream 415 may be generated based on given data requested by a given one of client devices 210. As one non-limiting example, in an embodiment in which server 220 provides cloud services in the form of public cloud services 310 (FIG. 3), serialized data stream 415 may correspond to a software data object requested by one of client devices 210, and serialized data source 410 may generate or obtain serialized data stream 415 representing the software data object for facilitating storage and transmission to a given one of client devices 210 via server 220.

[0147] In one or more other embodiments, the serialized data source 410 may transmit the serialized data stream 415 without receiving a request from the server 220 .

[0148] As a non-limiting example, the serialized data source 410 may be data from an IoT device (not shown) or may obtain serialized data from an IoT device. It will be appreciated that the type of IoT device is not limited and may include consumer IoT devices, commercial IoT devices, military IoT devices, industrial IoT devices, etc. The IoT device may be implemented as, for example, a smart speaker, a smart tracker, a satellite, a radar, one or more sensors, a surveillance robot, a drone, a smart vehicle (e.g., a smart car), a smart thermometer, a smart doorbell, a smart lock, a smart refrigerator, a phablet or tablet, a smart watch, a smart band, a smart keychain, smart glasses, etc.

[0149] In one or more other embodiments, the serialized data source 410 may be located and implemented within the electronic device that performs the data reduction and packing procedure 400, such as a component of the electronic device 100 connected to the processor 110 or GPU 111.

[0150] The serialized data stream 415 is a byte representation of one or more serialized data objects, including the object's data, information about the object's type, the type of data stored in the object, and additional metadata.

[0151] It will be appreciated that the serialized data stream 415 may be streamed to the data reduction and packing procedure 400 in the form of a byte representation or may be converted to a byte representation (e.g., by the serialized data source 410, the data reduction and packing procedure 400 or another procedure / device). It will be appreciated that the serialized data objects in the serialized data stream 415 may be obtained from regions of memory (i.e., memory locations) using memory addresses.

[0152] It will be appreciated that the information contained in serialized data stream 415 may depend on the type of serialization language. In one or more alternative embodiments, the data in serialized data stream 415 may have been serialized using an object-oriented programming language.

[0153] As one non-limiting example, for the JAVA programming language, the serialized data stream 415 may include a header including a version number, an object, a class description, a class name length, a string, a class serial version identifier, supported serialization, a number of fields, a field type, a field name length, a string value, a superclass description, a class name length, a string value, a class hierarchy, etc. It will be appreciated that one or more embodiments of the present invention may also be used with interpreted languages.

[0154] The serialized data stream 415 represents one or more (i.e., at least one) serialized data objects, each having a serialized object data type. The serialized data objects may contain other objects, controls, numbers, arrays, strings, and / or characters.

[0155] Serialized object data types may include one of the following: lists (also known as collections or arrays), objects (containing object properties and object values ​​also known as key-value pairs), and defined values ​​(also known as leaf values) (e.g., floats, booleans, integers, etc.).

[0156] As a non-limiting example, in the case of the JSON language, an object literal is represented as a {key;value} pair, where the key is a string and the value is another JSON object (i.e., number, string, array, object, boolean, null). Further details regarding JSON compatibility standards are available in the RFC8259 standard documentation and the ISO / IEC21778:2017 standard documentation, both of which are incorporated by reference. JSON text is formed from Unicode code points, and in the open ecosystem, JSON uses UTF-8 for character encoding (with UTF-16 surrogate pairs for characters not in the Basic Multilingual Plane).

[0157] As a non-limiting example, for the JSON language, the default includes six structural tokens (separated by semicolons): "[" (open square bracket); "{" (open curly bracket), "]" (close square bracket); "}" (close curly bracket); ":" (colon); and "," (comma). The JSON format includes the literal name tokens true, false, and null, and insignificant whitespace (which is allowed before or after any of the six structural characters: %x20 (space); %x09 (horizontal tab); %x0A (line feed or new line); %x0D (carriage return)).

[0158] As a non-limiting example, a JSON value may be one of an object, an array, a number, a string, true, false, or null. It will be appreciated that in the JSON language, an object structure is represented as a pair of curly bracket tokens surrounding zero or more name / value pairs, and an array structure is a pair of square bracket tokens surrounding zero or more values.

[0159] Serialized Data Reader The serialized data reader 420 is specifically configured to (i) obtain an input serialized data stream 415, (ii) generate a plurality of serialized data blocks 430 using the serialized data stream 415, and (iii) transmit the plurality of serialized data blocks 430, which are processed using a mutable reduction matrix 440.

[0160] In one or more embodiments, the serialized data reader 420 is executed by the server 220 during the data reduction and packing procedure 400. In one or more other embodiments, the serialized data reader 420 is executed by a separate processor that receives the serialized data stream 415 and generates and transmits a number of serialized data blocks 430 to the server 220.

[0161] In the context of the present invention, serialized data reader 420 reads serialized data stream 415 sequentially, and the size of the serialized data stream or the amount of data in the serialized data stream is not known in advance (i.e., serialized data stream 415 includes an additively infinite sequence of elements and is used to represent data elements that become available over time). Thus, serialized data stream 415 ends when the current session ends (or when there is an interruption in the current session). As one non-limiting example, serialized data stream 415 may include one or more JSON (or other serialized data format) files.

[0162] The serialized data reader 420 is configured to generate fixed size blocks using the serialized data stream 415 to obtain a number of serialized data blocks 430. It will be appreciated that the serialized data stream 415 may be divisible into a number of serialized data blocks, with each data block having a predetermined fixed size. As a non-limiting example, a given data block may have a size of 64 kilobytes.

[0163] As a non-limiting example, in the JSON language, a JSON object in a serialized data stream may be represented by blocks as detailed in Table 1 below. [Table 1]

[0164] Upon receiving the serialized data stream 415, the serialized data reader 420 continuously outputs a number of serialized data blocks 430. The data reduction and packing procedure 400 is configured to perform slicing of the number of serialized data blocks 430 by applying a mutable reduction matrix 440 to the number of serialized data blocks 430.

[0165] reduced matrix The reduced matrix 440 is stored in memory and is obtained by the data reduction and packing procedure 400 for use on multiple serialized data blocks 430 .

[0166] The data reduction and packing procedure 400 specifically (i) receives as input a given serialized data block of a plurality of serialized data blocks 430 from a serialized data reader 420, (ii) slices the given serialized data block into sliced ​​values ​​or sliced ​​controls associated with an index and transmits them for bit packing, and (iii) uses a reduction matrix 440 to recursively call the reduction matrix 440 until the plurality of serialized data blocks are completely processed (e.g., until the end of the stream or the end of the current session).

[0167] The data reduction and packing procedure 400 uses a reduction matrix 440 to perform a reduction based on the size of the elements in the data block 430. The reduction matrix 440 uses one or more variable length coding strategies to map the serialized data block to a variable number of bits which are then bit packed for further compression. The data reduction and packing procedure 400 continuously applies the reduction matrix 440 to data blocks successively obtained from the serialized data stream 415.

[0168] In one or more embodiments, reduction matrix 440 may be implemented as a polymorphic injection matrix that is used to perform slicing and reduction of the input serialized data based on the data type and context within the input serialized data block through the use of bit-aligned dynamic types. Reduction matrix 440 leverages standardized data structures within the data serialization format to generate the index.

[0169] The reduction matrix 440 is predefined by the operator of the present invention based on a predefined condition. The predefined condition may be based on a predefined serialized data type and / or a serialized data format. Thus, when the reduction matrix 440 is populated with a certain condition, the reduction matrix 440 can use a predefined reduction technique or update itself to refer to and execute an appropriate reduction technique, i.e., inject an appropriate reduction technique to obtain sliced ​​data. The reduction matrix 440 dynamically assigns reduction techniques based on the serialized sliced ​​data type and / or the serialized data format. A set of reduction techniques may be provided for a predefined condition.

[0170] It will be appreciated that in one or more embodiments, during execution, reduction matrix 440 does not evaluate the conditions itself, but rather performs polymorphic calls that are evaluated directly, thereby enabling reduced processing time that governs data reduction (i.e., compression). Reduction matrix 440 can replace a given reduction technique with another given reduction technique employed by modifying a custom in-memory structure that contains the index to write the other given reduction technique employed.

[0171] In one or more embodiments, an appropriate reduction technique can be obtained from the compression algorithm database 365 (FIG. 3) based on certain conditions; for example, when populated with a given serialized data format, the reduction matrix 440 can reference a particular compression technique.

[0172] In one or more embodiments, reduction matrix 440 may be in the form of a two-dimensional (i.e., two-level) array containing key:values, where the values ​​are processes, i.e., references (i.e., pointers), to reduction code blocks that may be stored in reduction matrix 440 itself or may be external (i.e., retrieved from memory or retrieved from the packed data stream itself). Because reduction matrix 440 is mutable and dynamic, the size of reduction matrix 440 may change at runtime during execution when iterating over serialized data block 410.

[0173] In one or more embodiments, the reduction matrix 440 is dynamic and mutable and operates by injection, i.e., the reduction matrix 440 is dynamically updated and modified in real time during the data reduction and packing procedure 400 so that the appropriate reduction technique can be invoked based on conditions (i.e., format or data type) when operating on the serialized data block 430.

[0174] In one or more alternative embodiments, the reduction matrix 440 may be static.

[0175] In one or more embodiments, reduction matrix 440 may be implemented as a mutable injected reduction matrix. In the context of the present invention, the injection utilized is similar to the concept in Inversion of Control (IoC) programming principles, where the code to be executed is modified at run-time when reduction matrix 440 is populated with predefined data types and / or data formats. Thus, reduction matrix 440 applies slicing and reduction techniques to multiple data blocks 430 based on the associated predefined data types and / or predefined data formats.

[0176] In one or more embodiments, reduction matrix 440 is configured to encode handling operation metadata using an overflow method for data serialized in JSON by mapping operations to a predefined table utilizing multiple possible unused IDs using an offset / overflow method, as detailed in Table 3 below.

[0177] As a non-limiting example, for a date field in a JSON data block where a string may contain a #### / ## / ## type field, reduction matrix 440 may process strings according to a predefined "date" type and reference a specific compression technique based on the assumptions that for a given context and transmitted sequence, dates are typically represented in a range of 64 or 128 years for a given transmitted sequence, months have fewer than 16 possible values, days have fewer than 32 possible values, and there can be only one date format, thereby allowing optimization of compression of such serialized data sequences in an adaptive manner.

[0178] As another non-limiting example, the age of an individual in an object may be represented using a number (e.g., an int16 or int32 in some languages). However, the range of an individual's age is typically between 0 and 100, requiring only 8 bits to represent. Thus, the "age" formatting conditions and reduction techniques may be stored accordingly in the reduction matrix 440.

[0179] The reduction matrix 440 can avoid redundant data field names when an instance exists within a list of objects.

[0180] The reduction matrix 440 dynamically operates on the input serialized data blocks 430 in real time by verifying conditions on each serialized data type and adaptively performing slicing of the data blocks, i.e., depending on the results of the conditions and the serialized data type, among others, to obtain multiple sliced ​​portions. The reduction matrix 440 operates recursively on the input serialized data stream blocks to generate optimal data structures that reduce memory space while preserving the information stored in the serialized data stream, thereby improving transmission speed. The reduction matrix 440 works by dynamic injection of "in-memory" mutable processes, which can be injected at any time whenever some condition is met during the processing of the data. As a result, the reduction matrix 440 keeps the handling of data simple, powerful, flexible, and efficient.

[0181] The reduction matrix 440 performs slicing of the input serialized data blocks 430 to generate data slices (also known as sliced ​​data portions), where each data slice is a data structure containing elements of the same type. The reduction matrix 440 dynamically assigns reduction techniques based on the type of data slice and the given context (where the context includes delimiters and conditions) by using bit-aligned dynamic data types. The reduction matrix 440 performs reduction based on the size of the data elements in the block, not the number of data elements. The reduction matrix 440 has access to one or more predefined and / or one or more injected reduction methods, which allows the reduction matrix to process information according to its own rules, thereby optimally custom compressing the streamed serialized data sequence.

[0182] The reduction matrix 440 performs slicing and reduction by successively and recursively operating on multiple data blocks and the resulting data slices until the serialized data stream 415 is terminated.

[0183] In one or more embodiments, reduced matrix 440 includes multiple reduced submatrices, each defined for a corresponding data type, and a given submatrix in the multiple submatrices can reference (i.e., call) another given submatrix in the multiple submatrices when the given submatrix is ​​populated with the data type for which it is defined.

[0184] The reduction matrix 440 initializes the index.

[0185] Index When operating on the serialized data block 430, the reduction matrix 440 dynamically generates an index (not shown) in memory and includes an indicator of the index as part of the incoming data stream 474. The index is generated and maintained in memory during the data reduction and packing procedure 400 and only for the duration of the session. During reconstruction of the incoming data stream 474 (i.e., during reconstruction by the reverse unpacking and reconstruction process), the index is dynamically reconstructed directly from the bit-packed data stream 474, and the processor performing the reverse unpacking and reconstruction process does not require prior knowledge of the index (i.e., the index is not stored or transmitted in the bit-packed data stream 474 prior to coding). The index is passed in the incoming data stream 474, which allows the index to be referenced with fewer bits than at the start of the incoming data stream 474 during the current session. It will be appreciated that this improves security since the bit-packed data stream 474 uniquely (or nearly uniquely) encodes the index for the current session based on the contents of the source serialized data stream.

[0186] An index is a container of mutable types that contains immutable elements of the form key:value type. It will be appreciated that an index may contain multiple indexes (i.e., sub-indexes), each associated with a separate index context.

[0187] An index context should be understood as the current index (i.e. subindex) in use with associated individual metadata (e.g. last entered URL, first entered date or any information that may be added to facilitate their processing). An index context can also be interpreted as the state of an automaton. An index context is a reference to an index (i.e. subindex).

[0188] Each index context is uniquely identified and is a global state by the data reduction and packing procedure 400. For a given sliced ​​serialized data portion, the index context can vary and depends on the sliced ​​serialized data type.

[0189] The index context may include one of a default index context, a key index context (i.e., an index context based on a sliced ​​key name or position), and an array index context (i.e., an index context based on a sliced ​​array name), which are described in more detail herein below.

[0190] The index and index context may be represented using a HashMap, vector, and / or array that is temporarily stored in memory during the current session. Thus, when there is an interruption in the transmission of data during the current session or when the current session ends, the index is deleted, but must be recreated during subsequent sessions. It will be appreciated that this allows the size of the index specific to the current session to be minimized (rather than adding information to a global index stored for multiple sessions).

[0191] It will be appreciated that the size of the index affects the number of bits that are transmitted and read, and the index size is used to dynamically calculate the bit array length instead of transmitting the bit array length.

[0192] In the context of the present invention, slicing the data reduces the index size, thereby reducing the number of bits required to represent the reference ID. It will be appreciated that the number of bits need not be the number of bits referenced.

[0193] A non-limiting example of a portion of reduced matrix 440 for string and character data types is provided below in the PYTHON programming language code in Table 2 below. [Table 2]

[0194] In one or more embodiments, for a given iteration, the reduced matrix 440 may result in one of a sliced ​​object 442 and a sliced ​​array 448 during the data reduction and packing procedure 400 .

[0195] Sliced ​​Objects Sliced ​​objects 442 are in the form of key:value pairs, where the key is a unique identifier and is associated with a value having a distinct data type (e.g., string, array, object, number, Boolean, null). In one or more embodiments, the key may be in the form of a string (i.e., a sequence of characters).

[0196] If the reduction matrix 440 outputs a sliced ​​object 442, the data reduction and packing procedure 400 iterates over the contents (i.e., nested contents) of the sliced ​​object 442 by returning the sliced ​​object 442 as an input to the reduction matrix 440. It will be appreciated that the sliced ​​object 442 may include one or more nested objects, numbers, strings, characters, controls, etc., that are successively sliced ​​by the reduction matrix 440.

[0197] The data reduction and packing procedure 400 passes along a key 452 associated with the sliced ​​object 442. It will be appreciated that the key 452 is unique and allows for the identification of a value (object property) associated with the key 452. The key 452 is one type of sliced ​​value that is processed by the reduction matrix 440.

[0198] The data reduction and packing procedure 400 sets an index context based on the key 452. In one or more embodiments, the data reduction and packing procedure 400 sets an index context to the name (i.e., expressed as a string) of the key 450. The index context should be understood as the currently used index with associated individual metadata (e.g., the last entered URL, the first entered date, or any information that may be added to facilitate their processing). As described below, if the key 452 is not present in the index, the data reduction and packing procedure 400 adds the key 452 to the index, and if the key 452 is present in the index, the data reduction and packing procedure 400 references the key 452 and changes the context so that the key 452 can be handled further.

[0199] The data reduction and packing procedure 400 passes the keys 450 as sliced ​​values ​​444 .

[0200] Sliced ​​arrays Once the reduction matrix 440 has obtained a sliced ​​array 448, the data reduction and packing procedure 400 iterates over the contents of the sliced ​​array 448 by returning the sliced ​​array 448 as an input to the reduction matrix 440. It will be appreciated that the sliced ​​array 448 contains elements that may be nested data types such as objects, numbers, strings, characters, controls, arrays, etc. The elements of the sliced ​​array 448 are returned to the reduction matrix 440 for further slicing and reduction. The reduction matrix 440 iterates over each of the elements in the sliced ​​array 448, nesting the elements in an iterative manner.

[0201] The reduced matrix 440 processes each given element having a data type in the sliced ​​array 448 accordingly by iterating over the given element and its nested elements in the sliced ​​array 448 .

[0202] In one or more embodiments, the data reduction and packing procedure 400 sets an index context for the sliced ​​array 448. It is understood that setting an index context for the sliced ​​array 448 is optional.

[0203] In one or more embodiments, for a sliced ​​array of strings (a list of text values), reduction matrix 440 detects the array based on the array context, i.e., the array starts with "[" and ends with "]". Reduction matrix 440 passes the string between the array delimiters to a string reduction technique adapted to reduce the string data type (values). In one or more embodiments, the string can be encoded using control overflow. The string reduction technique can index characters in the string as it is being transmitted and sends the characters back as input to reduction matrix 440 for further slicing.

[0204] In one or more embodiments, for sliced ​​sequences of multiple characters (strings), reduction matrix 440 detects sequences based on context, i.e., the string delimiter "and." Reduction matrix 440 passes the characters between the string delimiters to a character reduction technique, e.g., a UTF-8 reduction technique.

[0205] At each given iteration, reduction matrix 440 outputs one of sliced ​​values ​​444 and sliced ​​controls 446. It should be understood that sliced ​​values ​​and sliced ​​controls are "base" or "primitive" elements output by reduction matrix 440, and in some cases may also be associated with a parent element (i.e., may be derived from other data types).

[0206] Sliced ​​Values For a given iteration of the data reduction and packing procedure 400 , the reduced matrix 440 outputs sliced ​​values ​​444 .

[0207] In one or more embodiments, sliced ​​value 444 includes value metadata (not shown). Value metadata provides information about sliced ​​value 444 and / or allows for definition and / or description of sliced ​​value 444. Value metadata may include, for example, flag bits, prefix bits, etc. As a non-limiting example, for an API that supports date formats with different types of separators, e.g., MM / DD / YYYY and MM-DD-YY, 10212016+0 padded to one bit may be used to encode the date "10 / 21 / 2016" and 01012000+1 padded to one bit may be used to encode "01-01-2000", with the metadata corresponding to the format used with 0 bits for MM / DD / YYYY and 1 bit for MM-DD-YY. As another non-limiting example, for a list of dates such as ["Filming Ended: 10 / 10 / 1996", "Aired: 01 / 01 / 1997"], ["Filming Ended: 08 / 01 / 2011", "Aired: 11 / 07 / 2012", "Returned: 11 / 09 / 2012"], 0 could be used to qualify the filming end date, 1 could be used to qualify the aired date, and 2 could be used to qualify the return date. It will be appreciated that metadata can also be used to indirectly qualify elements to select a given injection method, e.g., 0 for free form text, 1 for URL, 2 for free form text with embedded numbers, 3 for CSV, etc.

[0208] In one or more embodiments, value metadata allows for the specification of data that is not included in the serialized data stream 410 .

[0209] For a given iteration, the data reduction and packing procedure 400 determines whether a sliced ​​value 444 requires an index ID. To do so, the data reduction and packing procedure 400 verifies whether the sliced ​​value 444 exists in the index.

[0210] If the data reduction and packing procedure 400 determines that the sliced ​​value 444 is not present in the index and requests an index ID, the data reduction and packing procedure 400 adds the sliced ​​value to the index as a new value.

[0211] As explained above, an index should be interpreted as a mutable container containing immutable elements of the form key:value. If the index is empty, no sliced ​​value 444 is referenced and the IDs of the sliced ​​controls can be used according to a predefined table of possible operations. In other words, if the index is empty for the current context, no new IDs are requested and are set to 0 by default.

[0212] The data reduction and packing procedure 400 is configured to use bit packing to bit pack the sliced ​​values ​​444. The bit packing procedure can be performed by the bit compressor plug-in instance 335 using triggered injection. Bit packing techniques are known in the art.

[0213] To add a sliced ​​value 444 to the index as a new value, the data reduction and packing procedure 400 bit-packs an unused ID associated with the given sliced ​​value 444 using a first number of bits n based on the current index length and encodes the sliced ​​data value 444 using a second number of bits x, thereby obtaining a given bit-packed sliced ​​value 462.

[0214] Bit n of the first number represents the minimum number of bits that can store the next unused ID, which corresponds to the current index length + 1. Bit x of the second number is the minimum number of bits required that can store the indexed value.

[0215] Referring briefly to FIG. 5, a new value 510 is shown added to the index using a first number of bits 514 and a second number of bits 518 .

[0216] If the data reduction and packing procedure 400 determines that the sliced ​​value 444 is present in the index, then the data reduction and packing procedure 400 references the ID present in the index by using bit n of the first number, in other words, no new ID is needed.

[0217] The data reduction and packing procedure 400 bit-packs the reference ID using the first number of bits n to obtain another given bit-packed sliced ​​value 466 .

[0218] In one or more embodiments, the reference ID may be associated with or include control metadata, which may be the sole control metadata or an offset to the last index ID.

[0219] Referring briefly to FIG. 5, an existing value 520 is indicated in the index using bits 524n of a first number.

[0220] Referring back to FIG. 4, for another given iteration, the data reduction and packing procedure 400 outputs sliced ​​controls 446 .

[0221] Sliced ​​Controls It should be noted that the sliced ​​controls 446 are not indexed and are not associated with values ​​in multiple data blocks 430. The sliced ​​controls 446 are inherent to the reduction technique used by the reduced matrix 440. The sliced ​​controls 446 are used to express logical operations.

[0222] In one or more embodiments, the sliced ​​control 446 may indicate a division of a portion of the serialized data stream depending on the type of serialization format, however, this need not be the case in each and every embodiment of the invention.

[0223] In one or more embodiments, the sliced ​​control 446 may be a reference to a reduction method that needs to be injected by the reduction matrix 440 based on the data type. Thus, during decoding, the sliced ​​control 446 may indicate that a given data type is present and that a corresponding appropriate method for reading the given data type must be used. It will be appreciated that this appropriate method can be referenced in the incoming data stream, which can then be read by a corresponding decoding matrix during the unpacking and reconstruction procedure, which updates itself with the above method and applies the method for decoding data associated with the given data type during decoding.

[0224] As a non-limiting example, in the case of the JSON language, sliced ​​controls 446 can be structural tokens within JSON (i.e., defining objects, arrays, or strings) and control characters within strings within JSON, such as "[", "]", "{", "}", ":", and ",".

[0225] In the context of the present invention, sliced ​​controls are handled using an overflow method. The overflow method is used to encode handling operation metadata by utilizing unused IDs to map operations to a predefined table. The overflow method uses an empty index number (of the same number of bits as the current index) to reduce the number of bits required to encode the default value. It will be appreciated that the predefined table is dependent on the serialized data language.

[0226] Table 3 provides a sample overflow scheme for serialized data in JSON to optionally encode handling operation metadata by mapping operations to predefined tables using an offset / overflow scheme leveraging possible unused IDs. [Table 3]

[0227] If the offset cannot be stored in the number of currently available bits, the maximum value is assigned as the key ID, and any further overflow offset is encoded in the minimum number of bits that can store the delta between the current index position and the last element in the calculation table.

[0228] As a non-limiting example, as shown in Figure 6, if the last ID + overflow offset (i) can be stored in the same number of bits as the last ID, and (ii) is not the maximum ID that can be stored in this number of bits, then the sum can be passed as the index ID. In Figure 6, the maximum ID 602 ​​is equal to 7 (111 in binary representation) and the last ID 604 is equal to 5 (101 in binary representation). The string append operation (shown in Table 3 above) adds a +1 offset to the last ID, resulting in a list if and overflow offset 608 of 5 (last ID) + 1 (string append operation) = 6 (101 in binary representation).

[0229] As another non-limiting example, if the Last ID + Overflow Offset is greater than or equal to the Max ID, then (i) the Max ID and Single Control - Number of Unused IDs is the minimum number of bits that can store the Max Single Offset - Number of Unused IDs, as shown in Figure 7. In Figure 7, the Max ID 702 is equal to 7 and the Last ID (101 in binary representation) is equal to 5 (101 in binary representation).

[0230] A numeric addition operation (shown in Table 3 above) adds a single control of 1 to the empty index, which results in a maximum ID of 7 (111 in binary representation) and a single control (number of unused IDs) of 1-1=0 (000 in binary representation). For an array addition operation (shown in Table 3 above), the maximum ID is 7 and the single control (number of unused IDs) for the end of the array is equal to 7-1=6 (110 in binary representation).

[0231] Table 4 presents a sample string bit-packing overflow table with a sample UTF-8 string technique. [Table 4]

[0232] It should be noted that 16 / 21-bit characters are preceded by a bit (0 for a 16-bit character, 1 for a 21-bit character) that quantifies the length of the 16 / 21-bit character.

[0233] Table 5 shows the character values ​​encoded with the significant bit for the UTF-8 binary format. [Table 5]

[0234] Table 6 presents a sample numeric technique for optionally encoding JSON numbers by utilizing a limited character set for each number part according to the rfc7159 standard. [Table 6]

[0235] Using the above technique, up to seven characters are indexed, leaving most of the reference stored in three bits, and leaving the eighth position free as an escape reference to the list of stored non-indexed characters.

[0236] As another non-limiting example, the following code represents a JSON object that includes an "ID List" object and a "Properties" object. JPEG2024546443000009.jpg108166

[0237] 8 further illustrates the above JSON object, which contains two objects. The first object has an "ID List" key 604 and an array 606 as a value, which contains five different MD5 hashes. The second object 624 has a "Properties" key and five objects 626 as values, each of which has a key that points to an individual MD5 hash in the ID list. Each of the five objects contains two individual objects: a first individual object key "active" with an individual value, and a second individual object key "links" that is an array of size 2.

[0238] In this example, the MD5 hash context can be detected by reduction matrix 440 based on the exemplary conditions that if a string has 32 characters and has more than four alphabetic to numeric switches, the string is assumed to be an MD5 hash. The MD5 hash can be referenced by using 3 bits (because the size of the index + 1 = 6, which is smaller than the size of 8 (3 bits)). Therefore, in this example, 3 bits are used to represent the MD5 hash instead of 256 bits, which corresponds to a reduction ratio of 85:1.

[0239] Bit-packed Data Aggregator The bit-packed data aggregator 470 is configured to aggregate all the bit-packed sliced ​​values ​​(i.e., including the given bit-packed sliced ​​value 462 and / or another given bit-packed sliced ​​value 466 generated in different iterations while processing the multiple data blocks 430) to obtain and transmit an output bit-packed data stream 474.

[0240] It will be appreciated that the index is not present in the output bit-packed data stream 474, but all the information necessary to dynamically recreate the index is stored in the bit-packed data stream 474. The size of the dynamically reconstructed index (which increases during the reconstruction process) is used to determine how many bits need to be read to reconstruct the serialized data stream.

[0241] In one or more embodiments, the output bit-packed data stream 474 is transmitted over the communications network 240 of FIG. 2 (eg, the Internet 340 of FIG. 3).

[0242] In one or more other embodiments, the output bit-packed data stream 474 may be transmitted to a processor that performs the data reduction and packing procedure 400, i.e., another processor operatively connected to another component within the same housing.

[0243] It will be appreciated that the bit-packed data stream 474 is decoded by a receiving computing device (e.g., a given one of the client devices 210). The receiving computing device can decode the bit-packed data stream 474 by dynamically reconfiguring the index, and the number of bits to be read by the receiving device to reconstruct the serialized data stream can be determined based on the current index size. Decompression of the bit-packed data stream 474 can be performed by using the reverse order of the data reduction and packing procedure 400 (i.e., the data unpacking and reconstruction procedure) to verify conditions and globally update the method for a given context or for a given type. Decompression includes reading n bits (representing the number of elements + 1 for the current context). If the ID is present in the index, the reconstruction procedure writes the value to the decompressed data stream value, and if the ID is not present in the current index context, the reconstruction procedure reads the value according to the conditions associated with the reduction technique for the current index context and writes the value to the index context and / or the decompressed data stream value. The reconstruction procedure validates the conditions and globally updates the reverse order reduction techniques for a context or data type.

[0244] Table 7 below presents the results of testing 47 public APIs of one or more embodiments of the present invention implemented in the PYTHON programming language. [Table 7]

[0245] It will be understood that the processing speed depends on the programming language used and the implementation of the code. The developers of the present invention have also implemented the present invention in the RUST programming language, which allows achieving processing speeds of 60-150 Mb / s (versus 1-5 Mb / s processing speeds in PYTHON).

[0246] FIG. 9 illustrates a finite state diagram 800 of a default data reduction and packing procedure in accordance with one or more non-limiting embodiments of the present invention.

[0247] 10 shows the finite state diagram of the data reduction and packing procedure of FIG. 9 in a modified state. The same reference numbers are assigned to the same states as in FIG.

[0248] In one or more embodiments, finite state diagram 800 represents data reduction and packing procedure 400 of Figure 4. Finite state diagram 800 represents the transitions of reduced matrix 830 (similar to reduced matrix 440). The states of reduced matrix 830 are represented by array for slicing state 832, literal for slicing state 834, key for slicing state 836, free form string for slicing state 838, date string for slicing state 840, number for slicing state 842, and object for slicing state 844.

[0249] An input serialized data block 802 is received by a reader which reads data 803 and outputs data block 804 .

[0250] The reduction matrix 830 receives the data block 804 and obtains one of the array 806 , the literal 808 , the key 810 , the number 816 , and the object 818 .

[0251] If the mutability condition 820 is satisfied (FIG. 10), the reduced matrix 830 can yield one of an array 806, a literal 808, a key 810, a free-form string 812, a date string 814, a number 816, and an object 818.

[0252] The array 806 changes the state of the reduced matrix 830 to an array state for slicing 832 and outputs a sliced ​​array 846 .

[0253] Literal 808 changes the state of reduced matrix 830 to slicing literal state 834 and outputs sliced ​​literal 848 .

[0254] The key 810 modifies the state of the reduced matrix 830 to the state of the slicing key 836 to output the sliced ​​key 850 .

[0255] If the mutability condition 820 is satisfied, the default state of the reduced matrix 830a (FIG. 9) is changed to a modified state 830b (FIG. 10).

[0256] If mutability condition 820 is satisfied (FIG. 10), free-form string 812 changes the state of reduced matrix 830 to the state of free-form string for slicing 838 and outputs sliced ​​free-form string 852.

[0257] If the mutability condition 820 is satisfied (FIG. 10), then the date string 814 changes the state of the reduced matrix 830 to the date string for slicing state 840, causing the sliced ​​date string 854 to be output.

[0258] Number 816 changes the state of reduced matrix 830 to the state of number for slicing 842 and outputs sliced ​​number 856 .

[0259] The object 818 changes the state of the reduced matrix 830 to the slicing object state 844 and outputs the sliced ​​object 858 .

[0260] Bit packer 864 receives sliced ​​literals 848, sliced ​​keys 850, sliced ​​free form strings 852, sliced ​​date strings 854, sliced ​​numbers 856, and sliced ​​objects 858 as inputs to sliced ​​array 846 and outputs bit-packed data 866.

[0261] The aggregator 868 receives the bit-packed data 866 and outputs the input data block 840 .

[0262] A sender 872 transmits an input data block 840 .

[0263] Show me how FIG. 11 illustrates a flow chart of a data reduction and bit packing method 900 in accordance with one or more non-limiting embodiments of the present invention.

[0264] In one or more embodiments, server 220 comprises a processor, such as processor 110 and / or GPU 111, operably connected to a non-transitory computer-readable storage medium, such as solid-state drive 120 and / or random access memory 130, that stores computer-readable instructions. The processor is configured or operable to perform method 900 upon execution of the computer-readable instructions.

[0265] The method 900 is performed for a current session in which a serialized data sequence is being streamed, but it is not known in advance when the serialized data sequence will end.

[0266] According to process step 902, a processor receives a serialized data sequence. In one or more embodiments, the serialized data sequence is the serialized data stream 410. The serialized data sequence is a byte representation of a number of serialized data objects that includes the object's data, information about the object's type, the type of data stored within the object, and additional metadata. It will be appreciated that serialized objects may be nested within other serialized objects within the serialized data sequence.

[0267] The serialized data sequence is streamed (i.e., made available over time) and multiple serialized data objects are processed one by one. As a non-limiting example, the serialized data sequence may include one or more JSON files having multiple serialized objects. The method 900 runs until the current session ends (i.e., until the end of the data stream) or is interrupted (e.g., due to loss of communication).

[0268] In one or more embodiments, the serialized data sequence is obtained in response to a request from a client device, such as one of the client devices 210. In one or more other embodiments, the serialized data sequence is obtained from another component within the same housing as the processor.

[0269] In one or more embodiments, the serialized data sequence is separated into a plurality of serialized data blocks 430. Each of the plurality of serialized data blocks 430 has a predetermined size. It is understood that the serialized data blocks are generated one by one as the serialized data sequence is received by the processor.

[0270] According to process step 904, the processor iteratively applies reduction matrix 440 to each of the serialized data sequence to obtain a plurality of sliced ​​values. It will be appreciated that since the serialized data sequence is streamed, the number of iterations is unknown a priori and process step 904 is applied continually to the serialized data sequence as it is received.

[0271] In one or more embodiments, prior to processing step 904, the processor initializes the reduced matrix 440 by obtaining the reduced matrix 440 from a non-transitory storage medium operably connected to the processor.

[0272] In one or more embodiments, reduction matrix 440 is a mutable reduction matrix (i.e., a two-dimensional array) that includes a set of predefined conditions based on the serialized data type and / or format. Each of the predefined conditions is associated with one or more slicing and reduction techniques. Reduction matrix 440 works by dynamic injection of mutable processes "in memory," such processes can be injected at any time during the processing of serialized data whenever some condition is met.

[0273] The processor initializes an index in memory, in one or more embodiments, the index is represented using one of a HashMap, a Vector, and an Array.

[0274] The reduction matrix 440 is continually applied to the plurality of serialized data blocks 430 until all elements present in the plurality of serialized data blocks 430 have been processed by the reduction matrix 440. Slicing the data reduces the index size, thereby reducing the number of bits required to represent the reference ID. It will be appreciated that the number of bits need not be referenced.

[0275] In one or more embodiments, reduction matrices 440 are defined for objects (corresponding to controls in the index), numbers (corresponding to values ​​in the index), strings (corresponding to values ​​in the index), keys (corresponding to values ​​in the index), literals (corresponding to values ​​in the index), and arrays (corresponding to values ​​in the index). Additionally, reduction matrices 440 may be defined for date strings (corresponding to values ​​in the index) and free form strings (corresponding to values ​​in the index). For objects, reduction matrix 440 obtains sliced ​​objects, for numbers, reduction matrix 440 obtains sliced ​​numbers, for strings, reduction matrix 440 obtains sliced ​​strings, for keys, reduction matrix 440 obtains sliced ​​keys, for literals, reduction matrix 440 obtains sliced ​​literals, and for arrays, reduction matrix 440 obtains sliced ​​arrays.

[0276] According to process step 906, for a given iteration of the plurality of iterations, the processor determines whether a corresponding given sliced ​​value of the plurality of sliced ​​values ​​exists in the index according to the reduction matrix 440. The processor proceeds to process step 908a or process step 908b.

[0277] According to process step 908 a , if the processor determines that the given sliced ​​value 444 is not present in the index, then the processor proceeds to process step 910 .

[0278] According to process step 910 , the processor adds the given sliced ​​value to the index and proceeds to process step 912 .

[0279] According to processing step 912, the processor bit packs an unused ID associated with the given sliced ​​value 444 using a first number of bits based on the current index length and encodes the sliced ​​data value using a second number of bits to obtain the given bit-packed sliced ​​value.

[0280] According to process step 908 b , if the processor determines that the given sliced ​​value 444 is present in the index, the processor proceeds to process step 914 .

[0281] According to process step 914 , if the given sliced ​​value is present in the index, the processor references the given sliced ​​value in the index 444 and proceeds to process step 916 .

[0282] According to processing step 916, the processor bit-packs the given sliced ​​value using a third number of bits representing the reference ID to obtain a given bit-packed sliced ​​value.

[0283] The processor continues to apply steps 904-918 when populated with sliced ​​values.

[0284] In one or more embodiments, after performing processing step 904, for a given iteration, the processor obtains a sliced ​​object 442.

[0285] In one or more embodiments, after performing process step 904, for a given iteration, the processor obtains a sliced ​​array 448.

[0286] According to processing step 918, the processor aggregates a plurality of bit-packed sliced ​​values ​​including a given bit-packed sliced ​​value obtained in a given iteration to obtain a bit data packed data sequence.

[0287] In one or more embodiments, after the mutable reduction matrix 100 is applied, the processor performs an aggregation of the bit-packed sliced ​​values ​​(i.e., each resulting from a corresponding iteration) and adds the bit-packed sliced ​​values ​​to a bit-packed data sequence in the form of a bit-packed data stream 474. The bit-packed data stream 474 stores all the information required to reconstruct the serialized data sequence. The bit-packed data stream 474 allows for dynamic reconstruction of the index (which increases during the reconstruction process), and the size of the index is used to determine how many bits need to be read to obtain the serialized data sequence. Thus, the index is not transmitted separately from the bit-packed data stream 474, but the information required to reconstruct the index is present in the bit-packed data stream 474.

[0288] According to process step 920, the processor transmits the bit-packed data sequence in the form of a bit-packed data stream 474. The bit-packed data sequence is generated and transmitted serially and continuously as a stream, i.e., process steps 904-920 are executed as long as the serialized data sequence is received by the processor in process step 902.

[0289] In one or more embodiments, the processor transmits the bit-packed data stream 474 over the communications network 240. In one or more other embodiments, the processor transmits the bit-packed data stream 474 to another component operatively connected to the processor.

[0290] It will be appreciated that the processor does not transmit a serialized data stream, but rather transmits a bit-packed data stream 474 that contains all the information necessary to reconstruct the data represented in the serialized data stream upon request to the device, and reads the bit-packed data stream 474 and uses it to reconstruct the serialized data sequence step by step.

[0291] In one or more other embodiments, the processor transmits the bit-packed data stream to another processor in the same housing as the processor performing method 900. In one or more alternative embodiments, the processor stores the bit-packed data stream 474 in a non-transitory recording medium, and the bit-packed serialized data can be unpacked and reconstructed upon retrieval.

[0292] The method 900 ends.

[0293] In one or more embodiments, method 900 terminates when the serialized data sequence ends or there is an interruption in receiving serialized data for the current session (e.g., due to a component / communication malfunction). Method 900 is applied in real-time as serialized data is being received in a stream and does not require prior knowledge of the complete structure of serialized data objects in the serialized data sequence.

[0294] One or more embodiments of the present invention operate over a transmission channel and can be adapted for use between components such as computing devices, routers, NVMe disk drivers, IoT devices, Docker or Kubernetes containers, mobile app services, NoSQL database engines, and cloud services.

[0295] It should be clearly understood that not all technical advantages mentioned herein need to be enjoyed in each and every embodiment of the present invention. For example, an embodiment of the present invention may be practiced such that a user does not enjoy some of these technical advantages, while other non-limiting embodiments may be practiced such that a user enjoys other technical advantages, or none at all.

[0296] Some of these steps and signal transmission and reception are well known in the art and therefore have been omitted in certain portions of this specification for simplicity. Signals can be transmitted and received using optical means (such as fiber optic connections), electronic means (such as electronic means using wired or wireless connections) and mechanical means (such as pressure-based mechanical means, temperature-based mechanical means or mechanical means based on any other suitable physical parameter).

[0297] Modifications and improvements to the above implementations of the invention may become apparent to those skilled in the art. The above description is intended to be illustrative, not limiting.

Claims

1. 1. A method for dynamically and adaptively reducing a streamed serialized data sequence to obtain a bit-packed data stream, the method being performed by at least one processor; receiving a streamed serialized data sequence including a plurality of serialized data objects, each having a serialized object data type; iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​based at least on the serialized object data type, wherein the iterative applying includes: Initialize the index, For a given iteration, determining whether a corresponding given sliced ​​value among the plurality of sliced ​​values ​​exists in an index; If the given sliced ​​value is not present in the index, adding the given sliced ​​value to the index, said adding comprising: Bit-packing an unused ID associated with the given sliced ​​value using a first number of bits based on a current index length and encoding the given sliced ​​value using a second number of bits, thereby obtaining a given bit-packed sliced ​​value. Adding Apply it in an iterative manner, including aggregating a plurality of bit-packed sliced ​​values, including the given bit-packed sliced ​​value, to obtain a bit-packed data stream; transmitting the bit-packed data stream including the index indicator and the given sliced ​​value; A method comprising:

2. 2. The method of claim 1, further comprising, before aggregating the plurality of bit-packed sliced ​​values ​​to obtain the bit-packed data stream: If the given sliced ​​value is present in the index, and further comprising referencing the given sliced ​​value, wherein the referencing comprises: bit-packing the given sliced ​​value using a third number of bits representing a reference ID to obtain the given bit-packed sliced ​​value. method.

3. 3. The method of claim 2, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​comprises obtaining a given sliced ​​object, determining an individual index context based on one of a position and a key associated with the given sliced ​​object, and passing the one of the position and the key as the sliced ​​value. method.

4. 4. The method of claim 3, passing the key as the sliced ​​value includes passing a sequence of characters representing the key as the sliced ​​value. method.

5. 10. The method of claim 1, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​includes determining a distinct current index context based at least on a format of the streamed serialized data sequence. method.

6. 10. The method of claim 1, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​comprises iteratively applying the reduction matrix to the sliced ​​objects to obtain different given sliced ​​values ​​associated with respective current index contexts. method.

7. 10. The method of claim 1, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​further comprises obtaining a given sliced ​​control that is not associated with a value in the serialized data sequence, the given sliced ​​control being processed using an overflow method. method.

8. 10. The method of claim 1, encoding the handling operation metadata based on a predetermined table, the predetermined table including, for each distinct data type and each distinct control, a distinct index offset to the last ID and a single control for a null index; method.

9. 10. The method of claim 1, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​to obtain a given sliced ​​array; and iteratively applying the reduction matrix to the streamed serialized data sequence to obtain further sliced ​​values. A method comprising:

10. 10. The method of claim 9, further comprising, after obtaining the given sliced ​​array: Determining an individual index context based on the given sliced ​​array. The method further comprises:

11. 10. The method of claim 1, the reduction matrix is ​​implemented as a mutable reduction matrix; method.

12. 1. A system for dynamically and adaptively reducing a streamed serialized data sequence to obtain a bit-packed data stream, comprising: at least one processor; a non-transitory storage medium containing computer-readable instructions operably connected to the at least one processor; Equipped with When the at least one processor executes the computer-readable instructions, receiving a streamed serialized data sequence including a plurality of serialized data objects, each having a serialized object data type; iteratively applying a reduction matrix to the streamed serialized data sequence to obtain a plurality of sliced ​​values ​​based at least on the serialized object data type, wherein the iterative applying includes: Initialize the index, For a given iteration, determining whether a corresponding given sliced ​​value among the plurality of sliced ​​values ​​exists in an index; If the given sliced ​​value is not present in the index, adding the given sliced ​​value to the index, said adding comprising: Bit-packing an unused ID associated with the given sliced ​​value using a first number of bits based on a current index length and encoding the given sliced ​​value using a second number of bits, thereby obtaining a given bit-packed sliced ​​value. Adding Apply it in an iterative manner, including aggregating a plurality of bit-packed sliced ​​values, including the given bit-packed sliced ​​value, to obtain a bit-packed data stream; transmitting the bit-packed data stream including the index indicator and the given sliced ​​value. Configured for system.

13. 13. The system of claim 12, wherein before the at least one processor aggregates the plurality of bit-packed sliced ​​values ​​to obtain the bit-packed data stream: If the given sliced ​​value is present in the index, and further configured to reference the given sliced ​​value, the reference comprising: bit-packing the given sliced ​​value using a third number of bits representing a reference ID to obtain the given bit-packed sliced ​​value. system.

14. 14. The system of claim 13, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​comprises obtaining a given sliced ​​object, determining an individual index context based on one of a key and a position associated with the given sliced ​​object, and passing the one of the key and the position as the sliced ​​value. system.

15. 15. The system of claim 14, passing the key as the sliced ​​value includes passing a sequence of characters representing the key as the sliced ​​value. system.

16. 13. The system of claim 12, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​includes determining a distinct current index context based at least on a format of the streamed serialized data sequence. system.

17. 13. The system of claim 12, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​comprises iteratively applying the reduction matrix to the sliced ​​objects to obtain different given sliced ​​values ​​associated with respective current index contexts. system.

18. 13. The system of claim 12, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​further comprises obtaining a given sliced ​​control that is not associated with a value in the streamed serialized data sequence, the given sliced ​​control being processed using an overflow method. system.

19. 13. The system of claim 12, the at least one processor is further configured to encode the handling operation metadata based on a predetermined table, the predetermined table including, for each distinct data type and each distinct control, a distinct index offset to a last ID and a separate control for a null index; system.

20. 13. The system of claim 12, iteratively applying the reduction matrix to the streamed serialized data sequence to obtain the plurality of sliced ​​values ​​comprises obtaining a given sliced ​​array, and iteratively applying the reduction matrix to the streamed serialized data sequence to obtain further sliced ​​values. system.

21. 21. The system of claim 20, wherein the at least one processor, after obtaining the given sliced ​​array, further configured to determine an individual index context based on the given sliced ​​array. system.

22. 13. The system of claim 12, the reduction matrix is ​​implemented as a mutable reduction matrix; system.

Images