Method and system for storing data structures

Abstract

The present invention relates to a method and system for storing a data structure. In one method, a processor partitions a data structure into a plurality of initial blocks. For each of the plurality of initial blocks, the processor determines a block key based on identifying a sequence of bits within the respective block that is unique in the data structure. The processor re-partitions the data structure into a plurality of final blocks, wherein the partitioning corresponds to locations of the identified sequences of bits.

Description

Technical Field

[0001] This invention relates to the field of data structures, and more specifically, to dividing a data structure into multiple blocks based on identifying a unique bit sequence within each block. Background Technology

[0002] Many applications are used to edit very large data structures, often exceeding several hundred megabytes. Such data structures can be JavaScript Object Notation (JSON) files, Rich Text Format (RTF) files, or code. For example, modern web-based editor applications are used to edit very large data structures. Therefore, saving and reloading data structures by these applications during the editing process often proves to be a time-consuming and computationally expensive process. Summary of the Invention

[0003] According to one embodiment of the present invention, a computer-implemented method, computer program product, and computer system are provided. A processor divides a data structure into a plurality of initial blocks. For each of the plurality of initial blocks, the processor determines a block key based on identifying a unique bit sequence within the data structure in the corresponding block. The processor then re-divides the data structure into a plurality of final blocks, wherein the division corresponds to the position of the identified bit sequence. Attached Figure Description

[0004] To better understand the invention and to more clearly show how the invention can be practiced, reference will now be made only by way of example to the accompanying drawings, in which:

[0005] Figure 1 This is a block diagram of an example system in which aspects of the illustrative embodiments can be implemented;

[0006] Figure 2 This is a flowchart of a method for storing data structures according to an embodiment;

[0007] Figure 3A and 3B This is a flowchart of a method for further processing the final block according to an embodiment;

[0008] Figure 4 This is a flowchart of a method for updating a data structure according to an embodiment;

[0009] Figure 5 This is a flowchart of a method for loading a data structure according to another embodiment;

[0010] Figure 6 This is a block diagram of an embodiment of a system for storing data structures;

[0011] Figure 7This is a simplified block diagram of a computer system, according to an embodiment, on which a method for populating a data input field with data can be executed. Detailed Implementation

[0012] Even when using known compression techniques, a significant amount of data still needs to be sent and received because the data structure is opened for editing and because the edits to the data structure are saved. Therefore, a method for storage and loading that exhibits low data throughput is needed.

[0013] One possible method to reduce the amount of data is to transmit a record of changes to the data structure when the edited data structure is saved. Thus, only the record of changes is sent to memory for storage, where it can be used to update and store the completely up-to-date data structure. However, embodiments of the present invention recognize that while such a method improves the performance of saving the data structure, it does not improve the performance of opening a data structure in which the entire data structure needs to be retrieved by an application.

[0014] In practice, the scenario could involve multiple users having data structures stored locally. If one user edits a data structure and subsequently updates the data structure in a shared storage location (possibly using a change record as described above), other users would need to load the entire data structure from the shared storage to obtain the latest version.

[0015] The invention will be described with reference to the accompanying drawings.

[0016] It should be understood that while the detailed description and specific examples indicate exemplary embodiments of the apparatus, system, and method, they are for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the invention will be better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are schematic only and not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0017] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude multiple. If the term "suitable" is used in the claims or description, it should be noted that the term "suitable" is intended to be equivalent to the term "configured as."

[0018] In the context of this application, when embodiments of the present invention constitute a method, it should be understood that such a method can be a process for execution by a computer, i.e., a computer-implementable method. Therefore, the various steps of the method can reflect various parts of a computer program, such as various parts of one or more algorithms.

[0019] Furthermore, in the context of this application, a system may be a single device or a collection of distributed devices suitable for performing one or more embodiments of the methods of the present invention. For example, a system may be a personal computer (PC), a server, or a collection of PCs and / or servers connected via a network such as a local area network, the Internet, etc., to collaboratively perform at least one embodiment of the methods of the present invention.

[0020] The implementations of this disclosure involve various techniques, methods, schemes, and / or solutions related to storing data structures. Specifically, the data structure is segmented to produce final data blocks with corresponding block keys. The segmentation of the final blocks is selected / configured to correspond to the position of a bit sequence (unique within the data structure for each of the multiple initial blocks), and the block key is based on the identified bit sequence.

[0021] By dividing the data structure into smaller final blocks, when loading the data structure, only the final blocks that are not stored locally (e.g., stored in the cache) need to be retrieved. Therefore, the amount of data that needs to be retrieved is reduced, resulting in improved performance in terms of both computational complexity and time consumption.

[0022] However, simply dividing the data structure into fixed-size blocks often proves problematic. Specifically, when bits are appended or removed from the data structure, not only the block where the bit was appended or removed needs to be updated, but every subsequent block does as well. Thus, whenever the data structure is edited (even in a minor way), many blocks will need to be updated. This can prove particularly problematic when many clients store blocks locally for use, as each client will have to reload many blocks to obtain the latest data structure.

[0023] Embodiments of the present invention recognize that the final block and block key generated by the present invention can overcome the above-mentioned problems. Specifically, the data structure is first divided into multiple initial blocks. Each initial block is then analyzed to identify a unique bit sequence. The unique bit sequence identified within each initial block is then used as the basis for the block key corresponding to each initial block. The data structure is subsequently re-divided at positions corresponding to the unique bit sequence positions. Thus, a block key array identifying the corresponding final block is obtained.

[0024] Therefore, when bits are added or removed from the data structure during editing, only the final block corresponding to those added / removed bits is updated. This is because the block key corresponds to the beginning of each final block, not because the final block has a specified length.

[0025] In other words, embodiments of the present invention provide a method for dividing a data structure into multiple final blocks, which allows for alteration of the "increment" applied to the final blocks to minimize the number of final blocks modified. Therefore, a local cache of the final blocks can then be used to reduce the burden of loading the data structure (e.g., by avoiding having the entire updated data structure on the network).

[0026] Other embodiments of the invention provide a computer-implemented method for loading a data structure. The method includes: obtaining a plurality of final blocks associated with the data structure and corresponding block keys, wherein the segmentation of the final blocks corresponds to the position of a unique bit sequence in the data structure, and wherein the block keys are based on the bit sequence; comparing the block keys with a plurality of locally stored blocks to identify block keys for which no locally stored blocks are associated; and loading the final blocks associated with the identified block keys.

[0027] In this way, only the final blocks not stored locally need to be loaded, thus reducing the burden of downloading many final blocks, or possibly even the entire data structure. This is achieved by dividing the data into final blocks corresponding to block keys, where the final blocks are partitioned at positions corresponding to positions of unique bit sequences within the data structure.

[0028] In some embodiments, the method for loading a data structure may further include merging multiple final blocks to obtain the data structure. This allows for the acquisition of the latest data structure.

[0029] Turning to another embodiment of the proposed concept for storing data structures, identifying bit sequences may include: sequentially scanning the data structure from the first bit to the last bit of the initial block to identify the bit sequence within the block that is unique in the data structure and is closest to the first bit.

[0030] By scanning the initial block from the first bit to the last bit, a unique bit sequence can be identified as being as close as possible to the first bit. In this way, the partitioning of the data structure that produces the final block ensures that the final block is as close as possible to the size of the initial block. This can prove particularly advantageous when a block of a given size is desired. Furthermore, scanning the initial block from the first bit to the last bit enables efficient and reliable identification of unique bit sequences within the data structure.

[0031] In other embodiments, the method of storing a data structure may further include: receiving a change record including changes made to the data structure; merging multiple final blocks to obtain a data structure; updating the data structure based on the change record; and dividing the updated data structure into multiple updated final blocks, wherein the division corresponds to the position of the identified bit sequence.

[0032] Because the change record is received instead of the complete updated data structure, the burden of data transfer is reduced. The change record can contain significantly less data than the updated data structure, especially when the changes made are relatively small compared to the size of the data structure. The updated data structure can then be obtained directly by merging the change record with the original data structure. The already generated block key can then be reused for delivering the final block, thus reducing the overall processing required to segment the updated data structure.

[0033] In some embodiments, the method of storing data structures may further include: for each of a plurality of final blocks, determining whether the size of the final block is greater than a predetermined upper limit threshold; and in response to determining that the size of the final block is greater than the predetermined upper limit threshold, segmenting the final block.

[0034] By effectively setting an upper limit threshold for the size of each final block, it is possible to avoid blocks that are so large that the benefits of segmentation are diminished. This can happen, for example, when the identified (unique) bit sequence of the first block is close to the first bit of that block and the identified (unique) bit sequence of the second sequential block is close to the last bit of that block. As a result, the segmentation of the final blocks can result in very large final blocks, thus requiring further segmentation.

[0035] In yet another embodiment, the method of storing the data structure may further include: for each of a plurality of final blocks, determining whether the size of the final block is less than a predetermined lower threshold; and in response to determining that the size of the final block is less than the predetermined lower threshold, merging the final block with an adjacent final block.

[0036] Similar to setting an upper threshold, setting a lower threshold can prevent blocks from becoming so small that they diminish the benefits of splitting. By merging adjacent blocks, blocks of a more suitable size can be obtained.

[0037] In an additional embodiment, the method of storing the data structure may further include: for each of a plurality of final blocks, calculating a hash of the data of the final block, and appending the hash and a delimiter to a block key corresponding to the final block.

[0038] Therefore, hashing can identify the final block corresponding to the block key.

[0039] In some embodiments, the method of storing the data structure further includes: compressing each of a plurality of final blocks; and storing the plurality of compressed final blocks and associated block keys on a server.

[0040] By compressing data structures and dividing them into blocks, more efficient data storage and transmission can be achieved.

[0041] In other embodiments, the data structure is a JavaScript Object Representation (JSON) data structure, a Rich Text Format (RTF) file, or a code file.

[0042] JSON data structures are often problematic for transmitting large amounts of data, especially when frequent saves, loads, and reloads are required. While JSON data structures are easy to understand and use, they are often not space-efficient; even small objects require many bytes to represent. Therefore, implementing methods to store JSON data structures by segmenting them can achieve efficient storage and transmission of JSON data structures.

[0043] In fact, this method can be extended to any uncompressed data structure. For example, RTF files or code files can benefit from being stored in the manner described above.

[0044] As a further explanation, given a very large data structure (e.g., a 100MB string), it can be advantageous to divide the string into manageable blocks (e.g., each 10MB). A simple solution would be to simply split the 100MB string into 10MB blocks. However, this means that inserting or deleting any number of bits at any position within the 100MB string will result in a change to every block preceding those bits. Consequently, a block mechanism that can produce blocks close to the desired length (e.g., 10MB) while minimizing the number of blocks changed due to insertions or deletions is desirable.

[0045] According to an embodiment of the present invention, this segmentation can be achieved through the following steps:

[0046] (i) Divide the data structure into initial blocks of (regular size);

[0047] (ii) For each initial block, scan from left to right to find a unique bit sequence (e.g., a unique 20-byte sequence) that has not appeared in any previous initial block;

[0048] (iii) Add each bit of the identified unique bit sequence to the block key array;

[0049] (iv) Re-segment the initial data structure at the position corresponding to the position of the identified block key to obtain the final block;

[0050] (v) If any generated final block is smaller than the minimum threshold size, append it to the previous final block and discard the corresponding block key;

[0051] (vi) If any resulting final block is larger than the maximum threshold size, split the final block using the same mechanism as in steps (i)-(iv); and

[0052] (vii) For each block key, append a delimiter and a hash of the block data of the final block corresponding to that block key. For example, MD5 or a secure hash algorithm (SHA) can be used.

[0053] The result of the above segmentation process is an array of block keys and the corresponding final data blocks. The block hash can be used as a cache key to cache the final data blocks locally in the browser. In fact, modern web development applications often support asynchronous storage for data blocks that can be used to store local caches.

[0054] By dividing the data structure in the manner described above, the final block and block key can be used in the following ways.

[0055] During initial data saving, all final blocks are compressed and sent to the server along with their corresponding block keys. The server can store the final blocks in its associated database. The server can also store the block keys. When data is subsequently loaded, the client can retrieve the block keys from the server. For each final block, the client can first check its local cache (and use the cached final block if available). If the final block is not available in the local cache, the client can retrieve it from the server and add it to the cache. The client can then decompress and merge the final blocks to obtain the complete data structure.

[0056] Furthermore, when saving edits to the data structure, the client can calculate the change record between the initial data structure and the updated version of the data structure, and then send the change record to the server. The server can then reconstruct the initial data structure by merging the stored final blocks, and then apply the change record to obtain the updated data structure. The server can then use the existing block key to segment the updated data structure and save the resulting updated final block. In other words, the server can segment the updated data structure according to steps (iv)-(vii) above.

[0057] This results in a set of updated final blocks, some of which match previous final blocks (depending on the scope of the changes). This helps clients load only the updated final blocks from the server, rather than loading unmodified ones. For example, an insertion at the beginning of the data structure will only affect the first final block, and all other final blocks will remain the same. Ultimately, this reduces the amount of data transfer required to obtain the latest data structure when saving to the server and when loading it by the client.

[0058] Turn now Figure 1 This diagram presents a block diagram of an example system 200 in which aspects of the illustrative embodiments can be implemented. System 200 is an example of a computer, such as a client in a distributed processing system, in which computer-usable code or instructions for implementing the processes of the illustrative embodiments of the invention may be located. For example, system 200 may be configured to implement a data manipulation unit and a block key generation unit according to embodiments.

[0059] In the depicted example, system 200 employs a central architecture including a Northbridge and Memory Controller Center (NB / MCH) 202 and a Southbridge and Input / Output (I / O) Controller Center (SB / ICH) 204. Processing unit 206, main memory 208, and graphics processor 210 are connected to NB / MCH 202. Graphics processor 210 can be connected to NB / MCH 202 via an Accelerated Graphics Port (AGP).

[0060] In the depicted example, a local area network (LAN) adapter 212 is connected to SB / ICH 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a hard disk drive (HDD) 226, a CD-ROM drive 230, a universal serial bus (USB) port and other communication ports 232, and a PCI / PCIe device 234 are connected to SB / ICH 204 via a first bus 238 and a second bus 240. The PCI / PCIe device may include, for example, an Ethernet adapter, an insert card, and a PC card for a notebook computer. PCI uses a card bus controller, while PCIe does not. ROM 224 may be, for example, a flash memory basic input / output system (BIOS).

[0061] HDD 226 and CD-ROM drive 230 are connected to SB / ICH 204 via a second bus 240. HDD 226 and CD-ROM drive 230 can use interfaces such as Integrated Drive Electronics (IDE) or Serial Advanced Technology Attachment (SATA). Super I / O (SIO) device 236 can be connected to SB / ICH 204.

[0062] The operating system runs on processing unit 206. The operating system coordinates and provides services to... Figure 2 The system 200 controls the various components within it. As a client, the operating system can be a commercially available operating system. Object-oriented programming systems (such as...) The programming system can run in conjunction with an operating system and provide execution from the system 200. A program or application's call to the operating system. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and / or its subsidiaries.

[0063] As a server, system 200 can, for example, run advanced interactive execution. Operating system or operating system eServer TM System p computer system. System 200 may be a symmetric multiprocessor (SMP) system, which includes multiple processors in processing unit 206. Alternatively, a single-processor system may be used. IBM, eServer, and AIX are trademarks of International Business Machines Corporation in the U.S. and / or other countries. Registered Trademarks It is used under a sublicense from the Linux Foundation, which is the exclusive licensee of Linus Torvalds and the owner of the trademark worldwide.

[0064] The operating system, programming system, and instructions for applications or programs reside on a storage device such as HDD 226 and can be loaded into main memory 208 for execution by processing unit 206. Similarly, one or more message processing programs according to the embodiments may be adapted to be stored by storage devices and / or main memory 208.

[0065] The process of the illustrative embodiment of the present invention can be executed by the processing unit 206 using computer-usable program code, which may be located in a memory, such as main memory 208, ROM 224 or one or more peripheral devices 226 and 230.

[0066] Bus systems (such as) Figure 2 The first bus 238 or the second bus 240 shown may include one or more buses. Of course, the bus system can be implemented using any type of communication structure or architecture that provides data transfer between different components or devices attached to a communication structure or architecture. Communication units (such as...) Figure 1 The modem 222 or network adapter 212 may include one or more devices for sending and receiving data. The memory may be, for example, main memory 208, ROM 224, or something similar to... Figure 1The cache found in NB / MCH 202.

[0067] Those skilled in the art will understand that Figure 1 The hardware within can vary depending on the implementation. (Except for or replacing...) Figure 1 The hardware shown can be replaced with other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disc drives. Furthermore, the processes of the illustrative embodiments can be applied to multiprocessor data processing systems other than those mentioned above, without departing from the spirit and scope of the invention.

[0068] Furthermore, system 200 can take the form of any of a variety of different data processing systems, including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants (PDAs), etc. In some illustrative examples, system 200 may be a portable computing device configured with flash memory to provide non-volatile storage for operating system files and / or user-generated data, etc. Therefore, system 200 can essentially be any known or later-developed data processing system without architectural limitations.

[0069] Now for reference Figure 2 A flowchart 100 of a method for storing a data structure according to an illustrative embodiment is presented.

[0070] In step 102, the processor divides the data structure into multiple initial blocks. The division can be performed at regular intervals, such that the data structure is divided into regular blocks (e.g., 10MB), or it can be selected individually. The block size can be set to a predetermined value or can be selected by the user.

[0071] A data structure can be any data structure in which the insertion or removal of data does not cause changes to the data structure at points before or after the insertion / deletion. For example, a data structure can be an uncompressed JSON data structure, a code file, or a text file. However, the data structure does not have to be a .doc file.

[0072] In step 104, for each of the plurality of initial blocks, the processor identifies a unique bit sequence within the data structure of that block. The bit sequence may be set to a predetermined length (e.g., 20 bytes).

[0073] One possibility is that identifying the bit sequence involves sequentially scanning the data structure from the first bit to the last bit of the initial block to identify the unique bit sequence within that block that is closest to the first bit. However, an alternative method of searching for (unique) bit sequences can be used. Another possibility is that the bit sequence is simply a sequence that has not appeared in any previous block.

[0074] In step 106, for each of the plurality of initial blocks, the processor determines the block key based on the identified bit sequence within that initial block.

[0075] In this way, the block key can be used to uniquely identify blocks of a data structure. The block key can be simply the identified bit sequence, or it can be another representation of the identified bit sequence, such as a compressed version of the identified bit sequence. Finally, the block key can be stored in a block array.

[0076] In step 108, the processor re-segments the data structure into multiple final blocks, wherein the segmentation corresponds to the position of the identified bit sequence.

[0077] In other words, multiple final blocks of the data structure are obtained, each with a corresponding block key. When data is inserted or removed from any individual's final block, the other final blocks remain unchanged. Therefore, when the data structure is edited, modified, or otherwise altered, it is possible that only some of the final blocks are changed. This ultimately has the effect that when an updated version of the data structure is downloaded to a client that has an older version of the data structure stored locally, only those blocks that have been modified are downloaded.

[0078] In some embodiments, method 100 may further include steps 110-112 and / or steps 114-116.

[0079] In step 110, for each of the plurality of final blocks, the processor computes a hash of the data in that final block. For example, the hash of the data can be an MD5 hash, an SHA hash, or any known hashing technique.

[0080] In step 112, for each of the plurality of final blocks, the processor appends the calculated hash and delimiter to the block key corresponding to that final block.

[0081] Therefore, the array of block keys can be obtained by hashing the final block data. This information can be cached locally. In web development applications, this information can be cached locally in the browser.

[0082] In step 114, the processor compresses each of the multiple final blocks. Final blocks can be compressed individually, thereby improving the speed and efficiency of transfer and storage. For example, LZW compression or other known compression techniques can be used to compress the final blocks.

[0083] In step 116, the processor stores multiple compressed final blocks and their associated block keys. In other words, the multiple compressed final blocks and block keys can be stored on a universally accessible server for use by many clients. This allows the data structure to be loaded by the client when needed. Updates can be sent to the server, and the client can then access the latest data structure from the server (by retrieving the updated final blocks).

[0084] Figure 3A and 3B A flowchart illustrating a method for further processing the final block according to an illustrative embodiment is presented. Both methods involve transmitting a final block of the desired size. More specifically, Figure 3A Ensure that each final block does not exceed a predetermined upper limit threshold, Figure 3B Ensure that each final block meets at least the predetermined lower threshold.

[0085] In step 120, the processor determines whether the size of the final block is greater than a predetermined upper limit threshold. The method terminates when it is determined that the size of the final block is not greater than the predetermined upper limit threshold.

[0086] In step 122, in response to determining that the size of the final block is greater than a predetermined upper threshold, the processor segments the final block. This can be achieved using information about... Figure 2 The described method splits the final block, thereby producing multiple sub-final blocks with corresponding block keys.

[0087] Therefore, excessively large final blocks suppress or lose the benefits associated with dividing the data structure into blocks.

[0088] In step 130, the processor determines whether the size of the final block is less than a predetermined lower threshold. The method terminates when it is determined that the size of the final block is not less than the predetermined lower threshold.

[0089] In step 132, in response to determining that the size of the final block is less than a predetermined lower threshold, the processor merges the final block with an adjacent final block. In some embodiments, the adjacent final block may be the previous final block. In this case, the block key associated with the final block that has been determined to be too small can be discarded. For excessively small final blocks, the increased complexity caused by re-merging and separate transmission outweighs the benefits associated with splitting the data structure into blocks.

[0090] Now go to Figure 4 A flowchart of a method 140 for updating (editing) a data structure is shown according to an illustrative embodiment.

[0091] In step 142, the processor receives a change record that includes changes made to the data structure. The change record may include bits to be added and their corresponding insertion positions, bits to be deleted and their corresponding positions, and bits to be changed and their corresponding positions.

[0092] In this way, by sending only the changes, the computational and bandwidth requirements of sending a completely updated data structure are avoided. This can be particularly beneficial when the number of changes made to the data structure is small. However, this requires that an older version of the data structure is available at the receiver.

[0093] In step 144, the processor merges multiple final blocks to obtain the data structure. In other words, by combining the stored final blocks, an older version of the data structure is obtained. This may require decompressing the final blocks.

[0094] In step 146, the processor updates the data structure based on the changed records.

[0095] The modification record provides a direct list of updates made to the data structure. Therefore, bits are added to or removed from the data structure, and bits can be modified. This transmits a completely up-to-date data structure from the previous data structure.

[0096] In step 148, the processor divides the updated data structure into multiple updated final blocks, wherein the division corresponds to the position of the identified bit sequence.

[0097] Because the updated data structure is re-segmented based on the positions of the previously identified bit sequences (reflected in the block key), updated final blocks can be obtained (because they contain the added, removed, and / or updated bits). Unchanged final blocks can also be obtained, as they do not contain the added, removed, and / or updated bits. Therefore, when a client with a previous version of the data structure attempts to load the updated data structure, it may only need to load the changed final blocks.

[0098] Figure 5 A flowchart is presented for a method 300 for loading a data structure according to another illustrative embodiment.

[0099] In step 302, the processor obtains multiple final blocks associated with the data structure and their corresponding block keys. The segmentation of the final blocks corresponds to the position of a unique bit sequence within the data structure, and the block keys are based on the bit sequence. In other words, according to the information about... Figure 2-4 Any method described for storing data structures.

[0100] In step 304, the processor compares the block key with multiple locally stored blocks to identify the block key of a block that is not associated with a local storage block.

[0101] In other words, locally stored blocks can correspond to previous versions of the data structure. If a block key exists that does not have an associated locally stored block, it means that the locally stored data structure is a previous version. The identified block key indicates which final blocks need to be loaded to obtain the latest version of the data structure.

[0102] In step 306, the processor loads the final block associated with the identified block key.

[0103] This avoids loading the entire data structure, thereby reducing data traffic over the network (regardless of the type of network, as long as this method of storing and loading data structures is applied).

[0104] Some embodiments of method 300 may also include step 308. In step 308, the processor merges multiple final blocks to obtain a data structure.

[0105] Therefore, it is possible to load fully updated data structures while minimizing the amount of data flowing through the system used to store, load, and update data structures.

[0106] Now for reference Figure 6 A simplified block diagram of a system 400 for storing data structures is depicted. System 400 includes at least a data manipulation unit 410 and a block key generation unit 420. Optionally, system 400 may include an interface unit 430.

[0107] First, the data manipulation unit 410 is configured to divide the data structure into multiple initial blocks. The block key generation unit 420 is configured (for each of the multiple initial blocks) to determine a block key based on a unique bit sequence identified within the data structure in that block.

[0108] Furthermore, the data manipulation unit 410 is configured to re-divide the data structure into multiple final blocks, wherein the division corresponds to the position of the identified bit sequence.

[0109] In some embodiments, the block key generation unit 420 is configured to identify a bit sequence by sequentially scanning the data structure from the first bit to the last bit of the initial block to identify the bit sequence within the block that is unique in the data structure and closest to the first bit.

[0110] Furthermore, system 400 may also include interface unit 430. Interface unit 430 may be configured to receive change records including changes made to the data structure. In this case, data manipulation unit 410 is further configured to: merge multiple final blocks to obtain a data structure, update the data structure based on the change records, and divide the updated data structure into multiple updated final blocks, wherein the division corresponds to the position of the identified bit sequence.

[0111] As another example, such as Figure 7 As shown, embodiments may include a computer system 70, which may form part of a networked system 7. For example, the computer system 70 may be used to implement a system for editing images to be displayed by a display device. Components of the computer system / server 70 may include, but are not limited to, one or more processing devices, such as a processor or processing unit 71, a system memory 74, and a bus 90 that couples different system components (including the system memory 74) to the processing unit 71.

[0112] System memory 74 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 75 and / or cache memory 76. Computer system / server 70 may also include other removable / non-removable, volatile / non-volatile computer system storage media 77. In such cases, each may be connected to bus 90 via one or more data media interfaces. Memory 74 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the proposed embodiments. For example, memory 74 may include a computer program product having a program executable by processing unit 71 to cause the system to perform a method for storing data structures according to the proposed embodiments.

[0113] A program / utility 78 having at least one set of program modules 79 may be stored in memory 74. The program modules 79 generally perform the functions and / or methods of the proposed embodiments for detecting scene transitions in video footage.

[0114] The computer system / server 70 can also communicate with one or more external devices 80 (e.g., keyboard, pointing device, display 85, etc.); with one or more devices that enable a user to interact with the computer system / server 70; and / or with any device that enables the computer system / server 70 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can occur via input / output (I / O) interface 72. Furthermore, the computer system / server 70 can communicate via network adapter 73 with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) (e.g., to deliver recreated content to the system or user).

[0115] In the context of this application, when embodiments of the present invention constitute a method, it should be understood that such a method is a process for execution by a computer, i.e., a computer-implementable method. Therefore, the various steps of the method reflect various parts of a computer program, such as various parts of one or more algorithms.

[0116] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0117] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0118] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0119] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0120] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0121] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0122] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0124] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: Divide the data structure into multiple initial blocks of regular size; For each of the plurality of initial blocks, a block key for the corresponding initial block is determined based on a unique bit sequence identified within the data structure in the corresponding initial block, wherein the bit sequence is a subset of the bits of the corresponding initial block; Add each block key to the block key array; and The data structure is re-divided into multiple irregularly sized final blocks, wherein the division of each final block corresponds to the position of the identified bit sequence.

2. The computer-implemented method according to claim 1, wherein, Identifying the bit sequence includes sequentially scanning the data structure from the first bit to the last bit of the corresponding initial block to identify the bit sequence within the initial block that is unique in the data structure and closest to the first bit.

3. The computer-implemented method according to claim 1 further includes: Receive change records that include changes made to the data structure; Merge the multiple final blocks to obtain the data structure; Based on the changed records, update the data structure; as well as The updated data structure is divided into multiple updated final blocks, where each division corresponds to the position of the identified bit sequence.

4. The computer-implemented method according to claim 1 further includes: For each of the plurality of final blocks, the size of the corresponding final block is determined to be greater than a predetermined upper limit threshold; as well as In response to determining that the size of the corresponding final block is greater than the predetermined upper limit threshold, the corresponding final block is segmented.

5. The computer-implemented method according to claim 1, further comprising: For each of the plurality of final blocks, the size of the corresponding final block is determined to be less than a predetermined lower threshold. as well as In response to determining that the size of the corresponding final block is less than the predetermined lower threshold, the corresponding final block is merged with the adjacent final block.

6. The computer-implemented method of claim 1, further comprising, for each of the plurality of final blocks: Calculate the hash of the data for the corresponding final block; and The hash and delimiter are appended to the block key corresponding to the respective final block.

7. The computer-implemented method according to claim 1, further comprising: Compress each of the plurality of final blocks; as well as Multiple compressed final blocks and their associated block keys are stored on the server.

8. The computer-implemented method according to claim 1, wherein, The data structure is selected from the group consisting of: JavaScript object representation JSON data structure, rich text format RTF file, and code file.

9. The computer-implemented method according to claim 1, further comprising: The block key is compared with multiple locally stored blocks to identify block keys that are not associated with locally stored blocks; as well as Load the corresponding final block associated with the identified block key.

10. The computer-implemented method according to claim 1, further comprising: The multiple final blocks are merged to obtain the data structure.

11. The computer-implemented method according to claim 1, further comprising: Obtain the plurality of final blocks and their corresponding block keys associated with the data structure, wherein: The final block segmentation corresponds to the position of a unique bit sequence in the data structure; and The block key is based on the bit sequence; The block key is compared with multiple locally stored blocks to identify block keys for blocks that are not associated with local storage; and Load the final block associated with the identified block key.

12. A computer program product, comprising: Program instructions, said program instructions being used to perform the method according to any one of claims 1 to 11.

13. A computer system, comprising: One or more computer processors, one or more computer-readable storage media, and program instructions commonly stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the program instructions including program instructions for performing the method according to any one of claims 1 to 11.

Images