A patch file splitting method and system based on BSDiff

Abstract

The application discloses a patch file splitting method based on BSDiff. A BSDiff algorithm is used to generate an uncompressed overall patch data according to a new file and an old file. A virtual equal division value is determined according to the available storage space of a device applying the patch file. The uncompressed overall patch data is split into multiple patch data blocks; the splitting is based on the fact that each patch data block can obtain a virtual equal division data block of the new file. One patch data block is received each time, and after decompression, a virtual equal division data block of the new file is generated according to the old file and the current patch data block, and is directly written into the storage area of the new file. The application reduces the consumption of network data and storage space in the differential upgrade process.

Description

Technical Field

[0001] This invention relates to a method for splitting differential upgrade (also known as incremental upgrade) files. Background Technology

[0002] A common method for software (including firmware) upgrades is differential upgrade. This method generates a differential upgrade file based on the differences between the new and old software versions and transmits this file. The receiving end performs the upgrade based on the old software version and the differential upgrade file, generating the new software version. The differential upgrade file is typically smaller than the new software version, thus significantly reducing the network bandwidth required for software upgrades and the storage space needed at the receiving end.

[0003] For embedded devices with limited storage space, even if the firmware is upgraded using the differential upgrade method, the size of the differential upgrade file may still put pressure on the network transmission and storage space of the embedded device. Therefore, it is desirable to reduce the pressure on the embedded device for each network transmission and each storage space occupation.

[0004] The BSDiff algorithm is a differential upgrade algorithm specifically designed for binary files. By finding similarities between the old and new files and recording the differences, this algorithm can significantly reduce the size of the generated patch file.

[0005] The BSDiff algorithm generates patch files as follows: It searches for similar segments in the old and new files, with each segment having a variable length. If found, it generates difference data between the two. If a segment in the new file cannot be found in the old file, that segment in the new file is added as new data. The control data, difference data, and new data are then compressed to generate the patch file.

[0006] The BSDiff algorithm applies the patch file as follows: The patch file is decompressed to obtain control data, difference data, and new data. Based on the control data, similar segments corresponding to each difference data point are read from the old file. Similar segments in the new file are then obtained based on the similar segments in the old file and their corresponding difference data. New data is then inserted into the specified positions in the new file based on the control data. The sum of the similar segments and the new data in the new file constitutes the new file.

[0007] Please see Figure 1The patch file generated by the BSDiff algorithm, after decompression, is divided into multiple control blocks and multiple data areas, with each control block corresponding one-to-one with each data area. Each control block is further divided into three parts: the size of the difference partition, the size of the new data area, and the offset of the old file. Each data area is further divided into two parts: the data content of the difference partition and the data content of the new data area. Within each control block, the size of the difference partition refers to the size of the "difference partition data content" of the corresponding data area; the size of the new data area refers to the size of the "new data content" of the corresponding data area; and the offset of the old file refers to the offset position of the next control block's operation on the old file. Control block one defaults to operating on the beginning position of the old file. Within each data area, the difference partition data content refers to the difference data content of a pair of similar segments between the old and new files; and the new data area data content refers to a segment of the new file that cannot be found in the old file.

[0008] The BSDiff algorithm is very memory-intensive. When processing large files (e.g., several GB in size), it is usually split into several data blocks of the same size, and then the BSDiff algorithm is applied to each pair of data blocks between the old and new files. This additional operation of dividing the data blocks can cause the size of the patch file generated by the BSDiff algorithm to increase dramatically.

[0009] Please see Figure 2 The only difference between the new and old files is the addition of a data segment (the gray-filled part). If the BSDiff algorithm is used directly to generate a patch file without dividing the data into blocks, the size of the patch file is mainly determined by the size of the added data. If the data is first divided into equal blocks and then the BSDiff algorithm is applied (dashed lines represent the boundaries between data blocks), then there will be differences between each data block in the old file and each data block in the new file. The patch generated by applying the BSDiff algorithm to each pair of data blocks from the old and new files will be large, significantly increasing the total size of the patch file. Furthermore, the patch file will contain many identical segments that exist in both the old and new files. Figure 2 This is just a simple example of inserting new data. In real-world scenarios, there are often numerous insertions, deletions, and differences between the new and old files, resulting in a large number of shifted and transposed segments between them. This further amplifies the disadvantages of first dividing the data into equal blocks and then applying the BSDiff algorithm. Summary of the Invention

[0010] The technical problem to be solved by this invention is: how to overcome the drawback of the patch file obtained by first dividing the data into equal blocks and then using the BSDiff algorithm, which contains redundant data.

[0011] To address the aforementioned technical problems, this invention proposes a BSDiff-based patch file splitting method, comprising the following steps: Step S1: Using the BSDiff algorithm, an uncompressed overall patch data is generated based on the new file and the old file. Step S2: A virtual equal division value is determined based on the available storage space of the device applying the patch file; the new file is virtually divided into multiple data blocks according to the virtual equal division value. Steps S1 and S2 may be performed simultaneously or arbitrarily first. Step S3: The uncompressed overall patch data is split into multiple block patch data, and each block patch data is compressed to obtain each block patch file; the splitting criterion is that applying the BSDiff algorithm to each block patch data can yield a virtual equally divided data block of the new file. Step S4: The device applying the patch file receives one segmented patch file at a time, decompresses it to obtain segmented patch data, and generates a virtual equally divided data block for the new file based on the old file and the current segmented patch data. This data block is then written directly to the storage area of ​​the new file. The corresponding segmented patch data is then discarded, and the reception of the next segmented patch file begins. This process is repeated until the device applying the patch file writes the last virtual equally divided data block of the new file to the storage area of ​​the new file.

[0012] Furthermore, in step S1, the last step of the BSDiff algorithm in generating the patch file is compression. This compression step is omitted to obtain uncompressed overall patch data; or, the BSDiff algorithm is fully executed to generate a compressed overall patch file, and then decompression is performed to obtain uncompressed overall patch data.

[0013] Furthermore, in step S2, the virtual equal score is less than or equal to the available storage space of the device for applying the patch file.

[0014] Furthermore, in step S2, when flash memory is used as the non-volatile memory, the virtual equal division value is an integer multiple of the flash memory erase block size.

[0015] Furthermore, in step S2, the size of the last virtual equal-division data block of the new file is allowed to be less than or equal to the virtual equal-division value.

[0016] Furthermore, in step S3, the number of block patch files = the size of the new file ÷ the virtual equal division value, and if there is a remainder, the quotient is incremented by one.

[0017] Furthermore, in step S3, the size of any block patch data is less than or equal to the available storage space of the device applying the patch file.

[0018] Furthermore, in step S4, all the virtual equally divided data blocks of the new file are naturally spliced ​​together in the memory of the new file.

[0019] Further, step S3 specifically includes the following sub-steps: Step S31: Create a new empty block patch file, which contains the new file offset position—used to record the starting position of a virtual partition data block in the new file generated by each block patch; the remaining space of the new file data that the current block patch file can generate is initialized to the virtual equal partition value. Step S32: Read each control block in the overall patch data in sequence, add the difference partition data size and the new partition data size in the current control block to obtain the new file data size that the current control block can generate. Step S33: If the new file data size that the current control block can generate is ≤ the remaining space of the new file data that the current block patch file can generate, then write the current control block and the corresponding data area directly into the current block patch file, and update the remaining space of the new file data that the current block patch file can generate—subtract the new file data size that the current control block can generate; then return to step S32 to read the next control block in the overall patch data. If the size of the new file data that can be generated from the current control block is greater than the remaining space of the new file data that can be generated from the current block patch file, then the current control block and its corresponding data area are decomposed into two control blocks and two corresponding data areas. The size of the new file data that can be generated from the first decomposed control block is equal to the remaining space of the new file data that can be generated from the current block patch file. The first decomposed control block and its corresponding data area are directly written into the current block patch file, and the remaining space of the new file data that can be generated from the current block patch file is zero. The current patch file is then compressed. The second decomposed control block and its corresponding data area are added to the overall patch data as the next control block and its corresponding data area. Then, the process returns to step S31 to create the next empty block patch file. Steps S31 to S33 are repeated until all control blocks and data areas in the overall patch data have been written into each block patch file, and each block patch file is compressed.

[0020] This invention also proposes a BSDiff-based patch file splitting system, including a differential processing module, a virtual equalization module, a patch splitting module, and a patch upgrade module. The differential processing module uses the BSDiff algorithm to generate an uncompressed overall patch data based on the new file and the old file. The virtual equalization module determines the virtual equalization value based on the available storage space of the device applying the patch file, and virtually divides the new file into multiple data blocks according to the virtual equalization value. The patch splitting module splits the uncompressed overall patch data into multiple block patch data, compresses each block patch data to obtain a block patch file; the splitting criterion is that applying the BSDiff algorithm to each block patch data yields a virtual equalization data block for the new file. The patch upgrade module receives a block patch file each time, decompresses it to obtain the current block patch data, and then generates a virtual equalization data block for the new file based on the old file and the current block patch data, directly writes it to the new file's storage area, discards the corresponding block patch data, and begins receiving the next block patch file; this process is repeated until the last virtual equalization data block of the new file is written to the new file's storage area.

[0021] The technical advantages achieved by this invention are as follows: First, the BSDiff algorithm is used to generate the overall patch data, and then the overall patch data is split into multiple block patch data. The sum of the volumes of the block patch data is only slightly larger than the overall patch data, with the addition of a header. Compared with existing methods that first equally divide the data into blocks and then apply the BSDiff algorithm, the block patch data generated by this invention has no redundant data, better supports differential upgrades in stages, better adapts to embedded devices with limited storage space, and reduces the network data and storage space consumption during software (including firmware) upgrades. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the data format of the patch file generated by the BSDiff algorithm after decompression.

[0023] Figure 2 This is a schematic diagram illustrating an example of first dividing data into equal blocks and then using the BSDiff algorithm to generate patch files.

[0024] Figure 3 This is a flowchart illustrating the patch file splitting method based on BSDiff proposed in this invention.

[0025] Figure 4 This is a schematic diagram illustrating an example of the patch file splitting method based on BSDiff proposed in this invention.

[0026] Figure 5 yes Figure 3 The detailed flowchart of step S3 is shown below.

[0027] Figure 6 This is a schematic diagram of the structure of the patch file splitting system based on BSDiff proposed in this invention.

[0028] The attached diagrams are labeled as follows: Differential processing module 1, Virtual equal division module 2, Patch splitting module 3, Patch upgrade module 4. Detailed Implementation

[0029] Please see Figure 3 The patch file splitting method based on BSDiff proposed in this invention includes the following steps.

[0030] Step S1: Use the BSDiff algorithm to generate an uncompressed overall patch data based on the new and old files. This process does not divide the new and old files into data blocks.

[0031] For example, the final step in the BSDiff algorithm to generate patch files is compression. This compression step can be omitted to obtain uncompressed overall patch data.

[0032] For example, by fully executing all the steps of generating the patch file using the BSDiff algorithm, a compressed overall patch file is obtained. Then, the first step of applying the patch file using the BSDiff algorithm is executed to decompress the compressed overall patch file, resulting in uncompressed overall patch data.

[0033] The data format of the uncompressed overall patch data is as follows: Figure 1 As shown. This step does not divide the old and new files into equal blocks before generating the patch file using the BSDiff algorithm, thus avoiding the drawback of dividing the data into equal blocks before applying the BSDiff algorithm. The resulting patch file is usually quite large, far exceeding the available storage space of the embedded device.

[0034] Step S2: Determine the virtual equal division value based on the available storage space of the device applying the patch file, where the virtual equal division value is less than or equal to the available storage space of the device applying the patch file. Embedded devices typically use flash memory as non-volatile storage, and the virtual equal division value is preferably an integer multiple of the flash memory erase block size. In subsequent steps, a virtual equal division data block of the new file is restored each time according to the virtual equal division value. This is equivalent to virtually dividing the new file into multiple data blocks according to the virtual equal division value. It is allowed that the size of the last virtual equal division data block of the new file is less than or equal to the virtual equal division value.

[0035] The order of steps S1 and S2 is not strictly limited; they can be performed simultaneously or either can be performed first or last.

[0036] Step S3: Split the uncompressed overall patch data into multiple chunked patch data. The splitting is based on the principle that applying the BSDiff algorithm to each chunked patch data results in a virtual equally divided data block for the new file. Each chunked patch data is then compressed to obtain a single chunked patch file.

[0037] The number of chunked patch files = new file size ÷ virtual partition value; if there is a remainder, add one to the quotient. The size of an uncompressed individual chunked patch data varies, determined by "one virtual partition data block that can generate a new file". Typically, the size of an uncompressed individual chunked patch data is ≤ the virtual partition value. Assuming a chunked patch data consists entirely of new data not present in the old file, considering that control data within the chunked patch data may occupy a small amount of space, its size is slightly larger than the virtual partition value. Overall, the size of any chunked patch data is ≤ the available storage space of the device applying the patch file.

[0038] Step S4: The sending end transmits one segmented patch file to the receiving device (the receiving end) each time. The receiving device decompresses the received segmented patch file to obtain the current segmented patch data, and then generates a virtual equally divided data block for the new file based on the old file and the current segmented patch data. The receiving device directly writes one virtual equally divided data block of the new file into the new file's storage area, then discards the corresponding segmented patch data and begins receiving the next segmented patch file. This process repeats until the receiving device writes the last virtual equally divided data block of the new file into the new file's storage area. All virtual equally divided data blocks of the new file are then naturally joined together in the new file's storage area. This invention is applicable to situations where, on the receiving device side, new files have a new file storage area, and old files have an old file storage area.

[0039] Please see Figure 4 This is the same as the one used here. Figure 2 For example, the only difference between the new and old files is the addition of a piece of data (the gray-filled part). After step S1, the overall patch data is obtained, and its size is mainly determined by the size of the newly added data. After step S2, the new file is virtually divided into multiple data blocks. After step S3, the overall patch data is split into multiple block patch files. Each block patch file, after being processed by the BSDiff algorithm, can reconstruct a virtual equally divided data block of the new file. Since all the newly added data in the new file is located in virtual equally divided data block one, and the remaining virtual equally divided data blocks of the new file all come from the old file, the block patch file corresponding to virtual equally divided data block one has the largest size, while the remaining block patch files corresponding to the other virtual equally divided data blocks are all very small.

[0040] Please see Figure 5 Step S3 specifically includes the following sub-steps.

[0041] Step S31: Create a new empty chunk patch file containing the new file offset position—used to record the starting position of a virtual partition data block in the new file generated by each chunk patch. The new file offset position of the first empty chunk patch file is zero. The new file offset position of subsequent newly created empty chunk patch files is the new file offset position of the previous chunk patch file plus the virtual partition value. The remaining space for generating new file data from the current chunk patch file is initialized to the virtual partition value.

[0042] Step S32: Read each control block in the overall patch data in sequence, add the difference partition data size and the new partition data size in the current control block to obtain the new file data size that can be generated by the current control block.

[0043] Step S33: If the size of the new file data that can be generated by the current control block is less than or equal to the remaining space of the new file data that can be generated by the current block patch file, then write the current control block and its corresponding data area directly into the current block patch file, and update the remaining space of the new file data that can be generated by the current block patch file—subtracting the size of the new file data that can be generated by the current control block. Then return to step S32 to read the next control block in the overall patch data.

[0044] If the size of the new file data that can be generated from the current control block is greater than the remaining space of the new file data that can be generated from the current chunked patch file, then the current control block and its corresponding data area are decomposed into two control blocks and two corresponding data areas. The size of the new file data that can be generated from the first decomposed control block is equal to the remaining space of the new file data that can be generated from the current chunked patch file. The first decomposed control block and its corresponding data area are directly written into the current chunked patch file. The remaining space of the new file data that can be generated from the current chunked patch file is zero, and the current patch file is compressed. The second decomposed control block and its corresponding data area are added to the overall patch data as the next control block and its corresponding data area. Then, return to step S31 to create the next empty chunked patch file.

[0045] Repeat steps S31 to S33 until all control blocks and data areas of the overall patch data have been written to the individual patch files, and then compress each patch file. At this point, the process of splitting a decompressed overall patch data into multiple patch files is complete.

[0046] Please see Figure 6 The patch file splitting system based on BSDiff proposed in this invention includes a differential processing module 1, a virtual equal division module 2, a patch splitting module 3, and a patch upgrade module 4. Figure 6 The system shown corresponds to Figure 3 The method shown.

[0047] The differential processing module 1 is used to generate an uncompressed overall patch data based on the new and old files using the BSDiff algorithm.

[0048] The virtual partitioning module 2 is used to determine the virtual partitioning value based on the available storage space of the device applying the patch file, wherein the virtual partitioning value is less than or equal to the available storage space of the device applying the patch file. The new file is then virtually divided into multiple data blocks according to the virtual partitioning value. Specifically, the size of the last virtual partitioned data block of the new file is allowed to be less than or equal to the virtual partitioning value.

[0049] The patch splitting module 3 is used to split the uncompressed overall patch data into multiple chunked patch data. The splitting is based on the principle that applying the BSDiff algorithm to each chunked patch data results in a virtual equally divided data block for the new file. Each chunked patch data is then compressed to obtain a single chunked patch file.

[0050] The patch upgrade module 4 receives a segmented patch file each time, decompresses it to obtain the current segmented patch data, and then generates a virtual equally divided data block for the new file based on the old file and the current segmented patch data. This data block is then directly written to the storage area of ​​the new file. The corresponding segmented patch data is then discarded, and the reception of the next segmented patch file begins. This process is repeated until the last virtual equally divided data block of the new file is written to the storage area of ​​the new file.

[0051] This invention first uses the BSDiff algorithm to generate overall patch data based on the old and new files without dividing them into data blocks. Then, it splits the overall patch data into multiple sub-patterns by virtually dividing the new file into equal data blocks. This method retains the advantages of the BSDiff algorithm while avoiding the disadvantages of first dividing the old and new files into equal data blocks and then applying the BSDiff algorithm. The sub-patterns contain no redundant data, have a small size, and reduce the network data required for transmission and the storage space required for differential upgrades.

[0052] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A patch file splitting method based on BSDiff, characterized in that, Includes the following steps; Step S1: Use the BSDiff algorithm to generate an uncompressed overall patch data based on the new and old files; Step S2: Determine the virtual equal division value based on the available storage space of the device applying the patch file; divide the new file into multiple data blocks according to the virtual equal division value; The order of steps S1 and S2 may be either simultaneous or either can be either one before the other. Step S3: Split the uncompressed overall patch data into multiple block patch data, and compress each block patch data to obtain each block patch file; the splitting basis is: so that applying the BSDiff algorithm to each block patch data can obtain a virtual equally divided data block of the new file; Step S4: The device applying the patch file receives one segmented patch file at a time, decompresses it to obtain segmented patch data, and generates a virtual equally divided data block for the new file based on the old file and the current segmented patch data. This data block is then written directly to the storage area of ​​the new file. The corresponding segmented patch data is then discarded, and the reception of the next segmented patch file begins. This process is repeated until the device applying the patch file writes the last virtual equally divided data block of the new file to the storage area of ​​the new file.

2. The patch file splitting method based on BSDiff according to claim 1, characterized in that, In step S1, the last step of the BSDiff algorithm to generate the patch file is compression. This compression step can be omitted to obtain uncompressed overall patch data; or, the BSDiff algorithm can be fully executed to generate a compressed overall patch file, and then decompression can be performed to obtain uncompressed overall patch data.

3. The patch file splitting method based on BSDiff according to claim 1, characterized in that, In step S2, the virtual equal score is less than or equal to the available storage space of the device for the application patch file.

4. The patch file splitting method based on BSDiff according to claim 1, characterized in that, In step S2, when flash memory is used as a non-volatile memory, the virtual equal division value is an integer multiple of the flash memory erase block size.

5. The patch file splitting method based on BSDiff according to claim 1, characterized in that, In step S2, the size of the last virtual equal-division data block of the new file is allowed to be less than or equal to the virtual equal-division value.

6. The patch file splitting method based on BSDiff according to claim 1, characterized in that, In step S3, the number of patch files is equal to the size of the new file divided by the virtual equal division value. If there is a remainder, the quotient is incremented by one.

7. The patch file splitting method based on BSDiff according to claim 1, characterized in that, In step S3, the size of any block patch data is less than or equal to the available storage space of the device applying the patch file.

8. The patch file splitting method based on BSDiff according to claim 1, characterized in that, In step S4, all the virtual equally divided data blocks of the new file are naturally spliced ​​together in the memory of the new file.

9. The patch file splitting method based on BSDiff according to claim 1, characterized in that, Step S3 specifically includes the following sub-steps; Step S31: Create a new empty block patch file containing the new file offset position—used to record the starting position of a virtual partition data block in the new file generated by each block patch; The remaining space for generating new file data from the current chunked patch file is initialized to the virtual equal division value; Step S32: Read each control block in the overall patch data in sequence, add the difference partition data size and the new partition data size in the current control block to obtain the new file data size that can be generated by the current control block; Step S33: If the size of the new file data that can be generated by the current control block is less than or equal to the remaining space of the new file data that can be generated by the current block patch file, then write the current control block and the corresponding data area directly into the current block patch file, and update the remaining space of the new file data that can be generated by the current block patch file by subtracting the size of the new file data that can be generated by the current control block. Then return to step S32 to read the next control block in the overall patch data; If the size of the new file data that can be generated from the current control block is greater than the remaining space of the new file data that can be generated from the current block patch file, then the current control block and its corresponding data area are decomposed into two control blocks and two corresponding data areas. The size of the new file data that can be generated from the first decomposed control block is equal to the remaining space of the new file data that can be generated from the current block patch file. The first decomposed control block and its corresponding data area are directly written into the current block patch file. The remaining space of the new file data that can be generated from the current block patch file is zero, and the current patch file is compressed. The second decomposed control block and its corresponding data area are added to the overall patch data as the next control block and its corresponding data area. Then return to step S31 to create the next empty block patch file; Repeat steps S31 to S33 until all control blocks and data areas in the overall patch data have been written into the individual patch files, and then compress the individual patch files.

10. A patch file splitting system based on BSDiff, characterized in that, This includes a differential processing module, a virtual equal division module, a patch splitting module, and a patch upgrade module; The differential processing module is used to generate an uncompressed overall patch data based on the new file and the old file using the BSDiff algorithm; The virtual division module is used to determine the virtual division value based on the available storage space of the device applying the patch file, and to virtually divide the new file into multiple data blocks according to the virtual division value. The patch splitting module is used to split the uncompressed overall patch data into multiple block patch data, and compress each block patch data to obtain each block patch file; the splitting basis is: so that applying the BSDiff algorithm to each block patch data can obtain a virtual equally divided data block of the new file; The patch upgrade module receives a block patch file each time, decompresses it to obtain the current block patch data, and then generates a virtual equally divided data block for the new file based on the old file and the current block patch data. This data block is then directly written into the storage area of ​​the new file. The corresponding block patch data is then discarded, and the reception of the next block patch file begins. This process is repeated until the last virtual equally divided data block of the new file is written into the storage area of ​​the new file.

Images