A method for generating a differential packet

By generating a hash set and matching it with the new version of the source file, the problem of high hardware requirements in the production of large program differential packages in the existing technology is solved, and efficient differential package generation is achieved.

CN115328525BActive Publication Date: 2026-11-10ABUP TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210847859.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-19
Publication Date
2026-11-10
Estimated Expiration
2042-07-19

AI Technical Summary

Technical Problem

Existing technologies require the complete loading of source files into the production environment's memory when creating differential packages for large programs, resulting in high hardware requirements and making it difficult to implement efficiently in environments with poor hardware conditions.

Method used

By generating a hash set of the old version source file and matching the hash set with the new version source file, differential intermediate data is generated. Only the hash set needs to be loaded into memory for matching, avoiding the complete loading of the old version source file.

Benefits of technology

This significantly reduces the memory requirements of the production environment and improves the production efficiency of creating differential packages.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115328525B_ABST
    Figure CN115328525B_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of embedded system, and particularly relates to a differential package generation method, comprising the following steps: S1, generating a hash set for an old version source file of a computer program; S2, matching a new version source file of the computer program by using the hash set to obtain differential intermediate data; S3, generating the differential package according to the differential intermediate data. The present application has the beneficial effect that in the process of making the differential package, the hash set is generated for the old version source file, and in the process of matching the new version source file, only the hash set needs to be matched, so that the differential intermediate data is found and the differential package is made. In the above process, only the hash set of the old version source file needs to be completely loaded in the memory, which greatly reduces the memory demand for the production environment compared with the scheme in the prior art that the source file needs to be completely loaded into the memory for matching, and improves the production efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data compression technology, and more specifically to a method for generating differential packets. Background Technology

[0002] A differential update patch, also known as a patch updater, is created by extracting the differences between a new and an older version of a computer system during the system update process. It is commonly used in computer systems to update released and commercially available systems. By creating differential updates, users only need to obtain the parts to be updated and combine them with existing software during the update process, avoiding the time-consuming problem of a full system rewrite. Especially in the field of large programs, using differential updates for updates such as games, in-vehicle infotainment systems, and video production software significantly saves bandwidth and processing time compared to traditional full installation packages, thus finding widespread application in embedded systems.

[0003] In existing technologies, there are quite a few technical solutions for creating difference packets based on difference algorithms. A typical difference algorithm (such as bsdiff) requires first sorting the source files by their suffixes to obtain a suffix array. Then, when matching a new file, both the source file and the suffix array need to be loaded into memory. The difference packet is then created by searching the suffix array.

[0004] However, in practice, the inventors found that most existing methods for creating differential packages require loading the entire source file into the production environment's memory. However, when creating differential packages for large programs, the source files are often very large, ranging from tens to hundreds of gigabytes. Therefore, these methods are difficult to implement in production environments with relatively poor hardware conditions, often requiring reliance on methods such as paging file swapping to read the source files, which significantly restricts production efficiency. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, a method for generating differential packets is provided.

[0006] The specific technical solution is as follows:

[0007] A method for generating differential packets, comprising:

[0008] Step S1: Generate a hash set for an older version of a computer program's source file;

[0009] Step S2: Use the hash set to match the new version source file of the computer program to obtain differential intermediate data;

[0010] Step S3: Generate the differential packet based on the differential intermediate data.

[0011] Preferably, step S1 includes:

[0012] Step S11: Divide the old version source file into sections to generate multiple substrings;

[0013] Step S12: Calculate the hash value for each of the substrings;

[0014] Step S13: Sort all the hash values ​​according to their size to generate the hash set.

[0015] Preferably, step S11 includes:

[0016] Step S111: Obtain the file size of the old version source file and the available memory of the operating environment;

[0017] Step S112: Generate the segment length based on the file size and the available memory;

[0018] Step S113: Divide the old version source file into sections according to the section length to generate multiple substrings.

[0019] Preferably, in step S112, the method for generating the segment length includes:

[0020]

[0021] Where k is the segment length, n is the file size in bytes, and m is the available memory.

[0022] Preferably, step S2 includes:

[0023] Step S21: Extract a matching substring from the position pointed to by the starting pointer in the new version source file, and generate a rolling hash of the matching substring;

[0024] Before executing step S21 for the first time, the start pointer is pre-set to point to the beginning position of the new version source file;

[0025] Step S22: Use the rolling hash to search the hash set to determine if a first matching result exists;

[0026] If so, proceed to step S23;

[0027] If not, based on the first matching result, choose to move the starting pointer or update the matched pointer, and then return to step S21;

[0028] Step S23: Match the new version source file and the old version source file according to the starting pointer, generate control triples and unmatched parts, and update the matched pointer;

[0029] Step S24: Use the control triplet and the unmatched portion as the intermediate difference data, then return to step S21 until the new version source file is traversed and then proceed to step S3;

[0030] The control triplet includes: a matching offset, a matching length, and an unmatched length.

[0031] Preferably, before step S21, the length of the substring is obtained from the hash set, and the length of the matching substring is equal to the length of the substring.

[0032] Preferably, in step S22, when there is no matching result, it is determined whether the matching length exceeds the distance of one substring length extending forward from the position pointed to by the starting pointer;

[0033] If so, update the matched pointer;

[0034] If not, move the starting pointer.

[0035] Preferably, step S23 includes:

[0036] Starting from the position pointed to by the starting pointer, perform forward and reverse matching on the new version source file based on the old version source file;

[0037] During the reverse matching process, the maximum length of the reverse matching is the difference between the starting pointer and the matched pointer, and the starting position of the reverse matching is the position pointed to by the starting pointer.

[0038] The termination condition for the reverse matching is: the second matching result of the reverse matching shows that there is no character in the new version source file that corresponds to the old version source file, or the maximum length is reached;

[0039] During the forward matching process, the starting position of the forward matching is a distance extending forward from the position pointed to by the starting pointer by one substring length;

[0040] The termination condition for the forward matching is: until the third matching result of the forward matching indicates that there is no character in the new version source file that corresponds to the old version source file;

[0041] After the forward matching and the reverse matching are completed, the control triplet and the unmatched part are generated based on the second matching result and the third matching result, and the matched pointer is updated.

[0042] Preferably, after performing step S22 and before performing step S23, the method further includes:

[0043] Step S221: Based on the first matching result, search for the original code string in the old version source file, and determine whether it is a true match based on the original code string and the matching substring;

[0044] If so, proceed to step S23;

[0045] If not, move the starting pointer or update the matched pointer, and then return to step S21.

[0046] Preferably, step S3 includes: compressing all the differential intermediate data to obtain the differential packet.

[0047] The above technical solution has the following advantages or beneficial effects: In the process of creating the differential package, a hash set is generated from the old version source file, and during the matching process with the new version source file, only the hash set needs to be matched to find the intermediate differential data and create the differential package. In this process, only the hash set of the old version source file needs to be fully loaded into memory. Compared to existing technologies that require loading the entire source file into memory for matching, this greatly reduces the memory requirements of the production environment and improves production efficiency. Attached Figure Description

[0048] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0049] Figure 1 This is an overall schematic diagram of an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of sub-step S1 in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of sub-step S11 in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of sub-step S2 in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of step S221 in an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0056] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0057] This invention includes:

[0058] A method for generating differential packets, such as Figure 1 As shown, it includes:

[0059] Step S1: Generate a hash set for an older version of a computer program's source file;

[0060] Step S2: Use a hash set to match the new version of the computer program's source file to obtain differential intermediate data;

[0061] Step S3: Generate a differential package based on the differential intermediate data.

[0062] Specifically, addressing the issue that existing differential algorithms require loading the entire old version source file into the production environment's memory, placing significant hardware demands on the production environment, this embodiment addresses this problem by generating a hash set for the old version source file during the differential package creation process. The matching process between the new and old version source files is then based on this hash set, generating intermediate differential data. During the comparison process, only the hash set and the hash value calculated from the new version source file need to be matched, without needing to load the entire old version source file into memory, significantly reducing the hardware requirements for the production environment.

[0063] In implementation, the old version source file refers to the source file of a computer program that is earlier than the new version source file in at least one of the parameters such as version number, release date, and compilation date. It is typically an embedded system or other software program that has been released and installed in an embedded device. In this invention, the computer program usually refers to a large program file, which often cannot be fully loaded into a typical computer production environment. For example, in the application scenario of an in-vehicle infotainment system, sometimes it is necessary to create a differential package for the complete image of the entire system, rather than just a module within it. In this case, the source file of the complete image of the system needs to be loaded into memory. To address this, a hash set can be generated and loaded into memory using the method described above, thereby reducing the memory requirements of the production environment. The hash set refers to a collection of multiple hash values ​​calculated based on the old version source file, with each hash value corresponding to a specific code segment in the old version source file. The differential intermediate data refers to the differences between the new and old versions of the source file found during the matching process, i.e., the parts that need to be updated by creating a differential package. Differential intermediate data includes control triples used to mark the locations of mismatched portions, and the mismatched portions (i.e., the code to be updated). A differential package refers to a differential update patch created based on the differential intermediate data, which is used to update devices that already have the software program corresponding to the old version of the source file to the software version corresponding to the new version of the source file.

[0064] In a preferred embodiment, such as Figure 2 As shown, step S1 includes:

[0065] Step S11: Divide the old version of the source file into sections to generate multiple substrings;

[0066] Step S12: Calculate the hash value for each substring;

[0067] Step S13: Sort all hash values ​​according to their size to generate a hash set.

[0068] Specifically, addressing the issue that existing differential algorithms require loading the entire old source file into the production environment's memory, placing significant hardware demands on the production environment, this embodiment divides the old source file into segments to generate multiple substrings. For each substring, its hash value is calculated, and an ordered hash set is generated and loaded into memory. This allows for comparison of the source code length for each substring during the actual matching process, thereby avoiding the need to load the entire old source file into memory, which imposes high requirements on the production environment.

[0069] In a preferred embodiment, such as Figure 3 As shown, step S11 includes:

[0070] Step S111: Obtain the file size of the old version source file and the available memory of the operating environment;

[0071] Step S112: Generate a segment length according to the file size and the available memory;

[0072] Step S113: Segment the old version source file according to the segment length to generate a plurality of substrings.

[0073] Specifically, in order to make the matching length for the new version source file in the matching process conform to the hardware conditions of the actual production environment, in this embodiment, each time a differential package is created and substrings are generated, the file size of the old version source file under the current condition and the available memory of the operating environment are respectively obtained, and the segment length in the current differential package creation process is calculated according to the file size and the available memory. The segment length is used to segment the old version source file, and the same length is used for matching in the subsequent matching process, so that each differential package creation process conforms to the hardware situation of the actual production environment.

[0074] In a preferred embodiment, in step S112, the method for generating the segment length includes:

[0075]

[0076] where k is the segment length, n is the file size in bytes, and m is the available memory.

[0077] Specifically, in order to make the matching length for the new version source file in the matching process conform to the hardware conditions of the actual production environment, in this embodiment, the segment length is comprehensively determined by combining the actual file size and available memory, which avoids exceeding the available memory of the production environment during the matching process, and also avoids the problem of low efficiency caused by excessive segments and excessive matching and searching times.

[0078] Specifically, let the file size be n (bytes), and the available memory be m (m < n). There are two parts of data that need to reside in the memory: the hash of the k-substrings of the source file and the result of hash sorting; the space required for the former is ( is the ceiling function), and the space required for the latter is (if 1G=2 30 ), or (if ). Then the k value, that is, the segment length, can be obtained based on the above formula, and the source file is segmented into substrings each with a length of k, and hash values are calculated respectively, so as to obtain a hash set.

[0079] As an optional implementation, the multiple hash values ​​in the hash set are arranged from largest to smallest or from smallest to largest, so as to improve the matching speed in step S2 by using binary search.

[0080] In a preferred embodiment, such as Figure 4 As shown, step S2 includes:

[0081] Step S21: Extract a matching substring from the position pointed to by the starting pointer in the new version of the source file, and generate a rolling hash of the matching substring;

[0082] Before executing step S21 for the first time, a zero value is pre-assigned to the starting pointer;

[0083] Step S22: Use rolling hashing to search the hash set and determine if a first matching result exists;

[0084] If so, proceed to step S23;

[0085] If not, based on the first matching result, choose to move the starting pointer or update the matched pointer, and then return to step S21;

[0086] Step S23: Match the new version source file and the old version source file according to the starting pointer, generate control triples and unmatched parts, and update the matched pointers;

[0087] Step S24: Use the control triples and unmatched parts as intermediate difference data, then return to step S21 until the new version source file is traversed and then proceed to step S3;

[0088] The control triple includes: match offset, match length, and unmatch length.

[0089] Specifically, addressing the issue in existing technologies that require loading the entire old version source file into memory when matching two versions of source files, resulting in significant memory demands, this embodiment addresses this problem by setting a start pointer and a matched pointer during the matching process of the new version source file. This involves performing a rolling hash match between the new version source file and the hash set already loaded in memory. This process identifies the different unmatched portions between the new and old versions of the source file and their locations. The locations of these unmatched portions can be used to generate control triples, which then serve as intermediate difference data. This achieves the recording and location marking of unmatched portions, thereby generating the corresponding difference packet.

[0090] During implementation, the starting pointer `p` represents the beginning of the match, and the matched pointer `q` represents the processed portion. Both the starting and matched pointers point to the beginning of the new version of the source file at the start of the match, i.e., `p = q = 0`. The control triple includes the matched offset, matched length, and unmatched length, set sequentially. The matched offset and matched length represent the length of the match between the current new file position and the source file starting from the specified offset. The unmatched length refers to the length of the remaining unmatched data, which is placed in the subsequent (unmatched) data area. Based on these settings, after each match between the corresponding positions of two versions of the source file, the position of the unmatched portion can be marked by recording the control triple. The rolling hash refers to the hash value of the matched substring whose length equals the segment length during the matching process. During the cyclical matching process, after each match, the starting pointer moves one byte forward, causing the entire matched substring to move one byte forward while maintaining its length, thus generating a new rolling hash.

[0091] As an optional implementation, there are two cases for traversing the new version source file in step S24:

[0092] When a match reaches the end of a new file: the matching ends after a new triple is generated;

[0093] Alternatively, if the starting pointer p reaches the end of the new file: at this point, the data between the starting pointer p and the matched pointer q is unmatched, the current triplet is updated to the unmatched length z = z + (pq), and the matching ends.

[0094] In a preferred embodiment, before step S21, the substring length is obtained from the hash set, and the length of the matched substring is equal to the substring length.

[0095] In a preferred embodiment, in step S22, when there is no first matching result, it is determined whether the matching length exceeds the length of a substring extended from the starting pointer.

[0096] If so, update the matched pointer;

[0097] If not, move the starting pointer.

[0098] Specifically, in order to achieve better matching results, in this embodiment, when it is determined that the rolling hash does not match any hash value in the hash set, it is further determined whether the matching length exceeds the substring length k. If it does, the matched pointer and the control triplet are updated so that the unmatched length in the matched pointer and the control triplet is increased by one byte. If it does not exceed, only the starting pointer is moved.

[0099] In a preferred embodiment, step S23 includes:

[0100] Starting from the position pointed to by the starting pointer, perform forward and reverse matching on the new version of the source file based on the old version of the source file;

[0101] During the reverse matching process, the maximum length of the reverse matching is the difference between the starting pointer and the matched pointer, and the starting position of the reverse matching is the position pointed to by the starting pointer.

[0102] The termination condition for reverse matching is: the second matching result of the reverse matching shows that there is no character in the new version source file that corresponds to the old version source file, or the maximum length is reached;

[0103] In the forward matching process, the starting position of the forward matching is the distance of one substring length extended forward from the position pointed to by the starting pointer;

[0104] The termination condition for forward matching is: until the third match result of the forward matching shows that there is no character in the new version source file that corresponds to the old version source file;

[0105] After forward and reverse matching are completed, control triples and unmatched parts are generated based on the second and third matching results, and the matched pointer is updated.

[0106] Specifically, to achieve better reading results for differential intermediate data, this embodiment loads the corresponding substring from the old version source file based on the hash value in the hash set when the hash values ​​in the rolling hash and hash set match. Then, in the new version source file, forward and reverse matching are performed based on the starting pointer position. The reverse matching length is at most pq characters, and the direction is from the starting pointer position towards the beginning of the new version source file. The forward matching needs to match as many characters as possible from position p+k towards the end of the new version source file. Let the lengths of the reverse and forward matching be l and r, respectively. The old triplet is updated to z = z + (pqr), and a new triplet is generated: x records the start (relative offset) of the current reverse matching, y = l + k + r, z = 0. p and q are moved to the end of the forward matching, p = p + k + l, q = p, thus realizing the matching process of the two versions of the source file, extracting the unmatched part to be updated, and marking the position of the unmatched part by controlling the triplet.

[0107] In a preferred embodiment, after performing step S22 and before performing step S23, the method further includes:

[0108] Step S221: Based on the first matching result, search for the original code string in the old version source file, and determine whether it is a real match based on the original code string and the matching substring;

[0109] If so, proceed to step S23;

[0110] If not, move the starting pointer or update the matched pointer, and then return to step S21.

[0111] Specifically, to improve the overall matching efficiency, in this embodiment, after a matching result exists, the source code string in the old version source file and the currently matched substring are searched to determine whether it is a real matching result, thus avoiding the problem of subsequent character matching being unable to be performed due to false matching.

[0112] In a preferred embodiment, step S3 includes: compressing all the differential intermediate data to obtain a differential packet.

[0113] The beneficial effects of this invention are as follows: In the process of creating a differential packet, a hash set is generated from the old version source file, and during the matching process with the new version source file, only the hash set needs to be matched to find the intermediate differential data and create the differential packet. In the above process, only the hash set of the old version source file needs to be fully loaded into memory. Compared with the existing technology that requires loading the entire source file into memory for matching, this greatly reduces the memory requirements of the production environment and improves production efficiency.

[0114] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating differential packets, characterized in that, include: Step S1: Generate a hash set for an older version of a computer program's source file; Step S2: Use the hash set to match the new version source file of the computer program to obtain differential intermediate data; Step S3: Generate the differential packet based on the differential intermediate data; Step S1 includes: Step S11: Divide the old version source file into sections to generate multiple substrings; Step S12: Calculate the hash value for each of the substrings; Step S13: Sort all the hash values ​​according to their size to generate the hash set; Step S11 includes: Step S111: Obtain the file size of the old version source file and the available memory of the operating environment; Step S112: Generate the segment length based on the file size and the available memory; Step S113: Divide the old version source file into sections according to the section length to generate multiple substrings; In step S112, the method for generating the segment length includes: ; in, The segment length is... The file size is in bytes. The available memory; Step S2 includes: Step S21: Extract a matching substring from the position pointed to by the starting pointer in the new version of the source file, and generate a rolling hash of the matching substring; Step S22: Use the rolling hash to search the hash set and determine if there is a first matching result. If yes, proceed to step S23; if no, move the starting pointer or update the matched pointer, and then return to step S21. Step S23: Match the new version source file and the old version source file according to the starting pointer, generate control triples and unmatched parts, and update the matched pointer; Step S24: Use the control triplet and the unmatched portion as the differential intermediate data; Step S23 includes: Starting from the position pointed to by the starting pointer, perform forward and reverse matching on the new version source file based on the old version source file; During the reverse matching process, the maximum length of the reverse matching is the difference between the starting pointer and the matched pointer, and the starting position of the reverse matching is the position pointed to by the starting pointer. The termination condition for the reverse matching is: the second matching result of the reverse matching shows that there is no character in the new version source file that corresponds to the old version source file, or the maximum length is reached; During the forward matching process, the starting position of the forward matching is a distance extending forward from the position pointed to by the starting pointer by one substring length; The termination condition for the forward matching is: until the third matching result of the forward matching indicates that there is no character in the new version source file that corresponds to the old version source file; After the forward matching and the reverse matching are completed, the control triplet and the unmatched part are generated based on the second matching result and the third matching result, and the matched pointer is updated. The control triplet includes: a matching offset, a matching length, and an unmatched length.

2. The generation method according to claim 1, characterized in that, Step S2 includes: Before executing step S21 for the first time, the starting pointer is pre-set to the starting position of the new version source file; In step S24, the control triplet and the unmatched portion are used as the differential intermediate data, and then the process returns to step S21 until the new version source file is traversed and then the process proceeds to step S3.

3. The generation method according to claim 2, characterized in that, Before step S21, the length of the substring is obtained from the hash set, and the length of the matching substring is equal to the length of the substring.

4. The generation method according to claim 3, characterized in that, In step S22, when there is no first matching result, it is determined whether the matching length exceeds the distance of one substring length extending forward from the position pointed to by the starting pointer; If so, update the matched pointer; If not, move the starting pointer.

5. The generation method according to claim 2, characterized in that, After performing step S22 and before performing step S23, the method further includes: Step S221: Based on the first matching result, search for the original code string in the old version source file, and determine whether it is a true match based on the original code string and the matching substring; If so, proceed to step S23; If not, move the starting pointer or update the matched pointer, and then return to step S21.

6. The generation method according to claim 1, characterized in that, Step S3 includes: compressing all the differential intermediate data to obtain the differential packet.

Citation Information

Patent Citations

  • Block-based differencing algorithm

    US20110202503A1

  • Automatic file differencing and updating system

    US5832520A