Method and apparatus for data processing and file updating

Abstract

The application provides a data processing and file updating method and device. The method is applied to a server and includes the following steps: obtaining an original file and a target file. The original file and the target file both include a header file and archive data. The archive data includes an archive header file, a hash table, a block table and a file data area. The file data of each original sub-file in the original file and the file data of each target sub-file in the target file in the respective file data areas are determined according to the hash table and the block table of the original file and the target file. The file data of each target sub-file is compared with the file data of each original sub-file to determine the incremental state and the differential file data of each target sub-file. An update data packet is generated according to the header file of the target file, the archive header file, the incremental state and the differential file data of each target sub-file. The update data packet is used to update the original file in a client.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a method and apparatus for data processing and file updating. Background Technology

[0002] In the field of incremental updates, BSDiff (a binary difference algorithm) is a widely used binary file difference algorithm that can efficiently calculate the differences between two files and generate a difference file. However, when performing the difference calculation, the BSDiff algorithm needs to load the old file data, the new file data, and the auxiliary data structures generated during the difference process into memory simultaneously. Its peak memory usage is typically about 20 times the size of the file to be differencing. For small files, this memory overhead is acceptable. But when the size of the file to be differencing is large (e.g., tens or even hundreds of megabytes), the memory consumption of BSDiff will increase dramatically, potentially leading to memory overflow or a severe degrade in system performance.

[0003] In applications such as virtual maps, map files are typically archived using archive formats. These files consist of a file header and embedded archive data. The embedded archive data uses the archive header, hash table, block table, and file data area to archive and store multiple sub-files (such as script files, terrain data, unit data, etc.). Specifically, the hash table establishes a mapping between filenames and block table entries using filename hash values ​​as indexes. The block table records the offset position, data size, and data attribute flags of each sub-file in the file data area, and the file data area stores the actual data of each sub-file. These types of archived data files are often quite large, and directly performing BSDiff differencing on the entire file would result in extremely high memory consumption. Summary of the Invention

[0004] This application provides a method and apparatus for data processing and file updating.

[0005] This application provides a data processing method, applied to a server, the method comprising: Obtain the original file and the target file, both of which include a header file and archive data. The archive data includes an archive header file, a hash table, a block table, and a file data area. The file data of each original sub-file in the original file and each target sub-file in the target file are determined in their respective file data areas based on the hash table and block table of the original file and the target file; The file data of each target sub-file is compared with the file data of each original sub-file to determine the incremental status and differential file data of each target sub-file. An update data packet is generated based on the header file of the target file, the archive header file, the incremental status of each target sub-file, and the differential file data. The update data packet is used to update the original file in the client.

[0006] The step of determining the file data of each original sub-file in the original file and each target sub-file in the target file in their respective file data areas based on the hash table and block table of the original file and the target file includes: The hash table and block table of the original file and the target file are determined based on the archive header files of the original file and the target file; Parse the hash tables of the original file and the target file to obtain the first position information of each original sub-file and each target sub-file in their respective block tables; Based on the first location information, determine the second location information of each original sub-file and each target sub-file in their respective file data areas from the block table; The file data of each original sub-file and each target sub-file are determined from the file data area based on the second location information.

[0007] The step of comparing the file data of each target sub-file with the file data of each original sub-file to determine the incremental state and difference file data of each target sub-file includes one of the following: If the original file corresponding to the target sub-file does not exist in the original file, then the incremental status of the target sub-file is marked as newly added; and the file data of the target sub-file is determined as differential file data. If the original file contains an original subfile corresponding to the target subfile, and the target subfile is identical to the original subfile, then the incremental status of the target subfile is marked as identical. If the original file contains an original subfile corresponding to the target subfile, and the target subfile is inconsistent with the original subfile, then the incremental status of the target subfile is marked as inconsistent; the differential file data is determined based on the target subfile and the original subfile.

[0008] Prior to parsing the hash tables of the original file and the target file, the method further includes: Parse the archive header files of the original file and the target file to obtain the third position information of the hash table of the original file and the target file in their respective archive data and the fourth position information of the block table in their respective archive data; The hash tables of the original file and the target file are determined from the archived data based on the third location information; The block tables of the original file and the target file are determined from the archived data based on the fourth location information.

[0009] The step of parsing the hash tables of the original file and the target file to obtain the first position information of each original sub-file and each target sub-file in their respective block tables includes: The entry data corresponding to each original sub-file and each target sub-file in their respective hash tables are determined based on the number of entries in the hash table. The number of entries is determined by parsing the archive header files of the original files and the target files. Data is read from the entry data to obtain the first position information of each original sub-file and each target sub-file.

[0010] Another embodiment of this application provides a file update method applied to a client, the method comprising: Based on the received update data packets sent by the server, determine the header file of the target file, the archive header file, the incremental status of each target subfile, and the differential file data; The file data of each original sub-file in the original file is determined in the file data area based on the hash table and block table in the original file. The file data of each original sub-file is updated based on the incremental status of each target sub-file and the differential file data to obtain the file data of each target sub-file; The hash table and block table of the original file are updated based on the file data of each target sub-file to obtain the hash table and block table of the target file; The target file is generated based on the header file, archive header file, hash table and block table of the target file, and file data of each target sub-file.

[0011] The step of updating the file data of each original sub-file based on the incremental status of each target sub-file and the differential file data includes one of the following: If the incremental status of the target sub-file is marked as newly added, the differential file data of the target sub-file is determined as the corresponding file data; If the incremental status of the target sub-file is marked as consistent, the file data of the original sub-file in the original file that corresponds to the target sub-file is determined as the file data of the target sub-file. If the incremental status of the target sub-file is marked as inconsistent, the file data of the original sub-file corresponding to the target sub-file in the original file is updated according to the differential file data of the target sub-file to obtain the file data of the target sub-file.

[0012] The step of updating the hash table and block table of the original file based on the file data of each target sub-file includes at least one of the following: If the incremental status of the target sub-file is marked as newly added or inconsistent, the block table is updated according to the data size of the file data of the target sub-file and its second position information in the file data area of ​​the target file; the hash table is updated according to the first position information in the block table based on the second position information. If the incremental status of the target sub-file is marked as consistent, the block table is updated according to the second position information of the file data of the target sub-file in the file data area of ​​the target file; the hash table is updated according to the first position information of the block table based on the second position information.

[0013] Another embodiment of this application provides a data processing apparatus, the apparatus comprising: The acquisition module is used to obtain the original file and the target file. Both the original file and the target file include a header file and archived data. The archived data includes an archived header file, a hash table, a block table, and a file data area. The first calculation module is used to determine the file data of each original sub-file in the original file and each target sub-file in the target file in their respective file data areas based on the hash table and block table of the original file and the target file; The first calculation module is further configured to compare the file data of each target sub-file with the file data of each original sub-file to determine the incremental status and differential file data of each target sub-file; The first processing module is used to generate an update data packet based on the header file of the target file, the archive header file, the incremental status of each target sub-file, and the differential file data. The update data packet is used to update the original file in the client.

[0014] Another embodiment of this application provides a file updating apparatus, the apparatus comprising: The communication module is used to determine the header file, archive header file, incremental status of each target subfile, and differential file data of the target file based on the update data packets sent by the server. The second calculation module is used to determine the file data of each original sub-file in the original file in the file data area based on the hash table and block table in the original file. The second calculation module is further configured to update the file data of each original sub-file based on the incremental status of each target sub-file and the differential file data, so as to obtain the file data of each target sub-file; The second calculation module is further configured to update the hash table and block table of the original file based on the file data of each target sub-file, so as to obtain the hash table and block table of the target file; The second processing module is used to generate the target file based on the header file, archive header file, hash table and block table of the target file, and file data of each target sub-file.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0016] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0017] Figure 1 A flowchart of a data processing method according to an embodiment of this application is shown; Figure 2 A flowchart of a data processing method according to another embodiment of this application is shown; Figure 3 A flowchart of a data processing method according to another embodiment of this application is shown; Figure 4 A flowchart of a data processing method according to another embodiment of this application is shown; Figure 5 A flowchart of a file update method according to an embodiment of this application is shown; Figure 6 A schematic diagram of the structure of a data processing apparatus according to an embodiment of this application is shown; Figure 7 A schematic diagram of a file update apparatus according to an embodiment of this application is shown; Figure 8 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To reduce memory consumption during incremental file calculation, one embodiment of this application provides a data processing method applied to a server, such as... Figure 1 As shown, the method includes: Step 101: Obtain the original file and the target file. Both the original file and the target file include a header file and archived data. The archived data includes an archived header file, a hash table, a block table, and a file data area.

[0020] Both the source and target files are files containing header files and archived data. The source file can be an older version, and the target file can be a newer version. Both the source and target files can be W3X (a file format), W3M (a file format), other files containing archived data structures, or other files suitable for data archiving.

[0021] The header file, located at the beginning of the file, stores the file's metadata. It may include file identifiers (such as magic numbers), version information, flags, maximum number of users, and other information. For example, the header file might include a 4-byte magic number to identify the file type, a map name string, map flags, and a maximum player number field.

[0022] Archived data follows the header file and is used to store multiple sub-files in a packaged format. The archived data uses an archive format to organize its internal data, allowing multiple sub-files to be transmitted and stored as a single, unified file. Archived data includes an archive header file, a hash table, a block table, and a file data area.

[0023] The archive header is the metadata area of ​​the archived data, located at the beginning of the archive. It stores information about the location and size of the hash table and the block table within the archive. The archive header is the entry point for parsing the hash table and block table. For example, the archive header may include fields such as archive signature, archive header size, total archive size, format version number, hash table offset, block table offset, number of hash table entries, and number of block table entries.

[0024] A hash table is an index structure in archived data, used to establish a mapping between subfile names and block table entries. Each entry in the hash table stores the hash value obtained by hashing the subfile name and the index of the corresponding entry in the block table. Hash tables can use open addressing or chaining to handle hash collisions.

[0025] The block table, also an index structure in the archived data, records the storage location and data attributes of each subfile within the file data area. Each entry in the block table stores the subfile's offset within the file data area, its data size, and data attribute flags. The data attribute flags indicate whether the subfile's data has been compressed and / or encrypted.

[0026] The file data area is the region within the archived data where the actual data of sub-files is stored. Data in the file data area can be compressed and / or encrypted, or stored in its raw form. Data for each sub-file in the file data area can be located and retrieved according to the offset and data size recorded in the block table.

[0027] For example, the platform manages different versions of virtual map files as archived data. The original file is the archived data file of version 1.0, and all clients accessing the platform have also downloaded this version 1.0 archived data file. When a new version of the map is released, the virtual map file in the platform is updated to the archived data file of version 2.0, meaning the target file is the archived data file of version 2.0. The original file includes a 512-byte header file followed by archived data, which contains five sub-files: a script file (10KB), a terrain data file (50KB), a unit data file (20KB), a trigger data file (15KB), and a sound effects data file (100KB). The total size of the original file is approximately 197KB. The target file consists of a 512-byte header file followed by archived data. The archived data contains six sub-files: a script file (12KB, modified content), a terrain data file (50KB, unchanged content), a unit data file (25KB, modified content), a trigger data file (15KB, unchanged content), a sound effects data file (100KB, unchanged content), and a model data file (30KB, a newly added file). The total size of the target file is approximately 232KB.

[0028] Step 102: Determine the file data of each original sub-file in the original file and each target sub-file in the target file in their respective file data areas based on the hash table and block table of the original file and the target file.

[0029] For both the original and target files, the file data of each subfile in the file data area is determined using their respective hash tables and block tables. Since the hash tables and block tables form a complete index link from the subfile name to the file data, the server can parse these two index structures to locate and extract the data of each subfile one by one without completely decompressing the entire archive data.

[0030] The process of determining the data in a subfile can be achieved in the following ways: The first method involves matching the hash values ​​in the hash table with a list of filenames (which can be preset files stored on the server or optional files stored in archived data within files) to determine the index of each sub-file in the block table. Then, based on the index, the offset and data size of the sub-file in the file data area are read from the block table to extract the file data.

[0031] The second method involves traversing all valid entries in the hash table to directly obtain the block table index corresponding to each sub-file, and then reading the position information from the block table based on the index to extract the file data.

[0032] Continuing with the previous example, for the original file, its hash table is parsed to determine the indices of its five sub-files in the block table. Then, based on these indices, the offset and data size of each sub-file in the file data area are read from the block table to extract the file data of each sub-file. For the target file, its hash table is parsed to determine the indices of its six sub-files in the block table. Then, based on these indices, the position information of each sub-file is read from the block table to extract the file data of each sub-file.

[0033] Step 103: Compare the file data of each target sub-file with the file data of each original sub-file to determine the incremental status and differential file data of each target sub-file.

[0034] The process iterates through each target subfile of the target file, comparing it with the corresponding original subfile in the original file. The incremental status of each target subfile is determined based on file existence and data consistency. The incremental status can include three types: newly added, consistent, and inconsistent.

[0035] File data can be compared in the following ways: The first method uses a byte-by-byte comparison approach, comparing the file data of the target sub-file with the file data of the original sub-file byte by byte. If all bytes are the same, the file is considered to be consistent; otherwise, it is considered to be inconsistent.

[0036] The second method uses hash digest comparison to calculate the hash digest values ​​of the target sub-file and the original sub-file (such as MD5 (a message digest algorithm), SHA-256 (a secure hash algorithm), etc.), and determines whether the file data is consistent by comparing whether the hash digest values ​​are the same.

[0037] The calculation of differential file data can be achieved in the following ways: The first method uses BSDiff to calculate the difference between the target subfile and the original subfile.

[0038] The second method uses rsync (a data synchronization differential algorithm) to calculate the difference between the target subfile and the original subfile.

[0039] The third method uses xdelta3 (an incremental encoding algorithm) to calculate the difference between the target subfile and the original subfile.

[0040] Continuing with the previous example, the six sub-files of the target file are compared with their corresponding sub-files in the original file. The script file has a corresponding sub-file in the original file, but the file data is inconsistent (size changes from 10KB to 12KB). Its incremental status is marked as inconsistent, and the BSDiff algorithm is used to calculate the difference file data. The terrain data file has a corresponding sub-file in the original file, and the file data is consistent; its incremental status is marked as consistent. The unit data file has a corresponding sub-file in the original file, but the file data is inconsistent (size changes from 20KB to 25KB). Its incremental status is marked as inconsistent, and the BSDiff algorithm is used to calculate the difference file data. The trigger data file and sound effect data file have corresponding sub-files in the original file, and the file data is consistent; their incremental status is marked as consistent. The model data file does not have a corresponding sub-file in the original file; its incremental status is marked as newly added, and the complete file data (30KB) of this sub-file is used as the difference file data.

[0041] Step 104: Generate an update data packet based on the header file of the target file, the archive header file, the incremental status of each target sub-file, and the differential file data. The update data packet is used to update the original file in the client.

[0042] An update data package is generated based on the comparison results. The update data package may include the header file and archive header file of the target file, the incremental status of each target subfile, the difference file data of the subfiles with inconsistent incremental status, and the complete file data of the subfiles with newly added incremental status.

[0043] The format of the update data packet can be achieved in the following ways: The first method is to use the ZIP (a compressed file format) format to package the data into a single ZIP file.

[0044] The second method is to use the TAR (an archive file format) format to package the data into a single TAR package.

[0045] The third method uses a custom binary format for packaging, writing each part of the data into a binary file according to a preset structure.

[0046] Continuing with the previous example, the incremental status of each target sub-file is determined, including inconsistent script files, consistent terrain data files, inconsistent unit data files, consistent trigger data files, consistent sound effect data files, and newly added model data files. The difference data between the incrementally inconsistent target sub-files and their corresponding original sub-files, the complete data of the incrementally added target sub-files, and the header and archive header files of the target sub-files are packaged into a ZIP-formatted update data package. This update data package is much smaller than the total size of the target file. After receiving this update data package, the client can reconstruct the target file based on the local original file.

[0047] In the above scheme, by parsing the hash table and block table inside the original file and the target file, the large archive file is broken down into sub-files, and differential calculation is performed on each sub-file one by one. This reduces the peak memory usage of differential calculation from about 20 times the total file size to several times the sum of the sizes of each sub-file, thereby effectively solving the memory bottleneck problem of differential calculation for large archive files, while ensuring the accuracy and completeness of the differential results.

[0048] This application also provides a data processing method in one example, such as Figure 2 As shown, determining the file data of each original sub-file in the original file and each target sub-file in the target file in their respective file data areas based on the hash table and block table of the original file and the target file includes: Step 201: Determine the hash table and block table of the original file and the target file based on the archive header files of the original file and the target file.

[0049] The archive header file records the location and size information of the hash table and block table within the archived data. By parsing the archive header file, the starting offset and number of entries for the hash table, as well as the starting offset and number of entries for the block table, can be determined within the archived data. Based on this location information, the complete data of the hash table and block table can be located and read from the archived data.

[0050] The archive header file may include the following fields: archive signature (a magic number used to identify the archive format), archive header size, total archive size, format version number, hash table offset, block table offset, number of hash table entries, and number of block table entries. The hash table offset indicates the starting position of the hash table within the archive data, and the block table offset indicates the starting position of the block table within the archive data.

[0051] Step 202: Parse the hash tables of the original file and the target file to obtain the first position information of each original sub-file and each target sub-file in their respective block tables.

[0052] Each entry in the hash table stores the hash value obtained by hashing the subfile name and the index of the corresponding entry in the block table. The first position information is the index of the subfile in the block table, which can be obtained by parsing the hash table entry.

[0053] Hash table entries may include the following fields: first hash value (the hash value obtained by calculating the sub-file name using the first hash function), second hash value (the hash value obtained by calculating the sub-file name using the second hash function), language identifier (indicating the language type of the sub-file), platform identifier (indicating the target platform of the sub-file), and block table index (indicating the position of the corresponding entry in the block table for the sub-file).

[0054] Hash tables can contain special entries, such as empty entries (entries that have not been used and can be identified by a specific identifier such as 0xFFFFFFFF) and deleted entries (entries that were once used but have been deleted and can be identified by a specific identifier such as 0xFFFFFFFE). When parsing a hash table, these special entries are skipped, and only valid entries are processed.

[0055] Step 203: Determine the second position information of each original sub-file and each target sub-file in their respective file data areas from the block table based on the first position information.

[0056] Based on the first location information, the corresponding entry in the block table is located, and the location information and data attributes of the sub-file in the file data area are read from that entry. The second location information is the storage location of the sub-file in the file data area, including the data offset and data size.

[0057] Block table entries may include the following fields: file data offset (the starting position of the sub-file data in the file data area), data size (the storage size occupied by the sub-file data in the file data area), and data attribute flags (indicating whether the sub-file data has been compressed and / or encrypted). Among them, the file data offset and the compressed data size constitute the second location information.

[0058] Step 204: Determine the file data of each original sub-file and each target sub-file from the file data area based on the second location information.

[0059] Based on the file data offset in the second location information, the starting position of the sub-file is located in the file data area, and data of the corresponding length is read according to the data size. Then, based on the data attribute flags, it is determined whether to decompress and / or decrypt the read data.

[0060] If the data attribute flags indicate that the subfile data is neither compressed nor encrypted, the read data is used directly as the subfile data. If the data attribute flags indicate that the subfile data is compressed, the read data is decompressed. If the data attribute flags indicate that the subfile data is encrypted, the read data is decrypted. If the data attribute flags indicate that the subfile data is both compressed and encrypted, decryption is performed first, followed by decompression.

[0061] In the above scheme, the hash table and block table are first located through the archive header file, then the hash table is indexed to the block table, and finally the sub-file data is extracted from the file data area. This achieves accurate extraction of sub-files inside the archive file based on the dual-table index structure, thus providing a data foundation for subsequent file-by-file differential calculation.

[0062] This application also provides a data processing method in one example, wherein comparing the file data of each target sub-file with the file data of each original sub-file to determine the incremental state and difference file data of each target sub-file includes one of the following: If the original file corresponding to the target sub-file does not exist in the original file, the incremental status of the target sub-file is marked as newly added; the file data of the target sub-file is determined as differential file data.

[0063] When the target subfile does not have a corresponding original subfile in the original file's file list, it indicates that the subfile is a newly added file within the target file. In this case, the incremental status of the target subfile is marked as newly added, and the complete file data of the subfile is used as the difference file data. When reconstructing the target file, the client can directly use this complete file data as the file data for the target subfile without performing difference calculations.

[0064] If the original file contains an original subfile corresponding to the target subfile, and the target subfile is identical to the original subfile, then the incremental status of the target subfile is marked as identical.

[0065] When a target subfile has a corresponding original subfile in the original file, and the file data of both is completely identical, it indicates that the target subfile has not changed during the version update. The incremental status of this target subfile is marked as consistent, and no differential file data is generated. When reconstructing the target file, the client can directly reuse the file data of the corresponding original subfile in the original file, thereby reducing data transfer volume and computational overhead.

[0066] If the original file contains an original subfile corresponding to the target subfile, and the target subfile is inconsistent with the original subfile, then the incremental status of the target subfile is marked as inconsistent; the differential file data is determined based on the target subfile and the original subfile.

[0067] When a target subfile has a corresponding original subfile in the original file, but their file data is inconsistent, it indicates that the target subfile has changed during a version update. The incremental status of this target subfile is marked as inconsistent, and the difference file data is calculated based on the file data of the target subfile and the original subfile. The difference file data can be calculated using difference algorithms such as BSDiff, rsync, or xdelta3. When the client reconstructs the target file, it updates the file data of the corresponding original subfile in the original file based on the difference file data to obtain the file data of the target subfile.

[0068] In the above scheme, by dividing the incremental status of the target sub-file into three types—new, consistent, and inconsistent—and adopting differentiated processing strategies according to different statuses, the scheme achieves accurate classification of changes to the target sub-file, thereby effectively reducing unnecessary data transmission and differential calculations.

[0069] This application also provides a data processing method in one example, such as Figure 3 As shown, before parsing the hash tables of the original file and the target file, the method further includes: Step 301: Parse the archive header files of the original file and the target file to obtain the third position information of the hash table of the original file and the target file in their respective archive data and the fourth position information of the block table in their respective archive data.

[0070] Before parsing the hash table, the archive header file needs to be parsed to obtain the location information of the hash table and block table within the archive data. The third location information is the starting offset and number of entries of the hash table within the archive data, and the fourth location information is the starting offset and number of entries of the block table within the archive data.

[0071] The archive header file may include hash table offset fields and block table offset fields, indicating the starting positions of the hash table and block table within the archived data, respectively. The archive header file may also include hash table entry count fields and block table entry count fields, indicating the total number of entries in the hash table and block table, respectively. By parsing these fields, third and fourth position information can be obtained.

[0072] Step 302: Determine the hash tables of the original file and the target file from the archived data based on the third location information.

[0073] Based on the hash table offset in the third location information, data is read starting from the corresponding position in the archived data. The length of data to be read is determined by the number of hash table entries (the size of each entry multiplied by the number of entries), thus obtaining the complete hash table data.

[0074] Step 303: Determine the block tables of the original file and the target file from the archived data based on the fourth location information.

[0075] Based on the block table offset in the fourth location information, data is read starting from the corresponding position in the archived data. The length of data to be read is determined by the number of block table entries (the size of each entry multiplied by the number of entries), thus obtaining the complete block table data.

[0076] In the above scheme, the location information of the hash table and block table is obtained by parsing the archive header file, and then the hash table and block table are located and extracted from the archive data based on the location information. This achieves fast location and extraction of the index structure, thereby improving the efficiency and accuracy of file parsing.

[0077] This application also provides a data processing method in one example, such as Figure 4 As shown, parsing the hash tables of the original file and the target file to obtain the first position information of each original sub-file and each target sub-file in their respective block tables includes: Step 401: Determine the entry data corresponding to each original sub-file and each target sub-file in their respective hash tables based on the number of entries in the hash table. The number of entries is determined by parsing the archive header files of the original files and the target files.

[0078] The number of entries is the total number of entries in the hash table, obtained by parsing the hash table entry count field in the archive header file. Based on the number of entries, the data range of the hash table (the range of entries from the beginning of the hash table multiplied by the size of each entry in bytes) can be determined, allowing traversal of all entries in the hash table.

[0079] Each entry in the hash table has a fixed size in bytes. When traversing the hash table, data for each entry is read sequentially, starting from the first entry. For each entry, it is checked whether it is a valid entry (i.e., a non-empty entry and not a deleted entry), and only valid entries are processed further.

[0080] Step 402: Read data from the entry data to obtain the first position information of each original sub-file and each target sub-file.

[0081] The block table index field is read from each valid entry data. The value of this field represents the first position information of the corresponding sub-file in the block table. Simultaneously, the first hash value and the second hash value can be read from the entry, which can be used to subsequently determine whether an original sub-file corresponding to each target sub-file exists in the original file.

[0082] In the above scheme, by determining the range of entry data based on the number of entries in the hash table, and then reading the first position information from the entry data, accurate traversal and parsing of the hash table are achieved, thereby ensuring the accuracy and completeness of the obtained sub-file index information.

[0083] To enable clients to efficiently reconstruct target files based on update data packets, one embodiment of this application provides a file update method applied to a client, such as... Figure 5 As shown, the method includes: Step 501: Determine the header file, archive header file, incremental status of each target subfile, and differential file data of the target file based on the update data packet sent by the server.

[0084] The client receives the update data packet sent by the server and parses the header file of the target file, the archive header file, the incremental status of each target subfile, and the differential file data from the update data packet.

[0085] Step 502: Determine the file data of each original sub-file in the file data area based on the hash table and block table in the original file.

[0086] The client stores the original files locally. Using the same parsing method as the server, it skips the header file of the original file, parses the archive header file to obtain the location information of the hash table and block table, and then locates and reads the hash table and block table based on the location information. It obtains the first position information of each original sub-file in the block table by parsing the hash table, then obtains the second position information of each original sub-file in the file data area based on the first position information, and finally extracts the file data of each original sub-file from the file data area based on the second position information.

[0087] Step 503: Update the file data of each original sub-file according to the incremental status of each target sub-file and the differential file data to obtain the file data of each target sub-file.

[0088] Based on the incremental status of each target sub-file, different strategies are adopted to update the file data of each original sub-file, thus obtaining the file data of each target sub-file.

[0089] Step 504: Update the hash table and block table of the original file according to the file data of each target sub-file to obtain the hash table and block table of the target file.

[0090] After obtaining the file data of each target sub-file, the hash table and block table of the original file are updated according to the storage location and data size of each target sub-file in the target file, resulting in the hash table and block table of the target file. Since the physical storage location of each sub-file in the target file will be rearranged as a whole due to the addition or modification of files, even target sub-files whose content remains unchanged may have different offsets in the file data area. Therefore, it is necessary to update each entry in the block table. Simultaneously, due to changes in block table entries, the indexes in the hash table pointing to these block table entries may also be updated accordingly.

[0091] Step 505: Generate the target file based on the header file, archive header file, hash table and block table of the target file, and file data of each target sub-file.

[0092] The header file, archive header file, updated hash table, updated block table, and file data of each target subfile are assembled according to the file format specifications to generate a complete target file. During assembly, the header file is located at the beginning of the file, followed by the archive header file, file data area, hash table, and block table in sequence. The hash table offset and block table offset fields in the archive header file are set according to the actual positions of the hash table and block table in the file.

[0093] In the above scheme, by updating the data of each original sub-file according to the incremental state and differential file data, and synchronously updating the hash table and block table, the client can completely reconstruct the target file by relying only on the incremental data packet, thereby effectively reducing the amount of data transmission and local computing overhead during the update process.

[0094] This application also provides a data processing method in one example, wherein updating the file data of each original sub-file based on the incremental status of each target sub-file and the differential file data includes one of the following: If the incremental status of the target sub-file is marked as newly added, the differential file data of the target sub-file is determined as the corresponding file data.

[0095] When the incremental status of the target subfile is "new," it means that the target subfile does not have a corresponding original subfile in the original file. The difference file data containing the target subfile in the update data packet is the complete file data of the target subfile. The client directly uses the complete file data of the target subfile in the update data packet as the file data of the target subfile, without performing any difference merging operation.

[0096] If the incremental status of the target sub-file is marked as consistent, the file data of the original sub-file corresponding to the target sub-file in the original file is determined as the file data of the target sub-file.

[0097] When the incremental status of the target subfile is consistent, it indicates that the target subfile has not changed during the version update process. The client directly reuses the file data of the original subfile corresponding to the target subfile in the local original file as the file data of the target subfile, without needing to obtain any data from the update data package.

[0098] If the incremental status of the target sub-file is marked as inconsistent, the file data of the original sub-file corresponding to the target sub-file in the original file is updated according to the differential file data of the target sub-file to obtain the file data of the target sub-file.

[0099] When the incremental status of the target subfile is inconsistent, it indicates that the target subfile has changed during the version update process. The difference file data of the target subfile in the update data packet is the difference file data between the target subfile and its corresponding original subfile in the original file. The client updates the file data of the original subfile corresponding to the target subfile in the local original file based on this difference file data to obtain the file data of the target subfile.

[0100] In the above scheme, by adopting different file data update strategies according to different incremental states, the file data of the target sub-file is efficiently reconstructed, thereby maximizing the reuse of existing data in the original file while maintaining data accuracy.

[0101] In one example of this application, a data processing method is also provided, wherein updating the hash table and block table of the original file based on the file data of each target sub-file includes at least one of the following: If the incremental status of the target sub-file is marked as newly added or inconsistent, the block table is updated according to the data size of the file data of the target sub-file and its second position information in the file data area of ​​the target file; the hash table is updated according to the first position information in the block table based on the second position information.

[0102] When the incremental status of the target subfile is "added" or "inconsistent," it indicates that the file data of the target subfile has been added or modified, and its storage location and data size in the file data area of ​​the target file may differ from those in the original file. Based on the data size of the target subfile and its second position information in the file data area of ​​the target file, the corresponding entry in the block table is updated. The update includes the file data offset, data size, and data attribute flags.

[0103] Simultaneously, the hash table is updated based on the first position information in the block table according to the second position information. For target subfiles with an incremental state of "new", a new entry is created in the hash table, and the first hash value, second hash value, language identifier, platform identifier, and block table index corresponding to the target subfile name are filled in. For target subfiles with an incremental state of "inconsistent", the block table index field of existing entries in the hash table is updated so that it points to the updated block table entry.

[0104] If the incremental status of the target sub-file is marked as consistent, the block table is updated according to the second position information of the file data of the target sub-file in the file data area of ​​the target file; the hash table is updated according to the first position information of the block table based on the second position information.

[0105] When the incremental status of the target subfile is consistent, although the file data of the target subfile itself has not changed, the storage location of the target subfile in the file data area of ​​the target file may change because the addition or modification of other subfiles may cause a complete reordering of the file data area. Based on the second position information of the target subfile in the file data area of ​​the target file, the file data offset of the corresponding entry in the block table is updated.

[0106] Simultaneously, the hash table is updated based on the first position information in the block table according to the second position information. Since the position of a block table entry may change due to the addition of a new entry, the index in the hash table pointing to that entry also needs to be adjusted accordingly. The block table index field for the corresponding entry in the hash table is updated to point to the correct block table entry.

[0107] In the above scheme, the block table and hash table are updated synchronously according to the incremental state, so as to achieve the correct reconstruction of the target file index structure, thereby ensuring that the reconstructed target file can be correctly indexed and read.

[0108] To implement the above data processing methods, such as Figure 6 As shown, an example of this application provides a data processing apparatus, including: The acquisition module 601 is used to obtain a raw file and a target file. Both the raw file and the target file include a header file and archived data. The archived data includes an archived header file, a hash table, a block table, and a file data area. The first calculation module 602 is used to determine the file data of each original sub-file in the original file and each target sub-file in the target file in their respective file data areas based on the hash table and block table of the original file and the target file; The first calculation module 602 is further configured to compare the file data of each target sub-file with the file data of each original sub-file to determine the incremental state and differential file data of each target sub-file; The first processing module 603 is used to generate an update data packet based on the header file of the target file, the archive header file, the incremental status of each target sub-file, and the differential file data. The update data packet is used to update the original file in the client.

[0109] The first calculation module 602 is further configured to determine the hash table and block table of the original file and the target file based on the archive header file of the original file and the target file; The first calculation module 602 is further configured to parse the hash tables of the original file and the target file to obtain the first position information of each original sub-file and each target sub-file in their respective block tables; The first calculation module 602 is further configured to determine, based on the first location information, the second location information of each original sub-file and each target sub-file in their respective file data areas from the block table; The first calculation module 602 is further configured to determine the file data of each original sub-file and each target sub-file from the file data area based on the second location information.

[0110] The first calculation module 602 is further configured to, when the original subfile corresponding to the target subfile does not exist in the original file, mark the incremental status of the target subfile as newly added; and determine the file data of the target subfile as differential file data. The first calculation module 602 is further configured to mark the incremental state of the target sub-file as consistent when the original file contains an original sub-file corresponding to the target sub-file and the target sub-file is consistent with the original sub-file; The first calculation module 602 is further configured to, when the original file contains an original sub-file corresponding to the target sub-file and the target sub-file is inconsistent with the original sub-file, mark the incremental state of the target sub-file as inconsistent; and determine the differential file data based on the target sub-file and the original sub-file.

[0111] The first calculation module 602 is further configured to parse the archive header files of the original file and the target file to obtain the third position information of the hash table of the original file and the target file in their respective archive data and the fourth position information of the block table in their respective archive data. The first calculation module 602 is further configured to determine the hash tables of the original file and the target file from the archived data based on the third location information; The first calculation module 602 is further configured to determine the block table of the original file and the target file from the archived data based on the fourth location information.

[0112] The first calculation module 602 is further configured to determine the entry data corresponding to each original sub-file and each target sub-file in their respective hash tables based on the number of entries in the hash table, wherein the number of entries is determined by parsing the archive header files of the original files and the target files; The first calculation module 602 is further configured to read data from the entry data to obtain the first position information of each original sub-file and each target sub-file.

[0113] To implement the above file update method, such as Figure 7 As shown, an example of this application provides a file updating apparatus, including: The communication module 701 is used to determine the header file, archive header file, incremental status of each target sub-file, and differential file data of the target file based on the update data packet sent by the server. The second calculation module 702 is used to determine the file data of each original sub-file in the original file in the file data area according to the hash table and block table in the original file. The second calculation module 702 is further configured to update the file data of each original sub-file according to the incremental status of each target sub-file and the differential file data, so as to obtain the file data of each target sub-file; The second calculation module 702 is further configured to update the hash table and block table of the original file according to the file data of each target sub-file, so as to obtain the hash table and block table of the target file; The second processing module 703 is used to generate the target file based on the header file, archive header file, hash table and block table of the target file, and file data of each target sub-file.

[0114] The second calculation module 702 is further configured to determine the differential file data of the target sub-file as the corresponding file data when the incremental status of the target sub-file is marked as newly added. The second calculation module 702 is further configured to, when the incremental status marker of the target sub-file is consistent, determine the file data of the original sub-file corresponding to the target sub-file in the original file as the file data of the target sub-file; The second calculation module 702 is further configured to update the file data of the original sub-file corresponding to the target sub-file in the original file according to the difference file data of the target sub-file when the incremental status of the target sub-file is marked as inconsistent, so as to obtain the file data of the target sub-file.

[0115] The second calculation module 702 is further configured to, when the incremental status of the target sub-file is marked as new or inconsistent, update the block table according to the data size of the file data of the target sub-file and its second position information in the file data area of ​​the target file; and update the hash table according to the first position information in the block table based on the second position information. The second calculation module 702 is further configured to, when the incremental status marker of the target sub-file is consistent, update the block table according to the second position information of the file data of the target sub-file in the file data area of ​​the target file; and update the hash table according to the first position information of the block table according to the second position information.

[0116] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0117] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0119] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods and / or file update methods. For example, in some embodiments, the data processing methods and / or file update methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing methods and / or file update methods described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform data processing methods and / or file update methods by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0126] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0127] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0128] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0129] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, Applied to the server side, the method includes: Obtain the original file and the target file, both of which include a header file and archive data. The archive data includes an archive header file, a hash table, a block table, and a file data area. The file data of each original sub-file in the original file and each target sub-file in the target file are determined in their respective file data areas based on the hash table and block table of the original file and the target file; The file data of each target sub-file is compared with the file data of each original sub-file to determine the incremental status and differential file data of each target sub-file. An update data packet is generated based on the header file of the target file, the archive header file, the incremental status of each target sub-file, and the differential file data. The update data packet is used to update the original file in the client.

2. The method according to claim 1, characterized in that, The step of determining the file data of each original sub-file in the original file and each target sub-file in the target file in their respective file data areas based on the hash table and block table of the original file and the target file includes: The hash table and block table of the original file and the target file are determined based on the archive header files of the original file and the target file; Parse the hash tables of the original file and the target file to obtain the first position information of each original sub-file and each target sub-file in their respective block tables; Based on the first location information, determine the second location information of each original sub-file and each target sub-file in their respective file data areas from the block table; The file data of each original sub-file and each target sub-file are determined from the file data area based on the second location information.

3. The method according to claim 2, characterized in that, The step of comparing the file data of each target sub-file with the file data of each original sub-file to determine the incremental status and difference file data of each target sub-file includes one of the following: If the original file corresponding to the target sub-file does not exist in the original file, then the incremental status of the target sub-file is marked as newly added; and the file data of the target sub-file is determined as differential file data. If the original file contains an original subfile corresponding to the target subfile, and the target subfile is identical to the original subfile, then the incremental status of the target subfile is marked as identical. If the original file contains an original subfile corresponding to the target subfile, and the target subfile is inconsistent with the original subfile, then the incremental status of the target subfile is marked as inconsistent; the differential file data is determined based on the target subfile and the original subfile.

4. The method according to claim 2, characterized in that, Before parsing the hash tables of the original file and the target file, the method further includes: Parse the archive header files of the original file and the target file to obtain the third position information of the hash table of the original file and the target file in their respective archive data and the fourth position information of the block table in their respective archive data; The hash tables of the original file and the target file are determined from the archived data based on the third location information; The block tables of the original file and the target file are determined from the archived data based on the fourth location information.

5. The method according to claim 4, characterized in that, The step of parsing the hash tables of the original file and the target file to obtain the first position information of each original sub-file and each target sub-file in their respective block tables includes: The entry data corresponding to each original sub-file and each target sub-file in their respective hash tables are determined based on the number of entries in the hash table. The number of entries is determined by parsing the archive header files of the original files and the target files. Data is read from the entry data to obtain the first position information of each original sub-file and each target sub-file.

6. A file update method, characterized in that, Applied to a client, the method includes: Based on the received update data packets sent by the server, determine the header file of the target file, the archive header file, the incremental status of each target subfile, and the differential file data; The file data of each original sub-file in the original file is determined in the file data area based on the hash table and block table in the original file. The file data of each original sub-file is updated based on the incremental status of each target sub-file and the differential file data to obtain the file data of each target sub-file; The hash table and block table of the original file are updated based on the file data of each target sub-file to obtain the hash table and block table of the target file; The target file is generated based on the header file, archive header file, hash table and block table of the target file, and file data of each target sub-file.

7. The method according to claim 6, characterized in that, The step of updating the file data of each original sub-file based on the incremental status of each target sub-file and the differential file data includes one of the following: If the incremental status of the target sub-file is marked as newly added, the differential file data of the target sub-file is determined as the corresponding file data; If the incremental status of the target sub-file is marked as consistent, the file data of the original sub-file in the original file that corresponds to the target sub-file is determined as the file data of the target sub-file. If the incremental status of the target sub-file is marked as inconsistent, the file data of the original sub-file corresponding to the target sub-file in the original file is updated according to the differential file data of the target sub-file to obtain the file data of the target sub-file.

8. The method according to claim 6, characterized in that, The step of updating the hash table and block table of the original file based on the file data of each target sub-file includes at least one of the following: If the incremental status of the target sub-file is marked as newly added or inconsistent, the block table is updated according to the data size of the file data of the target sub-file and its second position information in the file data area of ​​the target file; the hash table is updated according to the first position information in the block table based on the second position information. If the incremental status of the target sub-file is marked as consistent, the block table is updated according to the second position information of the file data of the target sub-file in the file data area of ​​the target file; the hash table is updated according to the first position information of the block table based on the second position information.

9. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to obtain the original file and the target file. Both the original file and the target file include a header file and archived data. The archived data includes an archived header file, a hash table, a block table, and a file data area. The first calculation module is used to determine the file data of each original sub-file in the original file and each target sub-file in the target file in their respective file data areas based on the hash table and block table of the original file and the target file; The first calculation module is further configured to compare the file data of each target sub-file with the file data of each original sub-file to determine the incremental status and differential file data of each target sub-file; The first processing module is used to generate an update data packet based on the header file of the target file, the archive header file, the incremental status of each target sub-file, and the differential file data. The update data packet is used to update the original file in the client.

10. A file updating device, characterized in that, The device includes: The communication module is used to determine the header file, archive header file, incremental status of each target subfile, and differential file data of the target file based on the update data packets sent by the server. The second calculation module is used to determine the file data of each original sub-file in the original file in the file data area based on the hash table and block table in the original file. The second calculation module is further configured to update the file data of each original sub-file based on the incremental status of each target sub-file and the differential file data, so as to obtain the file data of each target sub-file; The second calculation module is further configured to update the hash table and block table of the original file based on the file data of each target sub-file, so as to obtain the hash table and block table of the target file; The second processing module is used to generate the target file based on the header file, archive header file, hash table and block table of the target file, and file data of each target sub-file.

Images