A method for generating and parsing a data file

Abstract

The application provides a data file generation and analysis method, and relates to the technical field of embedded electronic equipment software download. The method introduces an algorithm identification and negotiation mechanism under the premise of complete compatibility with the existing sparse image format standard. An extension header structure and an additional check data section are added at the beginning of the sparse image file to record the check algorithm and check value. Before generating the file, the host obtains a check algorithm priority list supported by the target device, compares it with the local list, and selects the highest priority common algorithm. When making, in addition to generating the CRC check data section with the standard tool, the selected algorithm is used to generate the check data and save it in the additional check data section. After the device receives the file, the file type is determined through the format identification, the additional check data section is used to check the extension file, and the standard file is processed according to the original process. The application is completely compatible with the standard sparse image file, can dynamically adapt to the hardware capacity of the device, and maximizes the check speed under the premise of ensuring data correctness.

Description

Technical Field

[0001] This invention relates to the field of embedded electronic device software download technology, specifically to a method and apparatus for automatically selecting the optimal verification algorithm and verifying downloaded data in firmware download based on a sparse file format, and particularly to a method for generating and parsing data files. Background Technology

[0002] Data files, especially sparse image files, are widely used in embedded systems, mobile device firmware distribution, and large-scale data updates as an efficient image format for storage and transmission. Their core principle is to record the location and content of actual data blocks while ignoring zero-filled "holes," thereby significantly reducing file size and saving storage space and transmission bandwidth.

[0003] Currently, the Sparse Image format used by the Android system has become the de facto industry standard. Its structure typically includes a file header, data blocks, and corresponding Cyclic Redundancy Check (CRC32) blocks. The check block is used to verify the integrity of the corresponding data block during transmission or storage. However, existing sparse image formats have the following technical limitations. First, the check algorithm is singular; the current standard only supports CRC32. On systems lacking hardware CRC accelerators (such as some low-end embedded chips and older processors), this leads to a significant increase in check time, potentially making the check process a performance bottleneck during system startup or update. Second, there is a lack of algorithm negotiation mechanisms. The current standard does not provide the host and device sides with the ability to negotiate check algorithms, and cannot adaptively select the best algorithm based on the target device's hardware capabilities, resulting in insufficient utilization of hardware acceleration capabilities.

[0004] Therefore, a technical solution that can maintain compatibility with the original data file format, support multiple verification algorithms, make full use of hardware acceleration capabilities, and improve the verification efficiency of data files on different hardware platforms has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for generating and parsing data files.

[0006] According to the present invention, a method for generating and parsing data files is provided, the scheme is as follows: Firstly, a method for generating a data file is provided, the method comprising: Step S1: Obtain the verification algorithm capability information of the target device, and determine the target verification algorithm based on the verification algorithm capability information; Step S2: Obtain the original data file, which contains the original data and the original verification data generated based on the preset verification algorithm. The original data contains multiple data units. Step S3: Calculate the original data based on the target verification algorithm to generate new verification data; Step S4: Generate an extended data file, which includes a format identifier, the new verification data, and the original data file that is completely preserved. The format identifier is used to indicate the data structure of the extended data file.

[0007] Preferably, step S1 includes: obtaining a list of candidate verification algorithms supported by the target device and priority information corresponding to each candidate verification algorithm; Get a list of locally supported verification algorithms; Based on the intersection of the candidate verification algorithm list supported by the target device and the locally supported verification algorithm list, the candidate verification algorithm with the highest priority indicated by the priority information is determined as the target verification algorithm.

[0008] Preferably, generating the extended data file in step S4 includes: Generate an extended file header, which includes the format identifier and offset information. The offset information is used to indicate the storage location of the new checksum data and the original data file in the extended data file. The extended data file is obtained by concatenating the extended file header, the new checksum data, and the original data file in a preset order.

[0009] Preferably, the extended file header further includes at least one of the following metadata: Version identifier, used to indicate the structural version of the extended file header; Algorithm identifier, used to indicate the type of the target verification algorithm; Data length information is used to indicate the length of the verification value generated by the target verification algorithm; Total number of data units, used to indicate the number of data units contained in the original data; The header checksum is used to perform integrity verification on the extended file header itself.

[0010] Preferably, step S3 includes: The multiple data units are traversed according to a preset traversal order; For each of the data units traversed, the corresponding verification value is calculated using the target verification algorithm; The calculated check values ​​are arranged according to the preset traversal order to generate new check data containing an ordered check value sequence, wherein the Nth check value in the ordered check value sequence corresponds to the Nth data unit among the plurality of data units.

[0011] Secondly, a method for parsing data files is provided, the parsing method comprising: Read the header data of the data file to be parsed, and determine the file type of the data file to be parsed based on the format identifier in the header data; If the file type is an extended file type, the position of the new verification data is obtained based on the extended file header in the header data, the original data in the data file to be parsed is verified based on the new verification data, and the original verification data in the data file to be parsed is ignored. If the file type is a raw file type, then the raw data is verified based on the raw verification data in the data file to be parsed.

[0012] Preferably, validating the original data in the data file to be parsed based on the new verification data, and ignoring the original verification data in the data file to be parsed, includes: Parse the extended file header to obtain the storage locations of the new verification data and the original data; The original data is traversed in a preset traversal order, and the verification value of the current data unit is calculated using the verification algorithm specified in the extended file header. Read the corresponding pre-stored verification value from the storage location of the new verification data, and compare the calculated verification value with the pre-stored verification value; When the original verification data unit in the original data is encountered, the original verification data unit is skipped.

[0013] Preferably, skipping processing is performed on the original verification data unit, including: Identify the block type identifier of the original verification data unit; In response to the block type identifier matching a preset verification data type, the data content of the original verification data unit is not included in the verification calculation, and the original verification data unit is not compared with any verification value; The read pointer is moved directly to the next data unit after the original verification data unit.

[0014] Thirdly, an electronic device is provided, comprising: At least one processor; and, A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform either the data file generation method or the data file parsing method described in any one of the claims.

[0015] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data file generation method or the data file parsing method described in any one of the claims.

[0016] Compared with the prior art, the present invention has the following technical effects: 1. This invention, when generating extended sparse files, fully preserves the standard sparse file and its CRC32 checksum data, and sets a format identifier magic number in the file header. The device automatically identifies the file type by detecting the magic number: for devices supporting the extended format, a new checksum data segment is used for verification; for traditional devices that do not support the extended format, the extended portion can be ignored, and verification is performed according to the standard sparse file format. This design ensures full backward compatibility with the existing Android Sparse Image format standard and existing toolchains. 2. This invention utilizes an algorithm negotiation mechanism between the host and the device to enable the host to obtain a list of verification algorithms supported by the target device and their priority information before file generation, and automatically selects the algorithm that is supported by both parties and has the highest device priority as the verification algorithm. When the target device has a hardware verification accelerator, the system can prioritize the use of hardware-supported algorithms, thereby significantly improving the verification speed and preventing the verification process from becoming a performance bottleneck in the system startup or firmware update process; 3. When generating extended sparse files, this invention, in addition to retaining the original CRC32 checksum data of standard sparse files, adds an extra checksum data segment generated based on a negotiation-optimized algorithm. During device verification, the extended sparse file is verified using the newly added checksum data segment. Each checksum value in this checksum data segment corresponds one-to-one with the data block to be verified, ensuring the accuracy of the verification. By dynamically selecting a hardware-accelerated verification algorithm, verification efficiency is maximized while ensuring data integrity and correctness. 4. This invention defines metadata fields such as version number, verification algorithm identifier, and verification value length in the extended sparse file header. It can not only support a variety of mainstream hash algorithms, but also smoothly support new, more secure and efficient verification algorithms in the future through version evolution and the addition of algorithm identifiers, without modifying the overall framework of the file format, thus ensuring the life cycle and long-term evolution capability of the technical solution.

[0017] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0018] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0020] This invention provides a method for generating and parsing data files, at least addressing the problem of existing verification algorithms being singular and unable to adaptively select. It involves a standard data file (i.e., the original data file), an extended data file header, verification data blocks (i.e., new verification data), and methods for generating and parsing the standard data file, the extended data file header, and the verification data blocks.

[0021] The standard data file is any formatted data file conforming to industry standards. In a typical embodiment of the present invention, the standard data file is an Android Sparse file. The standard data file is generated using the Android standard tool img2simg. The generated sparse file has the structural characteristics of an Android Sparse file, containing original data and original verification data generated based on a preset verification algorithm (such as CRC32).

[0022] The extended data file header is used to identify that the file has an extended data file structure and content, and it contains various metadata information. In a typical embodiment of the present invention, the extended data file header specifically includes: Magic number: Used to identify the file as an extended data file format.

[0023] Version number: Used to identify the version of the extended data file format.

[0024] Header length: Used to indicate the size of the extended data file header.

[0025] Data file offset: Used to identify the starting position of the standard data file in the extended data file.

[0026] Algorithm identifier: Used to indicate the target verification algorithm corresponding to the verification data value in the verification data block.

[0027] Verification value length: Used to indicate the size of the verification value corresponding to the target verification algorithm.

[0028] Check data block offset: Used to indicate the starting position of the check data block in the extended data file.

[0029] Total number of data blocks: Indicates the total number of data units in the data file.

[0030] Algorithm Name: A string used to identify the target verification algorithm.

[0031] Header checksum: Used to store the integrity checksum of the extended data file header itself, such as the CRC value.

[0032] The magic number is a fixed-length integer value. When the embedded device starts up, it compares the read magic number with the fixed integer value to determine whether the current image file has the extended data file header structure and content.

[0033] The version number is used to identify the current version of the extended data file structure so that subsequent extensions and different processing methods can be used for different versions of the extended data file header.

[0034] Data file offset refers to the relative position of the standard data file header within the entire file. By knowing the data file offset, the position of the standard data file can be accurately located for subsequent processing and verification.

[0035] The verification data block (i.e., the new verification data) contains a sequence of verification values. The method for generating the verification value sequence is as follows: Following the original storage order of the data units to be verified in the standard data file, the valid data of each data unit to be verified is sequentially obtained; for each valid data unit to be verified, its corresponding verification value is calculated using the target verification algorithm specified in the extended data file header; the calculated verification values ​​are stored according to the original order of the data units to be verified, thereby forming an ordered list of verification values. The Nth verification value in the list corresponds to the Nth data unit to be verified in the standard data file.

[0036] This document supports data file generation and parsing methods for various verification algorithms, specifically including algorithm negotiation, file generation, and file processing steps. (See reference...) Figure 1 As shown, this embodiment of the invention provides a method for generating data files, as detailed below: Step S1: Obtain the verification algorithm capability information of the target device, and determine the target verification algorithm based on the verification algorithm capability information; Step S2: Obtain the original data file, which contains the original data and the original verification data generated based on the preset verification algorithm. The original data contains multiple data units. Step S3: Calculate the original data based on the target verification algorithm to generate new verification data; Step S4: Generate an extended data file. The extended data file contains a format identifier, new checksum data, and the original data file, which is fully preserved. The format identifier is used to indicate the data structure of the extended data file.

[0037] In this embodiment of the invention, file generation includes file format decision and extended data file generation.

[0038] File format decision refers to determining whether the target verification algorithm negotiated during the algorithm negotiation phase is valid (not empty); if valid, it is confirmed that the extended data file format will be used for this image file generation; otherwise, the standard data file format will be used.

[0039] Extended data file generation refers to the host performing the following operations after the file format is determined: 1. Generate a standard data file based on the original firmware data.

[0040] 2. If the standard data file format is confirmed to be used, file generation is complete. Otherwise, proceed to the next step.

[0041] 3. Generate an extended data file header, which records the identification information of the target verification algorithm.

[0042] 4. Based on the standard data file and the target verification algorithm, calculate and generate verification data blocks.

[0043] 5. Combine the extended data file header, the verification data block, and the standard data file in this order to form the final extended data file. The file generation is now complete.

[0044] In this embodiment of the invention, by performing algorithm negotiation before generating the extended data file and generating additional verification data based on the negotiation result, while completely preserving the original data file and its inherent original verification data, the generated extended data file can be efficiently verified by devices supporting the new format using hardware acceleration, and can also be verified by devices supporting only the old format according to the standard process, achieving full backward compatibility. The format identifier enables the device to automatically identify the file type and select the correct processing path.

[0045] In one possible implementation, step S1 includes: obtaining a list of candidate verification algorithms supported by the target device and priority information for each candidate verification algorithm; obtaining a list of locally supported verification algorithms; and, based on the intersection of the list of candidate verification algorithms supported by the target device and the list of locally supported verification algorithms, indicating the candidate verification algorithm with the highest priority based on the priority information, and determining it as the target verification algorithm. This negotiation process follows the principle of prioritizing device preferences to ensure that the selected algorithm can leverage the optimal hardware performance of the target device.

[0046] In one possible implementation, generating the extended data file in step S4 includes: generating an extended file header, the extended file header containing a format identifier and offset information, the offset information being used to indicate the storage location of the new check data and the original data file in the extended data file; and concatenating the extended file header, the new check data, and the original data file in a preset order to obtain the extended data file.

[0047] In one possible implementation, the step of generating the extended data file header includes: filling in the following metadata information in a predetermined order: 1. Fill in the format identification information, including the predefined magic number and version number, to identify the file format and version.

[0048] 2. Fill in the extended data file header length information and record the total byte length of the extended data file header itself.

[0049] 3. Fill in the verification algorithm information, including the target verification algorithm identifier determined during the algorithm negotiation phase, and the length of the verification value corresponding to the algorithm.

[0050] 4. Fill in the offset information, including: the starting storage offset of the check data block within the extended data file; and the starting storage offset of the original standard data file data within the extended data file.

[0051] 5. Fill in the quantity information and record the total number of data units contained in the original standard data file.

[0052] 6. Calculate and populate the integrity check value: Before calculation, initialize the field used to store the check value to a predetermined value (such as zero); calculate the cyclic redundancy check value based on the populated header data; and populate the calculated cyclic redundancy check value into the predetermined check value field in the header.

[0053] In one possible implementation, the steps for generating verification data blocks include: sequentially traversing each data unit requiring verification in the standard data file; for each data unit requiring verification, calculating the verification value of its data segment using a negotiated target verification algorithm; and creating a corresponding verification data block entry for the current original data unit. Each verification data block entry contains the verification value calculated based on the target verification algorithm. The verification data block entries are then stored sequentially into the verification data segment. By traversing all data units requiring verification within the standard data file and repeating the above steps, multiple verification data block entries, equal in number to the original data units, are finally generated and stored in the verification data segment.

[0054] In one possible implementation, the extended header also includes at least one of the following metadata: version identifier, header length, algorithm name identifier, etc. By setting a header checksum, the integrity of the extended header information during transmission can be ensured.

[0055] In one possible implementation, step S3 includes: traversing multiple data units in a preset traversal order; for each traversed data unit, obtaining its valid data and calculating the corresponding verification value using a target verification algorithm; arranging the calculated verification values ​​in a preset traversal order to generate new verification data containing an ordered sequence of verification values, wherein the Nth verification value in the ordered sequence of verification values ​​corresponds to the Nth data unit among the multiple data units.

[0056] Secondly, embodiments of the present invention provide a method for parsing data files, and a method for generating data files based on any one of the first aspects and their implementations described above, comprising: Read the header data of the data file to be parsed, and determine the file type of the data file to be parsed based on the format identifier in the header data; If the file type is an extended file type, the position of the new check data is obtained based on the extended file header in the header data. The original data in the data file to be parsed is checked based on the new check data, and the original check data in the data file to be parsed is ignored. If the file type is a raw file type, the raw data is validated based on the raw validation data in the data file to be parsed.

[0057] In this embodiment of the invention, file processing includes: after the device receives the firmware sent by the host, it performs the following verification process, including determining the file type and verifying the data block.

[0058] The file type determination process involves reading the firmware file's header data. By detecting a predefined format identifier, it determines whether the firmware file is a standard data file or an extended data file. The predefined format identifier is the magic number field in the extended data file header. If it is identified as an extended data file, the extended data file header is further verified. If the verification is successful, it is determined to be an extended data file type, and the extended data file verification process is executed; otherwise, it is determined to be a standard data file type, and the standard data file verification process is executed.

[0059] By automatically recognizing the format identifier, the device can complete efficient verification using the negotiated optimal algorithm and new verification data in one path, and fall back to the standard verification process in the other path, ensuring the correct processing of files of different formats.

[0060] In one possible implementation, verifying the extended data file header includes: retrieving and saving the value of the header checksum field in the extended data file header, and then initializing it to a predetermined value (such as zero); calculating the cyclic redundancy check value of the extended data file header; comparing the calculated cyclic redundancy check value with the saved header checksum field value; if the comparison results are consistent, the extended data file header is determined to be successfully verified; otherwise, the verification is determined to be unsuccessful.

[0061] In one possible implementation, the original data in the data file to be parsed is verified based on the new verification data, ignoring the original verification data in the data file to be parsed (i.e., the extended data file verification process). This includes the following steps: Parsing the starting position of the standard data file and the starting position of the verification data block from the data file offset and verification data block offset in the extended data file header, respectively. Starting from the starting position of the standard data file, each data unit is sequentially traversed. For the data unit that needs verification, the algorithm specified in the hash algorithm field in the extended data file header is used to calculate the verification value. Starting from the starting position of the verification data block, the corresponding verification data block entries are read in the same order. The calculated verification value is compared with the read verification data block entries. If they match, the current data unit is deemed to have passed verification. If they do not match, the current verification block is deemed to have failed verification, and consequently, the entire extended data file verification is deemed to have failed. When all data units have been traversed and all data units have passed verification, the entire extended data file is deemed to have been successfully verified.

[0062] In one possible implementation, the original data is verified based on the original verification data in the data file to be parsed (i.e., the standard data file verification process). This includes: starting from the beginning of the standard data file, sequentially traversing each data unit within it, and maintaining a cumulative CRC32 checksum for each data unit that needs to be verified. If the current data unit is of the type for which a checksum needs to be calculated, the cumulative CRC32 checksum is updated based on its data content. If the current data unit is a CRC32 checksum block, the cumulative checksum is compared with a preset checksum stored in the checksum block. If they match, the current checksum block is deemed to have passed verification, and the cumulative checksum is reset to its initial state to prepare for the verification of subsequent data units. If they do not match, the current checksum block is deemed to have failed verification, and consequently, the entire standard data file verification is deemed to have failed. When all data units have been traversed and all checksum blocks have passed verification, the overall verification of the standard data file is deemed successful.

[0063] In one possible implementation, skipping the original check data unit includes: identifying the block type identifier of the original check data unit; in response to the block type identifier matching a preset check data type, not including the data content of the original check data unit in the check calculation, and not comparing the original check data unit with any check value; and moving the read pointer directly to the next data unit of the original check data unit. This "ignore" mechanism ensures that the check process relies only on new check data, avoiding interference from old check blocks.

[0064] The present invention will now be described in more detail.

[0065] The technical solution of the present invention will be described in detail below with reference to specific embodiments. In a typical application scenario of the present invention, the data file is a sparse mirror file, the original data file is a standard sparse file, the preset verification algorithm is the CRC32 algorithm, the original verification data is the CRC check block in the standard sparse file, the data unit is the data block to be verified in the standard sparse file, and the format identifier is the magic number field in the extended sparse file header. However, those skilled in the art will understand that the above correspondence is not a limitation of the present invention, and the method of the present invention can be applied to any data file with similar structure and requirements.

[0066] like Figure 1 As shown, this embodiment provides a data file processing system that operates between a host and a target device to achieve, for example... Figure 1 The data stream shown is illustrated. The system mainly includes a negotiation and file generation module on the host side, and a file parsing and verification module on the target device side. The system completes the entire process of data file generation, transmission, and verification through four core steps: device capability negotiation, extended data file generation, format self-recognition, and adaptive verification.

[0067] Step 1: Equipment Capability Negotiation (Generation Method Step S1) Before the host computer downloads firmware to the target device, a verification algorithm negotiation is first performed. This step aims to determine a target verification algorithm that is supported by both parties and can optimize the performance of the target device's hardware.

[0068] Information Exchange: The host sends an algorithm query request to the target device via existing transmission control protocols (such as USB, UART, Ethernet, etc.). The target device responds with a list of candidate checksum algorithms. This list includes not only the checksum algorithm identifiers supported by the device (such as "CRC32", "SHA256", "SHA3-256", etc.), but also the priority information for each item. The priority is usually determined by hardware acceleration capabilities; for example, if the device includes a hardware SHA256 accelerator, then "SHA256" will have the highest priority. The host maintains a locally supported list of checksum algorithms.

[0069] Algorithm selection: After receiving the list of candidate verification algorithms, the host compares it with the list of locally supported verification algorithms to calculate the intersection set of algorithms, which represents the algorithms supported by both sides. Subsequently, based on the principle of "device preference priority," the host selects the algorithm with the highest priority in the candidate verification algorithm list from the intersection set, and determines it as the target verification algorithm used for this data file generation and verification.

[0070] Format Decision: If a valid target validation algorithm (non-empty) is successfully negotiated, the download will use the extended file type to generate an extended data file; otherwise, the original file type will be used to generate a raw, unextended data file. This decision will guide subsequent file generation steps.

[0071] Step 2: Generate extended data files (generation method steps S2-S4) When an extended file type is determined to be used, the host-side file generation module generates the final download file according to the following sub-steps.

[0072] First, step S2 is executed to obtain the raw data file. Specifically, the host uses a standard tool (such as img2simg for Android) to process the raw firmware data (such as an EXT4 file system image) to generate a raw data file. This raw data file conforms to industry standards and contains raw data and raw checksum data generated based on a preset checksum algorithm (such as CRC32). The raw data consists of multiple data units, and the file structure includes a file header and alternating data chunks of types such as RAW, FILL, DONT_CARE, and CRC32. Among them, the RAW type chunks are the data units that need to be checked, while the CRC32 type chunks are the raw checksum data units.

[0073] Next, step S3 is executed, which calculates the original data based on the target verification algorithm to generate new verification data (i.e., verification data blocks). Specifically, this includes: Traversal and Calculation: The host parses the original data file and, starting from the beginning of the file, traverses each data unit (e.g., RAW data unit) that requires a verification value to be calculated according to a preset traversal order. For each such data unit, its valid data is obtained, and its hash value is calculated using the target verification algorithm (e.g., SHA256) negotiated in step one.

[0074] Constructing a hash value sequence: All calculated hash values ​​are arranged sequentially according to the order in which their corresponding data units appear in the original data file, forming an ordered hash value sequence. This sequence, along with necessary descriptive information, constitutes the new verification data. The Nth hash value in this ordered hash value sequence uniquely corresponds to the Nth data unit to be verified in the original data file.

[0075] Then, step S4 is performed to generate the extended data file. This step further includes: Generate extended data file header (extended file header): The host creates an extended file header and populates the following metadata information in a predetermined order.

[0076] 1. Format identifier (magic number) and version identifier (version number): Write a fixed magic number (such as 0x4D415253) and version number (such as 0x0001) to identify the file as an extended file type and its structural version.

[0077] 2. Header information (file header length): Records the total length of the extended file header itself.

[0078] 3. Hash Algorithm Identifier (Hash Algorithm), Hash Value Size (Hash Size), and Algorithm Name (Hash Algorithm Name): This section contains the identifier of the target verification algorithm, the length of the hash value generated by the algorithm (e.g., 32 bytes for SHA256), and the name string that identifies the target verification algorithm.

[0079] 4. Offset Information: Calculate and write two key offsets: data file offset (the starting storage offset of the original data file within the extended data file); and check data block offset (the starting storage offset of the new check data within the extended data file).

[0080] 5. Total number of data units (total number of data blocks): Records the total number of data units in the original data file.

[0081] 6. Header Checksum: Before calculation, the reserved checksum field in the extended header is set to zero. Then, the CRC32 value is calculated for the entire extended header data, and the result is filled into the checksum field, thus enabling the extended header to have self-checking capability.

[0082] File Assembly: Finally, the host computer concatenates the extended file header, new checksum data, and original data file in binary format, strictly following the order of header-checksum-data, to form a complete and single extended data file. During this process, the original data file is preserved intact, and its internal checksum data units remain unchanged. This extended data file is the firmware package ultimately transmitted to the target device.

[0083] Step 3: File Transfer and Type Identification (File Type Determination Part of the Parsing Method) After the target device receives the data file sent by the host, the file parsing and verification module first determines its type.

[0084] Read format identifier (magic number): The device reads a predetermined number of bytes (e.g., 4 bytes) at the beginning of the file to obtain the format identifier.

[0085] Branch decision: If the read format identifier matches the predefined magic number of the extended file type, the device determines that the file is an extended file type and enters the extended file verification process (step four).

[0086] If there is a mismatch, the device determines that the file is of the original file type and proceeds to the original file verification process (step five). This design ensures full backward compatibility with existing systems and toolchains.

[0087] Step 4: Extended File Verification Process (Handling of Extended File Types in the Parsing Method) When the file type is identified as an extended file type, the device performs the following verification: Header verification: The device reads the complete extended file header; Extract the header checksum from the extended file header and store it temporarily, then set the field to zero in memory; Calculate the CRC32 value for the entire extended file header data in the current memory.

[0088] Compare the calculated CRC32 value with the temporarily stored original header checksum. If they do not match, the header is corrupted and verification fails; if they match, header verification passes and proceed to the next step.

[0089] Data validation (validation is performed based on the new validation data, ignoring the original validation data): Location: From the verified extended file header, parse out the data file offset (the starting position of the standard data file) and the check data block offset (the starting position of the new check data), and locate the starting positions of the original data file and the new check data in the received file, respectively.

[0090] Sequential traversal and comparison: The device starts from the data file offset and traverses each data unit in the original data in a preset traversal order.

[0091] For each data unit (such as a RAW block) that needs to be verified, the device uses the target verification algorithm specified in the hash algorithm field of the extended header to calculate the hash value of its valid data.

[0092] Simultaneously, the device reads the corresponding hash values ​​pre-stored in the new verification data in the same order, starting from the offset of the verification data block.

[0093] The calculated hash value is compared with the pre-stored hash value.

[0094] When traversing to a raw check data unit (such as a CRC32 block) in the raw data, the device performs a skip process: it identifies the block type identifier of the data unit, and in response to the type identifier matching the preset check data type, it does not include the data content of the data unit in the check calculation, and does not compare it with any hash value, and moves the read pointer directly to the next data unit.

[0095] Result determination: Short-circuit mechanism: If an inconsistency is found in any comparison, the entire verification process is immediately terminated, the file is determined to be corrupted, and the download fails.

[0096] Success condition: If all data units that need to be verified are processed sequentially and each comparison is consistent, then the entire extended data file is considered to have been successfully verified.

[0097] Step 5: Original File Verification Process (Handling of original file types and compatibility issues in the parsing method) When identified as an original file type, the device performs its inherent verification process based on a preset verification algorithm (such as CRC32). This process is existing technology and is briefly described below: The device parses data units sequentially, starting from the file header.

[0098] Maintain a dynamic CRC32 accumulator. For data units containing valid data patterns, such as RAW and FILL, update the accumulator using their data.

[0099] When a raw check data unit of type CRC32 is encountered, the current accumulator value is compared with the value stored in the raw check data unit.

[0100] If they are equal, the current segment verification passes, and the CRC32 accumulator is reset to zero to begin the verification of the next segment of data.

[0101] If the expected CRC32 block is not encountered before the end of the file, the verification fails.

[0102] If all CRC32 blocks pass verification, then the file verification is successful.

[0103] This invention provides a method for generating and parsing data files. By introducing an extensible algorithm identification and negotiation mechanism, the data file can automatically select the optimal verification algorithm based on the hardware capabilities of the target device, while also remaining compatible with standard data files. This effectively solves the problem of data file format verification algorithms being limited and unable to automatically adapt to platform hardware accelerators.

[0104] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0105] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for generating a data file, characterized in that, include: Step S1: Obtain the verification algorithm capability information of the target device, and determine the target verification algorithm based on the verification algorithm capability information; Step S2: Obtain the original data file, which contains the original data and the original verification data generated based on the preset verification algorithm. The original data contains multiple data units. Step S3: Calculate the original data based on the target verification algorithm to generate new verification data; Step S4: Generate an extended data file, which includes a format identifier, the new verification data, and the original data file that is completely preserved. The format identifier is used to indicate the data structure of the extended data file.

2. The method for generating a data file according to claim 1, characterized in that, Step S1 includes: obtaining a list of candidate verification algorithms supported by the target device and priority information corresponding to each candidate verification algorithm; Get a list of locally supported verification algorithms; Based on the intersection of the candidate verification algorithm list supported by the target device and the locally supported verification algorithm list, the candidate verification algorithm with the highest priority indicated by the priority information is determined as the target verification algorithm.

3. The method for generating a data file according to claim 1, characterized in that, The step S4 of generating the extended data file includes: Generate an extended file header, which includes the format identifier and offset information. The offset information is used to indicate the storage location of the new checksum data and the original data file in the extended data file. The extended data file is obtained by concatenating the extended file header, the new checksum data, and the original data file in a preset order.

4. The method for generating a data file according to claim 3, characterized in that, The extended file header also includes at least one of the following metadata: Version identifier, used to indicate the structural version of the extended file header; Algorithm identifier, used to indicate the type of the target verification algorithm; Data length information is used to indicate the length of the verification value generated by the target verification algorithm; Total number of data units, used to indicate the number of data units contained in the original data; The header checksum is used to perform integrity verification on the extended file header itself.

5. The method for generating a data file according to claim 1, characterized in that, Step S3 includes: The multiple data units are traversed according to a preset traversal order; For each of the data units traversed, the corresponding verification value is calculated using the target verification algorithm; The calculated check values ​​are arranged according to the preset traversal order to generate new check data containing an ordered check value sequence, wherein the Nth check value in the ordered check value sequence corresponds to the Nth data unit among the plurality of data units.

6. A method for parsing a data file, based on the data file generation method according to any one of claims 1-5, characterized in that, include: Read the header data of the data file to be parsed, and determine the file type of the data file to be parsed based on the format identifier in the header data; If the file type is an extended file type, the position of the new verification data is obtained based on the extended file header in the header data, the original data in the data file to be parsed is verified based on the new verification data, and the original verification data in the data file to be parsed is ignored. If the file type is a raw file type, then the raw data is verified based on the raw verification data in the data file to be parsed.

7. The data file parsing method according to claim 6, characterized in that, The original data in the data file to be parsed is validated based on the new validation data, and the original validation data in the data file to be parsed is ignored, including: Parse the extended file header to obtain the storage locations of the new verification data and the original data; The original data is traversed in a preset traversal order, and the verification value of the current data unit is calculated using the verification algorithm specified in the extended file header. Read the corresponding pre-stored verification value from the storage location of the new verification data, and compare the calculated verification value with the pre-stored verification value; When the original verification data unit in the original data is encountered, the original verification data unit is skipped.

8. The data file parsing method according to claim 7, characterized in that, Perform skip processing on the original verification data unit, including: Identify the block type identifier of the original verification data unit; In response to the block type identifier matching a preset verification data type, the data content of the original verification data unit is not included in the verification calculation, and the original verification data unit is not compared with any verification value; The read pointer is moved directly to the next data unit after the original verification data unit.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a data file generation method as described in any one of claims 1 to 5, or a data file parsing method as described in any one of claims 6 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for generating a data file as described in any one of claims 1 to 5, or the method for parsing a data file as described in any one of claims 6 to 8.

Images