Generation method and analysis method for data file of question bank system

Abstract

The invention discloses a generation method and an analysis method for a data file of a question bank system. The method comprises the steps of generating a file overall structure; determining the position of a file header structure based on the overall file structure, and generating the file header structure based on the position of the file header structure; determining the position of a table index area based on the file header structure, generating a table index area structure based on the position of the table index area, determining the position of each metadata table in the table data area based on the table index area structure, and generating the table data area based on the position of each metadata table; determining the position of a file index area based on the file header structure, generating a file index area structure based on the position of the file index area, determining the position of each file resource in the file data area based on the file index area structure, and generating the file data area based on the position of each file resource. The technical problems that in the prior art, when a large number of small files are frequently and randomly accessed, decompression efficiency is low, memory occupation is high, and access delay is large are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method for generating and parsing data files for a question bank system. Background Technology

[0002] In applications such as online education and qualification examinations, question bank systems typically need to manage and distribute a large number of small files (such as questions, images, and audio). Existing technologies generally use standard compression formats (such as ZIP) for data packaging and decompression. The basic process involves compressing each file in the question bank separately using a compression algorithm, and then packaging it into a ZIP file along with metadata such as filename, size, and offset. During access, the decompression program needs to read the central directory area, locate the file data blocks, and fully decode the compressed content to obtain the target file.

[0003] However, this type of general compression scheme suffers from problems such as low decompression efficiency, high memory consumption, large access latency, and potential security risks when facing frequent, random access scenarios. As the size of the question bank increases and the update frequency rises, the impact of these problems on system performance and user experience becomes increasingly apparent.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method for generating and parsing data files for a question bank system, which at least solves the technical problems of low decompression efficiency, high memory consumption, and large access latency in the prior art when frequently and randomly accessing a large number of small files.

[0006] According to one aspect of the present invention, a method for generating a data file for a question bank system is provided, comprising: generating an overall file structure, wherein the overall file structure includes functional areas, and a starting offset and an overall length of each functional area, wherein each functional area includes: a file header, a table index area, a file index area, a table data area, and a file data area; determining the position of a file header structure based on the starting offset and the overall length of the file header, and generating the file header structure based on the position of the file header structure, wherein the file header structure is used to locate the positions of each functional area; determining the position of the table index area based on the file header structure, and generating a table index area structure based on the position of the table index area; then determining the position of each metadata table in the table data area based on the table index area structure, and generating the table data area based on the position of each metadata table; determining the position of the file index area based on the file header structure, and generating a file index area structure based on the position of the file index area; then determining the position of each file resource in the file data area based on the file index area structure, and generating the file data area based on the position of each file resource.

[0007] According to another aspect of the present invention, a method for parsing data files for a question bank system is also provided, comprising: parsing the overall structure of the file to obtain the starting offset and overall length of each functional area in the overall structure of the file, wherein each functional area includes: a file header, a table index area, a file index area, a table data area, and a file data area; based on the starting offset and overall length of the file header, locating and parsing the file header structure to obtain the offsets and lengths of the table index area, the file index area, the table data area, and the file data area; based on the offset and length of the table index area, locating and parsing the table index area structure to obtain the table data area offset and table data length of the corresponding metadata table in the table data area, and locating and obtaining the corresponding metadata table based on the table data area offset and table data length of the corresponding metadata table; based on the offset and length of the file index area, locating and parsing the file index area structure to obtain the file data area offset and file data length of the corresponding file resource in the file data area, and locating and obtaining the corresponding file resource based on the file data area offset and file data length of the corresponding file resource.

[0008] According to another aspect of the present invention, an apparatus for generating data files for a question bank system is also provided, comprising: an overall structure generation module configured to generate an overall file structure, wherein the overall file structure includes functional areas, and a starting offset and an overall length of each functional area, wherein each functional area includes: a file header, a table index area, a file index area, a table data area, and a file data area; a file header structure generation module configured to determine the position of a file header structure based on the starting offset and the overall length of the file header, and generate the file header structure based on the position of the file header structure, wherein the file header structure is used to locate the positions of each functional area; and a table data generation module configured to determine the position of the table index area based on the file header structure, generate a table index area structure based on the position of the table index area, then determine the position of each metadata table in the table data area based on the table index area structure, and generate the table data area based on the position of each metadata table. The file data generation module is configured to determine the position of the file index area based on the file header structure, generate a file index area structure based on the position of the file index area, and then determine the position of each file resource in the file data area based on the file index area structure, and generate the file data area based on the position of each file resource.

[0009] According to another aspect of the present invention, a data file parsing apparatus for a question bank system is also provided, comprising: an overall structure parsing module configured to parse the overall structure of the file and obtain the starting offset and overall length of each functional area in the overall structure of the file, wherein each functional area includes: a file header, a table index area, a file index area, a table data area, and a file data area; a file header structure parsing module configured to locate the file header structure based on the starting offset and overall length of the file header, parse the file header structure, and obtain the offset and length of the table index area, the file index area, the table data area, and the file data area; and a table data parsing module configured to... To locate the table index area structure based on its offset and length, the system parses the table index area structure to obtain the table data area offset and table data length of the corresponding metadata table in the table data area. Then, based on the table data area offset and table data length of the corresponding metadata table, the system locates and retrieves the corresponding metadata table. The file data parsing module is configured to locate the file index area structure based on its offset and length, parse the file index area structure to obtain the file data area offset and file data length of the corresponding file resource in the file data area, and then locate and retrieve the corresponding file resource based on the file data area offset and file data length of the corresponding file resource.

[0010] In this embodiment of the invention, an overall file structure is generated, comprising functional areas, a starting offset, and an overall length for each functional area. Each functional area includes a file header, a table index area, a file index area, a table data area, and a file data area. Based on the starting offset and overall length of the file header, the position of the file header structure is determined, and the file header structure is generated based on its position. The file header structure is used to locate the positions of the functional areas. Based on the file header structure, the position of the table index area is determined, and a table index area structure is generated. Then, based on the table index area structure, the positions of the metadata tables in the table data area are determined, and the table data area is generated based on the positions of the metadata tables. Based on the file header structure, the position of the file index area is determined, and a file index area structure is generated. Then, based on the file index area structure, the positions of the file resources in the file data area are determined, and the file data area is generated based on the positions of the file resources. The above solution solves the technical problems of low decompression efficiency, high memory consumption, and large access latency in existing technologies when frequently and randomly accessing a large number of small files. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0012] Figure 1 This is a flowchart of an optional method for generating data files for a question bank system according to an embodiment of the present invention;

[0013] Figure 2 This is a flowchart of another method for generating data files for a question bank system according to an embodiment of this application;

[0014] Figure 3 This is a data structure diagram of an optional overall file structure according to an embodiment of the present invention;

[0015] Figure 4 This is a data structure diagram of an optional file header structure according to an embodiment of the present invention;

[0016] Figure 5 This is a data structure diagram of an optional key management area structure according to an embodiment of the present invention;

[0017] Figure 6 This is a data structure diagram of an optional table index area structure according to an embodiment of the present invention;

[0018] Figure 7This is a data structure diagram of an optional file index area structure according to an embodiment of the present invention;

[0019] Figure 8 This is a flowchart of a data file parsing method for a question bank system according to an embodiment of this application;

[0020] Figure 9 This is a flowchart of another method for parsing data files for a question bank system according to an embodiment of this application;

[0021] Figure 10 A schematic diagram of the structure of a computer device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] According to an embodiment of the present invention, a method embodiment for generating data files for a question bank system is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] Figure 1 This is a method for generating data files for a question bank system according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0026] Step S102: Generate the overall file structure, wherein the overall file structure includes each functional area, as well as the starting offset and overall length of each functional area. Each functional area includes: file header, table index area, file index area, table data area and file data area.

[0027] First, generate the starting offset and length of the file header and the key management area. For example, generate the starting offset and length of the file header, generate the starting offset of the key management area based on the starting offset and length of the file header, and generate the length of the key management area based on the actual length of the key.

[0028] Next, the starting offset and length of the table index area are generated. For example, based on the starting offset and length of the key management area, the starting offset of the table index area is generated, and the length of the table index area is generated based on the actual length of the table index area.

[0029] Next, the starting offset and length of the file index area are generated. For example, based on the starting offset and length of the table index area, the starting offset A of the file index area is generated, and the length of the file index area is generated based on the actual length of the file index area.

[0030] Then, the starting offset and length of the table data area are generated. For example, based on the starting offset A of the file index area and the length of the file index area, the starting offset B of the table data area is generated, and the length of the table data area is generated based on the actual length of the table data area.

[0031] Finally, the starting offset and length of the file data area are generated. For example, the starting offset C of the file data area is generated based on the starting offset B of the table data area and the length of the table data area, and the length of the file data area is generated based on the actual length of the file data area.

[0032] Step S104: Based on the starting offset and overall length of the file header, determine the position of the file header structure, and generate the file header structure based on the position of the file header structure, wherein the file header structure is used to locate the position of each functional area;

[0033] First, a file header structure is generated. For example, at the location of the file header structure, the following information is written sequentially to generate the file header structure: file identification information, used to identify the file format and version, including at least one of the following: magic number, version number, and global flag; key information, used to indicate the location of different keys used by the different data partitions, including at least one of the following: key offset and key length; index area information, used to indicate the location of the table index area and the file index area, including table index area offset, table index area length, file index area offset, and file index area length; data area information, used to indicate the location of the table data area and the file data area, including table data area offset, table data area length, file data area offset, and file data area length.

[0034] Next, a key management area structure is generated. For example, based on the starting offset and overall length of the key management area, a key management area structure is generated, wherein the key management area structure is used to use different key identifiers for different data partitions in each table data area and each file data area.

[0035] Step S106: Based on the file header structure, determine the location of the table index area, and generate the table index area structure based on the location of the table index area. Then, based on the table index area structure, determine the location of each metadata table in the table data area, and generate the table data area based on the location of each metadata table.

[0036] First, generate the table index area structure. At the location of the table index area, write the following information sequentially to generate the table index area structure: table index identification information, used to identify the type of the metadata table and its number of records, including at least one of the following: table type and number of records; table data location information, used to indicate the position of the metadata table in the table data area, including the table data area offset and table data length; table data encryption control information, used to indicate whether the metadata table is encrypted and the encryption parameter configuration, including encryption flags and initialization vectors.

[0037] Next, the table data area is generated. For example, information from the metadata table is added to the appropriate locations based on the information in the table index area structure.

[0038] Step S108: Based on the file header structure, determine the position of the file index area, and based on the position of the file index area, generate a file index area structure. Then, based on the file index area structure, determine the position of each file resource in the file data area, and based on the position of each file resource, generate the file data area.

[0039] First, a file index area structure is generated. For example, at the location in the file index area, the following information is written sequentially to generate the file index area structure: file index identification information, used to identify the type of file resource and its associated question, including at least one of the following: file type and associated question identifier; file positioning information, used to indicate the position of the file resource in the file data area, including the file data area offset and file data length; file encryption control information, used to indicate the encryption status and encryption parameter configuration of the file resource, including encryption flags and initialization vector.

[0040] Next, the file data area is generated. Based on the file index area structure, the location of each file resource in the file data area is determined, and the corresponding file resource is set based on the determined location.

[0041] Figure 2 This is another method for generating data files for a question bank system according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0042] Step S202: Generate the overall file structure.

[0043] During the data packaging stage, the system generates the overall file structure according to a preset format. The overall file structure is as follows: Figure 3 As shown, it includes, in sequence, a file header, a key management area, a table index area, a file index area, a table data area, and a file data area.

[0044] First, determine the starting offset and length of each functional area. The file header has a starting offset of 1 and a fixed length of 127 bytes. Based on the file header's offset and length, the starting offset of the key management area is determined to be 128 bytes, and the key management area's length is determined to be 1030 bytes based on its actual byte length (e.g., 1030 bytes). Following the above offset relationships, the starting offsets and lengths of the table index area, file index area, table data area, and file data area are calculated sequentially. The lengths of each index area and data area are variable values, depending on the number of metadata records and resources. Specifically, A = 1158 + table index area length, B = A + file index area length, and C = B + table data area length.

[0045] Step S204: Generate the file header structure.

[0046] The purpose of the file header structure is to provide a global mapping table for parsing the entire file, enabling rapid location of various functional areas later. The specific file header structure is as follows: Figure 4As shown, the header includes: file identification information (Magic = “BSQB”, Version, Flags); key information (KeyOffset, KeyLength); index area information (TableIndexOffset, TableIndexLength, FileIndexOffset, FileIndexLength); and data area information (TableDataOffset, TableDataLength, FileDataOffset, FileDataLength). The system writes the above header fields sequentially at the beginning of the file header structure, for example, at position 1. The file header structure is designed to be 127 bytes, but in this embodiment, only 81 bytes are used; the remaining bytes are reserved for future upgrades.

[0047] Step S206: Generate the key management area structure.

[0048] The key management area is used to store encryption keys and algorithm parameters. For example... Figure 5 As shown, the key management area structure includes the following fields: KeyType: Key type (e.g., 0 represents an internal key, 1 represents a general key); Algorithm: Encryption algorithm (e.g., AES / CTR / NoPadding); EKeyLen: Encryption key length; EncryptedKey: Key body (stored after encryption with the master key). The system writes the data corresponding to the above fields to the positions in the key management area structure according to the offset and length recorded in the file header structure.

[0049] Step S208: Generate the table index area structure.

[0050] The system writes multiple table index entries (TableIndexEntry) to the table index area. The table index area structure is as follows: Figure 6 As shown, each index entry contains: TableType (0 for subject, 1 for knowledge, 2 for question); RecordCount (the number of metadata records in the table); DataOffset and DataLength (offset and length within the table's data area); and an Encrypted flag and an IV (encryption flag and initialization vector). This structure enables fast location of different metadata tables within the table's data area.

[0051] Step S210: Generate the file index area structure.

[0052] Write several file index entries (DataFileIndexEntry) into the file index area. Each file index entry records information such as... Figure 7The following fields are shown: FileType (e.g., 0 for question stem, 1 for answer, 2 for analysis); RefID (corresponding question ID); DataOffset and DataLength (file data area offset and length); Encrypted and IV (encryption control information). This structure supports the mapping between questions and their resource files (images, audio, etc.).

[0053] Step S212: Generate the table data area and the file data area.

[0054] The system writes the actual metadata tables and file resources to their respective areas according to the offsets and lengths recorded in the table index and file index structures. Table data can be compressed JSON or binary serialized data (such as Protobuf format); file data can be encrypted and compressed rich text, image, or audio files. This ultimately forms a unified, efficient data file structure for rapid access by subsequent parsing methods.

[0055] In this embodiment, a fixed header and variable partitioning structure enables high-speed random access; independent encryption of each data area enhances security; an offset-based indexing structure avoids global scanning and improves file packaging efficiency; and a clear overall file structure facilitates subsequent memory mapping and parsing.

[0056] This application also provides a method for parsing data files in a question bank system. This parsing method can be applied to both the client and server sides. The following example uses the client side to demonstrate the rapid, on-demand parsing of encrypted and compressed data files transmitted from the server, enabling instantaneous access to question and resource data. Specifically, as... Figure 8 As shown, the method includes:

[0057] Step S802: Load file header and initialize memory mapping.

[0058] After receiving the encrypted and compressed data file, the client first reads a fixed-length header area (e.g., 128 bytes). The header area is as follows: Figure 3 The data includes fields used to describe the global structure of the file, such as the magic number, version number, and offset and length information for each functional area (KeyOffset, TableIndexOffset, FileIndexOffset, TableDataOffset, FileDataOffset, etc.).

[0059] Subsequently, the operating system's memory mapping interface (such as Linux's mmap() or Windows' CreateFileMapping()) is invoked to directly map the entire file or a selected region of the file into the virtual memory space. This process does not perform actual decompression or copying operations on the file; instead, it achieves "zero-copy" access by establishing a mapping relationship between the file content and the virtual address space, laying the foundation for subsequent efficient offset-based parsing.

[0060] Step S804: Parse the file header and extract the index area location.

[0061] Based on the offset and length information of each functional area recorded in the file header, the start and end addresses of the table index area and the file index area are calculated. For example: Table index area position = Table index area start offset (1158) + TableIndexOffset field in the file header structure; File index area position = Table index area start offset (A) + FileIndexOffset field in the file header structure. Based on this offset information, the logical range of the index area can be determined without reading the entire file content.

[0062] Step S806: Load only the index area.

[0063] Based on the offset address obtained in the previous step, the file's table index area and file index area are mapped into memory through memory mapping, without loading or parsing the key area and data area.

[0064] Since the index area typically occupies only a very small portion of the total file size, it can be loaded quickly. After loading, the data in the index area remains in a binary compressed state.

[0065] Step S808: Locate by offset based on the memory index table.

[0066] First, the index area is deserialized. Using lightweight binary deserialization protocols such as Protobuf, the raw binary stream in the index area is parsed into in-memory structured objects, forming an in-memory index table. The deserialized index table can be represented by two logical structures: a table index table (TableIndexList): each element contains the fields {TableType, RecordCount, DataOffset, DataLength, Encrypted, IV}; and a file index table (FileIndexList): each element contains the fields {FileType, RefID, DataOffset, DataLength, Encrypted, IV}.

[0067] Next, the system locates the resource file by offset based on the memory index table. When a client application needs to access a specific question (e.g., question ID=12345), it first looks up the corresponding record in the memory index table. Assuming the question metadata offset in the index table is [O1, L1] and the offset of the corresponding resource file in the file index table is [O2, L2], the system can calculate the virtual address of the memory mapping using the following formula: .in, This is the starting address of the file's virtual memory mapping; This is the relative offset of the record in the index table. This is the virtual address location of the target data. The system can directly access this address through a pointer without sequentially reading or loading preceding data blocks.

[0068] Step S810: Skip irrelevant data.

[0069] After locating the target offset, the parser does not need to scan or load irrelevant data areas. For non-target questions or irrelevant resource blocks, it skips them directly by updating the pointer position, achieving logical jump access. This implementation significantly reduces the number of I / O operations through the offset-based addressing mechanism, making random access performance close to linear addressing.

[0070] Step S812: Streaming decryption and decompression to obtain the target data.

[0071] Once the target data block is located, the corresponding decryption process is executed based on the encryption parameters recorded in the key management area. The system first reads the KeyOffset and KeyLength fields from the file header to locate the key management area, and then reads the corresponding key field: KeyType, Algorithm, EKeyLen, EncryptedKey, etc. Based on the key type, the corresponding decryption algorithm (such as AES-CTR mode) is selected, and streaming decryption is performed using the corresponding initialization vector (IV) and key identifier.

[0072] After decryption, the decryption result is streamed and decompressed (using either the DEFLATE or LZ4 algorithm). Because of the streaming processing method, the system can decrypt and decompress simultaneously, eliminating the need to load the entire data block into memory at once.

[0073] This embodiment can work in conjunction with the paging mechanism of the operating system: when an application accesses an undecompressed page, a page fault will be triggered, and the operating system will call the decompression engine to decompress only that page on the spot, thereby achieving true "on-demand decompression".

[0074] After the above streaming decryption and decompression process, the parser obtains the target question's metadata (such as the question stem text, knowledge points, answers, etc.) or file resources (images, audio, etc.). The decompressed data is directly mapped to an address space accessible to the application, which can read it instantly via pointers or API calls, without the need for intermediate files or secondary loading.

[0075] This embodiment, through memory mapping combined with offset indexing, offers the following advantages: 1) Employing an OS kernel-level mapping mechanism, the average latency for accessing any question or resource can be reduced to microseconds, significantly superior to traditional ZIP decompression schemes. 2) Only the index area and target data page are loaded during parsing, avoiding memory spikes caused by large-scale decompression. 3) The key management area is bound to the data partition, supporting independent encryption with multiple keys to prevent global data loss due to key leakage. 4) Through the offset addressing mechanism, central directory scanning is avoided, allowing direct jump to any question data location.

[0076] This application also provides a method for parsing data files in a question bank system, such as... Figure 9 As shown, the method includes the following steps:

[0077] Step S902: Obtain the index position and load the index area.

[0078] Similar to steps S802 to S806 in the above embodiments, they will not be described again here.

[0079] Step S904, hierarchical page mapping and predictive decompression.

[0080] The difference between this embodiment and the above embodiments is that, in the process of parsing data files, granular control of memory mapping and page decompression scheduling are adopted, so that the mapping granularity can be dynamically adjusted according to the data area type and access mode, and predictive page caching based on access behavior is introduced to achieve efficient parsing that combines on-demand and pre-read.

[0081] Specifically, after the index area is loaded, the access characteristics of each data area are first determined based on the type fields (TableType, FileType) of the records in the table index table and the file index table, as well as the size of the data area.

[0082] For high-frequency, small-volume table data areas, a page-level mapping strategy is adopted, with a default page granularity of 4KB, to achieve finer-grained random access control. For large-volume file data areas with strong sequential read characteristics, a block-level mapping strategy is adopted, with a default granularity of 64KB or 128KB, to reduce the number of page table entry switches. The mapping granularity is dynamically adjusted according to the device's memory capacity and the number of parallel threads. When there is ample available memory, the block-level mapping unit is automatically expanded to improve sequential read efficiency; when memory is limited, the page granularity is reduced to decrease unnecessary data loading.

[0083] After memory mapping is established, a Page Access Table is constructed. This table records the number of accesses, the most recent access time, and the access interval for each mapped page. Access weights are then calculated based on the data in the monitoring table.

[0084]

[0085] in, Let i be the frequency of visits to page i. The time interval between the last access. , These are the weighting coefficients. When the weights... When the threshold is exceeded, the page is identified as a "hot page" and a predictive decompression thread is started to pre-decompress and cache the hot page and its adjacent pages.

[0086] The predictive decompression thread works in conjunction with the operating system's page fault handling mechanism. When an application accesses a compressed data block, a page fault is triggered, immediately invoking the decompression engine to perform partial decompression on that page. For pages that have been predicted and cached, the decompression result is returned directly from the cache table, avoiding redundant I / O. Simultaneously, to prevent cache bloat, the least recently used page is automatically released when the cached page count exceeds a threshold.

[0087] Through the above improvements, the system achieves dynamic granular mapping while maintaining low memory usage. Compared with the fixed mapping method used in the previous embodiments, this embodiment can adaptively adjust the memory strategy according to the access pattern, significantly reducing random access latency.

[0088] Step S906: Streaming decompression.

[0089] The difference between this embodiment and the above embodiments is that it adopts a data block encryption and decryption mechanism and a decompression algorithm scheduling mechanism. By introducing a hash-derived key system at the index item layer and dynamically selecting the decompression algorithm according to the file type, it realizes integrated streaming processing and parallel optimization of decryption and decompression.

[0090] Specifically, after the parser locates the target data block based on the index, the system does not directly use the master key to decrypt the data. Instead, it calculates a local hash value based on the reference identifier (RefID) and initialization vector (IV) in the file index entry.

[0091] Then, the key derivation function (AES_KDF) is called, taking the master key MasterKey in the key management area as input, to generate a local key that is only valid for the current data block:

[0092]

[0093] The local key is destroyed immediately after the data block is decrypted to prevent key leakage due to memory residue.

[0094] For frequently accessed data blocks of the same type, the system establishes a two-level key cache (Key CacheLevel-2) in memory, recording the local key hash digest and the derived key index. When the same RefID is accessed again, the system first queries the cache table and verifies the matching result through the digest, thereby avoiding repeated derivation operations.

[0095] After decryption, the corresponding decompression algorithm is selected based on the file type recorded in the file index. When the file type is text data such as questions, knowledge points, and answers, the LZ4 algorithm is used to achieve fast streaming decompression; when the file type is large file data such as images, audio, or rich text, the ZSTD or DEFLATE algorithm is used to balance the compression ratio and CPU load.

[0096] This embodiment constructs an adaptive decompression pipeline, connecting the decryption stream output to the decompression thread pool to achieve pipelined parallel execution of decryption and decompression. Using this method, the decryption module continuously outputs data stream segments, which the decompression module consumes in real time and reconstructs the original data. The thread pool dynamically adjusts thread priorities and the number of parallel threads based on system load and access frequency: when a hot page is detected, the priority of the corresponding thread is increased and buffer space is reserved to achieve a fast response.

[0097] Furthermore, this embodiment maintains a decompression scheduling table across different algorithm threads to track the average decompression latency and CPU utilization for each algorithm. When the processing latency of a certain algorithm exceeds a threshold, some data blocks are automatically migrated to a lower-latency algorithm thread for execution, achieving load balancing and performance self-tuning. Through the above scheme, this embodiment can dynamically allocate decryption and decompression resources, achieving efficient streaming, parallel, and on-demand processing, ensuring optimal parsing performance for both large and small files within the same system.

[0098] This application also provides a data file generation apparatus for a question bank system, comprising: an overall structure generation module configured to generate an overall file structure, wherein the overall file structure includes functional areas, and the starting offset and overall length of each functional area, wherein each functional area includes: a file header, a table index area, a file index area, a table data area, and a file data area; a file header structure generation module configured to determine the position of a file header structure based on the starting offset and overall length of the file header, and to generate the file header structure based on the position of the file header structure, wherein the file header structure is used to locate the positions of each functional area; and a table data generation module. The file data generation module is configured to determine the location of the table index area based on the file header structure, generate a table index area structure based on the location of the table index area, then determine the location of each metadata table in the table data area based on the table index area structure, and generate the table data area based on the location of each metadata table; the file data generation module is configured to determine the location of the file index area based on the file header structure, generate a file index area structure based on the location of the file index area, then determine the location of each file resource in the file data area based on the file index area structure, and generate the file data area based on the location of each file resource.

[0099] This application also provides a data file parsing device for a question bank system. The parsing device includes: an overall structure parsing module configured to parse the overall file structure, obtaining the starting offset and overall length of each functional area in the overall file structure, wherein each functional area includes: a file header, a table index area, a file index area, a table data area, and a file data area; a file header structure parsing module configured to locate the file header structure based on the starting offset and overall length of the file header, parse the file header structure, and obtain the offsets and lengths of the table index area, the file index area, the table data area, and the file data area; and a table data parsing module configured to... The offset and length of the table index area are used to locate the table index area structure. The table index area structure is parsed to obtain the table data area offset and table data length of the corresponding metadata table in the table data area. Then, based on the table data area offset and table data length of the corresponding metadata table, the corresponding metadata table is located and obtained. The file data parsing module is configured to locate the file index area structure using the offset and length of the file index area. The file index area structure is parsed to obtain the file data area offset and file data length of the corresponding file resource in the file data area. Then, based on the file data area offset and file data length of the corresponding file resource, the corresponding file resource is located and obtained.

[0100] This application also provides a data file processing system for a question bank system, including the aforementioned file generation device and file parsing device.

[0101] It should be noted that the data file generation and parsing apparatus for the question bank system provided in the above embodiments are only illustrative examples of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the data file generation and parsing apparatus for the question bank system provided in the above embodiments belong to the same concept as the data file generation and parsing method embodiments for the question bank system. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0102] Figure 10 A schematic diagram of a computer device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 10 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0103] like Figure 10 As shown, the computer device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage section 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0104] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0105] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for generating a data file for a test bank system, characterized by, The method comprises: generating a file overall structure, wherein the file overall structure comprises functional areas, and starting offsets and overall lengths of the functional areas, the functional areas comprising a file header, a table index area, a file index area, a table data area, and a file data area; determining a position of a file header structure based on starting offsets and overall lengths of the file header, and generating the file header structure based on the position of the file header structure, wherein the file header structure is used to locate positions of the functional areas; determining a position of the table index area based on the file header structure, and generating a table index area structure based on the position of the table index area, then determining positions of metadata tables in the table data area based on the table index area structure, and generating the table data area based on the positions of the metadata tables; determining a position of the file index area based on the file header structure, and generating a file index area structure based on the position of the file index area, then determining positions of file resources in the file data area based on the file index area structure, and generating the file data area based on the positions of the file resources.

2. The method of claim 1, wherein the functional areas further comprise a key management area; and after the file header structure is generated, the method further comprises generating a key management area structure based on starting offsets and overall lengths of the key management area, wherein the key management area structure is used to identify different keys for different data partitions in the table data areas and the file data areas. The file header structure is generated based on the position of the file header structure, comprising: sequentially writing the following information at the position of the file header structure to generate the file header structure: file identification information used to identify a file format and a version, comprising at least one of the following: a magic number, a version number, and a global flag; 3. The method of claim 2, wherein, key information used to indicate positions of different keys for the different data partitions, comprising at least one of the following: a key offset and a key length; index area information used to indicate positions of the table index area and the file index area, comprising a table index area offset, a table index area length, a file index area offset, and a file index area length; data area information used to indicate positions of the table data area and the file data area, comprising a table data area offset, a table data area length, a file data area offset, and a file data area length. The table index area structure is generated based on the position of the table index area, comprising: sequentially writing the following information at the position of the table index area to generate the table index area structure: table index identification information used to identify a type of a metadata table and a number of records thereof, comprising at least one of the following: a table type and a number of records; 4. The method of claim 1, wherein, table data positioning information used to indicate a position of the metadata table in the table data area, comprising a table data area offset and a table data length; table data encryption control information used to indicate whether the metadata table is encrypted and encryption parameter configuration, comprising an encryption flag and an initial vector. ​ ​ 5. The method of claim 1, wherein, Based on the position of the file index area, the file index area structure is generated, including: in the position of the file index area, the following information is written in sequence to generate the file index area structure: File index identification information, used to identify the type of file resource and its associated topic, including at least one of the following: file type and associated problem identification; File positioning information, used to indicate the position of the file resource in the file data area, including file data area offset and file data length; File encryption control information, used to indicate the encryption state and encryption parameter configuration of the file resource, including encryption flag and initial vector.

6. The method of claim 2, wherein, The file overall structure is generated, including: The starting offset and length of the file header are generated, based on the starting offset and length of the file header, the starting offset of the key management area is generated, and based on the actual length of the key, the length of the key management area is generated; Based on the starting offset and length of the key management area, the starting offset of the table index area is generated, and based on the actual length of the table index area, the length of the table index area is generated; Based on the starting offset of the table index area and the length of the table index area, the starting offset A of the file index area is generated, and based on the actual length of the file index area, the length of the file index area is generated; Based on the starting offset A of the file index area and the length of the file index area, the starting offset B of the table data area is generated, and based on the actual length of the table data area, the length of the table data area is generated; Based on the starting offset B of the table data area and the length of the table data area, the starting offset C of the file data area is generated, and based on the actual length of the file data area, the length of the file data area is generated; Wherein, the length of the table index area, and the offset and length of the file index area, the table data area, and the file data area are variable.

7. A method for parsing a data file for a test bank system, the method comprising: Including: The file overall structure is parsed to obtain the starting offset and overall length of each functional area in the file overall structure, wherein the each functional area includes: file header, table index area, file index area, table data area and file data area; Based on the starting offset and overall length of the file header, the file header structure is positioned and parsed to obtain the offset and length of the table index area, the file index area, the table data area and the file data area; Based on the offset and length of the table index area, the table index area structure is positioned and parsed to obtain the table data area offset and table data length of the corresponding metadata table in the table data area, and based on the table data area offset and table data length of the corresponding metadata table, the corresponding metadata table is positioned and acquired; Based on the offset and length of the file index area, the file index area structure is positioned and parsed to obtain the file data area offset and file data length of the corresponding file resource in the file data area, and based on the file data area offset and file data length of the corresponding file resource, the corresponding file resource is positioned and acquired.

8. An apparatus for generating a data file for a test bank system, characterized by comprising: Including: The whole structure generating module is configured to generate a file whole structure, wherein the file whole structure comprises functional areas, and starting offsets and whole lengths of the functional areas, the functional areas comprising a file header, a table index area, a file index area, a table data area and a file data area; The file header structure generating module is configured to determine a position of a file header structure based on the starting offset and the whole length of the file header, and generate the file header structure based on the position of the file header structure, wherein the file header structure is used to locate positions of the functional areas; The table data generating module is configured to determine a position of the table index area based on the file header structure, and generate a table index area structure based on the position of the table index area, then determine positions of metadata tables in the table data area based on the table index area structure, and generate the table data area based on the positions of the metadata tables. The file data generating module is configured to determine a position of the file index area based on the file header structure, and generate a file index area structure based on the position of the file index area, then determine positions of file resources in the file data area based on the file index area structure, and generate the file data area based on the positions of the file resources.

9. An apparatus for parsing a data file for a test bank system, the apparatus comprising: a data file; a parser for parsing the data file; and a database for storing parsed data from the data file. The whole structure generating module is configured to generate a file whole structure, wherein the file whole structure comprises functional areas, and starting offsets and whole lengths of the functional areas, the functional areas comprising a file header, a table index area, a file index area, a table data area and a file data area; The file header structure generating module is configured to determine a position of a file header structure based on the starting offset and the whole length of the file header, and generate the file header structure based on the position of the file header structure, wherein the file header structure is used to locate positions of the functional areas; The table data generating module is configured to determine a position of the table index area based on the file header structure, and generate a table index area structure based on the position of the table index area, then determine positions of metadata tables in the table data area based on the table index area structure, and generate the table data area based on the positions of the metadata tables. The file data generating module is configured to determine a position of the file index area based on the file header structure, and generate a file index area structure based on the position of the file index area, then determine positions of file resources in the file data area based on the file index area structure, and generate the file data area based on the positions of the file resources. The whole structure generating module is configured to generate a file whole structure, wherein the file whole structure comprises functional areas, and starting offsets and whole lengths of the functional areas, the functional areas comprising a file header, a table index area, a file index area, a table data area and a file data area; 10. A processing system for data files of a test bank system, characterized by, The file header structure generating module is configured to determine a position of a file header structure based on the starting offset and the whole length of the file header, and generate the file header structure based on the position of the file header structure, wherein the file header structure is used to locate positions of the functional areas; The table data generating module is configured to determine a position of the table index area based on the file header structure, and generate a table index area structure based on the position of the table index area, then determine positions of metadata tables in the table data area based on the table index area structure, and generate the table data area based on the positions of the metadata tables. The file data generating module is configured to determine a position of the file index area based on the file header structure, and generate a file index area structure based on the position of the file index area, then determine positions of file resources in the file data area based on the file index area structure, and generate the file data area based on the positions of the file resources.

Images