Method for implementing large batch data decryption and encryption process pause and continue processing

Abstract

This invention provides a method for resuming processing after a pause in the encryption / decryption process of large amounts of data. First, a database index containing a hierarchical directory structure is constructed based on the target task. During processing, a bottom-up strategy is used to extract the current processing directory, and after the encryption / decryption loop of a single file is closed, the synchronization status of the file count variable is updated using database transactions. Upon capturing an interrupt signal, the system detects the memory stream buffer and forces a delayed response until the current single file completes its disk write-back operation and safely releases the underlying file handle, thereby locking the processing boundary and solidifying the paused state. When the program resumes execution, a Boolean error-proofing feature bit is used to verify the data write-to-disk status and calculate the effective skip amount. By performing an index span offset operation on the file traversal sequence, already processed files are precisely skipped, and subsequent tasks continue. This invention combines memory mapping, connection pooling, and multi-process concurrency techniques to effectively solve the problems of resource congestion and progress loss during massive data processing while ensuring the integrity of single files.

Description

Technical Field

[0001] This invention relates to the field of large-scale data processing, and in particular to a method for resuming processing after the encryption and decryption process of large-scale data has been paused. Background Technology

[0002] File encryption uses cryptographic techniques to transform file content according to a specific algorithm, hiding the true content and preventing unauthorized access, thus achieving security. In big data industries such as geographic information, project data volumes are typically enormous, encompassing complex structures such as high-resolution satellite imagery, vector maps, work reports, and realistic 3D models. Especially in accordance with national data management regulations requiring "off-site backup," important organizational data must be copied to designated locations for storage; the scale of such data often exceeds terabytes. To ensure the security of core information during distribution, transmission, or physical transfer, strict encryption is essential, followed by decryption and restoration upon final use or archiving.

[0003] In practice, data encryption and decryption heavily rely on the computer's underlying disk read / write performance and cryptographic computation capabilities. Once large-scale processing of massive amounts of files begins, the computer's disk throughput will be occupied for extended periods, resulting in extremely long processing cycles, often taking hours or even days to complete a single batch. During this time, system resources are completely consumed, preventing the computer from performing other tasks. When users encounter urgent work requiring computer access, they are either forced to wait for encryption / decryption to complete or forcibly terminate the task. Forcibly terminating the task may not only corrupt the current file but also result in the loss of previously processed data, necessitating a complete reprocessing and significantly reducing work efficiency.

[0004] To address the interruption and recovery issues in data processing, prior art US7646867B2 discloses a system and method for encrypting data. This primarily involves storing progress data related to the encryption process, allowing access to this data after an interruption to attempt to resume the encryption process. However, this prior art mainly targets single, continuous data streams or simple, flat file structures. When faced with complex directory trees containing millions of fragmented entity files and possessing extremely deep hierarchical structures, it lacks an efficient organization and indexing filtering mechanism for the massive underlying directories, easily leading to chaotic progress records. Furthermore, prior art CN113704277B discloses a database-based breakpoint resumption method and related apparatus. It stores breakpoint position identifiers in database breakpoint records and updates them during database transactions to represent processing progress. However, directly applying this prior art to encryption and decryption scenarios involving massive entity files means the system needs to create an index for each independent fragmented file in the database and frequently update the records. This will trigger extremely large-scale high-concurrency database read / write operations, causing not only high system memory and computing resource overhead but also a high risk of database deadlock.

[0005] Therefore, how to implement a processing mechanism that does not destroy the integrity of a single file, greatly reduces database concurrency overhead, and supports safe pause and precise breakpoint resumption during the encryption and decryption of large batches and multi-level data entity files is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to provide a method for resuming processing after pausing the encryption and decryption process of large batches of data. This solves the technical problems of existing technologies when processing massive, multi-level data files, where the database concurrency overhead is too high due to recording the progress of each individual file, and it is difficult to balance the integrity of individual files with secure and accurate breakpoint resumption.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for resuming processing after a large-scale data encryption / decryption process has been paused, comprising the following steps: S1. Construct a database index based on the target task. The database index includes a directory table for recording the hierarchical directory structure and a file table for recording the execution progress. S2. Extract the current processing directory from the directory table, synchronously write the path of the current processing directory into the file table, and initialize the number of completed files variable in the file table; S3. Call the operating system's file traversal API to perform file reading and encryption / decryption data conversion in the current processing directory, and write back to disk after the conversion of a single file is completed. Synchronously update the number of completed files in the file table by executing a database transaction. S4. When an interrupt signal is captured, the operation is safely stopped at the boundary where the current single file processing is completed, and the status record in the database index is persisted, keeping the number of completed files in the file table unchanged. S5. When the program resumes running, it extracts the current processing directory path and the number of completed files stored in the file table, calls the operating system's file traversal API to generate a file traversal sequence for the current processing directory, uses the number of completed files variable to perform an index skip operation in the file traversal sequence, and continues to load and perform encryption / decryption data conversion on the remaining files after skipping.

[0008] In the preferred scheme, the steps for building the database index in S1 include: The system recursively calls the operating system's underlying file search API to obtain the tree-like hierarchical data of the target task's root directory and all its subdirectories; Create directory tables and file tables by executing Data Definition Language (DDL) commands in a relational database; Parse the tree-structured data and extract the full path and directory depth values ​​for each directory. The primary key of the parent directory in the directory table and the total number of files in this directory. ( The extracted parameters are then stored in the directory table using SQL insert statements. Set a Status field in the directory table to identify the four enumerated states: not started, in progress, paused, and completed.

[0009] In the preferred embodiment, the step of retrieving the current processing directory in S2 includes: Use SQL query commands to retrieve the set of unprocessed directories from the directory table, and extract the maximum level depth value from the set. ; From the value with the maximum layer depth In the candidate subdirectories, extract the directory record with the smallest primary key ID as the current processing directory to achieve bottom-up directory hierarchy traversal; The path to the current processing directory and the total number of files are variables. Synchronously write to the file table, and initialize the completed file count variable in the file table to its initial state. ,in Indicates completion of the first The progress value after processing each file, among which and And update the Status field of the corresponding directory in the directory table to "in progress".

[0010] In the preferred scheme, the synchronous update of the file table in S3 includes: After completing the read, encryption / decryption algorithm conversion, and disk write loop for each individual file in the memory buffer, an explicit transaction commit operation is initiated to the database, updating the number of completed files variable to: ; Boolean condition calculations are performed in real time in system memory. When the result of this condition is true, an SQL delete command is triggered to clear the current directory record in the file table and change the Status field corresponding to the current processing directory in the directory table to the completed status. Using the parent directory primary key Perform a self-join SQL query upwards in the directory table, prioritizing the search for unprocessed sibling directories with the same parent directory primary key as the new current processing directory to enter the next round of file traversal loop.

[0011] In the preferred embodiment, the index skipping operation step in S5 includes: Extract the variable representing the number of currently completed files stored in the file table. ,in The length is the total number of files completed before the interruption, generated by calling the operating system's file traversal API. The current directory file list array FileList; Define array traversal index and During traversal, mathematical logic judgments are performed: When the judgment expression When established, the traversal index is... Perform a span offset operation with a step size of 1, skipping the file loading instruction; When the judgment expression When it is established, the index of the file list array FileList is... The file continues to load and performs encryption / decryption data conversion.

[0012] In the preferred scheme, algebraic backoff calculation of abnormal offsets is performed by combining Boolean logic variables with database transaction mechanisms, including: Add a Boolean error-proofing feature field to the file table. Set initial value ; After each file data stream is completely flushed to disk, while initiating an SQL transaction commit to update the number of completed files variable, a state flip is performed on the error-proofing feature field. This utilizes the atomicity of a single SQL transaction to synchronize commit status and count; When the program resumes operation, it reads the characteristic bits of the last record in the database. Progress value at the time of interruption Verify the theoretically derived feature bits; If the verification result indicates that the last file conversion was not completely written to disk, then calculate the actual effective skip amount before performing the file list array traversal. Calculated Replace the original judgment expression Then execute logical judgments. This allows the traversal offset to be rolled back by one step, reloading the original file at the location of the abnormal breakpoint.

[0013] In the preferred solution, a multi-threaded asynchronous concurrency model is used to improve the performance of processing large batches of files, including: The main thread initializes and maintains a memory-based thread-safe task queue to store subdirectory path tasks extracted from the directory table; A thread pool is constructed by pre-creating a fixed number of worker threads in the system process, and the worker threads continuously poll the thread-safe task queue. After a worker thread pops a task from the queue, it independently calls the operating system's file traversal API, performs memory read / write operations, and performs data encryption / decryption conversion. When the main thread catches an interrupt signal, it sets the global stop flag variable in memory to be active. After processing the current single file, the worker thread checks the global stop flag variable. If it is active, it commits the final database transaction and performs thread termination and recycling.

[0014] In the preferred embodiment, when performing the database transaction synchronization update step, database connection pooling technology is used to manage the underlying network connections, including: During the program startup phase, a connection pool set containing a preset number of pre-connected database instance objects is initialized in the system heap memory; Each time a worker thread completes the instruction to update the number of files, it directly requests an idle database instance object from the connection pool collection. Use the obtained database instance object to execute SQL update statements and transaction commit operations; After the operation is completed, the context variables of the database instance object are cleaned up and its status is marked as idle, and it is returned to the connection pool set to eliminate the system resources consumed by the frequent establishment and disconnection of database connections during high-frequency file encryption and decryption.

[0015] In the preferred scheme, during the encryption / decryption data conversion, the operating system's memory-mapped file API is called to optimize underlying read / write efficiency, including: The operating system's MMAP memory mapping interface is invoked to directly map the disk file to be processed into the virtual memory address space of the current process, and the starting address of the mapping is obtained. and the total file size (FileSize); Define a constant for the data block size, BlockSize, and use an iterative pointer address formula. Moving the underlying pointer in virtual memory, where The current data block number and ; The file data stream is read in fixed byte block slices and passed to the encryption / decryption algorithm function to perform in-situ cryptographic operations and transformations. The transformed byte blocks are then asynchronously synchronized to the disk using the operating system's underlying memory page dirty data write-back mechanism.

[0016] In the preferred solution, the operating system's multi-process concurrency model and the database's exclusive locking mechanism are used to accelerate the resumption of breakpoints in the underlying directory, including: The main process analyzes the parent-child node mapping relationships in the directory tables of the database and filters out all nodes that satisfy the dependency equation. A group of leaf directories at the bottom level with no subdirectories; The selected leaf directory groups are packaged into multiple independent process parameter packages, and the operating system's process derivation API is called to create multiple independently running child processes. The process parameter packages are then distributed through inter-process communication. Each child process concurrently calls the file traversal API and performs encryption / decryption data conversion in its own independent memory space; When executing the SQL statement that updates the number of files, the child process appends a row-level exclusive lock instruction to the SQL command to ensure that records in the same directory are calculated and updated serially during concurrent writes to the database, preventing data races from causing breakpoint count anomalies.

[0017] In the preferred embodiment, the steps in S4 to capture the interrupt signal and safely stop the operation include: when the event listener built into the main thread captures the asynchronous interrupt signal issued by the operating system, it checks whether there is a single file data stream in the memory stream buffer that has not yet been flushed to the physical disk; If a single file data stream that has not yet been flushed to the physical disk is detected, the asynchronous interrupt signal is added to the blocking queue to suspend it, forcibly delaying the interrupt response until the current single file completes the last disk I / O write-back operation and safely releases the underlying file handle, thereby locking the boundary of single file processing completion. After the underlying file handle is safely released, the update instruction of the structured query language is executed to switch the Status field corresponding to the current processing directory in the directory table to an enumerated constant representing the paused state. The final database transaction commit operation is triggered, and the latest number of completed files cached in memory is synchronized and persisted to the file table. Then, the underlying database network connection is actively disconnected and the worker thread pool in memory is destroyed, thereby maintaining the number of completed files in the file table to remain absolutely unchanged during program shutdown.

[0018] This invention provides a method for resuming large-scale data encryption / decryption processes after a pause. This application innovatively introduces a database index design based on a hierarchical directory structure, using directories as the smallest record granularity to manage the total number of files and the completion progress. This mechanism completely eliminates the enormous overhead of building databases for each of the massive individual files in traditional technologies, greatly reducing database concurrency pressure and system memory consumption.

[0019] This application generates a file sequence by calling the operating system's underlying file traversal application interface. When the program resumes, it performs a pure memory-level index span skipping operation based on the number of completed files. Users can safely pause at any time to release disk resources, and upon restarting, it precisely skips processed files with zero overhead and directly continues processing the remaining files, greatly improving the flexibility of device use.

[0020] This application combines error-proof feature bits with a state reversal mechanism to construct an abnormal offset rollback calculation mechanism. In the event of extreme circumstances such as power outages that result in incomplete data writing to disk, the system can automatically compare feature bits and calculate the effective skip amount, roll back the traversal offset by one step, reload the abnormal breakpoint file, completely prevent data corruption, and ensure the absolute security of the output data.

[0021] This application deeply integrates memory mapping technology into its underlying architecture to avoid redundant data copying losses. It also selects underlying leaf directories without subdirectories for independent multi-process distribution, supplemented by row-level exclusive locks to prevent concurrent data contention. While ensuring safe breakpoint resumption, it maximizes the multi-core concurrency potential of the underlying hardware, maintaining extremely excellent data transfer performance. Attached Figure Description

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of the method for resuming processing after a pause in this invention; Figure 2 This is a flowchart of the breakpoint recovery and exception rollback logic of the present invention. Detailed Implementation

[0023] Example 1 like Figure 1 As shown, a method for resuming large-scale data encryption / decryption processing after a pause includes the following steps: S1. Construct a database index based on the target task. The database index includes a directory table for recording the hierarchical directory structure and a file table for recording the execution progress. S2. Extract the current processing directory from the directory table, synchronously write the path of the current processing directory into the file table, and initialize the number of completed files variable in the file table; S3. Call the operating system's file traversal API to perform file reading and encryption / decryption data conversion in the current processing directory, and write back to disk after the conversion of a single file is completed. Synchronously update the number of completed files in the file table by executing a database transaction. S4. When an interrupt signal is captured, the operation is safely stopped at the boundary where the current single file processing is completed, and the status record in the database index is persisted, keeping the number of completed files in the file table unchanged. S5. When the program resumes running, it extracts the current processing directory path and the number of completed files stored in the file table, calls the operating system's file traversal API to generate a file traversal sequence for the current processing directory, uses the number of completed files variable to perform an index skip operation in the file traversal sequence, and continues to load and perform encryption / decryption data conversion on the remaining files after skipping.

[0024] In the preferred scheme, the steps for building the database index in S1 include: The system recursively calls the operating system's underlying file search API to obtain the tree-like hierarchical data of the target task's root directory and all its subdirectories; Create directory tables and file tables by executing Data Definition Language (DDL) commands in a relational database; Parse the tree-structured data and extract the full path and directory depth values ​​for each directory. The primary key of the parent directory in the directory table and the total number of files in this directory. ( The extracted parameters are then stored in the directory table using SQL insert statements. Set a Status field in the directory table to identify the four enumerated states: not started, in progress, paused, and completed.

[0025] In a specific implementation, the steps involved in constructing the database index are the core foundation for implementing a mechanism to resume processing large volumes of data from breakpoints. The system first calls the operating system's underlying file search application interface to recursively traverse and obtain the hierarchical tree data of the target task's root directory and all its subdirectories. This underlying file search application interface can directly interact with the operating system's file system kernel, thereby significantly improving the efficiency of scanning and retrieving massive amounts of fragmented files and ensuring that the system can completely and without omission grasp the physical storage structure of the data to be processed.

[0026] After acquiring the hierarchical tree data, the system executes data definition language instructions within the relational database using the database driver component to dynamically create directory tables and file tables for persistent storage. Employing a relational database leverages its atomicity and consistency characteristics to provide reliable underlying storage support for subsequent high-concurrency read / write operations and state changes. The directory table primarily maps and persists the multi-level file directory tree topology on the physical hard drive, while the file table is specifically used to dynamically track and record the progress boundaries of the currently processed files.

[0027] Subsequently, the system performs deep learning algorithms on the tree-like hierarchical data residing in computer memory to extract the key parameters required for constructing the database relational index. These key parameters specifically include the full path string of each independent directory and the directory level depth value indicating which level the directory is in the entire tree topology. The primary key used to refer to the parent directory of this directory within the directory table. And the total number of files contained within the directory, as calculated statistically. .

[0028] In this step, This mathematical expression strictly defines and constrains the legal data types and value space of the total number of files variable. The formula indicates that the total number of files variable... The value of this variable must strictly belong to the set of positive integers; that is, in the computer's underlying memory allocation and database field design, this variable is limited to unsigned integers greater than zero. This logical constraint ensures that only valid entity directories containing at least one data file are included in the substantive encryption or decryption task scheduling queue, automatically filtering out empty directories with no business value. The system will use the extracted full path and directory level depth values... Parent directory primary key and the total number of files variable Key parameters are persistently saved to the directory table in a structured manner using the insert statement of the structured query language, thereby completing the lossless conversion from the physical file level to the database relational model.

[0029] To achieve precise control and status management throughout the entire lifecycle of large-scale encryption / decryption tasks, the system has specifically included a Status field within the directory table's data structure. This Status field is rigorously defined as a control variable containing four fixed enumerated states, which precisely correspond to the subtask's unstarted, in progress, paused, and completed stages, respectively. Through this finite state machine design pattern, the system can lock the current job process in any underlying subdirectory in real-time and mutually exclusively, preventing dirty reads caused by inter-process communication.

[0030] The combination of steps described above for building a database index yields extremely significant benefits. By pre-calling the system's underlying application programming interface to parse the global tree structure, and abstracting the multi-level physical directory into numerical values ​​with accompanying hierarchical depths, the system achieves this. Parent directory primary key This application completely decouples the time-consuming file traversal and scanning phase from the high-intensity cryptographic encryption and decryption operations, using relational database records. This architectural design significantly reduces the peak memory usage at any given moment, preventing system crashes caused by memory overflow or disk I / O queue congestion when processing terabyte-level massive amounts of data. Simultaneously, a strict variable for the total number of files is introduced. The Status field, along with other status characteristics, enables computer systems to completely break free from reliance on volatile memory when faced with sudden power outages, kernel anomalies, or external interruptions caused by manual intervention. Relying solely on the persistent state dictionary and total count records in the persistent database, the system can instantly and losslessly restore to its precise position before the interruption upon program restart. This provides solid technical support for the robustness, fault tolerance, and high availability of the entire large-scale data encryption and decryption method, completely solving the underlying technical problem in existing technologies where the lack of a global hierarchical index leads to data truncation or state corruption during breakpoint continuation.

[0031] In the preferred embodiment, the step of retrieving the current processing directory in S2 includes: Use SQL query commands to retrieve the set of unprocessed directories from the directory table, and extract the maximum level depth value from the set. ; From the value with the maximum layer depth In the candidate subdirectories, extract the directory record with the smallest primary key ID as the current processing directory to achieve bottom-up directory hierarchy traversal; The path to the current processing directory and the total number of files are variables. Synchronously write to the file table, and initialize the completed file count variable in the file table to its initial state. ,in Indicates completion of the first The progress value after processing each file, among which and And update the Status field of the corresponding directory in the directory table to "in progress".

[0032] In a specific implementation, the step of extracting the currently processed directory is the core logical support for achieving ordered scheduling of multi-level file structures. The system first uses Structured Query Language (SCL) instructions to search the directory table stored in the database, filtering out all records that have not yet undergone encryption / decryption operations, thereby obtaining a set of unprocessed directories. Subsequently, the system uses aggregation functions to extract the maximum level depth value from this set of unprocessed directories. The level depth value represents the depth level of a specific node in the physical directory tree. Next, the system selects from all nodes with the highest level depth value... In the candidate subdirectories, the directory record with the smallest primary key identifier is extracted and established as the current processing directory. Through this scheduling strategy of first searching the deepest level and then finding the earliest entered primary key within the same level, the system establishes a strict bottom-up directory hierarchy traversal mechanism. The beneficial effect of this mechanism is that it ensures that when processing massive directory trees with complex parent-child dependencies, the computer prioritizes clearing the lowest-level leaf directories before gradually backtracking upwards. This not only avoids premature locking of the upper-level directory structure but also effectively prevents resource deadlock problems in multi-threaded concurrent environments, greatly improving the stability and logical rigor of hierarchical data traversal.

[0033] After determining the current processing directory, the system executes a database write transaction, storing the full path of the current processing directory and the total number of files within it. The progress is synchronously written to a file table specifically used to record dynamic progress. At the same time, the system initializes the number of completed files in the file table to its initial state. . formula This means that when the current processing directory has just been taken over by the background service and no underlying single file has yet completed encryption / decryption operations, its corresponding progress baseline is safely and accurately reset to zero. Furthermore, the system employs the discrete state equation... To indicate the completion of the first The progress value after processing each file. Variables in the formula. It belongs to the set of natural numbers, that is, it satisfies the formula This restricts the number of files processed to a non-negative integer. Furthermore, this progress value is strictly subject to inequalities throughout the entire processing lifecycle. The logical constraints. This inequality clarifies that regardless of how the encryption / decryption process proceeds, the minimum number of completed records will not be lower than zero, and the maximum will absolutely not exceed the total number of files in the directory. By introducing the aforementioned rigorous mathematical algebraic expressions and boundary constraints, this application effectively avoids the risks of memory overflow or counter overflow that are prone to occur in traditional pointer increment algorithms. Its beneficial effect is that it provides an extremely accurate and mathematically sound state tracking benchmark for breakpoint continuation, completely eliminating the hidden danger of chaotic progress records under extreme abnormal conditions.

[0034] Finally, while completing the above synchronization and initialization operations, the system leverages the atomicity of a single database transaction to immediately update the Status field in the directory table corresponding to the currently processed directory to a specific value representing the in-process status. This status update step is equivalent to adding a business-exclusive lock to the directory at the system-wide level. Its beneficial effect is that once the status field is marked as in progress, any other background polling process or concurrent worker thread will automatically filter out that directory when performing a search of the unprocessed directory set. This lightweight concurrency control method based on database status field effectively eliminates data races and duplicate encryption / decryption problems caused by multiple processes simultaneously contending for the same directory, thus optimizing the concurrency scheduling performance of the underlying multi-core hardware while ensuring the correctness of large-scale file data flow.

[0035] In the preferred scheme, the synchronous update of the file table in S3 includes: After completing the read, encryption / decryption algorithm conversion, and disk write loop for each individual file in the memory buffer, an explicit transaction commit operation is initiated to the database, updating the number of completed files variable to: ; Boolean condition calculations are performed in real time in system memory. When the result of this condition is true, an SQL delete command is triggered to clear the current directory record in the file table and change the Status field corresponding to the current processing directory in the directory table to the completed status. Using the parent directory primary key Perform a self-join SQL query upwards in the directory table, prioritizing the search for unprocessed sibling directories with the same parent directory primary key as the new current processing directory to enter the next round of file traversal loop.

[0036] In a specific implementation, the synchronous update of the file table is a core execution step to ensure accurate recording of encryption / decryption progress and secure state transition. The computer system, within its memory buffer, rigorously executes a complete closed-loop operation for each individual file: reading from the physical disk, performing core data conversion via encryption / decryption algorithms, and then securely writing the converted target data stream back to the physical disk. Each time such a single-file processing loop is fully completed, the system immediately initiates an explicit transaction commit operation to the underlying relational database. The core logic of this transaction commit operation lies in the rigorous mathematical recursive update of the number of completed files; the specific update calculation follows discrete state transition equations. In this mathematical formula, the variables... This is a status node representing the number of files that the system has successfully processed. The variable on the right-hand side of the equation... This represents the total number of valid files completed in the current directory before the system begins processing the latest single-unit file. The system algebraically adds this historical cumulative value to a constant 1 to calculate the latest total number of completed files after processing the current single-unit file, and assigns this latest result to the variable on the left side of the equation. This recursive update mechanism, deeply integrated with explicit database transactions and discrete state transition equations, offers significant advantages. From its underlying architecture, this mechanism ensures absolute atomic synchronization between the data write-to-disk state of individual physical files and the progress updates of database records. In extreme scenarios such as sudden power outages, system crashes, or kernel failures, it completely avoids fatal data inconsistencies, such as file data being converted and written back but the progress count not being recorded, or the progress count increasing prematurely but the physical file not yet written back to disk. This provides an absolutely reliable data foundation for subsequent precise breakpoint continuation.

[0037] As the closed-loop encryption and decryption of single-file applications is executed frequently, the system will calculate and compare Boolean judgments in real time in the efficient system memory. In this Boolean condition, the variable on the left side of the equals sign... The variable on the right side of the equals sign represents the latest number of completed files, dynamically updated in real time by the system based on the aforementioned steps. This represents the total number of physical files statically stored in the database under the current processing directory during the initial indexing phase. The system performs high-frequency equality checks in memory. Once the Boolean expression outputs a true result, it logically indicates that all underlying single-file units in the currently extracted processing directory have been encrypted / decrypted and securely written to disk, signifying the complete termination of the processing task for the current directory. At this point, the system's main control program sends a trigger command to the database, executing a delete command in Structured Query Language to completely clear the current directory record data in the file table. Simultaneously, the system updates the Status field in the directory table corresponding to the current processing directory with a specific value representing the completed state. This series of automated cleanup and status change steps brings significant system load reduction benefits. Its advantage lies in maintaining the file table used to record high-frequency dynamic progress at a consistently small size. The system immediately destroys the high-frequency update records in the file table after processing each underlying directory. This not only completely frees up the valuable space in the database buffer and greatly reduces the data redundancy accumulated over a long period, but also fundamentally eliminates the problem of severe performance degradation in database queries and updates caused by the infinite expansion of data in a single table as encryption and decryption tasks progress.

[0038] After confirming that the current processing directory has been completely processed and the database state has been cleaned up, the system needs to seamlessly transition to the next unprocessed task node. To do this, the system extracts the primary key of the parent directory corresponding to the current processing directory. As the anchor point for the next stage of retrieval, a structured query language operation that performs a self-join is initiated upwards from the persistent directory table. Through this self-join query mechanism unique to relational databases, the system utilizes the extracted parent directory primary key. The system prioritizes searching the global directory tree for sibling directories with the same parent directory primary key and whose status characteristic field indicates an unprocessed state. Once a match is found in the underlying database, the system directly extracts this sibling directory as the new current processing directory and drives the overall encryption / decryption program smoothly into the next round of file sequence traversal and encryption / decryption conversion loop. This scheduling and addressing mechanism, which traces upwards based on the parent directory primary key and searches horizontally at the same level, significantly improves the system's workflow efficiency. Its beneficial effect is that the system does not need to re-call the time-consuming operating system low-level application interface to scan the massive physical disk tree structure; instead, it directly and instantly completes the smooth transfer of task processing objects within the already structured database relational tables through high-dimensional index comparison. This mechanism achieves rapid switching between massive fragmented directories with extremely low CPU computational overhead and minimal network query latency, powerfully ensuring the efficient and uninterrupted operation of large-scale, multi-level data encryption / decryption pipelines.

[0039] In the preferred embodiment, the index skipping operation step in S5 includes: Extract the variable representing the number of currently completed files stored in the file table. ,in The length is the total number of files completed before the interruption, generated by calling the operating system's file traversal API. The current directory file list array FileList; Define array traversal index and During traversal, mathematical logic judgments are performed: When the judgment expression When established, the traversal index is... Perform a span offset operation with a step size of 1, skipping the file loading instruction; When the judgment expression When it is established, the index of the file list array FileList is... The file continues to load and performs encryption / decryption data conversion.

[0040] In a specific implementation, the index skipping operation is the core control mechanism that enables the system to accurately and efficiently resume the encryption / decryption process after an interruption. When the system resumes operation from a paused state or restarts, the main control program first retrieves the currently completed file count variable from the persistent file table. ,in This represents the total number of files that have been completely encrypted / decrypted and securely copied to disk before the system is interrupted or paused. Accurate extraction of this variable establishes an absolute progress benchmark for system recovery from breakpoints. Subsequently, the system calls the underlying operating system's file traversal application interface to perform a full scan of the currently processed target directory, thereby generating a variable in computer memory with a length strictly equal to the total number of files. The current processing directory file list array is named FileList. This step maps the complex, discrete files on the physical disk into an ordered array structure in memory that can be accessed linearly at high speed, providing the necessary data carrier for subsequent rapid location.

[0041] To accurately locate and traverse the generated file list array, the system defines a dedicated array traversal index in memory. From a rigorous mathematical logic perspective, this traversal index must satisfy... The condition is that the index is set to an element in the set of positive integers, ensuring the validity of the underlying array addressing. Simultaneously, this index is subject to the inequality... Strict constraints. This mathematical inequality explicitly states that traversing the index... The initial value starts from 1, and its maximum traversal range will never exceed the total number of files in the current directory. By setting such extremely strict boundary control conditions, the system completely eliminates the risk of memory out-of-bounds access or illegal address reading from the underlying code logic, providing extremely high program stability and security for the continuous flow of large batches of files.

[0042] During the iterative traversal of the file list array, the system frequently executes mathematical logic judgments based on the aforementioned variables through the central processing unit. Specifically, when the mathematical logic judgment expression... When it is true, it means that the current array traversal index is true. The file it points to had already completed all cryptographic conversions and been successfully written to disk before the system interruption occurred. In this case, the system only performs index traversal at the pure memory level. The system performs a stride offset operation with a step size of 1, directly incrementing the index variable while completely skipping the disk load instructions, memory read instructions, and encryption / decryption algorithm call instructions for the file corresponding to that index at the underlying instruction scheduling level. This mechanism brings extremely significant benefits, replacing the time-consuming physical disk seeks and complex cryptographic floating-point operations with a memory integer variable increment operation that takes only nanoseconds. When faced with hundreds of thousands or even millions of processed files, the system can instantly traverse this historical data within milliseconds, achieving true zero-overhead breakpoint jumping and greatly saving the computer's central processing unit computing power and disk throughput bandwidth.

[0043] As the span offset operation continues, when the array is traversed to the index... Incrementing until the mathematical logic judgment is made When the critical point is reached, the system logic seamlessly switches. The validity of this conditional statement precisely indicates that the system's traversal pointer has crossed the boundary of all historically processed data and accurately landed on the first target file that was unprocessed at the time of the interruption. At this point, the system immediately resumes the full encryption / decryption pipeline, targeting the file list array FileList at index... The system sends a read command to the target file, loads its data stream into the memory buffer, and calls the underlying cryptographic algorithm engine to perform the actual encryption and decryption data conversion operation. This precise and seamless mechanism ensures that even after any form of external interruption, no unprocessed confidential file will be missed, and encrypted or decrypted files will never be rewritten in a destructive manner. Its substantial benefit lies in completely overcoming the problem of data loss and state misalignment that easily occurs when resuming operations on large batches of complex directory files, powerfully guaranteeing the absolute security, integrity, and extremely high continuity of core industry data such as geographic information.

[0044] In the preferred scheme, algebraic backoff calculation of abnormal offsets is performed by combining Boolean logic variables with database transaction mechanisms, including: Add a Boolean error-proofing feature field to the file table. Set initial value ; After each file data stream is completely flushed to disk, while initiating an SQL transaction commit to update the number of completed files variable, a state flip is performed on the error-proofing feature field. This utilizes the atomicity of a single SQL transaction to synchronize commit status and count; When the program resumes operation, it reads the characteristic bits of the last record in the database. Progress value at the time of interruption Verify the theoretically derived feature bits; If the verification result indicates that the last file conversion was not completely written to disk, then calculate the actual effective skip amount before performing the file list array traversal. Calculated Replace the original judgment expression Then execute logical judgments. This allows the traversal offset to be rolled back by one step, reloading the original file at the location of the abnormal breakpoint.

[0045] In specific implementations, the step of performing algebraic backoff calculations of abnormal offsets by combining Boolean logic variables with database transaction mechanisms is the core data error prevention and self-healing mechanism designed in this application for extreme failure scenarios. In actual large-scale file encryption and decryption processes, extreme abnormal interruptions such as sudden power outages or system kernel crashes may occur. To prevent corruption of dirty data files due to physical disk write latency and asynchronous database state updates, the system specifically adds a Boolean-type error prevention feature field to the file table. From a mathematical logic perspective, the data type of this field is strictly limited to... This means that the error-proofing feature bit can only take the integer value of zero or one in a relational database. When the current processing directory is first taken over and initialized, and no underlying file data stream transformation has yet occurred, the system sets the initial value of this error-proofing feature bit field to... This serves as the baseline verification starting point for the entire error-proofing state machine.

[0046] During the continuous operation of the encryption / decryption pipeline, after each encrypted or decrypted data stream of a single file is completely flushed to the physical disk, the system initiates a Structured Query Language (SCL) transaction commit operation to the underlying database to update the number of completed files in real time. Simultaneously, the system performs a state inversion calculation on the error-proofing feature field in memory. This state inversion calculation strictly follows algebraic equations. In this algebraic equation, the variables... This is a status node representing the number of files that the system has successfully processed. This represents the historical feature bit state before processing the latest monolithic file, while This represents the latest feature bit state calculated after processing the current single-entity file. The system utilizes the underlying atomicity of single-transaction Structured Query Language transactions to calculate the error-proofing feature bit state. The updated count of completed files is synchronized and packaged for submission to the database. This transaction-level binding mechanism ensures that the increment of progress and the flipping of status bits are absolutely bound and inseparable at the database physical level. If any step fails, the entire processing record of a single file will be automatically rolled back.

[0047] When the system experiences an unexpected interruption and triggers program recovery, the main control program performs strict breakpoint consistency checks to ensure data integrity. The system first reads the actual characteristic bits of the last record in the file table through the database query interface. Subsequently, the system retrieves the progress value that was frozen in the database at the moment of system interruption. The system then uses the parity of this progress value to calculate the theoretically existing state of the feature bits. Specifically, since the initial value is zero and each file processing triggers the feature bits to alternate between zero and one, there is a very strict mathematical mapping relationship between the parity of the breakpoint progress value and the theoretical state of the feature bits. The system then uses the feature bits actually recorded in the database. The data is rigorously compared and verified with the theoretical characteristic bits calculated based on the progress value to determine whether the data write state of the underlying disk at the moment of interruption is perfectly consistent with the database record.

[0048] If the above comparison and verification results indicate a discrepancy between the theoretical and actual feature bits—that is, explicitly indicating that the last file data conversion was not fully committed to disk but the database progress has been abnormally committed—or indicating other forms of file stream truncation or corruption, the system will automatically initiate the algebraic rollback repair procedure. Before traversing the file list array of the current processing directory, the system performs algebraic calculations based on the formula... The actual effective skip amount is calculated. In this calculation formula, the system subtracts an integer from the progress value recorded at the time of the interruption, and uses a maximum value function to ensure the calculated actual effective skip amount. The value will not be less than zero, thus strictly preventing negative numbers from exceeding the bounds during subsequent addressing of underlying arrays and causing system crashes. Next, the system uses the calculated actual effective skip amount... Replace the progress value in the original logical expression This leads to the execution of a new loop logic judgment. In its physical macroscopic form, this algebraic substitution operation forces the file sequence offset that should have been skipped back by one unit step. Therefore, the system no longer skips the breakpoint file that is on the verge of corruption; instead, it treats it as unprocessed dirty data, reloads the original file data stream at the abnormal breakpoint location into memory, and performs the complete cryptographic encryption / decryption conversion and disk overwrite / write-back operation again.

[0049] The aforementioned algebraic backoff mechanism for abnormal offsets delivers significant system-level benefits. This mechanism cleverly transforms the complex problem of physical disk power failure detection into a pure Boolean algebraic flip and parity check verification problem, greatly reducing resource overhead during abnormal recovery. By introducing lightweight state flip equations and a maximum backoff formula with lower bound protection, the system possesses extremely strong data self-correction and dirty data self-healing capabilities. In the face of any form of abnormal physical downtime, the system can not only safely maintain progress data but also automatically identify damaged data blocks caused by asynchronous input / output within milliseconds during recovery and restart, accurately implementing backoff of the traversal step size and overlapping overwriting of damaged files. This fundamentally eliminates the risk of file corruption, garbled text, or permanent data loss that is easily generated during the continuation of breakpoints in massive data processing, elevating the industrial-grade reliability, disaster recovery performance, and confidentiality integrity of nationally important data encryption and decryption methods to a new technological level.

[0050] In the preferred solution, a multi-threaded asynchronous concurrency model is used to improve the performance of processing large batches of files, including: The main thread initializes and maintains a memory-based thread-safe task queue to store subdirectory path tasks extracted from the directory table; A thread pool is constructed by pre-creating a fixed number of worker threads in the system process, and the worker threads continuously poll the thread-safe task queue. After a worker thread pops a task from the queue, it independently calls the operating system's file traversal API, performs memory read / write operations, and performs data encryption / decryption conversion. When the main thread catches an interrupt signal, it sets the global stop flag variable in memory to be active. After processing the current single file, the worker thread checks the global stop flag variable. If it is active, it commits the final database transaction and performs thread termination and recycling.

[0051] In this specific implementation, to overcome the computational and underlying disk read / write bottlenecks of single-threaded processing when dealing with massive amounts of entity files, this application introduces a multi-threaded asynchronous concurrency model to significantly improve the overall processing performance of large batches of files. During system startup and pre-processing, the main thread first initializes and maintains a thread-safe task queue in system memory. This task queue is specifically used to store subdirectory path tasks extracted by the main thread from the underlying database directory table. By residing the task queue in high-speed memory and endowing it with strict thread-safe mutual exclusion characteristics, data races and memory pointer corruption issues caused by multiple worker threads simultaneously competing for tasks can be effectively prevented. This provides a highly flexible and high-throughput data buffer center for task distribution by the main thread and task acquisition by worker threads.

[0052] After the task queue is initialized, the system process pre-creates a fixed number of worker threads to build a memory-resident thread pool. This pre-setting and creation of a fixed number of threads is crucial. It avoids the extremely high system scheduling overhead caused by frequently creating and destroying threads when dealing with massive amounts of fragmented files. It also strictly controls the system's concurrency within the safe threshold of the underlying CPU and disk I / O hardware, effectively preventing system memory exhaustion or operating system kernel crashes due to unlimited concurrency. In the completed thread pool, all worker threads continuously poll the thread-safe task queue asynchronously and non-blockingly, ready to take over newly assigned encryption / decryption tasks at any time.

[0053] When a worker thread detects an unprocessed directory task in the task queue through polling, it proactively removes the specific subdirectory path task from the queue. Subsequently, each worker thread, in its independent execution context, concurrently and independently calls the operating system's file traversal application interface to generate the physical file sequence under the corresponding subdirectory. Next, each worker thread independently performs read / write operations to load the file data stream from the physical disk into memory, and directly calls the underlying cryptographic engine in memory to perform encryption and decryption data conversion. This multi-threaded concurrent independent processing mechanism brings extremely significant benefits. It completely parallelizes the originally single-line serial file read / write operations and the extremely computationally intensive cryptographic data conversion operations, maximizing the parallel computing potential of modern multi-core CPUs and the multi-channel concurrent read / write bandwidth of underlying storage devices. This significantly reduces the overall time for terabyte-level large-volume data transfer, greatly improving the industrial-grade throughput efficiency of data processing.

[0054] During the high-speed operation of the concurrent processing pipeline, the system employs a highly secure and smooth exit response mechanism to handle interrupt signals triggered manually or by system anomalies. When the main thread detects an externally issued interrupt signal, the system will not resort to abrupt process termination. Instead, it will directly activate the global stop flag variable residing in shared memory. After completing the encryption / decryption and disk write loop of the currently processed single file, each high-speed worker thread will proactively check the value of this global stop flag variable before the start of the next cycle. Once a worker thread detects that the variable is active, it will immediately stop requesting new tasks from the task queue and refuse to read any new single files. At this point, the worker thread will submit the final database transaction for the file it just processed to the relational database to ensure the absolute accuracy of the breakpoint progress record. Subsequently, it will safely and orderly perform thread self-termination and resource reclamation operations. This asynchronous collaborative interruption mechanism based on state variables has the advantage of ensuring that the computer system can quickly and gently respond to user pause requests to release hardware resources, while also absolutely locking the integrity of the data stream of the currently processed single file at the underlying logic level. This mechanism completely eliminates the serious risks of encrypted file corruption, data garbling, or damage to underlying disk file handles caused by forcibly terminating threads, providing the most solid underlying guarantee for the subsequent safe and accurate breakpoint continuation of the entire large-scale data processing system.

[0055] In the preferred embodiment, when performing the database transaction synchronization update step, database connection pooling technology is used to manage the underlying network connections, including: During the program startup phase, a connection pool set containing a preset number of pre-connected database instance objects is initialized in the system heap memory; Each time a worker thread completes the instruction to update the number of files, it directly requests an idle database instance object from the connection pool collection. Use the obtained database instance object to execute SQL update statements and transaction commit operations; After the operation is completed, the context variables of the database instance object are cleaned up and its status is marked as idle, and it is returned to the connection pool set to eliminate the system resources consumed by the frequent establishment and disconnection of database connections during high-frequency file encryption and decryption.

[0056] In a specific implementation, the step of performing database transaction synchronization updates is a crucial step in ensuring that the processing progress of massive single-file applications is efficiently and stably solidified onto the physical medium. Since the encryption and decryption process of large volumes of data involves the state changes of hundreds of thousands or even millions of fragmented entity files, if the traditional method of temporarily establishing database connections is used for the progress update of each single file, it will cause extremely serious network input / output bottlenecks and system overhead. Therefore, this application innovatively introduces database connection pool technology to centrally manage underlying network connections. During the startup and warm-up phase of the entire application, the main control program centrally initializes a connection pool set containing a preset number of pre-connected database instance objects in the heap memory address space of the computer system. This preset number is set to a constant. ,in This represents the maximum number of underlying network transport control protocol connections that the system can concurrently accommodate. During this warm-up phase, the system pre-completes the network handshake, authentication, and persistent session channel establishment operations between all database instance objects and the underlying relational database. Residing these pre-connected database instance objects in the system heap memory ensures that these valuable network connection resources are not accidentally reclaimed and destroyed by the operating system's garbage collection mechanism throughout their subsequent high-frequency call lifecycle, thus building a highly resilient long-connection resource pool.

[0057] With the high-speed operation of the multi-threaded asynchronous concurrency model, when any worker thread completes the encryption / decryption loop of a single file and is ready to execute the instruction to update the file count variable, it will never initiate any new network socket creation request to the operating system kernel. Instead, the worker thread will directly request a connection from the connection pool in the system heap memory, retrieving a database instance object that is currently idle. After successfully obtaining the database instance object, the worker thread will use this readily available underlying network channel to directly send a Structured Query Language update statement to the database engine to update the progress count in the file table and immediately trigger the subsequent transaction commit operation. This process completely eliminates the network connection establishment steps that originally required a long time, transforming them into microsecond-level memory object reference and retrieval operations, ensuring that the progress increment instruction at the single file level can be seamlessly transmitted to the underlying database with extremely low latency.

[0058] After a single Structured Query Language (SCL) update and transaction commit operation is completely completed, the worker thread does not disconnect or destroy the underlying network connection. Instead, it immediately executes a strict resource reclamation and state reset process. The worker thread thoroughly cleans up the context variables within the database instance object, clearing any residual cursor state, parameter binding traces, and memory cache results from the previous SCL execution. After ensuring that the database instance object is restored to its absolutely clean initial session state, the worker thread changes its internal state flag from occupied to idle and safely returns it to the connection pool, allowing other queued worker threads to request and reuse the network channel at any time.

[0059] The complete steps described above for managing underlying network connections using database connection pooling technology yield significant system-level benefits. This mechanism, through a space-for-time system architecture optimization approach, moves the overhead of establishing and destroying database connections—originally scattered across hundreds of thousands of file encryption / decryption loops—to the front end and centralizes it during the program startup phase. This approach completely eliminates the unnecessary consumption of CPU computing power and system network port resources by frequent underlying network communication handshakes, security protocol verifications, and connection destruction during high-frequency file encryption / decryption processes. More importantly, the limited and reusable connection pool mechanism provides a robust traffic protection barrier for the underlying relational database. This mechanism effectively isolates and mitigates instantaneous high-concurrency write surges generated by concurrent worker threads, preventing database memory overflows or denial-of-service paralysis caused by massive numbers of threads simultaneously initiating connection requests to the database. From the underlying communication architecture perspective, it greatly improves the continuous throughput limit and industrial-grade stability of the system for resuming interrupted large-scale data processing under extremely high loads.

[0060] In the preferred scheme, during the encryption / decryption data conversion, the operating system's memory-mapped file API is called to optimize underlying read / write efficiency, including: The operating system's MMAP memory mapping interface is invoked to directly map the disk file to be processed into the virtual memory address space of the current process, and the starting address of the mapping is obtained. and the total file size (FileSize); Define a constant for the data block size, BlockSize, and use an iterative pointer address formula. Moving the underlying pointer in virtual memory, where The current data block number and ; The file data stream is read in fixed byte block slices and passed to the encryption / decryption algorithm function to perform in-situ cryptographic operations and transformations. The transformed byte blocks are then asynchronously synchronized to the disk using the operating system's underlying memory page dirty data write-back mechanism.

[0061] In a specific implementation, the step of calling the operating system's memory-mapped file API to optimize underlying read / write efficiency during the data encryption / decryption conversion is the core underlying architecture optimization mechanism of this application to address the performance bottleneck of reading and writing terabyte-level massive data. The system first calls the operating system's underlying MMAP memory mapping interface to directly map the physical disk file to be processed to the virtual memory address space of the current process. Through this underlying mapping operation, the system successfully obtains the mapping start address. And the total length of the physical file, FileSize. Mapping start address. This represents the logical starting point for reading file data in the process's virtual memory space, while the total file size (FileSize) strictly defines the legal data addressing boundaries of the file within the memory-mapped area. This mechanism changes the traditional disk input / output model, allowing applications to access disk file data directly, just like accessing ordinary memory data structures. The direct benefit is that it completely eliminates the redundant overhead of copying massive amounts of data back and forth between the operating system kernel space buffer and the user process space buffer, greatly reducing the frequency of context switching between kernel and user modes for the central processing unit, significantly freeing up system bus bandwidth, and paving the way for subsequent high-speed encryption and decryption operations.

[0062] After successfully establishing memory mapping, to prevent physical memory overflow caused by loading large files all at once, the system predefines a data block size constant `BlockSize` in its code logic. This constant represents the fixed byte block length that the system sends to the encryption / decryption algorithm engine for cryptographic operations each time. Subsequently, the system uses an iterative pointer address formula... Precisely move the underlying data pointer within the virtual memory address space. In this mathematical iterative formula, the variable... Represents the sequence number of the data block currently being processed, and satisfies... The mathematical condition is that the data block number is strictly limited to a positive integer that increments from one. According to this formula, when the data block number... When the value is equal to one, the current pointer address is exactly equal to the mapping start address. ; with the data block sequence number As the data block size continuously increases, the underlying pointer performs smooth step-size offset addressing in virtual memory with a fixed span of the data block size, BlockSize. This pointer addressing mechanism based on algebraic iteration formulas has significant beneficial effects. Through extremely lightweight memory pointer arithmetic operations, this mechanism achieves equal-length slicing and precise address positioning for extremely large files, ensuring that no matter how large the file size is, the actual working memory occupied by the system is always maintained at a very low level matching the data block size constant, thus providing extremely high memory usage safety and concurrency stability for large-scale data processing.

[0063] After accurately locating the data using low-level pointer formulas, the system reads the file data stream in fixed byte block slices and directly passes the slice data block pointers to the underlying encryption / decryption algorithm functions to perform in-place cryptographic transformations. In-place transformation means that the encrypted or decrypted output byte stream is directly overwritten to the original virtual memory address, without needing to allocate a new memory buffer in the heap to temporarily store the transformation results. After completing a single byte block memory overwrite, the system fully utilizes the operating system's underlying dirty data write-back mechanism. Virtual memory pages modified in-place are automatically marked as dirty data pages by the operating system kernel's memory management unit. Subsequently, the operating system's underlying page scheduling daemon automatically and transparently writes these transformed byte blocks back to the underlying physical disk in an asynchronous, non-blocking manner. This combination of in-place transformation and asynchronous write-back mechanism brings a significant performance leap. Its substantial benefit lies in completely decoupling the CPU-intensive cryptographic floating-point operation logic from the slow physical disk write latency. Once the worker thread completes the conversion of the current memory data block, it can immediately perform pointer calculations and encryption operations for the next data block, without needing to hibernate and wait for the underlying disk hardware to respond to the write operation. This mechanism pushes the sequential read / write performance of modern solid-state drives or hard disk drives to their physical limits, significantly shortening the overall encryption / decryption conversion cycle for large amounts of confidential file data.

[0064] In the preferred solution, the operating system's multi-process concurrency model and the database's exclusive locking mechanism are used to accelerate the resumption of breakpoints in the underlying directory, including: The main process analyzes the parent-child node mapping relationships in the directory tables of the database and filters out all nodes that satisfy the dependency equation. A group of leaf directories at the bottom level with no subdirectories; The selected leaf directory groups are packaged into multiple independent process parameter packages, and the operating system's process derivation API is called to create multiple independently running child processes. The process parameter packages are then distributed through inter-process communication. Each child process concurrently calls the file traversal API and performs encryption / decryption data conversion in its own independent memory space; When executing the SQL statement that updates the number of files, the child process appends a row-level exclusive lock instruction to the SQL command to ensure that records in the same directory are calculated and updated serially during concurrent writes to the database, preventing data races from causing breakpoint count anomalies.

[0065] In specific implementations, leveraging the operating system's multi-process concurrency model and the database's exclusive lock mechanism to accelerate the process of resuming interrupted tasks in the underlying directory is the core underlying scheduling architecture designed in this application to overcome the performance bottleneck of a single computing process and achieve rapid flow of massive amounts of data. The system's main process first assumes the role of hub for global task allocation. Through in-depth analysis of the parent-child node mapping relationships pre-built in the relational database's directory tables, the system can accurately grasp the topology of the entire massive file system. Based on this topological relationship, the main process executes strict mathematical filtering logic, specifically selecting all tasks that satisfy the dependency equation. A group of underlying leaf directories without subdirectories. In this dependency equation, the variables... This represents the total number of direct child directory nodes attached to a specific parent directory node in the directory tree structure. The value of this variable is zero only if the current directory structure is at the lowest physical level, containing only the entity data file to be processed and no nested subfolders. Through this precise algebraic equation filtering, the system logically completely removes complex hierarchical dependencies, extracting the smallest basic task unit that can be processed in absolute parallelism.

[0066] After successfully extracting the underlying leaf directory groups, the main process scientifically packages these non-overlapping leaf directory groups into multiple completely independent process parameter packages based on the number of CPU cores and memory capacity of the underlying hardware. Subsequently, the main process directly calls the operating system kernel-level process-derived application programming interface (SDIA) to simultaneously create multiple independently running child processes within the computer system. The main process uses efficient inter-process communication channels to precisely distribute the pre-packaged process parameter packages to the corresponding child processes. Upon receiving their dedicated task instructions, each child process concurrently calls the file traversal application programming interface within an absolutely independent and physically isolated memory address space allocated by the operating system, and performs high-speed, computationally intensive cryptographic encryption and decryption data conversion operations.

[0067] In a high-speed pipeline where multiple processes concurrently perform data transformations, a highly targeted concurrency control mechanism is introduced to ensure the absolute accuracy of breakpoint continuation recording. When any child process completes the data transformation of a single file and prepares to execute a structured query language statement to update the number of completed files, the child process forcibly appends a row-level exclusive lock instruction to the end of the standard data update instruction. The mechanism of the row-level exclusive lock instruction is that when a child process is incrementing and overwriting the number of completed files for a specific directory record in a database table, the database engine completely locks that data row at the underlying level. If other child processes happen to attempt to update the same directory record due to overlapping task assignments, they must wait in a queue outside the database until the previous child process's transaction is safely committed and the lock resource is released before they can continue execution. This micro-level queuing mechanism ensures that, in the complex environment of extremely high-concurrency database writing at the macro level, the cumulative calculation of progress for the same directory record is forcibly transformed into an absolutely safe serial update.

[0068] The combined scheduling steps of the aforementioned multi-process concurrency model and database exclusive lock bring about extremely significant system-level benefits. First, the bottom-up leaf node extraction strategy based on dependency equations, combined with the operating system's multi-process derivation mechanism, completely breaks through the memory addressing limits and computational resource ceilings of a single process when facing massive amounts of fragmented files. This mechanism perfectly and evenly distributes the massive load of cryptographic floating-point operations across all available hardware processor cores, enabling large-scale data encryption / decryption pipelines to possess truly industrial-grade high-throughput concurrent processing capabilities. Second, each child process runs in an independent memory space, naturally possessing extremely strong fault isolation characteristics. When a single child process crashes due to a specific corrupted file, it will absolutely not affect the main process or other running child processes, greatly improving the system's disaster recovery and resilience in harsh data environments. Finally, the cleverly introduced database row-level exclusive lock instruction perfectly resolves the concurrent write conflicts that inevitably arise when multiple processes concurrently share the same database table with minimal performance loss. This mechanism fundamentally prevents data races and breakpoint count anomalies caused by multiple processes simultaneously reading and overwriting progress variables. It ensures that no matter how intense the underlying concurrency competition is, the breakpoint resume progress recorded by the system always maintains absolute mathematical and logical accuracy, laying an impeccable foundation for data integrity for any subsequent pause and resume operations.

[0069] In the preferred embodiment, the steps in S4 to capture the interrupt signal and safely stop the operation include: when the event listener built into the main thread captures the asynchronous interrupt signal issued by the operating system, it checks whether there is a single file data stream in the memory stream buffer that has not yet been flushed to the physical disk; If a single file data stream that has not yet been flushed to the physical disk is detected, the asynchronous interrupt signal is added to the blocking queue to suspend it, forcibly delaying the interrupt response until the current single file completes the last disk I / O write-back operation and safely releases the underlying file handle, thereby locking the boundary of single file processing completion. After the underlying file handle is safely released, the update instruction of the structured query language is executed to switch the Status field corresponding to the current processing directory in the directory table to an enumerated constant representing the paused state. The final database transaction commit operation is triggered, and the latest number of completed files cached in memory is synchronized and persisted to the file table. Then, the underlying database network connection is actively disconnected and the worker thread pool in memory is destroyed, thereby maintaining the number of completed files in the file table to remain absolutely unchanged during program shutdown.

[0070] In a specific implementation, step S4 employs a robust asynchronous signal interception and state protection mechanism to ensure absolute security and consistency of the large-scale data encryption / decryption task during pauses. When the event listener built into the main thread captures an asynchronous interrupt signal from the operating system, the system does not immediately terminate the process. Instead, it first enters a sensitive data state detection phase, scanning the memory stream buffer in real time to check for any single-file data streams that have not yet been flushed to the physical disk. The benefit of this step is that it provides the system with a "window" to perceive the underlying I / O state, preventing damage to the single-file structure that could result from a forced power outage or process termination during data writing, thus laying a physical integrity foundation for subsequent breakpoint recovery.

[0071] Based on the detection results, if the system determines that there is still residual data in the memory stream buffer that has not yet been flushed to disk, it will activate a signal blocking strategy, placing the asynchronous interrupt signal into a specially designed blocking queue for suspension. At this time, the system will forcibly delay its response to the interrupt command until the currently processed single-file completes its final disk I / O write-back operation, and the operating system kernel safely releases the underlying file handle of that file. The beneficial effects of this delayed response mechanism are obvious: it establishes a buffer zone between the chaotic asynchronous interrupt requests and the orderly synchronous file write-back, ensuring that every file marked as "processed" is in a closed-loop and usable state on the physical disk by locking the logical boundary of single-file processing completion, fundamentally preventing the generation of fragmented bad files during encryption or decryption.

[0072] After the underlying file handles are safely released and the physical data stream is completely solidified, the system enters the final locking phase of the database state. The main control program issues update commands in Structured Query Language (SMQ) to switch the Status field corresponding to the currently processed directory in the directory table from an in-process state to an enumerated constant representing a paused state. Subsequently, the system triggers the final database transaction commit operation, synchronously solidifying the latest completed file count variable, which resides in the memory cache, to the persistent file table. After completing this series of actions, the system proactively disconnects the network connection to the underlying database and executes the destruction and resource reclamation commands for the worker thread pool in memory, thereby ensuring that the completed file count variable in the file table remains absolutely constant during program shutdown.

[0073] Example 2 Further explanation in conjunction with Example 1, such as Figure 1 The structure shown is described in one specific embodiment of this application. The method is deployed on a computer system equipped with a multi-core central processing unit and a high-speed disk array to process massive, multi-level results data at the terabyte level in the geographic information industry.

[0074] During the initialization phase of task startup, the system executes step S1 to build a global database index. The main process first calls the operating system's underlying file search application interface to recursively scan the specified task root directory. This process generates a complete directory tree structure. Here, Root represents the user-specified starting node for the data to be processed. Subsequently, the system executes Data Definition Language (DDL) instructions in the relational database to create task tables, directory tables, and file tables. The system parses the directory tree structure in memory, extracting the full path of each subdirectory and the numerical value representing the directory hierarchy depth. The primary key pointing to its parent directory record and the total number of files contained in this directory. In mathematical logical constraints, This indicates that the total number of files must be a positive integer, thus filtering out empty directories containing no files. These parameters are stored in the directory table using Structured Query Language instructions, forming a table containing... An ordered set of directory records Meanwhile, the system configures a status feature field, Status, for each directory record, with an initial value of 0, representing a state that has not started.

[0075] Subsequently, the system proceeds to step S2 to extract the first processing object. The task scheduler sends a query command with an aggregation function to the database to retrieve the maximum level depth value from the set where the status feature bit field is 0. If multiple directories with the same depth exist, the system will determine the appropriate directory based on inequality logic. The record with the smallest primary key identifier is extracted as the current processing directory. This bottom-up extraction strategy ensures that the leaf directories at the bottom level are processed first. The system uses the full path of this directory and the total number of files as variables. Synchronously write to the file table and initialize the completed file count variable to its initial state. At this point, the Status field in the directory table is updated to 1, indicating that it has entered the in-process state.

[0076] In the core execution phase S3, the system initiates a multi-threaded asynchronous concurrency model. The main thread maintains an in-memory thread-safe task queue, and multiple worker threads in the thread pool continuously poll this queue and pop tasks from the list. Worker threads obtain idle database instance objects through a database connection pool, the size of which is a constant. To limit the overhead of frequent network handshakes, worker threads call the operating system's memory mapping interface MMAP for each individual file. For large files, the system uses an iterative pointer address formula. Move the underlying pointer in virtual memory. (In the formula...) Represents the starting address of the mapping. `BlockSize` represents the sequence number of the currently processed data block, and `BlockSize` represents the preset fixed data block size. Worker threads read the data stream block by block and call encryption / decryption algorithm functions to perform in-situ cryptographic transformations. The transformed dirty pages are asynchronously synchronized to disk by the operating system. After completing the processing loop for each individual file, the system executes a database transaction, updating the number of completed files to `[value]`. .in This represents the number of files that have been completed. To prevent errors, the system synchronously updates the boolean error-prevention feature field. The execution state reversal is calculated using the following formula: .

[0077] When a user triggers a pause command or the system captures an interrupt signal, step S4 is executed. The main thread's event listener intercepts the asynchronous interrupt signal and places it in the blocking queue. The system checks the memory stream buffer; if there is data that has not yet been written to disk, it forcibly delays the interrupt response until the current single file releases its underlying file handle and completes the last disk I / O write-back operation, thus locking the processing boundary. Subsequently, the system executes a Structured Query Language update command, switching the Status field to the enumeration constant 2, i.e., the paused state. At this time, the progress value in the file table... Securely fixed, among which This represents the total number of files that were completed before the interruption.

[0078] When the system resumes operation and executes step S5, the program first reads the progress value from the database. With error prevention feature The system executes a verification algorithm to verify the actual records. The result is compared with the theoretical characteristic bits calculated based on the parity of the progress values. If it is found that the last conversion was not completely committed to disk, an abnormal rollback calculation is initiated, and the calculation formula is as follows: The system utilizes the calculated... Replace the existing progress value. Then, the system calls the file traversal application interface to generate a file of length [length missing]. Create an array of files named FileList and define the traversal index. During the loop, when the judgment expression... When established, the system only checks the index. Perform a span offset operation with a step size of 1, without triggering any disk load. When the index increments to... At that point, the system accurately located the breakpoint, reloaded the original file, and continued the encryption / decryption conversion.

[0079] To further optimize hardware performance, the system analyzes dependency equations in the main process. The process filters out leaf directories without subdirectories. The main process calls the process derivation interface to create multiple independent child processes. Each child process concurrently processes different parameter packets in physically isolated memory spaces. During progress updates, the child processes append row-level exclusive locks to the instructions to ensure that the completed file count variable for the same directory is updated serially and accurately during concurrent database writes by multiple processes, effectively preventing counting anomalies caused by data races. This complete technical process ensures extremely high robustness and processing efficiency for massive data encryption and decryption in complex business scenarios.

[0080] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A method for resuming processing after pausing the encryption / decryption process of large amounts of data. Its characteristics include: steps: S1. Construct a database index based on the target task. The database index includes a directory table for recording the hierarchical directory structure and a file table for recording the execution progress. S2. Extract the current processing directory from the directory table, synchronously write the path of the current processing directory into the file table, and initialize the number of completed files variable in the file table; S3. Call the operating system's file traversal API to perform file reading and encryption / decryption data conversion in the current processing directory, and write back to disk after the conversion of a single file is completed. Synchronously update the number of completed files in the file table by executing a database transaction. S4. When an interrupt signal is captured, the operation is safely stopped at the boundary where the current single file processing is completed, and the status record in the database index is persisted, keeping the number of completed files in the file table unchanged. S5. When the program resumes running, it extracts the current processing directory path and the number of completed files stored in the file table, calls the operating system's file traversal API to generate a file traversal sequence for the current processing directory, uses the number of completed files variable to perform an index skip operation in the file traversal sequence, and continues to load and perform encryption / decryption data conversion on the remaining files after skipping.

2. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 1, characterized in that: The steps for building a database index in S1 include: The system recursively calls the operating system's underlying file search API to obtain the tree-like hierarchical data of the target task's root directory and all its subdirectories; Create directory tables and file tables by executing Data Definition Language (DDL) commands in a relational database; Parse the tree-structured data and extract the full path and directory depth values ​​for each directory. The primary key of the parent directory in the directory table and the total number of files in this directory. ( The extracted parameters are then stored in the directory table using SQL insert statements. Set a Status field in the directory table to identify the four enumerated states: not started, in progress, paused, and completed.

3. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 2, characterized in that: The steps for retrieving the current processing directory in S2 include: Use SQL query commands to retrieve the set of unprocessed directories from the directory table, and extract the maximum level depth value from the set. ; From the value with the maximum layer depth In the candidate subdirectories, extract the directory record with the smallest primary key ID as the current processing directory to achieve bottom-up directory hierarchy traversal; The path to the current processing directory and the total number of files are variables. Synchronously write to the file table, and initialize the completed file count variable in the file table to its initial state. ,in Indicates completion of the first The progress value after processing each file, among which and And update the Status field of the corresponding directory in the directory table to "in progress".

4. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 3, characterized in that: The steps for synchronizing and updating file tables in S3 include: After completing the read, encryption / decryption algorithm conversion, and disk write loop for each individual file in the memory buffer, an explicit transaction commit operation is initiated to the database, updating the number of completed files variable to: ； Boolean condition calculations are performed in real time in system memory. When the result of this condition is true, an SQL delete command is triggered to clear the current directory record in the file table and change the Status field corresponding to the current processing directory in the directory table to the completed status. Using the parent directory primary key Perform a self-join SQL query upwards in the directory table, prioritizing the search for unprocessed sibling directories with the same parent directory primary key as the new current processing directory to enter the next round of file traversal loop.

5. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 3, characterized in that: The steps for performing index skip operations in S5 include: Extract the variable representing the number of currently completed files stored in the file table. ,in The length is the total number of files completed before the interruption, generated by calling the operating system's file traversal API. The current directory file list array FileList; Define array traversal index and During traversal, mathematical logic judgments are performed: When the judgment expression When established, the traversal index is... Perform a span offset operation with a step size of 1, skipping the file loading instruction; When the judgment expression When it is established, the index of the file list array FileList is... The file continues to load and performs encryption / decryption data conversion.

6. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 5, characterized in that: Combining Boolean logic variables with database transaction mechanisms to perform algebraic backoff calculations for abnormal offsets, including: Add a Boolean error-proofing feature field to the file table. Set initial value ; After each file data stream is completely flushed to disk, while initiating an SQL transaction commit to update the number of completed files variable, a state flip is performed on the error-proofing feature field. This utilizes the atomicity of a single SQL transaction to synchronize commit status and count; When the program resumes operation, it reads the characteristic bits of the last record in the database. Progress value at the time of interruption Verify the theoretically derived feature bits; If the verification result indicates that the last file conversion was not completely written to disk, then calculate the actual effective skip amount before performing the file list array traversal. Calculated Replace the original judgment expression Then execute logical judgments. This allows the traversal offset to be rolled back by one step, reloading the original file at the location of the abnormal breakpoint.

7. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 1, characterized in that: Improve the performance of large-volume file processing through a multi-threaded asynchronous concurrency model, including: The main thread initializes and maintains a memory-based thread-safe task queue to store subdirectory path tasks extracted from the directory table; A thread pool is constructed by pre-creating a fixed number of worker threads in the system process, and the worker threads continuously poll the thread-safe task queue. After a worker thread pops a task from the queue, it independently calls the operating system's file traversal API, performs memory read / write operations, and performs data encryption / decryption conversion. When the main thread catches an interrupt signal, it sets the global stop flag variable in memory to be active. After processing the current single file, the worker thread checks the global stop flag variable. If it is active, it commits the final database transaction and performs thread termination and recycling.

8. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 1, characterized in that: When performing database transaction synchronization updates, database connection pooling technology is used to manage the underlying network connections, including: During the program startup phase, a connection pool set containing a preset number of pre-connected database instance objects is initialized in the system heap memory; Each time a worker thread completes the instruction to update the number of files, it directly requests an idle database instance object from the connection pool collection. Use the obtained database instance object to execute SQL update statements and transaction commit operations; After the operation is completed, the context variables of the database instance object are cleaned up and its status is marked as idle, and it is returned to the connection pool set to eliminate the system resources consumed by the frequent establishment and disconnection of database connections during high-frequency file encryption and decryption.

9. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 1, characterized in that: During the encryption / decryption data conversion, the operating system's memory-mapped file API is called to optimize underlying read / write efficiency, including: The operating system's MMAP memory mapping interface is invoked to directly map the disk file to be processed into the virtual memory address space of the current process, and the starting address of the mapping is obtained. and the total file size (FileSize); Define a constant for the data block size, BlockSize, and use an iterative pointer address formula. Moving the underlying pointer in virtual memory, where The current data block number and ; The file data stream is read in fixed byte block slices and passed to the encryption / decryption algorithm function to perform in-situ cryptographic operations and transformations. The transformed byte blocks are then asynchronously synchronized to the disk using the operating system's underlying memory page dirty data write-back mechanism.

10. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 1, characterized in that: Utilizing the operating system's multi-process concurrency model and the database's exclusive locking mechanism to accelerate breakpoint continuation in the underlying directory includes: The main process analyzes the parent-child node mapping relationships in the directory tables of the database and filters out all nodes that satisfy the dependency equation. A group of leaf directories at the bottom level with no subdirectories; The selected leaf directory groups are packaged into multiple independent process parameter packages, and the operating system's process derivation API is called to create multiple independently running child processes. The process parameter packages are then distributed through inter-process communication. Each child process concurrently calls the file traversal API and performs encryption / decryption data conversion in its own independent memory space; When executing the SQL statement that updates the number of files, the child process appends a row-level exclusive lock instruction to the SQL command to ensure that records in the same directory are calculated and updated serially during concurrent writes to the database, preventing data races from causing breakpoint count anomalies.

11. The method for resuming processing after pausing the encryption / decryption process of large batches of data according to claim 1, characterized in that: The steps for capturing an interrupt signal and safely stopping the operation in S4 include: when the event listener built into the main thread captures an asynchronous interrupt signal issued by the operating system, it checks whether there is a single file data stream in the memory stream buffer that has not yet been flushed to the physical disk; If a single file data stream that has not yet been flushed to the physical disk is detected, the asynchronous interrupt signal is added to the blocking queue to suspend it, forcibly delaying the interrupt response until the current single file completes the last disk I / O write-back operation and safely releases the underlying file handle, thereby locking the boundary of single file processing completion. After the underlying file handle is safely released, the update instruction of the structured query language is executed to switch the Status field corresponding to the current processing directory in the directory table to an enumerated constant representing the paused state. The final database transaction commit operation is triggered, and the latest number of completed files cached in memory is synchronized and persisted to the file table. Then, the underlying database network connection is actively disconnected and the worker thread pool in memory is destroyed, thereby maintaining the number of completed files in the file table to remain absolutely unchanged during program shutdown.

Images