Multi-level concurrency control method and system based on file lock and state file

Abstract

The invention provides a multi-level concurrency control method and system based on a file lock and a state file. The method comprises the following steps: traversing a directory of a data file to be processed; creating a table-level lock file corresponding to the target source data file; the processing node obtains the exclusive processing right of the target source data file and stores the state file set in a directory where the target source data file is located; whether abnormity exists in the state file set or not is judged, if yes, it is judged that the target source data file does not need to be processed so as to release the corresponding table-level lock file, and local isolation is conducted on a source data text file corresponding to the table-level lock file; and all the processing nodes identify the processed files or the local isolation files according to the state files existing in the catalog where each target source data file is located. The method can prevent the single file problem from affecting the overall task execution, multiple nodes can work cooperatively through the shared file system, and the method is easy to integrate with an existing big data ecosystem.

Description

Technical Field

[0001] This invention belongs to the technical field of file data processing, and specifically relates to a multi-level concurrent control method and system based on file locks and status files. Background Technology

[0002] Traditional data synchronization solutions typically rely solely on database-level locking mechanisms, which can lead to problems such as high database pressure, difficulties in collaborative processing in distributed environments, and the inability to quickly recover after task interruption.

[0003] Among existing technologies, database locking mechanisms rely on row locks and table locks to achieve concurrency control, but this can put a lot of pressure on the database; distributed lock services use Redis, ZooKeeper, etc. to implement distributed locks, but require additional middleware support; message queue sequential consumption ensures the sequential execution of tasks through message queues, but cannot effectively handle task interruption recovery. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-level concurrent control method and system based on file locks and status files, thereby overcoming the shortcomings of the prior art.

[0005] In a first aspect, the present invention provides a multi-level concurrency control method based on file locks and status files, the method comprising: The data files to be processed are stored according to a preset multi-level directory structure, and the directories of the data files to be processed are traversed. When any processing node performs a parsing and import operation on a target source data file, a table-level lock file corresponding to the target source data file is created, wherein the storage area of ​​the table-level lock file corresponds to the multi-level directory structure of the target source data file; The table-level lock file is created based on the atomic operation of file creation. If the creation is successful, the processing node obtains exclusive processing rights to the target source data file and maintains the table-level lock by attaching a preset validity period to the table-level lock file, starting a timer, and sending a renewal request within the validity period. Based on the processing node, detect the multi-level state semantic set associated with the target source data file, and store the multi-level state semantic set in the directory where the target source data file is located; When the existence of the error status file is detected, the error type and error count recorded in the error status file are read; Determine whether there is an anomaly in the multi-level state semantic set. If so, determine that the target source data file does not need to be processed to release the corresponding table-level lock file, and perform local isolation on the source data text file corresponding to the table-level lock file. All processing nodes identify processed files or partially isolated files based on the multi-level state semantic set existing in the directory where each target source data file is located.

[0006] Compared with existing technologies, the beneficial effects of this invention are: through a multi-level status file mechanism, the status of each processing step is traceable; the error isolation mechanism prevents single file problems from affecting the overall task execution; file system locks have lower resource overhead than database locks; they support parallel processing of data from different tables, improving overall throughput; they naturally support distributed deployment; multiple nodes can work collaboratively through a shared file system; they are easy to integrate with existing big data ecosystems; the status files are visualized, facilitating monitoring and troubleshooting; and no additional distributed lock service is required, reducing system complexity.

[0007] Furthermore, the multi-level directory structure includes a first-level database identifier, a second-level database identifier, and a third-level date identifier.

[0008] Furthermore, after the step of performing the table-level lock file creation attempt based on the file creation atomic operation, the method further includes: If the creation of the table-level lock file fails, it is determined that there are other processing nodes processing the target source data file, and the current processing node is made to skip the target source data file being processed. Traverse other source data files in the same directory as the target source data file.

[0009] Furthermore, the multi-level state semantic set includes an end state file for identifying processing completion, an error state file for identifying processing failure, and a progress anchor file for recording the current processing position.

[0010] Furthermore, after the step of reading the error type and error count recorded in the error status file, the method further includes: If the number of errors is less than the preset retry threshold, the error status file is cleared, and the target source data file is parsed and imported again. If the number of errors is greater than or equal to the preset retry threshold, the circuit breaker isolation mechanism is activated, and the target source data file is determined to be isolated. The model expression for the dynamic circuit breaker isolation is: ; In the formula, This indicates the preset retry threshold. This indicates the current average system load. Indicates the system target load. , These represent the load adjustment factor and the error rate adjustment factor, respectively. Indicates an index. This indicates the percentage of erroneous files within the most recent preset time period.

[0011] Furthermore, after the step of determining whether there is an anomaly in the multi-level state semantic set, the method further includes: If no anomalies are found in the set of status files, then the target source data file is confirmed to be in a pending state. Analyze whether any abnormal information occurs during the parsing and importing process. If any abnormal information occurs, generate an error status file in the directory where the target source data file is located, store the abnormal information in the error status file using a structured format, and delete or release the table-level lock file.

[0012] Furthermore, before the step of all the processing nodes identifying processed files or partially isolated files based on the multi-level state semantic set existing in the directory where each of the target source data files is located, the method further includes: Each processing node is assigned a unique node identifier through an ownership and claiming mechanism, and the node identifier is written into the table-level lock file it creates; If a processing node fails and restarts, a self-healing takeover process is initiated. Based on the node identifier, it checks for any zombie table-level lock files created by the node itself but not properly released. If found, the zombie table-level lock files are forcibly released, and breakpoint resumption is performed according to the corresponding progress anchor file. The model expression for breakpoint resumption is: ; In the formula, This indicates the efficiency improvement of resuming interrupted downloads. This indicates the total number of files interrupted due to a fault. Represents a file The weighting coefficients, Represents a file Total workload to be processed Represents a file The progress anchor value that was completed at the time of the interruption. Represents a file The processing speed.

[0013] Secondly, the present invention also provides a multi-level concurrent control system based on file locks and status files, the system comprising: The storage traversal module is used to store the data files to be processed according to a preset multi-level directory structure and traverse the directory of the data files to be processed. The import creation module is used to create a table-level lock file corresponding to the target source data file when any processing node performs a parsing and import operation on a target source data file. The storage area of ​​the table-level lock file corresponds to the multi-level directory structure of the target source data file. A creation and acquisition module is used to perform an atomic operation based on file creation to attempt to create the table-level lock file. If the creation is successful, the processing node acquires exclusive processing rights to the target source data file and maintains the table-level lock by attaching a preset validity period to the table-level lock file, starting a timer, and sending a renewal request within the validity period. The detection and storage module is used to detect the multi-level state semantic set associated with the target source data file based on the processing node, and store the multi-level state semantic set in the directory where the target source data file is located; The reading module is used to read the error types and error counts recorded in the error status file when the existence of the error status file is detected; The judgment module is used to determine whether there is an anomaly in the multi-level state semantic set. If so, it determines that the target source data file does not need to be processed to release the corresponding table-level lock file, and performs local isolation on the source data text file corresponding to the table-level lock file. The identification module is used by all the processing nodes to identify processed files or locally isolated files based on the multi-level state semantic set existing in the directory where each target source data file is located.

[0014] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the above-described multi-level concurrent control method based on file locks and status files.

[0015] Fourthly, the present invention also provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described multi-level concurrent control method based on file locks and status files. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a multi-level concurrent control method based on file locks and status files in the first embodiment of the present invention; Figure 2 This is a structural block diagram of a multi-level concurrent control system based on file locks and status files in the second embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device in the third embodiment of the present invention.

[0018] Explanation of key component symbols: 10. Storage Traversal Module; 20. Import and Creation Module; 30. Creation and Acquisition Module; 40. Storage Detection Module; 50. Reading Module; 60. Judgment Module; 70. Recognition Module; 80. Bus; 81. Processor; 82. Memory; 83. Communication interface.

[0019] The embodiments of the present invention will be further described below with reference to the accompanying drawings. Detailed Implementation

[0020] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0021] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] Example 1 Please see Figure 1 The figure shows a multi-level concurrent control method based on file locks and status files in the first embodiment of the present invention, the method including steps S1 to S7: S1, Store the data files to be processed according to a preset multi-level directory structure, and traverse the directory of the data files to be processed; It should be noted that the multi-level directory structure includes a first-level database identifier, a second-level database identifier, and a third-level date identifier. This three-level directory structure forms a storage path of " / data / {database name} / {table name} / {date} / {file name}", where the file name is used to uniquely identify a source data file. The constructed multi-level directory structure is used to store the source data files to be processed hierarchically. The program traverses the directory where the data files are stored: / data / {database name} / {table name} / {date}.

[0024] S2, when any processing node performs a parsing and import operation on a target source data file, a table-level lock file corresponding to the target source data file is created, wherein the storage area of ​​the table-level lock file corresponds to the multi-level directory structure of the target source data file; It should be noted that when any processing node prepares to perform a parsing and import operation on a target source data file, it first attempts to create a table-level lock file corresponding to the target source data file in a preset lock file storage area. The lock file storage area has a multi-level directory structure corresponding to the source data file storage area, and the filename of the table-level lock file is generated by appending a preset lock file suffix (e.g., ".lock") to the filename of the target source data file. Its complete path is in the form of " / lock / {database name} / {table name} / {date} / {filename}.lock". By generating a table-level lock file, a lightweight table-level lock is implemented, thereby preventing multiple processes from processing the same data text file simultaneously.

[0025] S3, based on the atomic operation of file creation, the table-level lock file is created. If the creation is successful, the processing node obtains exclusive processing rights to the target source data file and maintains the table-level lock by attaching a preset validity period to the table-level lock file, starting a timer, and sending a renewal request within the validity period. It should be noted that table-level locks need to be determined by a decision function. In this embodiment, the expression of the valid decision function for a table-level lock is: ; In the formula, Indicates the processing node identifier. Indicates the target source data file. Indicates the target source data file The corresponding table-level lock file, Indicates the current time point, , , , , These represent the current system time, the creation time of the table-level lock file, the lock's validity period, the time interval for lock renewal requests, and the maximum allowed number of renewals, respectively. The table-level lock validity determination function not only checks for lock timeouts but also limits the number of renewals. When the number of renewals exceeds... Even if the absolute expiration date has not yet arrived, the lock will be deemed invalid to prevent indefinite renewal due to network jitter and the formation of a "permanent lock".

[0026] Specifically, step S3 further includes steps S31 to S32: S31, if creating the table-level lock file fails, it is determined that there are other processing nodes processing the target source data file, and the current processing node skips the target source data file being processed. S32, Traverse other source data files in the same directory as the target source data file; It should be noted that the creation of the table-level lock file is attempted based on the atomic file creation operation provided by the underlying file system. If the creation operation is successful, it means that the current processing node has successfully acquired exclusive processing rights to the target source data file. If the creation operation fails (for example, an exception is thrown because the lock file already exists), it is determined that there are other processing nodes processing the target source data file. The current processing node immediately skips the target source data file and continues to traverse other source data files in the same directory. It's worth noting that if the table-level lock file is generated successfully, it means that no process is currently processing the corresponding data text file. If generation fails, it means that another process is processing the data text file; the process will skip this data file and continue processing other text files.

[0027] S4, based on the processing node, detect the multi-level state semantic set associated with the target source data file, and store the multi-level state semantic set in the directory where the target source data file is located; It should be explained that the multi-level state semantic set includes an end state file to identify processing completion, an error state file to identify processing failure, and a progress anchor file to record the current processing position. The multi-level state semantics expresses three different processing semantics—"success," "failure," and "in progress"—through these three state files: the end state file, the error state file, and the intermediate state file (progress anchor). These three state files together constitute the complete file processing lifecycle state machine. The progress anchor file refers to the current progress identifier of the parsing and import operation recorded in the intermediate state file. It acts as an anchor point, marking the precise position of the processing flow (such as the number of lines processed, byte offset, or transaction sequence number), providing a precise basis for resuming interrupted downloads.

[0028] It should be noted that after successfully acquiring exclusive processing rights, the current processing node further checks the set of status files associated with the target source data file. The set of status files is stored in the directory where the source data file is located. It includes: an end status file (with the suffix ".end") used to indicate that the file processing has been completed normally, an error status file (with the suffix ".error") used to indicate that an error occurred during the file processing, and a lock status file (with the suffix ".lock") used to indicate that the file is being processed. The lock status file is the table-level lock file that was created.

[0029] S5, when the existence of the error status file is detected, read the error type and error count recorded in the error status file; Specifically, after step S5, steps S51 to S52 are included: S51, if the number of errors is less than the preset retry threshold, then clear the error status file and re-parse and import the target source data file; It should be noted that after the processing node completes the parsing and importing of the target source data file, if an exception occurs during the process, the system will generate an error status file (with the .error extension) in the directory where the file is located. This error status file records detailed information about the failure, including but not limited to: error type, number of errors, timestamp, and error stack. When the file is scanned again, its error status file will be read, the error count field will be extracted, and compared with a preset retry threshold (e.g., 3 times).

[0030] S52, if the number of errors is greater than or equal to the preset retry threshold, then the circuit breaker isolation mechanism is activated, and the target source data file is determined to be isolated; It should be noted that when the cumulative number of errors recorded in the error status file reaches or exceeds the preset retry threshold, the file is determined to be a "permanent error" or a "stubborn file," and further retries are meaningless. At this point, the system activates the circuit breaker isolation mechanism, permanently excluding the file from the processing queue. The circuit breaker isolation mechanism refers to the mechanism whereby when the number of error retries of the target source data file reaches the preset threshold, the system automatically marks it as isolated, stops processing it, and releases the relevant lock resources.

[0031] The model expression for the dynamic circuit breaker isolation is as follows: ; In the formula, This indicates the preset retry threshold. This indicates the current average system load. Indicates the system target load. , These represent the load adjustment factor and the error rate adjustment factor, respectively. Indicates an index. This represents the percentage of erroneous files within the most recent preset time period. Traditional fixed retry thresholds cannot adapt to dynamic system changes. When the system load is high, the number of retries should be reduced to avoid wasting resources; when the error rate increases, it indicates a possible systemic problem, and rapid circuit breaking should be implemented. The model expression for dynamic circuit breaking isolation achieves adaptive adjustment of the circuit breaking isolation threshold through exponential decay and load feedback.

[0032] S6, determine whether there is an anomaly in the multi-level state semantic set. If so, determine that the target source data file does not need to be processed to release the corresponding table-level lock file, and perform local isolation on the source data text file corresponding to the table-level lock file. Specifically, step S6 includes steps S61 to S62: S61, if there are no anomalies in the set of status files, then the target source data file is confirmed to be in a pending state; S62, analyze whether any abnormal information occurs during the parsing and importing process. If any abnormal information occurs, generate an error status file in the directory where the target source data file is located, store the abnormal information in the error status file using a structured format, and delete or release the table-level lock file. It should be noted that if an end-of-life file or an error-state file is detected, the target source data file is determined to be unnecessary to process (it has been terminated or isolated due to an error), and the acquired table-level lock file is immediately released (e.g., the lock file is directly deleted). If only a table-level lock file is detected and no end-of-life file or error-state file is detected, it is determined to be an abnormal state (e.g., lock residue caused by an unexpected interruption in the previous processing). In this case, you can choose to perform a wait operation (wait for the lock file to be released automatically) or skip the operation and record the log. If no status file is detected, the target source data file is confirmed to be in a pending state, and the parsing and import process for the file is then started. Furthermore, an error status file is written to store the exception information in a structured format. During the parsing and import process, if any type of exception occurs (including but not limited to network interruption, data format error, database connection failure, etc.), an error status file is immediately generated in the directory where the target source data file is located, and the exception information (including timestamp, error type, stack trace, etc.) is written to the file. At the same time, the table-level lock file is deleted or released to ensure that other nodes do not process the exception file repeatedly. It is worth noting that if the parsing and import process is successfully completed, the table-level lock file will be renamed to the end status file, that is, the ".lock" suffix will be changed to ".end" in the same directory to persistently identify that the target source data file has been processed. Furthermore, when multiple processing nodes compete for the same file lock simultaneously, if all nodes retry immediately, it will lead to severe lock contention. The exponential backoff mechanism randomly staggers the retry times of each node. In this embodiment, the expression for the exponential backoff mechanism is: ; In the formula, Indicates the maximum number of backoff attempts. Indicates the first The probability of acquiring the lock on the first attempt. , This represents the backoff attenuation coefficient. This represents the base probability of acquiring the lock on the first attempt. Indicates the basic backoff time interval. This indicates the time taken by the lock creation operation itself. Indicates the first The probability of successfully acquiring the lock on the first attempt. This indicates the expected time to successfully acquire the table-level lock.

[0033] S7, all the processing nodes identify processed files or partially isolated files based on the multi-level state semantics existing in the directory where each target source data file is located; It should be noted that when the distributed data synchronization system restarts due to a fault or resumes a periodic task, all processing nodes uniformly identify processed or isolated files based on the status files (i.e., end-of-life and error-state files) existing in the directories where the source data files are located. Processing is only restarted for source data files that have neither end-of-life nor error-state files, thus achieving a breakpoint resumption mechanism. Furthermore, target source data files are stored in time-sharded chunks, and expired source data files and their associated status files are automatically archived or cleaned up. It should be noted that source data files are stored daily, supporting a multi-level directory structure of database / table / date directory / data file, and supporting time-sharded data management, facilitating data archiving and retrieval.

[0034] Specifically, step S6 further includes steps S61 to S62: S61, assign a unique node identifier to each processing node through the ownership and claiming mechanism, and write the node identifier into the table-level lock file it creates; S62, if a processing node restarts after a failure, the self-healing takeover process is initiated. Based on the node identifier, it is detected whether there is a zombie table-level lock file created by itself but not properly released. If so, the zombie table-level lock file is forcibly released, and the interrupted transmission is resumed according to the corresponding progress anchor file. It's important to note that ownership assignment refers to writing a unique node identifier into the table-level lock file it creates, giving the lock file a clear "owner" attribute. When multiple processing nodes collaborate in a shared file system, the ownership assignment mechanism ensures that only the lock holder can modify, renew, or release the lock, preventing unauthorized preemption. The self-healing takeover process refers to the ability of a processing node to proactively detect and clean up its own residual zombie lock files based on its node identifier upon restarting after a failure, and to resume processing tasks according to the corresponding progress anchor point. This achieves rapid fault detection, accurate identification, and automatic repair in distributed data processing systems, offering significant advantages over traditional solutions that rely on passive timeouts or manual intervention for recovery.

[0035] The model expression for resuming interrupted downloads is as follows: ; In the formula, This indicates the efficiency improvement of resuming interrupted downloads. This indicates the total number of files interrupted due to a fault. Represents a file The weighting coefficients, Represents a file Total workload to be processed Represents a file The progress anchor value that was completed at the time of the interruption. Represents a file Processing speed; Furthermore, in order to know the recovery time after the faulty node restarts, a recovery time prediction model is used during the self-healing takeover process. The expression for the recovery time prediction model is: ; In the formula, Indicates the self-healing takeover recovery time. This represents the set of zombie table-level lock files belonging to the faulty node that are detected when the node restarts. Represents a file Total workload to be processed Represents a file The progress anchor value that was completed at the time of the interruption. This indicates the processing rate of the current node. Indicates the cleanup of zombie lock files The required time. The total time for a node to self-heal and take over is predicted, including two parts: the time for reprocessing unfinished work and the time for clearing legacy locks. This model allows operations personnel to estimate the impact of fault recovery on the overall system schedule.

[0036] In summary, the multi-level concurrent control method based on file locks and status files in the above embodiments of the present invention ensures the traceability of the status of each processing stage through a multi-level status file mechanism, prevents single file problems from affecting the overall task execution through an error isolation mechanism, has lower resource overhead than database locks, supports parallel processing of data from different tables, improves overall throughput, naturally supports distributed deployment, allows multiple nodes to work collaboratively through a shared file system, is easy to integrate with existing big data ecosystems, provides visualization of status files for easy monitoring and troubleshooting, and eliminates the need for additional distributed lock services, thus reducing system complexity.

[0037] Example 2 Please see Figure 2 The figure shows a multi-level concurrent control system based on file locks and status files in the second embodiment of the present invention. The system includes: The storage traversal module 10 is used to store the data files to be processed according to a preset multi-level directory structure and traverse the directory of the data files to be processed. The multi-level directory structure includes a first-level database identifier, a second-level database identifier, and a third-level date identifier. Import creation module 20 is used to create a table-level lock file corresponding to the target source data file when any processing node performs a parsing and import operation on a target source data file. The storage area of ​​the table-level lock file corresponds to the multi-level directory structure of the target source data file. The creation and acquisition module 30 is used to perform the creation attempt of the table-level lock file based on the atomic operation of file creation. If the creation is successful, the processing node acquires exclusive processing rights to the target source data file and maintains the table-level lock by attaching a preset validity period to the table-level lock file, starting a timer, and sending a renewal request within the validity period. The detection and storage module 40 is used to detect the multi-level state semantic set associated with the target source data file based on the processing node, and store the multi-level state semantic set in the directory where the target source data file is located. The multi-level state semantic set includes an end state file for identifying the completion of processing, an error state file for identifying the failure of processing, and a progress anchor file for recording the current processing position. The reading module 50 is used to read the error type and error count recorded in the error status file when the existence of the error status file is detected; The judgment module 60 is used to determine whether there is an anomaly in the multi-level state semantic set. If so, it determines that the target source data file does not need to be processed to release the corresponding table-level lock file, and performs local isolation on the source data text file corresponding to the table-level lock file. The identification module 70 is used by all the processing nodes to identify processed files or locally isolated files based on the multi-level state semantic set existing in the directory where each target source data file is located.

[0038] In some alternative embodiments, the creation and acquisition module 30 includes: If the creation of the table-level lock file fails, the creation unit determines that there are other processing nodes processing the target source data file and causes the current processing node to skip the target source data file being processed. The traversal unit is used to traverse other source data files in the same directory as the target source data file.

[0039] In some alternative embodiments, the reading module 50 includes: The clearing unit is used to determine if the number of errors is less than a preset retry threshold, then clear the error status file and re-parse and import the target source data file; An isolation unit is used to determine whether the target source data file is isolated if the number of errors is greater than or equal to the preset retry threshold. The model expression for the dynamic circuit breaker isolation is: ; In the formula, This indicates the preset retry threshold. This indicates the current average system load. Indicates the system target load. , These represent the load adjustment factor and the error rate adjustment factor, respectively. Indicates an index. This indicates the percentage of erroneous files within the most recent preset time period.

[0040] In some optional embodiments, the determination module 60 includes: The confirmation unit is used to determine whether the target source data file is in a pending state if there is no abnormality in the set of status files. The analysis and generation unit is used to analyze whether any abnormal information occurs during the parsing and importing process. If any abnormal information occurs, an error status file is generated in the directory where the target source data file is located. The abnormal information is stored and written into the error status file in a structured format, and the table-level lock file is deleted or released.

[0041] In some alternative embodiments, the identification module 70 includes: The allocation unit is used to assign a unique node identifier to each processing node through an ownership and claiming mechanism, and write the node identifier into the table-level lock file it creates; A creation unit is established to initiate a self-healing takeover process if a processing node fails and restarts. This unit checks for zombie table-level lock files created by the node itself but not properly released based on the node identifier. If such files exist, the zombie table-level lock files are forcibly released, and breakpoint resumption is performed based on the corresponding progress anchor file. The model expression for breakpoint resumption is: ; In the formula, This indicates the efficiency improvement of resuming interrupted downloads. This indicates the total number of files interrupted due to a fault. Represents a file The weighting coefficients, Represents a file Total workload to be processed Represents a file The progress anchor value that was completed at the time of the interruption. Represents a file The processing speed.

[0042] The functions or operation steps implemented by the above modules and units are largely the same as those in the above method embodiments, and will not be repeated here.

[0043] The multi-level concurrent control system based on file locks and status files provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0044] Example 3 The third embodiment of the present invention also proposes an electronic device, please refer to [link / reference]. Figure 3 The image shows an electronic device according to a third embodiment of the present invention.

[0045] The electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0046] Specifically, the processor 81 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement this application.

[0047] The memory 82 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include removable or non-removable (or fixed) media. Where appropriate, the memory 82 may be internal or external to a data processing device. In a particular embodiment, the memory 82 is non-volatile memory. In a particular embodiment, the memory 82 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0048] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0049] The processor 81 reads and executes the computer program instructions stored in the memory 82 to implement the multi-level concurrent control method based on file locks and status files in the first embodiment described above.

[0050] In some embodiments, the electronic device may further include a communication interface 83 and a bus 80. For example, Figure 3 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0051] The communication interface 83 is used to enable communication between the various modules, devices, units, and / or equipment in this application. The communication interface 83 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0052] Bus 80 includes hardware, software, or both, that couples the components of a device together. Bus 80 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although this application describes and illustrates a specific bus, this application considers any suitable bus or interconnection.

[0053] The electronic device can acquire a multi-level concurrent control system based on file locks and status files, and execute the multi-level concurrent control method based on file locks and status files in this embodiment.

[0054] Furthermore, in conjunction with the multi-level concurrent control method based on file locks and status files in Embodiment 1 above, this application can provide a storage medium for implementation. This storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement the multi-level concurrent control method based on file locks and status files in Embodiment 1 above.

[0055] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0056] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A multi-level concurrency control method based on file locks and status files, characterized in that, The method includes: The data files to be processed are stored according to a preset multi-level directory structure, and the directories of the data files to be processed are traversed. When any processing node performs a parsing and import operation on a target source data file, a table-level lock file corresponding to the target source data file is created, wherein the storage area of ​​the table-level lock file corresponds to the multi-level directory structure of the target source data file; The table-level lock file is created based on the atomic operation of file creation. If the creation is successful, the processing node obtains exclusive processing rights to the target source data file and maintains the table-level lock by attaching a preset validity period to the table-level lock file, starting a timer, and sending a renewal request within the validity period. Based on the processing node, detect the multi-level state semantic set associated with the target source data file, and store the multi-level state semantic set in the directory where the target source data file is located; When the existence of the error status file is detected, the error type and error count recorded in the error status file are read; Determine whether there is an anomaly in the multi-level state semantic set. If so, determine that the target source data file does not need to be processed to release the corresponding table-level lock file, and perform local isolation on the source data text file corresponding to the table-level lock file. All processing nodes identify processed files or partially isolated files based on the multi-level state semantic set existing in the directory where each target source data file is located.

2. The multi-level concurrent control method based on file locks and status files according to claim 1, characterized in that, The multi-level directory structure includes a first-level database identifier, a second-level database identifier, and a third-level date identifier.

3. The multi-level concurrent control method based on file locks and status files according to claim 1, characterized in that, After the step of performing the table-level lock file creation attempt based on the file creation atomic operation, the method further includes: If the creation of the table-level lock file fails, it is determined that there are other processing nodes processing the target source data file, and the current processing node is made to skip the target source data file being processed. Traverse other source data files in the same directory as the target source data file.

4. The multi-level concurrent control method based on file locks and status files according to claim 1, characterized in that, The multi-level state semantic set includes an end state file for identifying processing completion, an error state file for identifying processing failure, and a progress anchor file for recording the current processing position.

5. The multi-level concurrent control method based on file locks and status files according to claim 1, characterized in that, After the step of reading the error type and error count recorded in the error status file, the method further includes: If the number of errors is less than the preset retry threshold, the error status file is cleared, and the target source data file is parsed and imported again. If the number of errors is greater than or equal to the preset retry threshold, the circuit breaker isolation mechanism is activated, and the target source data file is determined to be isolated. The model expression for the dynamic circuit breaker isolation is: ； In the formula, This indicates the preset retry threshold. This indicates the current average system load. Indicates the system target load. , These represent the load adjustment factor and the error rate adjustment factor, respectively. Indicates an index. This indicates the percentage of erroneous files within the most recent preset time period.

6. The multi-level concurrent control method based on file locks and status files according to claim 1, characterized in that, After the step of determining whether there is an anomaly in the multi-level state semantic set, the method further includes: If no anomalies are found in the set of status files, then the target source data file is confirmed to be in a pending state. Analyze whether any abnormal information occurs during the parsing and importing process. If any abnormal information occurs, generate an error status file in the directory where the target source data file is located, store the abnormal information in the error status file using a structured format, and delete or release the table-level lock file.

7. The multi-level concurrent control method based on file locks and status files according to claim 1, characterized in that, Before the step of all the processing nodes identifying processed files or partially isolated files based on the multi-level state semantic set existing in the directory where each of the target source data files is located, the method further includes: Each processing node is assigned a unique node identifier through an ownership and claiming mechanism, and the node identifier is written into the table-level lock file it creates; If a processing node fails and restarts, a self-healing takeover process is initiated. Based on the node identifier, it checks for any zombie table-level lock files created by the node itself but not properly released. If found, the zombie table-level lock files are forcibly released, and breakpoint resumption is performed according to the corresponding progress anchor file. The model expression for breakpoint resumption is: ； In the formula, This indicates the efficiency improvement of resuming interrupted downloads. This indicates the total number of files interrupted due to a fault. Represents a file The weighting coefficients, Represents a file Total workload to be processed Represents a file The progress anchor value that was completed at the time of the interruption. Represents a file The processing speed.

8. A multi-level concurrent control system based on file locks and status files, characterized in that, The system includes: The storage traversal module is used to store the data files to be processed according to a preset multi-level directory structure and traverse the directory of the data files to be processed. The import creation module is used to create a table-level lock file corresponding to the target source data file when any processing node performs a parsing and import operation on a target source data file. The storage area of ​​the table-level lock file corresponds to the multi-level directory structure of the target source data file. A creation and acquisition module is used to perform an atomic operation based on file creation to attempt to create the table-level lock file. If the creation is successful, the processing node acquires exclusive processing rights to the target source data file and maintains the table-level lock by attaching a preset validity period to the table-level lock file, starting a timer, and sending a renewal request within the validity period. The detection and storage module is used to detect the multi-level state semantic set associated with the target source data file based on the processing node, and store the multi-level state semantic set in the directory where the target source data file is located; The reading module is used to read the error types and error counts recorded in the error status file when the existence of the error status file is detected; The judgment module is used to determine whether there is an anomaly in the multi-level state semantic set. If so, it determines that the target source data file does not need to be processed to release the corresponding table-level lock file, and performs local isolation on the source data text file corresponding to the table-level lock file. The identification module is used by all the processing nodes to identify processed files or locally isolated files based on the multi-level state semantic set existing in the directory where each target source data file is located.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-level concurrent control method based on file locks and status files as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the multi-level concurrent control method based on file locks and status files as described in any one of claims 1 to 7.

Images