Batch file polling loading method, device, equipment and program product

Abstract

The application discloses a batch file polling loading method and device, equipment and program product, and is applied to a file loading service of a target area, and relates to the field of financial technologies.The method comprises the following steps: based on a preset polling period, performing polling scheduling on each target batch in the target area; wherein the preset polling period is determined based on resource consumption of single polling of the target area, file arrival rate of each target batch, and file processing efficiency; for a single target batch polled, it is detected whether there is a remaining unloaded file in the target batch; if there is no remaining unloaded file in the target batch, it is checked whether all current batch files are complete; and when all current batch files are complete, a file loading operation is performed on the current batch files.The technical scheme of the embodiment of the application avoids cross-area resource preemption and local fault global diffusion, and improves the success rate of batch loading and data integrity.

Description

Technical Field

[0001] This invention relates to the field of financial technology, and in particular to a batch file polling loading method, apparatus, device, and program product. Background Technology

[0002] With the development of cross-regional business, multi-regional, multi-timezone, and multi-active geographically deployed architectures have been widely adopted, covering branches in multiple regions. The system needs to receive and load massive amounts of business data pushed by various branches daily. Due to the large number of branches involved, the large scale of file processing, and the highly dispersed arrival times of files in different regions caused by time zone differences, high demands are placed on the scheduling efficiency, resource utilization efficiency, and operational stability of batch file loading.

[0003] Current conventional batch file loading technologies mostly adopt a random processing mode that triggers loading as soon as the file arrives. During the loading process, problems such as duplicate loading, loading failure, and incomplete data are prone to occur. At the same time, under the globally shared loading service architecture, loading tasks in different areas compete for shared resources in an disorderly manner, and local anomalies can easily spread through links such as shared databases. Summary of the Invention

[0004] This invention provides a batch file polling loading method, apparatus, device, and program product, which avoids cross-regional resource contention and the global spread of local faults, thereby improving the success rate and data integrity of batch loading.

[0005] According to one aspect of the present invention, a batch file polling loading method is provided, applied to a file loading service for a target region, the method comprising: Based on a preset polling period, each target batch within the target area is polled and scheduled; wherein, the preset polling period is determined based on the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency. For each target batch polled, check whether there are any remaining unloaded files in the target batch; If there are no remaining unloaded files in the target batch, then verify whether all files in the current batch are complete. If all files in the current batch are complete, then a file loading operation is performed on the current batch of files.

[0006] According to another aspect of the present invention, a batch file polling loading apparatus is provided for a file loading service applied to a target region, the apparatus comprising: The batch polling scheduling module is used to poll and schedule each target batch within the target area based on a preset polling period; wherein, the preset polling period is determined based on the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency. The remaining unloaded file detection module is used to detect whether there are any remaining unloaded files in a single target batch that has been polled. The current batch file completeness detection module is used to verify whether all files in the current batch are complete if there are no remaining unloaded files in the target batch. The file loading operation execution module is used to perform a file loading operation on the current batch of files if all the files in the current batch are complete.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the batch file polling loading method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the batch file polling loading method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the batch file polling loading method described in any embodiment of the present invention.

[0010] The technical solution of this invention adopts an independent file loading service architecture based on target regions, achieving regional-level scheduling isolation and resource isolation. This avoids cross-regional resource contention and the global spread of local faults at the architectural level. By performing ordered polling scheduling on each target batch within a region based on a preset polling period, a preset polling period is introduced. This preset polling period is determined based on the resource consumption of a single poll in the target region, the file arrival rate of each target batch, and the file processing efficiency. This balances processing time and system resource overhead, reducing invalid polling losses. For each polled target batch, a dual pre-checking step is set up: detecting remaining unloaded files and verifying file completeness. This sequentially triggers the file loading operation of the target batch, avoiding duplicate loading and starting loading before files are complete from the data entry side, thus improving batch loading success rate and data integrity. Serial loading of a single region and a single batch prevents resource contention and solves concurrency conflicts such as missing file loading, data corruption, and lock waiting caused by parallel execution of multiple batches, further improving batch loading success rate and data integrity.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a batch file polling loading method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a batch file polling loading method provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of a batch file polling loading method provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of a batch file polling and loading device according to Embodiment 4 of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements the batch file polling loading method of this invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart illustrating a batch file polling loading method according to Embodiment 1 of the present invention. This embodiment of the invention is applicable to cross-region batch file loading. The method can be executed by a batch file polling loading device, which can be implemented in hardware and / or software. This device can be configured in an electronic device that carries batch file polling loading functionality, such as in a file loading service for the target region.

[0017] See Figure 1 The batch file polling loading method shown is applied to the file loading service of the target region, including: S101. Based on the preset polling period, perform polling scheduling for each target batch within the target area.

[0018] The target region is the region to which the branch office belongs. There can be multiple target regions. Optionally, the branch office can be divided into multiple target regions based on its time zone. The branch office is used for business processing. The branches are located in different time zones.

[0019] The file loading service is used to load batches of files for corresponding target regions. Each target region can deploy the file loading service independently. Therefore, the execution of file loading tasks in different target regions does not interfere with each other; that is, each target region independently performs thread scheduling and business caching, which physically avoids the problem of cross-regional resource contention across time zones and eliminates the pain point of asynchronous tasks in multiple time zones interfering with each other. Optionally, each target region only shares a unified scheduling and management platform and a global state database (i.e., a shared database). This achieves both isolation between target regions and ensures global controllability.

[0020] The preset polling period is a pre-defined trigger interval for the file loading service of the target region to perform polling scheduling for each target batch. The preset polling period characterizes the polling scheduling frequency of the file loading service for the target region. The preset polling period is determined based on the resource consumption of a single poll in the target region, the file arrival rate of each target batch, and the file processing efficiency. Specifically, the resource consumption of a single poll in the target region is the total system resources consumed by the file loading service of the target region to complete a single round of polling and loading of all target batches. The file arrival rate of the target batch is the number of files arriving in the target batch per unit time. The file processing efficiency of the target batch is the number of files successfully loaded by the target batch per unit time. Optionally, the preset polling period can be determined by comprehensively considering the resource consumption of a single poll in the target region, the file arrival rate of each target batch, and the file processing efficiency.

[0021] A target batch is the smallest independent scheduling unit for loading batch files within a target region, divided by region. The target batch is the direct execution object of polling scheduling actions. It also serves as the basic granularity for file verification, status control, and fault isolation. Target batches have a mapping relationship with regions within the target region. Optionally, a target batch can correspond to one or more regions within the target region. It can be agreed with the file sender to distribute files according to target batches. By using target batches corresponding to regions as units for file transfer, complex many-to-many coupling mappings are eliminated, file ownership is clearly defined from the source, the traceability difficulties caused by many-to-many coupling are solved, file traceability is improved, and operation and maintenance costs are significantly reduced.

[0022] Specifically, at each preset polling cycle, the file loading service for the target region initiates a complete round of polling scheduling for each target batch within the target region. The polling scheduling can be performed sequentially, following the batch order of each target batch. For example, the batch order can include priority order or a fixed order.

[0023] S102. For a single target batch that has been polled, check whether there are any remaining unloaded files in the target batch.

[0024] Remaining unloaded files are those belonging to a single target batch, which have arrived at the corresponding loading queue for that target batch, but whose loading process has not yet been completed. The reasons for remaining unloaded files include interruptions during the previous round of polling or a large file size in the previous single target batch. Specifically, a large file size in a single target batch may prevent all files from being loaded within a single poll, thus resulting in remaining unloaded files for that single target batch.

[0025] Specifically, for a single target batch that has been polled, the loading status of each file in the loading queue of the target batch is checked. If the loading status of any file in the loading queue is detected as "polled", it is determined that there are any remaining unloaded files in the target batch; if the loading status of all files in the loading queue is detected as "loaded", it is determined that there are no remaining unloaded files in the target batch.

[0026] The loading status is used to characterize the polling or loading status of individual files within the target batch. Optionally, if the target batch has not been polled, the loading status of each file within the target batch is kept as "not polled". When the target batch is polled, the loading status of each file within the target batch is updated to "polled". When a file in the target batch has arrived in the loading queue and has been loaded, the loading status of each file within the target batch is updated to "loaded".

[0027] For example, the loading status field is updated after each round of polling. The loading status field is used to characterize the loading status of a single file within the target batch. There are three loading status fields: 0 (indicating the target batch has not been polled), 1 (indicating the files in the target batch have been loaded), and 2 (indicating the target batch has been polled). Specifically, after each poll, it is determined whether the files in the target batch have arrived and been loaded. If the files in the target batch have not arrived, the loading status field of that target batch is updated to 2 after each poll; if the files in the target batch have arrived and been loaded, the loading status field of that target batch is updated to 1. Optionally, after the file data of the target batch is loaded into the table (i.e., after the file data of the target batch is written to the local file), the loading date can be incremented by 1. Target batches that have been loaded will no longer participate in the polling scheduling of subsequent batches within a single preset polling cycle. Optionally, after all target batches have been polled once, the loading status field of the target batches whose files have not all arrived can be updated from 2 to 0, and the next round of polling can begin again.

[0028] In an optional embodiment of the present invention, after detecting whether there are any remaining unloaded files in a single target batch that has been polled, the method further includes: if there are any remaining unloaded files in the target batch, terminating the current round of polling for each target batch in the target area and prioritizing the loading of the remaining unloaded files; and restarting the polling scheduling for each target batch in the target area after all the remaining unloaded files have been loaded.

[0029] The current round is the full batch traversal process triggered by the file loading service corresponding to the target region according to the preset polling cycle. The complete execution range of a single polling round is from the start of the first target batch to the completion of the traversal of all batches within the target region.

[0030] Specifically, if there are any remaining unloaded files in the target batch, the file loading service terminates the subsequent traversal of each target batch in the target area for the current round. Other target batches not yet polled in the current round will not undergo polling scheduling and loading operations in the current round; instead, the remaining unloaded files in the target batch will be loaded first. After all remaining unloaded files in the target batch have been loaded, the file loading service restarts a new round of polling scheduling for each target batch in the target area.

[0031] This solution ensures that only one batch performs file loading operations at a time by first checking whether there are any unloaded files in the target batch before each new target batch is cycled. If so, the cycle of the new target batch is skipped until all remaining unloaded files in the target batch have been loaded before continuing to cycle the new target batch. This prevents data corruption caused by multiple batches executing concurrently.

[0032] S103. If there are no remaining unloaded files in the target batch, verify whether all files in the current batch are complete.

[0033] The current batch of files is a complete set of files that should be delivered to the target batch within the preset polling period of the current round. The current batch of files belongs to the target batch of the current round. The file scope of the current batch of files is predetermined based on business processing rules and the file issuer. The current batch of files can be business files generated in the region corresponding to the target batch between two preset polling periods. The current batch of files is a collection of all business files that should be received and loaded in the current round.

[0034] Specifically, if there are no remaining unloaded files in the target batch, all business cycles under the target batch that have not yet been loaded are retrieved. These business cycles are then sorted from earliest to latest to form a verification sequence. Each business cycle has a pre-configured list of files that should be delivered. This list pre-defined the file identifier, file quantity, business cycle identifier, and target batch identifier for the files to be delivered within the business cycle.

[0035] Specifically, the current business cycle is filtered out from all business cycles in chronological order. Based on the current business cycle identifier, the current business cycle file corresponding to the current business cycle is filtered out from the files in the current batch's loading queue within the target batch. For the current business cycle, the current business cycle file is matched against the corresponding list of files that should be present. If a file for the current business cycle is missing compared to the list of files that should be present, it is determined that the current business cycle file is incomplete. If the current business cycle file completely matches the list of files that should be present, it is determined that the current business cycle file is complete, and the process returns to the step of filtering the current business cycle in chronological order until all business cycles are matched.

[0036] Specifically, if all business cycle documents for each business cycle are complete, then the current batch of documents is considered complete; if any business cycle document is incomplete, then the current batch of documents is considered incomplete.

[0037] Optionally, when the file sender distributes files in batches, each file corresponds to a verification file. The verification file is used to verify the integrity of the corresponding distributed file. The verification file may have a verification reference value for the corresponding distributed file. For example, the verification reference value may include a cyclic redundancy check value and a hash check value, etc. Optionally, the filename of the distributed file and the filename of the corresponding verification file may be the same; correspondingly, the file extension of the distributed file and the file extension of the corresponding verification file may be different. For example, the filename may include the data date and the target batch identifier.

[0038] Optionally, if there are no remaining unloaded files in the target batch, the verification method corresponding to the verification file is used to perform verification calculations on each individual file in the current batch, obtaining the verification calculation value for each individual file in the current batch. A consistency comparison is then performed between the verification calculation value and the verification reference value for each individual file in the current batch. If the verification calculation value and the verification reference value are consistent, then the individual file in the current batch is considered complete; if they are inconsistent, then the individual file in the current batch is considered incomplete. If all files in the current batch are complete, then the current batch is considered fully complete; if any file in the current batch is incomplete, then the current batch is considered incomplete.

[0039] Optionally, if there are no remaining unloaded files in the target batch, check whether all the required files for the current batch are present; if all files for the current batch are present, perform a file loading operation on the required files for the current batch.

[0040] The current batch of files includes optional files and mandatory files. Mandatory files are those that must be fully delivered before the file loading operation can begin. These mandatory files represent the core business data for the region corresponding to the target batch. Optional files are those that do not affect the initiation of the file loading operation. These optional files enrich the dimensions of the business data. For example, optional files may include supplementary descriptions and additional detail files for the region corresponding to the target batch.

[0041] Specifically, during the polling process, if all the required files for the current batch in the target batch have arrived, the file loading operation can be performed without waiting for all the non-required files in the current batch to arrive. If not all the required files for the current batch in the target batch have arrived, the required files for the current batch in the target batch can only be loaded after all the required files for the current batch have arrived.

[0042] By using the files that are guaranteed to arrive in the current batch as the criterion for completeness verification and loading initiation, the loading of core business data can be started in advance, which can improve the processing efficiency of the main process, avoid invalid polling caused by the lag of non-guaranteed files in the current batch, reduce the waste of system resources, weaken the impact of abnormal supplementary files on the main business, enhance the stability and fault tolerance of business operation, and at the same time be compatible with different file distribution rhythms, improving the adaptability and scalability of the business.

[0043] S104. If all files in the current batch are complete, then perform a file loading operation on the current batch of files.

[0044] Whether the current batch of files is complete is a condition for initiating the file loading operation on the current batch of files in the target batch. The file loading operation is used to load the current batch of files in the target batch.

[0045] Specifically, if all files in the current batch are complete, then a file loading operation is performed on the current batch of files.

[0046] The technical solution of this invention adopts an independent file loading service architecture based on target regions, achieving regional-level scheduling isolation and resource isolation. This avoids cross-regional resource contention and the global spread of local faults at the architectural level. By performing ordered polling scheduling on each target batch within a region based on a preset polling period, a preset polling period is introduced. This preset polling period is determined based on the resource consumption of a single poll in the target region, the file arrival rate of each target batch, and the file processing efficiency. This balances processing time and system resource overhead, reducing invalid polling losses. For each polled target batch, a dual pre-checking step is set up: detecting remaining unloaded files and verifying file completeness. This sequentially triggers the file loading operation of the target batch, avoiding duplicate loading and starting loading before files are complete from the data entry side, thus improving batch loading success rate and data integrity. Serial loading of a single region and a single batch prevents resource contention and solves concurrency conflicts such as missing file loading, data corruption, and lock waiting caused by parallel execution of multiple batches, further improving batch loading success rate and data integrity.

[0047] Example 2 Figure 2 This is a flowchart of a batch file polling loading method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment of the present invention specifies the "determination of the preset polling period based on the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency" as "periodically acquiring the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency; calculating the latency cost of a single poll in the target area based on the file arrival rate and file processing efficiency of each target batch; and determining the preset polling period for the target area with resource consumption and latency cost as dual constraint objectives." The preset polling period can be dynamically adapted to the real-time business load of the target area. During off-peak periods, it can automatically reduce the scanning frequency, saving system overhead; during peak periods, it can automatically increase the scanning frequency, shortening the average waiting time for files, significantly reducing the risk of file backlog, and achieving flexible peak shaving and valley filling of system load. While balancing file processing timeliness and system resource overhead, it improves the adaptability of the scheduling strategy to scenarios with asynchronous file arrival in multiple time zones. It should be noted that parts not detailed in this embodiment of the present invention can be found in the descriptions of other embodiments.

[0048] See Figure 2 The batch file polling loading method shown includes: S201. Periodically obtain the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency.

[0049] Since files from different branches are generated at different times, the preset polling period for the target area is periodically updated. When the number of files in the target area decreases during a certain time period or all files in the target batch are loaded, the preset polling period can be automatically increased, which can reduce the frequency of file polling traversal and thus save system resources; when there is a peak in files, the preset polling period can be automatically decreased, which can improve response speed.

[0050] Specifically, the total system resources consumed by the file loading service in the target region to complete a single polling loading process of the full target batch are periodically obtained, thus obtaining the resource consumption of the target region per polling. The number of files arriving in each target batch within a unit of time is periodically obtained, thus obtaining the file arrival rate of each target batch. The number of files successfully loaded completely in each target batch within a unit of time is periodically obtained, thus obtaining the file processing efficiency of each target batch. For example, the preset polling period can be updated hourly.

[0051] S202. Calculate the latency cost of a single poll in the target area based on the file arrival rate and file processing efficiency of each target batch.

[0052] The latency cost is the quantifiable system service loss caused by the processing delay after the current batch of files arrives at the target batch's loading queue.

[0053] Specifically, a latency cost calculation formula can be used to calculate the latency cost of a single polling of the target region based on the file arrival rate and file processing efficiency of each target batch: ; In the formula, The latency cost of a single poll for the target area; The inherent waiting time for polling is given; where T is the polling period to be solved. The queuing waiting time is denoted by ; where i is the target batch number; and N is the total number of target batches. Let i be the file arrival rate of the i-th target batch; Let i be the system utilization rate of the i-th target batch. In the formula, The efficiency of file processing for the i-th target batch; This represents the remaining processing capacity for the i-th target batch.

[0054] S203. Determine the preset polling cycle for the target area based on the dual constraints of resource consumption and latency cost.

[0055] Specifically, the total system cost for a single poll of the target area is obtained by comprehensively considering resource consumption and latency costs.

[0056] For example, the total system cost for a single poll of the target region can be represented by the following formula: ; In the formula, denoted as the total system cost for a single poll of the target region; T represents the polling period to be solved. As a weight for latency cost; The latency cost of a single poll for the target area; Weighted by resource consumption; is the resource consumption of a single poll in the target region; i is the target batch number; N is the total number of target batches; Let i be the file arrival rate of the i-th target batch; Let i be the system utilization rate of the i-th target batch. In the formula, The efficiency of file processing for the i-th target batch; This represents the remaining processing capacity for the i-th target batch.

[0057] Specifically, the preset polling period for the target region is determined with the goal of minimizing the total system cost of a single poll in the target region.

[0058] For example, the first derivative of the total system cost for a single poll of the target region can be obtained as follows: ; In the formula, denoted as the total system cost for a single poll of the target region; T represents the polling period to be solved. is the latency cost weight; i is the target batch number; N is the total number of target batches; Let i be the file arrival rate of the i-th target batch; Let i be the system utilization rate of the i-th target batch. In the formula, The efficiency of file processing for the i-th target batch; This represents the processing capacity margin for the i-th target batch; Weighted by resource consumption; This represents the resource consumption for a single poll of the target area.

[0059] For example, setting the first derivative to 0 (a necessary condition for finding the minimum point), we obtain the following formula: ; In the formula, As a weight for latency cost; is the latency cost of a single poll of the target region; i is the target batch number; N is the total number of target batches; Let i be the file arrival rate of the i-th target batch; Let i be the system utilization rate of the i-th target batch. In the formula, The efficiency of file processing for the i-th target batch; This represents the processing capacity margin for the i-th target batch; Weighted by resource consumption; denoted as , where is the resource consumption per poll in the target region; T is the polling period to be solved.

[0060] For example, after processing, the following formula can be used to represent the preset polling period for the target region: ; In the formula, The preset polling period for the target region; Weighted by resource consumption; The resource consumption for a single poll of the target area; is the latency cost weight; i is the target batch number; N is the total number of target batches; Let i be the file arrival rate of the i-th target batch; Let i be the system utilization rate of the i-th target batch. In the formula, The efficiency of file processing for the i-th target batch; This represents the remaining processing capacity for the i-th target batch.

[0061] In an optional embodiment of the present invention, determining the preset polling period of the target region with resource consumption and latency cost as dual constraints includes: determining the candidate polling period of the target region with resource consumption and latency cost as dual constraints; and selecting the larger value between the candidate polling period and the safe polling period as the preset polling period of the target region with the safe polling period as the lower limit constraint.

[0062] The candidate polling period is the optimal polling period determined with the goal of balancing resource consumption and latency costs. The safe polling period is a pre-set lower threshold for the polling period.

[0063] The secure polling period serves as a safety net to ensure the stable operation of each branch office. It can be understood as preventing situations where a surge in business traffic results in an excessively short candidate polling period, leading to overly frequent polling, system resource overload, and even service instability. Optionally, the secure polling period can be pre-configured and adjusted by technical personnel based on the system capacity of each branch office. The secure polling period can be a fixed value, meaning it doesn't dynamically change with business traffic. For example, a secure polling period of 1 minute is suitable.

[0064] Specifically, the total system cost for a single polling of the target region is obtained by comprehensively considering resource consumption and latency costs. With the goal of minimizing the total system cost for a single polling of the target region, candidate polling periods are determined. The candidate polling periods and safe polling periods are compared, and the larger value between the two is selected as the preset polling period for the target region.

[0065] For example, the following formula can be used to represent selecting the larger value between the candidate polling period and the safe polling period as the preset polling period for the target region, with the safe polling period as the lower limit constraint: ; In the formula, The preset polling period for the target region; For safe polling cycle; Weighted by resource consumption; The resource consumption for a single poll of the target area; is the latency cost weight; i is the target batch number; N is the total number of target batches; Let i be the file arrival rate of the i-th target batch; Let i be the system utilization rate of the i-th target batch. In the formula, The efficiency of file processing for the i-th target batch; This represents the remaining processing capacity for the i-th target batch.

[0066] This solution solves for candidate polling cycles based on the dual constraints of resource consumption and latency cost. It introduces a safe polling cycle as a safety lower limit constraint and selects the larger of the two as the preset polling cycle for the target area. By dynamically adapting to business load and balancing scheduling efficiency and resource consumption, the safety lower limit constraint can avoid system overload and concurrency conflicts caused by excessively high polling frequency, thus balancing scheduling optimization and operational stability.

[0067] S204. Based on the preset polling period, perform polling scheduling for each target batch within the target area.

[0068] S205. For a single target batch that has been polled, check if there are any remaining unloaded files in the target batch.

[0069] S206. If there are no remaining unloaded files in the target batch, then verify whether all files in the current batch are complete.

[0070] S207. If all files in the current batch are complete, then perform a file loading operation on the current batch of files.

[0071] The technical solution of this invention differs from fixed-period polling. It periodically collects the resource consumption, arrival rate of each batch of files, and processing efficiency of a single poll in the target area, and calculates the latency cost of a single poll accordingly. With resource consumption and latency cost as dual constraints, the preset polling period of the target area is obtained. The preset polling period can be dynamically adapted to the real-time business load of the target area. During off-peak periods, the scanning frequency can be automatically reduced to save system overhead. During peak periods, the scanning frequency can be automatically increased to shorten the average waiting time of files, significantly reducing the risk of file backlog. This achieves flexible peak shaving and valley filling of system load. By balancing file processing timeliness and system resource overhead, it improves the adaptability of the scheduling strategy to scenarios with asynchronous file arrival in multiple time zones.

[0072] Example 3 Figure 3 This is a flowchart of a batch file polling loading method provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment of the present invention specifies the "determination of the preset polling period based on the resource consumption of a single poll in the target region, the file arrival rate of each target batch, and the file processing efficiency" as "periodically acquiring the resource consumption of a single poll in the target region, the file arrival rate of each target batch, and the file processing efficiency; calculating the latency cost of a single poll in the target region based on the file arrival rate and file processing efficiency of each target batch; and determining the preset polling period for the target region with resource consumption and latency cost as dual constraint objectives." This enables fault classification early warning and proactive blocking, avoids the continuous consumption of resources through invalid retries, prevents local faults from spreading across regions through links such as shared databases, effectively curbs fault expansion, and improves the overall stability and risk controllability of the batch loading system. It should be noted that parts not detailed in this embodiment of the present invention can be referred to in the descriptions of other embodiments.

[0073] See Figure 3 The batch file polling loading method shown includes: S301. Based on the preset polling period, poll and schedule each target batch within the target area.

[0074] S302. For a single target batch that has been polled, check whether there are any remaining unloaded files in the target batch.

[0075] S303. If there are no remaining unloaded files in the target batch, then verify whether all files in the current batch are complete.

[0076] S304. If all files in the current batch are complete, then perform a file loading operation on the current batch of files.

[0077] S305. Real-time monitoring of the file loading status of each target batch.

[0078] The file loading status of the target batch is used to characterize the execution result of the overall file loading of the target batch. The file loading status of the target batch is statistically granular at the level of a single target batch. For example, the file loading status of the target batch includes all files loading successfully and some files in the batch failing to load.

[0079] Specifically, a globally unique task identifier can be bound to each business file to be loaded in the target batch. During the file loading operation of an individual file within the target batch, the loading status of each file is reported back in real time through a periodic heartbeat reporting mechanism, marking files in the loading execution stage as "loading in progress". If the loading task corresponding to an individual file times out or the loading process is abnormally interrupted, the loading status of that file is marked as "loading failed". If an individual file is loaded successfully, its loading status is marked as "loading successful". Based on the loading status of all files in the target batch, the overall file loading status of the target batch is determined.

[0080] S306. When all files in the target batch are successfully loaded, mark the target batch as normal and maintain the normal polling loading process.

[0081] The "normal state" characterizes the target batch as being in a stable running state of batch file loading scheduling. A target batch in the normal state can be understood as all files within the target batch loading normally without any abnormalities. The normal polling loading process is the polling loading process based on a preset polling period as described in the above embodiments.

[0082] Specifically, when all files in the target batch are successfully loaded, the target batch is determined to be in a normal state, and the normal polling loading process based on the preset polling cycle is maintained.

[0083] S307. When a target batch fails to load a file in a single instance, mark the target batch as being in a sub-healthy state and simultaneously count the number of consecutive failures of the target batch.

[0084] The term "sub-healthy state" indicates that the batch file loading process of the target batch is in a slightly abnormal state. A target batch in a sub-healthy state can be understood as experiencing a single file loading failure and entering a limited retry phase.

[0085] The consecutive failure count refers to the number of times the target batch has consecutively failed to load files in the current round. Simultaneously counting the consecutive failure count of the target batch can be understood as continuing to monitor the loading status of each file in the target batch when it is in a sub-optimal state.

[0086] Specifically, when a target batch experiences a single file loading failure, it is determined that the target batch is in a sub-healthy state. During the continued file loading operation on the target batch, a timed heartbeat reporting mechanism is used to transmit the loading status of individual files in real time, and the number of consecutive file loading failures in the target batch is counted to determine the consecutive failure count for the target batch.

[0087] S308. For a target batch that is in a sub-healthy state, determine the failure propagation probability of the target batch based on the failure propagation coefficient, the number of consecutive failures, and the preset failure tolerance threshold.

[0088] The fault propagation coefficient is an adjustable quantification parameter used to quantify the sensitivity of a loading anomaly in a target batch to its outward propagation. The fault propagation coefficient controls the rate of increase in the probability of fault propagation. Optionally, the fault propagation coefficient can be predetermined and adjusted based on historical data.

[0089] The preset failure tolerance threshold is a pre-defined lower limit for the number of consecutive failures when switching from a sub-healthy state to a fault isolation state. For example, the preset failure tolerance threshold can be 3 times.

[0090] The probability of fault propagation is the probability that, when the target batch is in a sub-healthy state, it will switch from a sub-healthy state to a fault isolation state as the number of consecutive failures accumulates.

[0091] Specifically, for target batches in a sub-healthy state, the fault propagation probability calculation formula can be used to calculate the fault propagation probability of the target batch based on the fault diffusion coefficient, the number of consecutive failures, and the preset failure tolerance threshold.

[0092] For example, the following formula can be used to predefine the one-step transition probability matrix of a four-state Markov: ; In the formula, P is the four-state Markov one-step transition probability matrix, where the four states include normal state, sub-health state, fault isolation state and recovery state; This represents the probability that all files in the current round will load successfully and the target batch will remain in a normal state. This represents the probability of switching to a sub-healthy state due to a single file loading failure in the current round. and This indicates that the normal state cannot directly jump to the fault isolation state or the recovery state; The probability of switching back to normal state after the current round of anomalies is eliminated and all files are loaded successfully; The probability of the current round still experiencing single file loading failures and continuing to remain in a sub-healthy state is high. The probability of fault propagation from a sub-healthy state to a fault isolation state; A state of sub-health cannot be directly switched to a state of recovery; and This indicates that the fault-isolated state cannot directly revert to the normal or sub-healthy state; The probability that the root cause of the current failure has not been eliminated and the failure remains in an isolated state. The probability of switching to the recovery state for verification and trial operation after eliminating the root cause of the fault; To verify that there are no abnormalities during the trial run and to determine the probability of successfully switching back to normal status; This indicates that the recovered state cannot directly regress to the sub-healthy state; The probability of reverting to a fault isolation state if a batch of anomalies recurs during trial operation; This represents the probability that the current round of verification has not been completed and the system will remain in the recovery state for trial operation.

[0093] For example, the following formula can be used to represent the probability of fault propagation in the target batch: ; In the formula, The probability of fault propagation in the target batch; This represents the number of consecutive failures. This is a preset failure tolerance threshold.

[0094] S309. When the number of consecutive failures reaches the preset failure tolerance threshold, or the failure propagation probability exceeds the preset failure propagation probability threshold, the target batch is switched from the sub-health state to the fault isolation state, the file writing link of the target batch is cut off simultaneously, and the polling scheduling of the target batch is suspended.

[0095] The preset fault propagation probability threshold is a pre-defined lower limit of the fault propagation probability when switching from a sub-healthy state to a fault isolation state. For example, the preset fault propagation probability threshold can be 0.85.

[0096] The fault isolation state is the circuit breaker protection operating state of the target batch polling loading process. When the target batch is in the fault isolation state, it is determined that the target batch is a critical propagation node for fault propagation, and a blocking barrier is injected to cut off the target batch's external connections.

[0097] The file write link is the file write link in the loading queue of the target batch in the file sending direction.

[0098] Specifically, when a target batch is in a sub-healthy state, if the number of consecutive failures is greater than or equal to a preset failure tolerance threshold, or if the failure propagation probability is greater than a preset failure propagation probability threshold, the target batch is switched from a sub-healthy state to a fault isolation state. Simultaneously, the file write link between the target batch and the shared database is severed, and polling scheduling of the target batch is suspended. This prevents the loading anomalies of the target batch from propagating globally.

[0099] In an optional embodiment of the present invention, after switching the target batch from a sub-healthy state to a fault isolation state, the method further includes: locking the baseline file offset of the target batch based on log stub records for the most recent successful loading, and simultaneously obtaining the actual write offset of the local file corresponding to the target batch; performing a consistency check on the baseline file offset and the actual write offset; if the check is consistent, then in the isolation environment, using the baseline file offset as the boundary, extracting local file segments from the target batch, and simultaneously obtaining reference file segments corresponding to the boundary from the shared database; performing hash calculations on the local file segments and the reference file segments respectively to obtain the hash values ​​of the local segments and the reference segments; performing a bitwise XOR operation on the hash values ​​of the local segments and the reference segments; if the result of the bitwise XOR operation is all zeros, then determining that the integrity check of the local file segments has passed; using the baseline file offset as the starting breakpoint for resuming transmission, performing breakpoint resuming transmission on the target batch; and after successful resuming transmission, switching the target batch from the fault isolation state to the recovery state.

[0100] Log stubs are persistent log collection nodes pre-embedded throughout the entire file loading process. Each time the target batch completes the loading of all files, log stubs are synchronously and persistently stored as points for file tracing.

[0101] The shared database is a remote database shared by all target regions. The baseline file offset is the file byte offset in the shared database corresponding to the most recent successful full load of the target batch. The baseline file offset is used to identify the file boundary of the most recent successful full load of the target batch in the shared database.

[0102] Local files are business files loaded onto the local disk through file loading operations but belonging to the target batch. The actual write offset is the file byte offset in the local file corresponding to the most recent successful complete load of the target batch. The actual write offset is used to identify the file boundary of the most recent successful complete load of the target batch in the local file.

[0103] The isolated environment is a temporary running sandbox that operates independently of the normal polling loading schedule and is isolated from other resources. The isolated environment does not consume global read / write resources and does not affect the polling loading process of other target batches that are in a normal state.

[0104] The local file segment is a byte segment of the target batch that was most recently successfully loaded, extracted from the local file using the base file offset as the dividing boundary. The reference file segment is a byte segment of the target batch that was most recently successfully loaded, extracted from the shared database using the base file offset as the dividing boundary.

[0105] The local fragment hash value is the hash calculation result of the local file fragment. The reference fragment hash value is the hash calculation result of the reference file fragment. The resume start breakpoint is the starting read position for resuming from a breakpoint. The recovery state is the transition verification state after the fault isolation state.

[0106] Specifically, after the target batch is in a fault isolation state, based on the log stub record corresponding to the target batch, the log stub record of the most recently successfully loaded batch is parsed, and the file write position corresponding to the most recently successfully loaded batch in the shared database is extracted to obtain the baseline file offset.

[0107] Specifically, the file write position corresponding to the most recently successfully loaded round is read in the local file to obtain the actual write offset.

[0108] Specifically, the baseline file offset and the actual write offset are compared. If the baseline file offset and the actual write offset are the same, the local file fragments of the target batch are extracted from the local file, using the actual write offset (i.e., the baseline file offset) as the starting boundary. Reference file fragments of the target batch are extracted from the shared database, using the baseline file offset as the starting boundary.

[0109] Specifically, the same hash algorithm can be used to perform hash calculations on both the local file fragment and the reference file fragment, obtaining the local fragment hash value and the reference fragment hash value respectively. A bitwise XOR operation is then performed on the local fragment hash value and the reference fragment hash value. If the result of the bitwise XOR operation is all zeros, the local file fragment integrity check is considered passed; if the result of the bitwise XOR operation contains a non-zero value, the local file fragment integrity check is considered failed.

[0110] For example, the following formula can be used to perform hash calculations on the local file fragment and the reference file fragment respectively to obtain the local fragment hash value and the reference fragment hash value, and then perform a bitwise XOR operation on the local fragment hash value and the reference fragment hash value: ; In the formula, The result of the bitwise XOR operation between the local fragment hash value and the reference fragment hash value is used to characterize the integrity of the local file fragment. This indicates that the local file fragment integrity check has passed. This indicates that the local file fragment integrity check failed; This is the hash value of the local fragment. This is a local file fragment; This is a bitwise XOR operation; For reference fragment hash value; This is a reference document fragment.

[0111] Specifically, when the local file fragment integrity check passes, the target batch is resumed from the baseline file offset. After successful resumption, the target batch is switched from the fault isolation state to the recovery state.

[0112] Existing technologies often employ simple, unlimited, or limited retry mechanisms when handling abnormal batches. When a region encounters persistent failures such as database deadlocks or network congestion, repeated retries not only fail to restore business operations but also continuously consume the independent system resources of that region. This can even cause the congestion to propagate to other regions through the shared database, escalating a local failure into a global paralysis.

[0113] This solution, in a fault-isolated state, obtains the baseline offset and the actual local write offset through log stubs. It first performs an offset consistency check, and after the offset consistency check passes, it performs a hash XOR operation to complete the local file fragment integrity check. After the check passes, it performs breakpoint resume and switches to the recovery state. This solves the problem of duplicate writing or data loss caused by the need to load abnormal batches from the beginning in traditional solutions. At the same time, it ensures the accuracy of the breakpoint position and the integrity of the file content, improves the success rate of breakpoint resume, avoids data corruption and loss, realizes autonomous and reliable recovery of fault batches, and reduces manual operation and maintenance costs.

[0114] In an optional embodiment of the present invention, after switching the target batch from the fault isolation state to the recovery state, the method further includes: when the target batch remains in the recovery state for a preset duration, attempting to rebuild the file write link between the target batch and the shared database, and continuously monitoring the connection status; if the connection status is stable, gradually releasing the traffic limit of the target batch and removing the blocking barrier; switching the target batch from the recovery state to the normal state, re-incorporating it into the global polling scheduling queue, and continuing to execute the unfinished polling loading process.

[0115] The preset duration is the observation buffer period after fault repair. The preset duration is the minimum duration forcing the target batch to be in the recovery state. The connection status is the connectivity status of the file write link between the target batch and the shared database. The global polling scheduling queue is the polling scheduling queue for the target region. The global scheduling queue contains all target batches that can be polled and scheduled. The polling loading process is the polling loading process based on the preset polling period in the above embodiment.

[0116] Specifically, continuously monitor the duration for which the target batch remains in the recovery state. Compare this duration with a preset duration. When the preset duration is reached, re-establish the file write link between the target batch and the shared database, and continuously monitor the operational metrics of the file write link. Compare the operational metrics of the file write link with preset normal threshold ranges. When the operational metrics of the file write link are within the preset normal threshold range, the connection status of the file write link is considered stable.

[0117] Specifically, if the file write link connection is stable, the traffic restrictions on the target batch are gradually lifted and the blocking barrier is removed. The target batch is then switched from the recovered state back to the normal state, re-included in the global polling scheduling queue, and the unfinished polling loading process of the target batch continues.

[0118] This solution achieves a smooth transition in fault recovery by switching the target batch to the recovery state and maintaining it for a preset time, then rebuilding the shared database connection and testing its stability. Once the connection is stable, traffic restrictions are gradually lifted and blocking barriers are removed, and finally the process switches back to the normal state and resumes the polling loading process. This avoids the secondary impact and fault recurrence caused by direct unblocking, ensures the stability of the system recovery process, and improves the self-healing capability of batch loading services.

[0119] Optionally, for non-critical errors (such as network jitter), the original retry logic is executed. For critical faults that trigger blocking and isolation, the alarm level is directly escalated, and SMS and platform alarms are pushed, marked "Entered isolation and self-healing process" to avoid blind intervention by operation and maintenance personnel.

[0120] The technical solution of this invention establishes a batch full lifecycle status management mechanism, dividing the system into three levels: normal state, sub-healthy state, and fault isolation state. In the sub-healthy state, the probability of fault propagation is calculated based on the fault propagation coefficient and the number of consecutive failures. When the threshold is reached, the system actively switches to the fault isolation state, cuts off the file writing link, and suspends polling scheduling. This enables fault classification early warning and active blocking, avoids the continuous consumption of resources by invalid retries, prevents local faults from spreading across regions through links such as shared databases, effectively curbs the expansion of faults, and improves the overall stability and risk controllability of the batch loading system.

[0121] Example 4 Figure 4 This is a schematic diagram of a batch file polling loading device provided in Embodiment 4 of the present invention. This embodiment of the invention is applicable to cross-region batch file loading. The device can execute a batch file polling loading method and can be implemented in hardware and / or software. The device can be configured in an electronic device that carries the batch file polling loading function, such as in a file loading service for a target region.

[0122] See Figure 4 The batch file polling and loading device shown is applied to file loading services in a target area and includes: a batch polling scheduling module 401, a remaining unloaded file detection module 402, a current batch file completeness detection module 403, and a file loading operation execution module 404. The batch polling scheduling module 401 is used to poll and schedule each target batch in the target area based on a preset polling period; the preset polling period is determined based on the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency. The remaining unloaded file detection module 402 is used to detect whether there are any remaining unloaded files in a single polled target batch. The current batch file completeness detection module 403 is used to verify whether all files in the current batch are complete if no remaining unloaded files are found in the target batch. The file loading operation execution module 404 is used to perform a file loading operation on the current batch if all files in the current batch are complete.

[0123] The technical solution of this invention adopts an independent file loading service architecture based on target regions, achieving regional-level scheduling isolation and resource isolation. This avoids cross-regional resource contention and the global spread of local faults at the architectural level. By performing ordered polling scheduling on each target batch within a region based on a preset polling period, a preset polling period is introduced. This preset polling period is determined based on the resource consumption of a single poll in the target region, the file arrival rate of each target batch, and the file processing efficiency. This balances processing time and system resource overhead, reducing invalid polling losses. For each polled target batch, a dual pre-checking step is set up: detecting remaining unloaded files and verifying file completeness. This sequentially triggers the file loading operation of the target batch, avoiding duplicate loading and starting loading before files are complete from the data entry side, thus improving batch loading success rate and data integrity. Serial loading of a single region and a single batch prevents resource contention and solves concurrency conflicts such as missing file loading, data corruption, and lock waiting caused by parallel execution of multiple batches, further improving batch loading success rate and data integrity.

[0124] In an optional embodiment of the present invention, the device further includes: a word polling index acquisition module, used to periodically acquire the resource consumption of a single polling of the target region, the file arrival rate of each target batch, and the file processing efficiency; a latency cost calculation module, used to calculate the latency cost of a single polling of the target region based on the file arrival rate and file processing efficiency of each target batch; and a preset polling period determination module, used to determine the preset polling period of the target region with resource consumption and latency cost as dual constraint objectives.

[0125] In an optional embodiment of the present invention, the preset polling period determination module includes: a candidate polling period determination unit, used to determine the candidate polling period of the target region with resource consumption and latency cost as dual constraints; and a preset polling period determination unit, used to select the larger value between the candidate polling period and the safe polling period as the preset polling period of the target region with the safe polling period as the lower limit constraint.

[0126] In an optional embodiment of the present invention, the apparatus further includes: a remaining unloaded file loading module, configured to, if there are no remaining unloaded files in the target batch, verify whether all files in the current batch are complete, and if there are remaining unloaded files in the target batch, terminate the current round of polling for each target batch in the target area and prioritize loading the remaining unloaded files; and a target batch polling scheduling restart module, configured to, after all remaining unloaded files have been loaded, restart the polling scheduling for each target batch in the target area.

[0127] In an optional embodiment of the present invention, the device further includes: a file loading status fact monitoring module, used to monitor the file loading status of each target batch in real time; a normal state marking module, used to mark the target batch as normal when all files in the target batch are successfully loaded, and maintain the normal polling loading process; a sub-healthy state marking module, used to mark the target batch as sub-healthy when a single file loading failure occurs in the target batch, and simultaneously count the number of consecutive failures of the target batch; a fault propagation probability determination module, used to determine the fault propagation probability of the target batch in the sub-healthy state based on the fault propagation coefficient, the number of consecutive failures, and a preset failure tolerance threshold; and a fault isolation state switching module, used to switch the target batch from the sub-healthy state to the fault isolation state when the number of consecutive failures reaches the preset failure tolerance threshold, or the fault propagation probability exceeds the preset fault propagation probability threshold, simultaneously cut off the file writing link of the target batch, and suspend the polling scheduling of the target batch.

[0128] In an optional embodiment of the present invention, the apparatus further includes: an offset acquisition module, configured to, after switching the target batch from a sub-healthy state to a fault isolation state, lock the offset of the most recently successfully loaded baseline file of the target batch based on log stub records, and simultaneously acquire the actual write offset of the local file corresponding to the target batch; an offset consistency verification module, configured to perform consistency verification on the baseline file offset and the actual write offset; and a file fragment extraction module, configured to, if the verification is consistent, extract local file fragments from the target batch in the isolation environment with the baseline file offset as the boundary, and simultaneously acquire the corresponding boundary from the shared database. The system includes: a reference file fragment; a hash calculation module, used to perform hash calculations on the local file fragment and the reference file fragment respectively to obtain the local fragment hash value and the reference fragment hash value; a bitwise XOR operation module, used to perform a bitwise XOR operation on the local fragment hash value and the reference fragment hash value; an integrity detection module, used to determine that the local file fragment integrity detection has passed if the result of the bitwise XOR operation is all zeros; a target batch breakpoint resume module, used to perform breakpoint resume for the target batch with the base file offset as the start breakpoint; and a recovery state switching module, used to switch the target batch from the fault isolation state to the recovery state after successful resume.

[0129] In an optional embodiment of the present invention, the device further includes: a connection status continuous monitoring module, used to attempt to rebuild the file write link between the target batch and the shared database and continuously monitor the connection status when the target batch is maintained in the recovery state for a preset duration after switching the target batch from the fault isolation state to the recovery state; a blocking barrier removal module, used to gradually release the traffic limit of the target batch and remove the blocking barrier if the connection status is stable; and a normal state switching module, used to switch the target batch from the recovery state to the normal state, re-include it in the global polling scheduling queue, and continue to execute the unfinished polling loading process.

[0130] The batch file polling loading device provided in the embodiments of the present invention can execute the batch file polling loading method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0131] In the technical solutions of this invention, the information collected is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0132] Example 5 According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0133] Figure 5 A schematic diagram of an electronic device 500 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0134] like Figure 5 As shown, the electronic device 500 includes at least one processor 501 and a memory, such as a read-only memory 502 or a random access memory 503, communicatively connected to the at least one processor 501. The memory stores computer programs executable by the at least one processor. The processor 501 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 502 or loaded from storage unit 508 into the random access memory 503. The random access memory 503 can also store various programs and data required for the operation of the electronic device 500. The processor 501, read-only memory 502, and random access memory 503 are interconnected via a bus 504. An input / output interface 505 is also connected to the bus 504.

[0135] Multiple components in electronic device 500 are connected to input / output interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0136] Processor 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 501 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 501 performs the various methods and processes described above, such as the batch file polling loading method.

[0137] In some embodiments, the bulk file polling loading method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 500 via read-only memory 502 and / or communication unit 509. When the computer program is loaded into random access memory 503 and executed by processor 501, one or more steps of the bulk file polling loading method described above can be performed. Alternatively, in other embodiments, processor 501 can be configured to perform the bulk file polling loading method by any other suitable means (e.g., by means of firmware).

[0138] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), complex programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0143] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system. This addresses the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0144] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A batch file polling loading method, characterized in that, A file loading service applied to a target region, the method comprising: Based on a preset polling period, each target batch within the target area is polled and scheduled; wherein, the preset polling period is determined based on the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency. For each target batch polled, check whether there are any remaining unloaded files in the target batch; If there are no remaining unloaded files in the target batch, then verify whether all files in the current batch are complete. If all files in the current batch are complete, then a file loading operation is performed on the current batch of files.

2. The method according to claim 1, characterized in that, The preset polling period is determined based on the resource consumption of a single poll in the target region, the file arrival rate of each target batch, and the file processing efficiency, including: Periodically obtain the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency; Calculate the latency cost of a single polling of the target region based on the file arrival rate and file processing efficiency of each target batch. The preset polling period for the target region is determined by using the resource consumption and latency cost as dual constraints.

3. The method according to claim 2, characterized in that, The step of determining the preset polling period for the target region, using the resource consumption and latency cost as dual constraints, includes: The candidate polling period for the target region is determined by using the resource consumption and the latency cost as dual constraints. Using the safe polling period as a lower limit constraint, the larger value between the candidate polling period and the safe polling period is selected as the preset polling period for the target region.

4. The method according to claim 1, characterized in that, After detecting whether there are any remaining unloaded files in a single target batch that has been polled, the method further includes: If there are any remaining unloaded files in the target batch, then the current round of polling for each target batch in the target area is terminated, and the remaining unloaded files are loaded first. After all the remaining unloaded files have been loaded, the polling scheduling of each target batch within the target region is restarted.

5. The method according to claim 1, characterized in that, Also includes: Real-time monitoring of the file loading status of each target batch; When all files in the target batch are successfully loaded, the target batch is marked as normal, and the normal polling loading process is maintained. When the target batch fails to load a file in a single instance, the target batch is marked as being in a sub-healthy state, and the number of consecutive failures of the target batch is counted simultaneously. For the target batch that is in a sub-healthy state, the failure propagation probability of the target batch is determined based on the failure propagation coefficient, the number of consecutive failures, and a preset failure tolerance threshold. When the number of consecutive failures reaches a preset failure tolerance threshold, or the failure propagation probability exceeds a preset failure propagation probability threshold, the target batch is switched from a sub-healthy state to a fault isolation state, the file writing link of the target batch is simultaneously cut off, and the polling scheduling of the target batch is suspended.

6. The method according to claim 5, characterized in that, After switching the target batch from a sub-healthy state to a fault-isolated state, the method further includes: Based on the log stub record, lock the offset of the baseline file that was most recently successfully loaded for the target batch, and at the same time obtain the actual write offset of the local file corresponding to the target batch; Perform a consistency check on the baseline file offset and the actual write offset; If the verification is consistent, then in the isolated environment, local file segments in the target batch are extracted with the baseline file offset as the boundary, and reference file segments corresponding to the boundary are obtained from the shared database. Perform hash calculations on the local file fragment and the reference file fragment respectively to obtain the local fragment hash value and the reference fragment hash value; Perform a bitwise XOR operation on the local fragment hash value and the reference fragment hash value; If the result of the bitwise XOR operation is all zeros, then the integrity check of the local file segment is deemed to have passed. Using the baseline file offset as the starting breakpoint for resuming transmission, the target batch is resumed from the breakpoint. After successful retransmission, the target batch is switched from the fault isolation state to the recovery state.

7. The method according to claim 6, characterized in that, After switching the target batch from the fault isolation state to the recovery state, the method further includes: When the target batch remains in the recovery state for a preset duration, an attempt is made to rebuild the file write link between the target batch and the shared database, and the connection status is continuously monitored; If the connection status is stable, gradually release the traffic restrictions on the target batch and remove the blocking barrier; The target batch is switched from the recovered state to the normal state and re-included in the global polling scheduling queue to continue the unfinished polling loading process.

8. A batch file polling and loading device, characterized in that, A file loading service applied to a target region, the apparatus comprising: The batch polling scheduling module is used to poll and schedule each target batch within the target area based on a preset polling period; wherein, the preset polling period is determined based on the resource consumption of a single poll in the target area, the file arrival rate of each target batch, and the file processing efficiency. The remaining unloaded file detection module is used to detect whether there are any remaining unloaded files in a single target batch that has been polled. The current batch file completeness detection module is used to verify whether all files in the current batch are complete if there are no remaining unloaded files in the target batch. The file loading operation execution module is used to perform a file loading operation on the current batch of files if all the files in the current batch are complete.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the batch file polling loading method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the batch file polling loading method according to any one of claims 1-7.

Images