Queuing scheduling warehousing method for discrete data

Abstract

The application relates to the technical field of data management, and discloses a queuing scheduling warehousing method for discrete data, which converts high-concurrency database writing into orderly queuing through the establishment of a task queue and an intelligent scheduling mechanism, effectively avoids table locking and system blocking caused by resource competition, and improves data warehousing stability. Meanwhile, standardized data access and task registration processes, combined with precise management of task states (waiting, execution, success, and failure), build a task life cycle system that is monitorable and traceable, support rapid fault positioning and retry, and improve processing reliability and operation efficiency. In addition, the data submission is decoupled from the actual warehousing execution, and the underlying database operation is encapsulated, which not only reduces the operation complexity and development cost of the data provider, but also unifies the data writing permission, thereby enhancing the security of the data warehouse.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, and more particularly, to a queuing and scheduling method for discrete data BACKGROUND

[0002] In modern enterprise informatization construction, as the core of enterprise data assets, the data warehouse bears the key mission of storing, integrating, analyzing and mining massive business data. Conventional business data can be stored in the data warehouse in real time or quasi-real time through mature data flow schemes. However, in complex business operations, there is also a large amount of "discrete data" storage demand, such as offline activity statistical data upload, external source data periodic or irregular reception, data batch initialization when switching business systems, and data batch correction when data correction, etc. This kind of data has diverse sources, different formats, and significant unpredictability in generation time and data volume, but it is also crucial for forming a complete business view and supporting accurate decision-making. Therefore, how to efficiently and reliably integrate these discrete data into a unified data warehouse system has become a technical problem to be solved in enterprise data management.

[0003] In order to cope with such needs, some existing technologies have emerged to unify the data import process. For example, patent application CN106933948A discloses a unified storage method for financial data, which can analyze financial data from different financial tools and save data information in the database to a plurality of preset data tables, realizing the unified collection of data from different sources. Another patent application CN114896197A discloses a unified storage method for multi-format documents, which can import documents of different formats such as Excel, Word or TXT into the system platform, and use a preset template and data normalization algorithm to analyze and process the data, extracting usable interface tables. These existing technical solutions to some extent solve the problem of diversified data sources and non-uniform formats, and through standardized analysis and conversion, achieve the purpose of importing heterogeneous data into the target database.

[0004] However, the above prior art solutions mainly focus on data level analysis and format unification, but ignore the task management and resource scheduling problems in the data warehousing process, so there are still significant defects in practical application. Specifically, these solutions do not provide a mechanism to effectively manage concurrent and burst warehousing requests. When multiple discrete data queuing and scheduling warehousing tasks are initiated at the same time, due to the lack of unified scheduling and resource control, each task will directly and disorderly compete for database write permissions, which can easily cause the database target table to be locked for a long time, which not only blocks the data processing and query of the downstream system, but also can significantly prolong the daily data task processing time, especially in the scenario of data traffic surge such as promotion activities, the system stability and data processing efficiency will face serious challenges. SUMMARY

[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application propose a queuing and scheduling warehousing method for discrete data to overcome the problem that the prior art solutions only focus on data format conversion, but ignore the task management and resource scheduling in the warehousing process, resulting in database table locking and system blocking.

[0006] According to one aspect of the present application, a queuing and scheduling warehousing method for discrete data is provided, comprising: data standardization and warehousing task registration on the original data source and the warehousing metadata to obtain a standardized data file and warehousing task registration information; based on the warehousing task registration information, data block persistence and task queue initialization are performed on the standardized data file to obtain a waiting state task pool, the state of each warehousing task in the waiting state task pool is waiting; based on system configuration, task scheduling and resource locking are performed on the waiting state task pool to obtain dispatched tasks; based on the dispatched tasks, data acquisition and analysis verification are performed to obtain processed data, and the state of the warehousing task is updated to warehousing execution; based on the target database information in the dispatched tasks, the processed data is written into the target database.

[0007] In one possible implementation, data standardization and warehousing task registration on the original data source and the warehousing metadata to obtain a standardized data file and warehousing task registration information, comprising: template application and format conversion on the original data source to obtain the standardized data file; file fingerprint generation on the standardized data file to obtain a file hash value; metadata verification and fusion on the warehousing metadata to obtain verified metadata; fusion of the file hash value and the verified metadata to obtain the warehousing task registration information.

[0008] In one possible implementation, applying templates and converting formats to the original data source to obtain the standardized data file includes: performing encoding unification processing on the original data source to obtain encoded unified data; performing structured conversion on the encoded unified data to obtain a structured data stream; and performing content compliance processing on the structured data stream to obtain the standardized data file.

[0009] In one possible implementation, based on the inbound task registration information, standardized data files are segmented and persisted, and task queues are initialized to obtain a waiting task pool. The waiting task pool contains inbound tasks in a waiting state. This includes: defining data packets and logically segmenting standardized data files based on the inbound task registration information to obtain data packet numbers and a list of temporary data block definitions; instantiating the definitions of each temporary data block in the temporary data block definition list based on the metadata of the inbound task registration information to obtain an initialized task group; and activating the initialized task group in batches to obtain the waiting task pool.

[0010] In one possible implementation, based on the metadata of the inbound task registration information, the definitions of each temporary data block in the temporary data block definition list are instantiated to obtain an initialized task group. This includes: performing block iteration and physical generation on the definitions of each temporary data block in the temporary data block definition list to obtain a physical data block temporary file; uploading the physical data block temporary file to a static object storage service to obtain a data block object URL; and generating the initialized task group based on the metadata, data packet number, and data block object URL of the inbound task registration information.

[0011] In one possible implementation, based on system configuration, task scheduling and resource locking are performed on the waiting task pool to obtain dispatched tasks, including: based on system configuration, selecting the best task from the waiting tasks based on system load and priority to obtain a sorted candidate task list; and performing competitive locking iteration on the sorted candidate task list to obtain the dispatched tasks.

[0012] In one possible implementation, the system configuration includes a global system state and a basic system configuration. The global system state includes the number of currently active tasks, and the basic system configuration includes parallelism limits and priority rules.

[0013] Compared with existing technologies, the queuing and scheduling method for discrete data storage provided in this application transforms high-concurrency database writes into ordered queuing by establishing task queues and intelligent scheduling mechanisms. This effectively avoids table locks and system blocking caused by resource contention, improving data storage stability. Simultaneously, the standardized data access and task registration process, combined with precise management of task status (waiting, executing, successful, failed), constructs a monitorable and traceable task lifecycle system, supporting rapid fault location and retries, improving processing reliability and operational efficiency. Furthermore, decoupling data submission from actual storage execution and encapsulating underlying database operations not only reduces the operational complexity and development costs for data providers but also uniformly reclaims data write permissions, enhancing data warehouse security. Attached Figure Description

[0014] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 The illustration shows a schematic flowchart of a method for queuing, scheduling, and storing discrete data according to an embodiment of this application.

[0016] Figure 2 The figure shows a schematic flowchart of step S1 in the queuing and scheduling method for storing discrete data according to an embodiment of this application.

[0017] Figure 3 The figure shows a schematic flowchart of step S11 in the queuing and scheduling method for storing discrete data according to an embodiment of this application.

[0018] Figure 4 The figure shows a schematic flowchart of step S2 in the queuing and scheduling method for storing discrete data according to an embodiment of this application.

[0019] Figure 5 The illustration shows a schematic flowchart of step S22 in the queuing and scheduling method for storing discrete data according to an embodiment of this application.

[0020] Figure 6 The figure shows a schematic flowchart of step S3 in the queuing and scheduling method for storing discrete data according to an embodiment of this application. Detailed Implementation

[0021] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0022] Figure 1 The illustration shows a schematic flowchart of a queuing and scheduling method for storing discrete data according to an embodiment of this application. Figure 1 As shown, this application provides a method for queuing and scheduling discrete data for database entry, including: S1, standardizing the original data source and database entry metadata and registering database entry tasks to obtain standardized data files and database entry task registration information; S2, based on the database entry task registration information, performing data block persistence and task queue initialization on the standardized data files to obtain a waiting task pool, wherein the status of each database entry task in the waiting task pool is "waiting"; S3, based on system configuration, performing task scheduling and resource locking on the waiting task pool to obtain dispatched tasks; S4, based on the dispatched tasks, performing data acquisition, parsing and verification to obtain processed data, and updating the status of the database entry tasks to "data entry in progress"; S5, based on the target database information in the dispatched tasks, writing the processed data into the target database.

[0023] For example, in step S1, the original data source and import metadata are standardized and the import task is registered to obtain standardized data files and import task registration information. It should be understood that discrete data sources are diverse and have different formats. If not standardized, subsequent data parsing, verification, and writing processes will require customized development for each data source, which is not only inefficient but also makes unified monitoring and management difficult. Therefore, this application requires all data sources to follow a consistent data file format and registration information template by agreeing on an external data import agreement; it also specifies detailed data registration information requirements, including necessary metadata and descriptive information, to enhance data traceability and consistency, thereby establishing a stable and standardized entry point for the entire import process.

[0024] In one embodiment, such as Figure 2 As shown, data standardization and entry task registration are performed on the original data source and entry metadata to obtain standardized data files and entry task registration information, including: S11, applying templates and converting formats on the original data source to obtain the standardized data file; S12, generating file fingerprints on the standardized data file to obtain file hash values; S13, verifying and fusing the entry metadata to obtain verified metadata; S14, fusing the file hash values ​​and verified metadata to obtain the entry task registration information.

[0025] Specifically, firstly, such as Figure 3As shown, the process of applying templates and converting formats to obtain the standardized data file from the original data source includes: S111, performing encoding unification processing on the original data source to obtain encoded data, that is, converting various possible source file encodings (such as GBK, ANSI) into the system's internal standard encoding format (such as UTF-8) to avoid garbled characters in subsequent processing; S112, performing structured conversion on the encoded data to obtain a structured data stream, that is, parsing unstructured or semi-structured file content (such as Excel worksheets) into a standardized data stream (such as JSON arrays or CSV format data streams) that is easy for the system to process; S113, performing content compliance processing on the structured data stream to obtain the standardized data file, that is, according to the preset data template or rules, verifying and cleaning the field type, format, and business logic of each record in the data stream to ensure data quality.

[0026] After generating standardized data files, the system generates file fingerprints using hash algorithms such as MD5 or SHA256 to calculate a unique hash value for the complete file content. This hash value is used to verify data integrity during subsequent transmission and processing, preventing data tampering. Simultaneously, the system verifies and integrates the user-submitted metadata for database entry. This includes checking the availability of the target database connection, the existence of the target data table, and whether the specified user has write permissions. Metadata that passes verification forms verified metadata. Finally, the system integrates the generated file hash value and the verified metadata into a complete, structured information body, which serves as the final database entry task registration information.

[0027] To clarify how data standardization and data entry task registration are performed, a specific implementation example is provided. In this example, an e-commerce agency needs to input an Excel file named "Offline Activity Sales Records.xlsx" into a data warehouse. In this scenario, the process of applying templates and converting formats on the original data source to obtain the standardized data file is as follows: First, when applying templates and converting formats on the original data source, the system receives the "Offline Activity Sales Records.xlsx" file encoded in GBK. Its content is then uniformly converted to UTF-8 encoding. Next, the Excel worksheet is parsed and converted into an internal standard structured data stream. Based on a preset template, the system checks whether the sales time field is in YYYY-MM-DD HH:mm:ss format and whether the sales quantity is a positive integer. After processing, a standardized data file, standard_sales_data_20231027.csv, is generated. Subsequently, the system generates a file fingerprint for the newly generated CSV file, using the MD5 algorithm to calculate its content and obtain a file hash value, such as e10adc3949ba59abbe56e057f20f883e. Simultaneously, the system receives the data entry metadata provided by the data entry personnel, including information such as the target database being MySQL and the target table being fact_offline_sales, and performs metadata verification. After confirming that the database connection is normal, the table exists, and user permissions are correct, verified metadata is generated. Finally, the system combines the file hash value e10adc3949ba59abbe56e057f20f883e with the verified metadata into a complete JSON object. This object serves as the unique credential for this data entry request, i.e., the data entry task registration information, and is stored in the task registration table of the scheduling system. This completes the entire data standardization and data entry task registration process.

[0028] In a preferred embodiment, the inbound metadata may further include one or more callback URLs. These callback URLs point to a network interface provided by the data user or downstream system for receiving status notifications after the inbound task is completed (whether successfully or not).

[0029] For example, in step S2, based on the database entry task registration information, the standardized data file is segmented and persisted, and a task queue is initialized to obtain a waiting task pool. The status of each database entry task in the waiting task pool is "waiting." It should be understood that when a discrete data file reaches hundreds of megabytes or even gigabytes in size, processing it as an indivisible whole will face many problems. First, the single processing cycle is lengthy—from data download, memory loading, parsing and verification to final writing to the database, the entire process is time-consuming and continuously occupies the computing and memory resources of a single execution unit during this period. More seriously, this monolithic processing mode has extremely poor fault tolerance. Once an anomaly occurs in the later stages of processing (such as network jitter or database connection timeout), the entire task will fail, all previously completed work will be wasted, and it must be re-executed, resulting in huge resource waste and time delays. Furthermore, the large task granularity also limits fine-grained resource scheduling. The system cannot easily split tasks and allocate them to multiple idle executors for parallel processing based on the current load, thus failing to effectively utilize the computing power of the distributed system. Therefore, it is necessary to monitor and evaluate the original discrete data. When the data volume exceeds a preset threshold, it should be reasonably divided into multiple appropriately sized, independently processable data blocks, and each data block should be transformed into an independent, schedulable data ingestion task. In this way, a large and high-risk overall data ingestion request can be decomposed into a series of lightweight, independent, parallel executable subtasks with independent fault tolerance capabilities, laying a solid foundation for achieving efficient, stable, and scalable data ingestion.

[0030] In one embodiment, such as Figure 4 As shown, based on the inbound task registration information, standardized data files are segmented and persisted, and task queues are initialized to obtain a waiting task pool. The waiting task pool contains inbound tasks in a waiting state. The process includes: S21, defining data packets and logically segmenting standardized data files based on the inbound task registration information to obtain data packet numbers and a list of temporary data block definitions; S22, instantiating the definitions of each temporary data block in the temporary data block definition list based on the metadata of the inbound task registration information to obtain an initialized task group; and S23, activating the initialized task group in batches to obtain the waiting task pool.

[0031] Specifically, firstly, based on the registration information of the data entry task, the standardized data file is defined into data packets and logical blocks to obtain a list of data packet numbers and temporary data block definitions. That is, the system reads the metadata of the standardized data file to obtain its total size and compares it with the preset single data block size limit in the system configuration. If the file size exceeds the limit, the system calculates the number of data blocks that need to be split according to the limit and generates a unified data packet number for all future data blocks originating from the same file. This number serves as their identifier of shared origin. Subsequently, the system generates a logical block definition list, which details the start and end positions of each data block in the original standardized data file (e.g., byte offsets or line number ranges), but at this point, physical file splitting has not yet been performed.

[0032] Next, based on the metadata of the inbound task registration information, the definitions of each temporary data block in the temporary data block definition list are instantiated to obtain an initialized task group. In one embodiment, such as... Figure 5 As shown, based on the metadata of the inbound task registration information, the definitions of each temporary data block in the temporary data block definition list are instantiated to obtain an initialized task group. This includes: S221, performing block iteration and physical generation on the definitions of each temporary data block in the temporary data block definition list to obtain a physical data block temporary file. Here, the system traverses the temporary data block definition list, accurately reads the corresponding data fragment from the original standardized data file according to the start and end positions described in each definition, and writes this fragment into a new, independent physical data block temporary file. After this operation, a large standardized data file is physically divided into multiple small, content-independent data block files; S222, uploading the physical data block temporary file to the static object storage service to obtain the data block object URL. Here, the data is separated from the computation through uploading, realizing data persistence and location independence. After successful uploading, the static object storage service returns a globally unique, network-accessible data block object URL for each data block file. This URL becomes the unique data pointer of the data block in the system; S223, generating an initialized task group based on the metadata, data packet number, and data block object URL of the inbound task registration information. Specifically, the system creates a brand new task record in the inbound scheduling system's task table for each data block object URL. This record inherits all metadata from the original inbound task registration information, such as target database information, priority, callback address, etc., and is additionally associated with a global data packet number and its unique data block object URL. At this time, the state of these newly created task records is uniformly set to initialization.

[0033] Finally, the initialized task groups are activated in batches to obtain the waiting task pool. Specifically, all tasks in the task table that are in the initialized state are periodically scanned. For each scanned task, the system performs a final verification, such as verifying whether its corresponding data block object URL is valid and accessible. After all verifications are successful, the system updates the status of these tasks from initialized to waiting in batches and atomically.

[0034] To clarify how data block persistence and task queue initialization are performed, a specific implementation example is provided. In this example, the system obtains a standardized data file, standard_sales_data_20231027.csv, with a size of 500MB. For this standardized data file, the system first defines data packets and performs logical block partitioning: the system detects that the 500MB file size exceeds the 200MB limit for single processing, therefore it decides to divide it into three logical data blocks (the first two 200MB, the last 100MB), and generates a unified data packet number for this batch of data blocks, such as PKG_20231027_001. Next, the system instantiates the task: based on the definition of logical blocks, it iteratively reads data from the source file, generating three physical temporary data block files: block_1.csv, block_2.csv, and block_3.csv. Then, these three physical data block files are uploaded to a static object storage service (such as Alibaba Cloud OSS), and a unique data block object URL is generated for each file, e.g., oss: / / my-bucket / PKG_20231027_001 / block_1.csv. Finally, based on the original inbound task registration information, data packet number, and object URL of each data block, the system creates three new task records in the inbound task table and sets the status of these tasks to initialized. These three records together constitute the initialized task group. Finally, batch task activation is performed: the system's task queue script periodically scans these three tasks with the initialized status, performs final confirmation, and batch updates their status to waiting. At this point, these three independent inbound tasks officially enter the waiting task pool, ready to receive the next step of scheduling and execution.

[0035] For example, in step S3, based on system configuration, task scheduling and resource locking are performed on the waiting task pool to obtain dispatched tasks. It should be understood that in real-world applications, the waiting task pool may accumulate a large number of inbound tasks from different business lines and with different priorities within a short period. If execution units are allowed to freely and indiscriminately grab tasks from the task pool and execute them, it will inevitably lead to excessive concurrent access to downstream shared resources, especially the target database. This disordered concurrent writing will trigger intense resource contention, easily causing the target database table to be locked for a long time, thereby blocking other normal business queries and data processing flows. In severe cases, it may even lead to the exhaustion of the database connection pool or deadlock, causing a sharp decline in the availability of the entire data service. Furthermore, the lack of a scheduling mechanism also means that the system cannot distinguish the urgency of tasks. A high-priority urgent data correction task may not be processed in a timely manner because it is scheduled after a low-priority task importing massive amounts of historical data, failing to meet the timeliness requirements of the business. Therefore, this application further introduces a centralized, rule-based intelligent scheduling and resource management mechanism, which transforms the chaotic task execution mode into an orderly task dispatch mode that conforms to preset strategies by real-time monitoring and control of the system's global load and specific resource load, thereby ensuring the stability, efficiency and predictability of the data entry process.

[0036] In one embodiment, such as Figure 6 As shown, based on the system configuration, task scheduling and resource locking are performed on the waiting task pool to obtain dispatched tasks, including: S31, based on the system configuration, the best task selection for waiting tasks based on system load and priority is performed to obtain a sorted candidate task list; S32, competitive locking iteration is performed on the sorted candidate task list to obtain the dispatched tasks.

[0037] Specifically, the core of the system's task scheduling and resource locking mechanism for the waiting task pool, based on system configuration, is the scheduler. It makes decisions based on a series of predefined system configurations. These system configurations form the rule basis for the entire scheduling behavior and mainly include: global system state and basic system configuration. First, the global system state refers to dynamically changing data reflecting the current real-time load during system operation. This includes the number of currently active tasks, accurately recording the total number of tasks in the system at any given moment that are in a download, data entry, or write state. This dynamic indicator provides the scheduler with the immediate feedback needed for decision-making, enabling it to perceive the true load of the system. Second, the basic system configuration refers to a set of static rules pre-set by the system administrator and relatively fixed over a period of time. This includes parallelism limits and priority rules. The parallelism limit defines the upper limit of the total number of concurrent tasks allowed to be executed in the entire discrete data entry system at the same time. It acts as a global throttling valve, designed to protect the entire system and its dependent downstream services (such as databases and network bandwidth) from overload due to instantaneous task surges. Priority rules define how the scheduler should determine the execution order of multiple tasks when they simultaneously meet the execution conditions. These rules can be based on various factors, such as static priority levels set for different business data sources, or combined with task submission timestamps to implement more complex scheduling algorithms such as First-In-First-Out (FIFO) or weighted round-robin.

[0038] Specifically, at the start of a scheduling cycle, the scheduler's primary task is to assess system capacity. It first reads the number of currently active tasks from the global system state and then retrieves the preset parallelism limit from the basic system configuration. By subtracting these two values ​​(parallelism limit minus the number of currently active tasks), the scheduler can accurately calculate how many available concurrent execution slots remain in the system. If the result is less than or equal to zero, it means the system is already at full or overloaded, and the scheduler will abort the current scheduling cycle, waiting for the next cycle to attempt again, thus avoiding further stress on the system.

[0039] If available concurrent slots are found, the scheduler proceeds to the candidate task filtering and sorting phase. It queries the task management database to retrieve all tasks in a waiting state, forming an initial candidate task set. Next, the scheduler applies priority rules defined in the underlying system configuration to sort this candidate task set. For example, if the rules are based on static priority and submission time, the scheduler will first sort them in descending order of priority level, and then, if priorities are the same, in ascending order of task creation time. Through this sorting operation, the scheduler obtains an ordered queue of candidate tasks that reflects business urgency and fairness. Subsequently, starting from the head of this ordered queue, the scheduler extracts the same number of tasks as the number of available concurrent slots, forming the final candidate task list to be dispatched.

[0040] Finally, the scheduler starts from the head of the sorted candidate task queue and checks each candidate task one by one to see if it meets the dispatch conditions. For each candidate task, the scheduler performs a series of verification checks. First is a global concurrency check: the scheduler compares the total number of currently executing tasks with the global maximum concurrency in the system configuration. If the former has reached or exceeded the latter, it means that the overall system load is saturated, and the scheduler will immediately abort the current round of scheduling and wait for the next scheduling cycle. If there is still spare global concurrency capacity, the scheduler will perform a target resource concurrency check. It parses the metadata of the current candidate task to determine the target resource to which it will be written, then queries the current concurrent execution count of that specific resource and compares it with the concurrency limit set for that resource in the system configuration. If the concurrency count of that resource has also reached the limit, this candidate task cannot be dispatched, and the scheduler will skip it and continue to check the next candidate task in the queue.

[0041] A candidate task is considered dispatchable only if it passes both the global concurrency check and the target resource concurrency check. Once a task is determined to be dispatchable, the scheduler immediately performs a resource locking operation. This resource locking is not a traditional database lock, but a logical lock, accomplished through two atomic operations. The first operation updates the task status; the scheduler immediately updates the task's status in the task database from "waiting" to "dispatched." This status change is exclusive, ensuring that in a distributed environment, no other scheduler instance will dispatch the same task repeatedly. The second operation updates the system load count; the scheduler atomically increments both the global concurrent execution count and the corresponding target resource's concurrent execution count. These two operations together complete the logical locking of an execution slot. After locking, the scheduler pushes this task record, containing complete metadata and the data block URL—the dispatched task—to a message queue available for consumption by the executor cluster, or directly assigns it to an idle executor through other mechanisms. The scheduler will repeat this decision and dispatch cycle until the candidate task queue has been checked, or the system's concurrency capacity (whether global or specific resource-based) is completely full, then end the current round of scheduling and wait for the next cycle.

[0042] To clarify how task scheduling and resource locking are performed, a specific implementation example is provided. In this example, the system has a waiting task pool containing three waiting tasks. First, the scheduler performs optimal task selection: it obtains the global system status and learns that the number of active tasks is currently 15. Simultaneously, it reads the basic system configuration and learns that the parallelism limit is 20. Based on this, it calculates that the system currently has 20-15=5 available concurrent slots. Next, the scheduler scans the waiting task pool and sorts them according to priority rules. Since the three tasks in this example are marked as "high" priority because their target table `fact_offline_sales`, they are prioritized before other ordinary priority tasks. Because the available slots (5) are greater than the number of high-priority tasks to be processed (3), all three tasks are selected, forming a sorted candidate task list. Subsequently, the competitive locking iteration phase begins: the scheduler traverses this list containing the three tasks and attempts to apply a task lock to the database for each task (e.g., by updating a specific field in the task record and utilizing the atomicity of the database). For the three tasks in this example, the scheduler successfully acquired task locks for all of them. These three successfully locked tasks are now dispatched tasks, officially removed from the waiting queue, and ready to be assigned to idle executor processes, thus completing the entire scheduling and locking process.

[0043] For example, in step S4, data acquisition and parsing verification are performed based on the dispatched task to obtain the processed data, and the status of the inbound task is updated to inbound execution. It should be understood that a dispatched task is essentially a set of instructions containing metadata and data location pointers (URLs). While the data blocks it points to are logically ready, they are still physically stored in a remote static object storage service, and the eventual consistency and compliance of their content have not yet been verified at the execution end. Directly using this remote data for database write operations carries significant risks: the data may be corrupted during storage or network transmission; or, although verification was performed in the initial standardization phase, unforeseen problems may be introduced during sharding or storage. Without the rigorous verification in this step, directly attempting to write potentially problematic data to the database can easily lead to write exceptions at the database level. This not only causes the entire task to fail but also results in expensive operations such as database transaction rollback, wasting valuable database connections and computing resources. Therefore, this application further performs data acquisition and parsing verification based on the dispatched task to obtain the processed data.

[0044] Specifically, the process of acquiring, parsing, and verifying data based on dispatched tasks to obtain processed data is handled by an independent executor. First, the executor retrieves a dispatched task from the task dispatch channel (such as a message queue). This task record contains all the information required for execution. The executor's primary action is to immediately update the task's status in the central task database. Specifically, the executor changes the task status from "dispatched" to "downloading," explicitly announcing to the monitoring system that the task has begun its first physical operation in its lifecycle: data acquisition. Subsequently, the executor parses the task metadata, extracts the data block object URL, and uses this URL to request a static object storage service to download the corresponding physical data block file completely to the executor's local temporary storage space or directly into memory. During this process, the executor implements a strict error handling mechanism. If the download fails (e.g., due to network interruption, invalid URL, or non-existent file), the task status is immediately updated to "failed," and a detailed reason for the failure is recorded. The task's execution flow is then terminated.

[0045] Once the data block file is successfully downloaded, the executor updates the task status again, changing it from "Downloading" to "Download Successful," signifying the successful completion of the data acquisition phase. Immediately afterwards, the executor updates the task status to "Database Ingestion." This status announces to the entire system that the task has moved beyond the preparation phase and officially entered the computationally intensive parsing and verification stage. All work until the data is written to the database will be completed in this state. Afterward, the executor begins parsing and verifying the downloaded data block file. It reads the file content according to the data format defined in the task metadata (such as the delimiter for CSV files, the structure for JSON files, etc.) and parses it into a structured data representation in memory, such as a list of records. The parsing process itself is a preliminary form of structured verification.

[0046] After the data is successfully loaded into memory, the executor initiates a more in-depth content validation loop. It iterates through every data record in memory and performs rigorous compliance checks on each field within the record, based on the data import protocol and target table structure definition carried in the task metadata. These checks include, but are not limited to: data type validation, ensuring that fields that should be numeric can be correctly converted to numbers, and that fields that should be date conform to the preset format; non-null constraint validation, checking that all fields requiring non-null constraints actually contain valid values; length and range validation, confirming that the string length or numerical size is within the allowed range; and more complex business logic validation. Throughout the validation process, if any field in any record is found to be inconsistent with the preset specifications, the executor will immediately interrupt processing and mark the entire data block as validation failed. At this point, the final status of the task will be set to "failed," accompanied by precise error information, such as line 105, where the value N / A of the order_amount field cannot be converted to a numeric type. Only when all records in a data block have passed all parsing and verification checks without any errors can this clean and structured data set loaded into the executor's memory be finally confirmed as processed data.

[0047] In a preferred embodiment, while parsing and verifying the data, the system can also perform preliminary data feature analysis, such as analyzing the cardinality or data distribution of fields. Based on these analysis results, the system can automatically generate index optimization suggestions for the target database table and record them in the task log, thereby helping to improve the query performance of downstream applications after the data is stored.

[0048] To clarify how data acquisition, parsing, and verification are performed, a specific implementation example is provided. In this example, the executor receives a dispatched task to process the block_1.csv data block. The executor begins the actual data processing. First, it updates the task status to "Downloading," and then downloads the block_1.csv file from the static object storage service based on the data block object URL in the task information. Upon successful download, the task status is updated to "Download Successful." Next, the executor updates the task status to "Ingestion in Progress" and begins parsing the block_1.csv file. During this process, it verifies that the file's headers and the number of fields match the definitions in the metadata, and performs a final format check on the data content. All verified data is loaded into the executor's memory, forming the processed data. If any parsing or verification failure occurs during this process, such as an incorrect field format, the task status is immediately set to "Failed," and a detailed error log is recorded, thus terminating the subsequent process of the task.

[0049] For example, in step S5, the processed data is written to the target database based on the target database information in the dispatched tasks. It should be understood that by writing the processed data to the target database, the data is transformed from a temporary, volatile executor memory state to a permanent, stable database storage state.

[0050] Specifically, the executor parses detailed target database information from the metadata of dispatched tasks. This includes the database type, host address, port, database name, and the username and password used for authentication. To improve performance and avoid the high overhead of frequently creating and destroying database connections, the executor typically does not create a new connection for each task. Instead, it obtains an available connection from a pre-configured database connection pool. This connection reuse mechanism significantly reduces the latency of establishing communication with the database.

[0051] After successfully acquiring a database connection, the executor immediately initiates a database transaction. This is a crucial operation to ensure the atomicity of data writes. By sending a transaction start instruction (such as BEGIN TRANSACTION in SQL), the executor declares to the database that subsequent write operations will be executed as an indivisible logical unit. This means that all data write operations are invisible to other concurrent database sessions until this transaction is committed.

[0052] Next, the executor transforms the processed data in memory (typically a list of records or a similar data structure) into a format that the target database can efficiently receive. A simple but inefficient method is to iterate through each record and execute a separate SQL INSERT statement, but this incurs significant network round trips and database overhead. Therefore, the implementation of this application preferably employs a batch write strategy. The executor assembles multiple records in memory into a single, large database instruction. For example, it can generate an extended INSERT statement containing multiple value tuples (such as INSERT INTO table_name (col1, col2) VALUES (val1a, val2a), (val1b, val2b), ...), or format the data into the format required by a specific database batch loading tool (such as the data stream required by PostgreSQL's COPY command). This "batch processing" approach, which combines multiple writes into a single execution, significantly reduces communication overhead and the database's parsing and execution burden, thereby significantly improving the throughput of data ingestion.

[0053] After the data is assembled, the executor sends the batch processing instructions to the database through the established database connection. Upon receiving the instructions, the database attempts to write the data to the target table. The executor synchronously waits for the database's execution result. Two possible outcomes are possible. The first is successful execution, where the database returns a success status code indicating that all data has been written correctly. In this case, the executor immediately sends a "commit transaction" instruction (e.g., COMMIT) to the database. Once the COMMIT instruction executes successfully, all data modifications made in this transaction will be permanently applied and visible to other database sessions. The second outcome is execution failure, where the database returns an error message. This could be due to various reasons, such as a violation of unique key constraints, data type mismatch (although previous steps have validated, database-level constraints may still exist), or a problem with the database server itself. In this case, the executor immediately sends a "rollback transaction" instruction (e.g., ROLLBACK) to the database. The ROLLBACK instruction undoes all write operations already performed in this transaction, restoring the database to its state before the transaction began, thus ensuring data integrity.

[0054] Finally, regardless of whether the write operation is successful or not, the executor needs to complete status feedback and resource release. If the transaction is successfully committed, the executor will update the task's status in the central task database to "success". If the transaction is rolled back, the executor will update the task status to "failure" and record the specific error information received from the database in the task's log for subsequent troubleshooting. After completing the status update, the executor will release the database connection obtained from the connection pool, making it reusable by other tasks. At the same time, it will notify the scheduling center that the task has been completed, allowing the scheduler to release the concurrent resource locks occupied by the task (i.e., decrement the corresponding concurrency count), making execution space available for new waiting tasks.

[0055] To clarify how the data is written to the target database, a specific implementation example is provided. In this example, the executor has successfully processed the data block of block_1.csv and formed the processed data in memory. At this point, the executor begins parsing the target database information from the task metadata: the database type is MySQL, and the target table is fact_offline_sales. The executor obtains a connection to this MySQL instance from the connection pool. Then, it immediately updates the task's status in the central task database to "writing" and sends a START TRANSACTION instruction to the MySQL server to start a transaction. Next, the executor constructs a single, large INSERT SQL statement containing multiple sets of values ​​from all the sales records in block_1.csv in memory. It then executes this SQL statement through the established connection. The MySQL server successfully executes the insert operation and returns a success response. Upon receiving the success response, the executor immediately sends a COMMIT instruction to the MySQL server to commit the transaction, officially making this batch of data part of the fact_offline_sales table. Finally, the executor updates the task's final status to success and closes the task. If any problem occurs at any stage of the process, such as an INSERT statement failing due to a violation of a unique key constraint, the executor will send a ROLLBACK instruction to roll back the transaction, update the task status to failure, and record detailed database error information.

[0056] After a task's status is updated to success or failure, the executor checks if a callback URL is registered in the task's metadata. If so, the executor sends an HTTP (or HTTPS) request to all registered callback URLs. The request body typically contains a JSON object with the task ID, packet number, final status (success / failure), completion time, and error message upon failure. This callback mechanism allows downstream systems to be triggered automatically in real time. Furthermore, the system can monitor all inbound tasks under the same packet number. When all tasks belonging to the same packet have completed successfully, the system triggers a packet-level completion notification, ensuring that downstream systems only begin processing after the data is fully prepared, thus avoiding the risk of data inconsistency.

[0057] In a preferred embodiment, when a task's status is updated to failed, the system includes a failed task analysis and retry module. This module periodically scans tasks with a failed status, parses error information from their logs, and matches it against preset retry rules (such as database connection timeout or task execution timeout). If the cause of failure matches successfully, the system automatically updates the task's status from failed to retry and puts it back into the waiting task pool for rescheduling. For failures that do not meet the retry conditions, the task will remain in a failed state, awaiting manual intervention.

[0058] In summary, the discrete data queuing and scheduling method for data ingestion provided in this application transforms unordered, high-concurrency direct database write operations into ordered, controlled queuing execution by establishing task queues and introducing an intelligent scheduling mechanism. This effectively avoids database target table locking and system blocking issues caused by resource contention, significantly improving the stability of data ingestion and the overall availability of the system. Simultaneously, the standardized data access and task registration process, along with the clear definition and updating of the status of each stage of data processing (such as waiting, execution, success, and failure), constructs a complete, monitorable, and traceable task lifecycle management system. This ensures that every ingestion operation is traceable, and in the event of a failure, it allows for rapid location and effective retries, significantly improving the reliability of data processing and operational efficiency. Furthermore, decoupling data submission from actual ingestion execution and encapsulating the underlying database operations not only reduces the operational complexity and development costs for data providers but also uniformly reclaims data write permissions, enhancing the security of the data warehouse. Through a callback notification mechanism, real-time feedback on task status and process automation were achieved, constructing an end-to-end automated data processing pipeline. This significantly improved the response speed and overall efficiency of data processing, forming an effective closed-loop management system. The intelligent failure retry mechanism significantly enhanced the system's fault tolerance and reliability, enabling it to autonomously recover from temporary failures, reducing unnecessary manual intervention, and ensuring a high final success rate for data entry.

[0059] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the queuing, scheduling, and storage method for discrete data provided in the above embodiments.

[0060] In this application, the system, computer-readable storage medium, or computer program product provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0061] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments.

[0062] The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, with reference to each other for similar or identical parts. Each embodiment focuses on highlighting its differences from other embodiments.

Claims

1. A method for queuing and scheduling discrete data for data storage, characterized in that, Includes the following steps: Data standardization and inbound task registration are performed on the original data source and inbound metadata to obtain standardized data files and inbound task registration information; Based on the registration information of the inbound task, the standardized data file is segmented and persisted in blocks and the task queue is initialized to obtain a waiting task pool. The status of each inbound task in the waiting task pool is waiting. Based on system configuration, task scheduling and resource locking are performed on the waiting task pool to obtain dispatched tasks. Based on the dispatched tasks, data is acquired, parsed, and verified to obtain the processed data, and the status of the inbound task is updated to inbound execution. Based on the target database information in the dispatched tasks, the processed data is written into the target database; Specifically, based on the registration information of the inbound tasks, standardized data files are segmented and persisted, and task queues are initialized to obtain a waiting task pool. The status of each inbound task in the waiting task pool is "waiting," including: Based on the inbound task registration information, standardized data files are defined and logically divided into data packets to obtain a list of data packet numbers and temporary data block definitions. Based on the metadata of the inbound task registration information, the definition of each temporary data block in the definition list of temporary data blocks is instantiated to obtain an initialized task group. Batch task activation is performed on the initialized task group to obtain the waiting task pool; Specifically, based on the metadata of the inbound task registration information, the definitions of each temporary data block in the temporary data block definition list are instantiated to obtain an initialized task group. This includes: performing block iteration and physical generation on the definitions of each temporary data block in the temporary data block definition list to obtain a physical data block temporary file; uploading the physical data block temporary file to a static object storage service to obtain a data block object URL; and generating an initialized task group based on the metadata, data packet number, and data block object URL of the inbound task registration information.

2. The method for queuing, scheduling, and storing discrete data according to claim 1, characterized in that, Data standardization and inbound task registration are performed on the original data source and inbound metadata to obtain standardized data files and inbound task registration information, including: The original data source is subjected to template application and format conversion to obtain the standardized data file; File fingerprints are generated from standardized data files to obtain file hash values; Perform metadata verification and fusion on the incoming metadata to obtain verified metadata; The file hash value and verified metadata are fused to obtain the registration information for the inbound task.

3. The method for queuing, scheduling, and storing discrete data according to claim 2, characterized in that, The standardized data file is obtained by applying a template and converting the format of the original data source, including: The original data source is encoded to obtain the encoded data. The data after unified encoding is transformed into a structured data stream. The structured data stream is processed to achieve content compliance in order to obtain the standardized data file.

4. The method for queuing, scheduling, and storing discrete data according to claim 1, characterized in that, Based on system configuration, task scheduling and resource locking are performed on the waiting task pool to obtain dispatched tasks, including: Based on system configuration, the best task for waiting tasks is selected based on system load and priority to obtain a sorted list of candidate tasks. Competitive locking iteration is performed on the sorted candidate task list to obtain the dispatched tasks.

5. The method for queuing, scheduling, and storing discrete data according to claim 4, characterized in that, The system configuration includes the global system status and the basic system configuration. The global system status includes the number of currently active tasks, and the basic system configuration includes parallelism limits and priority rules.

Images