Data processing method and device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment, storage medium and program product. The method comprises: generating a task instance identifier of a target task according to a task type and a scheduling time window of the target task. Attempting to insert a task instance corresponding to the target task into a database with the task instance identifier as a database primary key. If the attempt to insert the task instance into the database fails, terminating task processing and outputting a prompt information, the prompt information being used to represent that the task instance already exists within the scheduling time window. If the attempt to insert the task instance into the database succeeds, triggering an execution engine to execute the task instance. The method is used to reduce the waste of system resources of a distributed system.

Description

Technical Field

[0001] This application relates to the field of distributed technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In a distributed scheduling architecture that eliminates the registry center (such as ZooKeeper), there is a business requirement for multiple master nodes to concurrently write task instances to the same task. Currently, when multiple master nodes compete to schedule the same task, there is a concurrency window in key stages such as task status determination, instance creation, and resource allocation. Within this window, each master node may make independent scheduling decisions based on local state information, leading to anomalies such as the same task being repeatedly generated into multiple instances or the same task instance being triggered multiple times. This can result in problems such as wasted system resources, disordered business logic, and data consistency risks.

[0003] Therefore, how to reduce the waste of system resources in distributed systems is an urgent problem to be solved. Summary of the Invention

[0004] This application provides data processing methods, apparatus, electronic devices, storage media, and program products to reduce the waste of system resources in distributed systems.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Based on the task type and scheduling time window of the target task, generate the task instance identifier of the target task;

[0007] An attempt was made to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key;

[0008] If the attempt to insert the task instance into the database fails, the task processing is terminated and a prompt message is output, which indicates that the task instance already exists within the scheduling time window.

[0009] If the attempt to insert the task instance into the database is successful, the execution engine will be triggered to execute the task instance.

[0010] Optionally, generating the task instance identifier of the target task based on the task type and scheduling time window of the target task includes:

[0011] Determine the task type identifier based on the task type of the target task;

[0012] The scheduling time window identifier is determined based on the aforementioned scheduling time window;

[0013] The task type identifier and the scheduling time window identifier are combined to generate the task instance identifier.

[0014] Optionally, the step of attempting to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key includes:

[0015] Obtain the globally unique identifier of the target task in the distributed system;

[0016] Using the task instance identifier as the database primary key, a record of the task instance is inserted into the task instance table of the database to attempt to insert the task instance corresponding to the target task into the database. The record of the task instance includes the task instance identifier, the scheduling time window, and the globally unique identifier. The task instance table includes a uniqueness constraint on the task instance identifier.

[0017] Optionally, if the attempt to insert the task instance into the database fails, the task processing is terminated and a prompt message is output, including:

[0018] If a primary key conflict exception is received from the database, it is determined that the attempt to insert the task instance into the database has failed.

[0019] Terminate the task processing and output the prompt message.

[0020] Optionally, if the attempt to insert the task instance into the database is successful, the execution engine is triggered to execute the task instance, including:

[0021] If no primary key conflict exception is received from the database, it is determined that the attempt to insert the task instance into the database was successful.

[0022] The execution engine is triggered to execute the task instance.

[0023] Optional, also includes:

[0024] The scheduling time window granularity is determined based on the task type of the target task;

[0025] The scheduling time window for the target task is determined based on the granularity of the scheduling time window.

[0026] Optional, also includes:

[0027] The historical task scheduling records within a preset time period are cached, and the historical task scheduling records include the historical task type and the historical task scheduling time window;

[0028] After generating the task instance identifier, the task instance identifier is matched with the historical task scheduling record;

[0029] If the historical task scheduling record does not contain a target record that conflicts with the task instance identifier, then the task instance corresponding to the target task will be inserted into the database using the task instance identifier as the database primary key.

[0030] Secondly, embodiments of this application provide a data processing apparatus, including:

[0031] The processing module is used to generate a task instance identifier for the target task based on the task type and scheduling time window of the target task.

[0032] The first control module is used to attempt to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key;

[0033] The second control module is used to terminate task processing and output a prompt message if the attempt to insert the task instance into the database fails. The prompt message is used to indicate that the task instance already exists within the scheduling time window.

[0034] The third control module is used to trigger the execution engine to execute the task instance if the attempt to insert the task instance into the database is successful.

[0035] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0036] The memory stores computer-executed instructions;

[0037] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0039] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0040] The data processing method, apparatus, electronic device, storage medium, and program product provided in this application generate a task instance identifier for the target task based on the task type and scheduling time window of the target task. Using the task instance identifier as the database primary key, an attempt is made to insert the task instance corresponding to the target task into the database. If the attempt to insert the task instance into the database fails, the task processing is terminated, and a prompt message indicating that the task instance already exists within the scheduling time window is output. If the attempt to insert the task instance into the database succeeds, the execution engine is triggered to execute the task instance. Compared with existing technologies, this method can ensure the global uniqueness of task instances within the same scheduling batch by generating a unique task instance identifier and utilizing the uniqueness constraint of the database primary key, thus guaranteeing the consistency of business logic and data. Furthermore, this method eliminates the need for querying whether task instances are duplicated, reduces the write path, improves system response speed, and does not rely on external components such as ZooKeeper, distributed locks, and message queues, thereby reducing system resource waste and improving overall system performance and stability. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0043] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0044] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0045] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0046] Figure 5 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0047] Figure 6 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0048] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0050] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0052] The distributed scheduling system involved in this application is widely used in scenarios requiring high-concurrency task processing, such as big data processing platforms, task distribution in microservice architectures, real-time computing frameworks, and transaction coordination in distributed databases. In these scenarios, the distributed scheduling system needs to handle the generation and scheduling of massive numbers of task instances simultaneously, and these task instances must be globally unique to avoid duplicate execution. For example, in a real-time data analysis system, millions of task instances may need to be generated every minute to process real-time data streams; in a distributed database, transaction scheduling needs to ensure that the same transaction is not committed repeatedly across multiple nodes.

[0053] Traditional solutions manage the uniqueness of task instances through a centralized registry (such as ZooKeeper) or a globally distributed lock. However, such solutions suffer from performance bottlenecks in large-scale concurrency scenarios and their reliance on external components (such as databases or message queues) increases architectural complexity. Furthermore, in a decentralized architecture, multiple scheduling nodes may simultaneously attempt to generate the same task instance. If a "query first, write later" strategy is adopted, network latency or lock contention can lead to a "concurrency window" problem, causing duplicate task instances to be generated, resulting in data inconsistency or resource waste.

[0054] In view of this, this application provides a data processing method that generates a task instance identifier for the target task based on the task type and scheduling time window of the target task. Using the task instance identifier as the database primary key, an attempt is made to insert the corresponding task instance into the database. If the attempt to insert the task instance into the database fails, the task processing is terminated and a prompt message indicating that the task instance already exists within the scheduling time window is output. If the attempt to insert the task instance into the database succeeds, the execution engine is triggered to execute the task instance. Compared with existing technologies, this method can ensure the global uniqueness of task instances within the same scheduling batch by generating a unique task instance identifier and utilizing the uniqueness constraint of the database primary key, thus guaranteeing the consistency of business logic and data. Furthermore, this method eliminates the need for querying whether task instances are duplicated, reduces the write path, improves system response speed, and does not rely on external components such as ZooKeeper, distributed locks, and message queues, thereby reducing system resource waste and improving overall system performance and stability.

[0055] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0056] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0057] S101. Generate a task instance identifier for the target task based on the task type and scheduling time window of the target task.

[0058] The target task refers to the specific business processing unit that needs to be scheduled and executed, such as data cleaning tasks, order processing tasks, user analysis tasks, etc.

[0059] Task type refers to the classification identifier used to distinguish the target task. It can include types categorized by time period (e.g., hourly tasks, daily tasks, weekly tasks, monthly tasks, etc.), types categorized by business function (e.g., report generation tasks, data synchronization tasks, system monitoring tasks, etc.), and types categorized by data processing object (e.g., user data processing tasks, product data processing tasks, transaction data processing tasks, etc.). For example, the following description uses the hourly report (hourly_report) task type as an example.

[0060] A scheduling time window refers to a pre-defined time interval used to trigger the execution of a target task. The start and end times of this interval can be configured according to business needs and system resource load. For example, the scheduler of a distributed system can be used to calculate the planned execution time of the current round of tasks based on a Cron expression or periodic configuration, and the scheduling time window can be determined based on the planned execution time. For example, if the planned execution time is 2025-11-27 14:00:00, the scheduling time window can be [2025-11-27 14:00:00, 2025-11-27 14:00:00].

[0061] A task instance identifier is a string or code combination that can uniquely represent an execution instance of a target task within a specific scheduling time window. The generation rules of this identifier can ensure that there is only one execution instance of a target task of the same type within the same scheduling time window.

[0062] Specifically, the task type characteristics of the target task can be combined and transformed with the scheduling time window characteristics to generate a globally unique identifier.

[0063] One possible implementation is to directly concatenate the task type identifier and scheduling time window identifier of the target task to generate a task instance identifier.

[0064] Another possible implementation is to perform a hash operation on the feature encoding corresponding to the task type of the target task and the time range feature of the scheduling time window, and use the fixed-length hash value obtained by the operation as the task instance identifier. The algorithm used for the hash operation can ensure that the same combination of input features corresponds to a unique output hash value, and different combinations of input features correspond to different output hash values.

[0065] S102. Using the task instance identifier as the database primary key, attempt to insert the task instance corresponding to the target task into the database.

[0066] The task instance corresponding to the target task refers to a structured data set containing all parameters and attributes required for the execution of the task. It may include the task execution priority, the range of data to be processed, the identifiers of the prerequisite tasks, and the retry strategy after execution failure.

[0067] A database refers to a structured storage system used to persistently store information about task instances, and can include relational databases, distributed key-value databases, etc.

[0068] A database primary key is a field used to uniquely identify each record in a database table. It has a uniqueness constraint, meaning that two records with the same primary key field value are not allowed in the same database table.

[0069] Specifically, the task instance identifier can be used as the primary key field, and all the structured data of the task instance can be written as a record to a specified storage table in the database. The database's primary key uniqueness constraint can be used to automatically verify whether the same task instance already exists within the current scheduling time window.

[0070] The specified storage table in the database can be created in advance. The task instance identifier is set as the primary key field in the table structure, and the primary key uniqueness constraint is enabled. Other fields used to store task instance attributes are also set, such as the start and end time of the scheduling time window, task execution priority, task dependencies, etc.

[0071] Insertion operations can be performed using the database's native insert statement. There is no need to perform a query operation beforehand to check for duplicate records. The generated task instance records are directly written to the task instance table, and the database automatically verifies the uniqueness of the primary key.

[0072] One possible implementation is to first obtain the globally unique identifier of the target task in the distributed system, and then use the task instance identifier as the database primary key to insert a task instance record containing the task instance identifier, scheduling time window, and globally unique identifier into the task instance table in the database.

[0073] Another possible implementation is to combine the task instance identifier with all the execution parameters of the task instance to generate a complete record containing primary key fields and business fields, and then directly insert the record into the task instance table of the database. In addition to the task instance identifier, the inserted record can also include information such as task type, task execution status, and task creation time.

[0074] If the attempt to insert a task instance into the database fails, it indicates that the insertion operation triggers the database's primary key uniqueness constraint verification, which means that the same task instance already exists within the current scheduling time window, and step S103 is executed.

[0075] If the task instance is successfully inserted into the database, it indicates that the insertion operation did not trigger constraint verification. This means that the current task instance is being inserted for the first time and there is no duplicate task instance corresponding to the target task within the current scheduling time window. Then, proceed to step S104.

[0076] S103. Terminate task processing and output a prompt message.

[0077] The prompt message indicates that a task instance already exists within the scheduling time window.

[0078] Terminating task processing means stopping the scheduling and execution process of the current target task and preventing subsequent task execution operations from being triggered.

[0079] The prompt information refers to the structured or unstructured information used to inform the relevant processing unit or operation and maintenance personnel of the reason for the failure of the current task instance insertion. For example, it may include the task instance identifier, task type, scheduling time window, failure reason code, failure reason description, etc.

[0080] Specifically, upon receiving an insertion failure response from the database, the execution of the current task scheduling thread can be stopped directly, releasing the occupied system resources. Simultaneously, information including the task instance identifier, task type, scheduling time window, and reason for failure can be generated according to a preset prompt message format. The prompt message format can be set to, for example, JSON format for easy machine parsing, or text format for easy human viewing.

[0081] The generated notification messages can be sent to the system's monitoring and alarm module, log storage module, or via preset notification channels to the terminal devices of maintenance personnel. For example, the generated JSON format notification message can include fields such as task instance identifier, task type, scheduling time window, and failure reason. The generated text format notification message can be, for example, stated as "Task of type XXX with task instance identifier XXX already exists within the scheduling time window from XXX to XXX; task processing has been terminated."

[0082] S104, Trigger the execution engine to execute task instances.

[0083] The execution engine refers to the business processing unit used to actually execute the target task, which may include distributed computing engines, task scheduling engines, business process execution engines, etc.

[0084] Triggering the execution engine to execute a task instance means sending the execution parameters of the task instance to the execution engine, which then processes the task instance according to preset business logic. Specifically, after confirming that the task instance has been successfully inserted into the database, all execution parameters of the task instance can be passed to the corresponding execution engine to start the actual execution process of the task instance and synchronize the execution status of the task to the database in real time.

[0085] The method provided in this application generates a task instance identifier for the target task based on the task type and scheduling time window. Using the task instance identifier as the database primary key, an attempt is made to insert the corresponding task instance into the database. If the insertion attempt fails, task processing is terminated and a prompt indicating that the task instance already exists within the scheduling time window is output. If the insertion attempt succeeds, the execution engine is triggered to execute the task instance. Compared to existing technologies, this method ensures the global uniqueness of task instances within the same scheduling batch by generating a unique task instance identifier and utilizing the uniqueness constraint of the database primary key, thus guaranteeing the consistency of business logic and data. Furthermore, this method eliminates the need for querying whether task instances are duplicated, reduces the write path, improves system response speed, and eliminates the need for external components such as ZooKeeper, distributed locks, and message queues, thereby reducing system resource waste and improving overall system performance and stability.

[0086] The following section provides a detailed explanation of how step S101 above generates a task instance identifier for the target task based on its task type and scheduling time window. Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 2 As shown, step S101 may specifically include:

[0087] S201. Determine the task type identifier based on the task type of the target task.

[0088] A task type identifier is a standardized string or code used to uniquely map the task type to which a target task belongs. Its generation rules ensure that the same task type corresponds to a unique identifier, and different task types correspond to different identifiers. For example, in the hourly report task type scenario of this embodiment, the task type identifier can be generated based on rules such as the English abbreviation of the task type or business code. For instance, the hourly report task type can be mapped to "hourly_report" as the task type identifier, or it can be converted into a fixed-length numeric code, such as "001" as the task type identifier.

[0089] Specifically, a set of mapping relationships between task types and task type identifiers can be pre-established. This set can be stored in a distributed configuration center, a database configuration table, or a local configuration file, and can be dynamically updated according to business needs in the future.

[0090] When determining the task type identifier of a target task, the task type can be identified by reading the target task's metadata configuration, parsing the tag fields of the task execution logic, or matching the type description in the task configuration file. Then, the corresponding identifier content can be matched from the mapping relationship set. For example, if the task type marked in the target task's metadata configuration is "generate business operation reports every hour," it can be identified as belonging to the intersection of hourly tasks divided by time period and report generation tasks divided by business function. In this case, a combined identifier rule can be used to generate "hourly_report_gen" as the task type identifier, which reflects both the time period characteristic and the business function characteristic.

[0091] S202. Determine the scheduling time window identifier based on the scheduling time window.

[0092] A scheduling time window identifier is a standardized string or code used to uniquely represent a specific scheduling time window. Its generation rules ensure that the same scheduling time window corresponds to a unique identifier, and different scheduling time windows correspond to different identifiers. In the scheduling time window scenario described in the above embodiment, the start and end times of the scheduling time window are the same, i.e., [2025-11-27 14:00:00, 2025-11-27 14:00:00]. In this case, the scheduling time window identifier can be generated based on the timestamp of that time point, a formatted time string, etc.

[0093] Specifically, the generation of the scheduling time window identifier can adopt any of the following rules:

[0094] The first type is a formatted time string rule, which can convert the start and end times of the scheduling time window into strings in the format "YYYYMMDDHHMMSS", and then concatenate them with a preset separator. If the start and end times are the same, the formatted string is used directly as the identifier. For example, the scheduling time window in this embodiment can generate "20251127140000" as the identifier.

[0095] The second method is the timestamp rule, which converts the start time of the scheduling time window into a millisecond-level timestamp, and then into a string as an identifier. For example, the millisecond-level timestamp corresponding to 2025-11-27 14:00:00 can be used as an identifier after being converted into a string.

[0096] The third method is the time window numbering rule, which assigns a unique numerical number to each time window according to the creation order of the time windows, and then converts it into a string as an identifier. For example, the hourly time windows of each day are numbered sequentially from "00" to "23". In this embodiment, the 2 PM time window corresponds to the number "14", and "2025112714" is generated as an identifier by combining the date information.

[0097] The generation of scheduling time window identifiers can be based on appropriate rules selected according to database storage requirements and query efficiency requirements. For example, if it is necessary to quickly query task instances based on a time range, a formatted time string rule can be selected to facilitate range queries through string prefix matching. If it is necessary to minimize the storage length of the identifier, a timestamp rule can be selected to convert the timestamp into a hexadecimal string to further shorten the length.

[0098] S203, combine task type identifier, scheduling time window identifier, and generate task instance identifier.

[0099] The core of concatenating the task type identifier and the scheduling time window identifier is to merge the two identifiers into a globally unique string according to a preset combination rule, ensuring that different task types and scheduling time windows generate different task instance identifiers. In the hourly report task scenario of this embodiment, the task type identifier is "hourly_report" and the scheduling time window identifier is "20251127140000". Task instance identifiers can be generated through different concatenation rules.

[0100] One possible implementation is to directly concatenate the task type identifier and the scheduling time window identifier in the order of "task type identifier + scheduling time window identifier" without adding a separator. In this embodiment, "hourly_report20251127140000" can be generated as the task instance identifier.

[0101] Another possible implementation is to add a preset special character as a separator between the task type identifier and the scheduling time window identifier, such as an underscore or a hyphen. In this embodiment, “hourly_report_20251127140000” or “hourly_report-20251127140000” can be generated as the task instance identifier.

[0102] Another possible implementation is to add a preset fixed prefix or suffix to the beginning and end of the concatenated string to distinguish task instances from different business lines. For example, adding the business line prefix "Y" can generate "Y_hourly_report_20251127140000" as a task instance identifier.

[0103] The choice of concatenation rules needs to consider factors such as the length limit of the database primary key field, the readability of the identifier, and subsequent query efficiency. For example, if there is a length limit to the database primary key field, a direct concatenation rule can be chosen to reduce the character length occupied by the delimiter. If it is necessary to improve the readability of the identifier, a delimiter concatenation rule can be chosen to make the characteristics of the task type and time window clearer. If there are tasks from multiple business lines in the system, a prefix-suffix rule can be chosen to quickly distinguish task instances from different business lines through the prefix.

[0104] The method provided in this application converts task types into standardized task type identifiers and scheduling time windows into standardized scheduling time window identifiers. Then, it concatenates these two identifiers according to preset rules to generate a task instance identifier, ensuring the uniqueness and scalability of the task instance identifier. This approach not only clearly reflects the core attributes of a task but also adapts to different business scenarios by adjusting the mapping relationship set and concatenation rules. Furthermore, the standardized identifier format improves the efficiency of database storage and retrieval, facilitating subsequent task management and monitoring.

[0105] The following section provides a detailed explanation of how, in step S102, the task instance identifier is used as the database primary key to attempt to insert the task instance corresponding to the target task into the database. Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 3 As shown, step S102 may specifically include:

[0106] S301. Obtain the globally unique identifier of the target task in the distributed system.

[0107] A globally unique identifier is a string or code that can uniquely represent a target task across all nodes and services in a distributed system. Its generation rules can ensure that it will not be duplicated with the identifiers of other tasks or business entities with a very high probability.

[0108] Optionally, the globally unique identifier can be generated using algorithms such as Snowflake Algorithm, UUID, or a distributed ID generation service. For example, using the Snowflake Algorithm to generate a 64-bit ordered integer ID includes fields such as timestamp, machine identifier, and sequence number, ensuring both uniqueness and incrementality, facilitating subsequent sorting and querying by generation time. Alternatively, the globally unique identifier can be obtained directly from the source node of the target task, where the source node has pre-generated a globally unique identifier corresponding to the target task.

[0109] S302. Using the task instance identifier as the database primary key, insert a record of the task instance into the task instance table in the database to attempt to insert the task instance corresponding to the target task into the database.

[0110] The task instance record includes the task instance identifier, scheduling time window, and globally unique identifier. The task instance table includes a unique constraint on the task instance identifier.

[0111] The task instance table refers to a structured data table in the database specifically used to store task instance information. Its table structure design can be customized according to the storage requirements of task attributes.

[0112] For example, continuing with the hourly reporting task scenario in the above embodiment, the fields of the task instance table may include: a task instance identifier field as the primary key, used to store "hourly_report_20251127140000"; a scheduling time window start time field, used to store "2025-11-27 14:00:00"; a scheduling time window end time field, used to store "2025-11-27 14:00:00"; a globally unique identifier field, used to store "1426300800000000000" generated by the Snowflake algorithm; a task type field, used to store "hourly_report"; and a task execution status field, with a default value of "pending execution", etc.

[0113] Specifically, inserting task instance records can be performed using the database's native INSERT statement. For example, using the MySQL INSERT statement, where `instance_id` is the globally unique identifier of the target task, `instance_code` is the task instance identifier, `task_name` is the task name, and `schedule_time` is the scheduling time window, the statement format can be as follows:

[0114] INSERT INTO task_instance (instance_id, instance_code, task_name,schedule_time) VALUES ('?', '?', '?', '?');

[0115] When this statement is executed, the database will automatically check the uniqueness of the instance_id field. If the same field value already exists in the table, a primary key conflict error will be returned; if the same value does not exist, the record will be written to the table and the result of successful insertion will be returned.

[0116] To ensure the atomicity and reliability of insert operations, database transaction support can be enabled, wrapping the insert operation within a transaction. If an exception occurs during the insert process, the transaction is rolled back, preventing the generation of incomplete task instance records. Additionally, a database retry mechanism can be configured. If insert failures occur due to network fluctuations, temporary database unavailability, or other reasons, a preset number of retries can be automatically performed, improving the success rate of insert operations.

[0117] The method provided in this application obtains a globally unique identifier and inserts it into the database along with information such as the task instance identifier and scheduling time window. This leverages the uniqueness constraint of the database primary key to verify duplicate tasks and enables task tracking in a distributed environment through the globally unique identifier. This approach requires no additional distributed locks or coordination services; it relies solely on the database's native functionality to guarantee the uniqueness of task instances. Furthermore, the stored globally unique identifier provides a unified tracking identifier for subsequent cross-node task monitoring and troubleshooting, improving system maintainability.

[0118] The following section details the method for determining whether an attempt to insert a task instance into the database has failed, as described in the foregoing embodiments. Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 4 As shown, the method may specifically include:

[0119] S401. Determine if a primary key conflict exception has been received from the database.

[0120] A primary key conflict exception refers to a standardized error message returned by the database when an insert operation violates the primary key uniqueness constraint of a database table. Taking MySQL as an example, the error code for a primary key conflict exception is 1062, and the error description contains the keyword "Duplicateentry 'XXX' for key 'PRIMARY'", where "XXX" represents a duplicate primary key value.

[0121] Specifically, the response after an insert operation can be monitored through the exception handling mechanism provided by the database driver. For example, in Java, try-catch statements can be used to catch SQL exceptions, and in Python, try-except statements can be used to catch MySQLdb exceptions.

[0122] Then, predefine the matching rules for primary key conflict exceptions, including error codes, keywords in error descriptions, etc. These rules can be stored in a configuration file for easy adjustment according to different database systems.

[0123] Finally, the captured exception information is compared with the matching rules. If the error code of the exception is consistent with the code in the rule and the error description contains the preset keywords, it is determined that a primary key conflict exception has been received; otherwise, it is determined that the exception has not been received.

[0124] If the insertion operation is executed successfully, the database will not throw an exception, but will only return a result that the number of rows affected by the insertion is 1. At this time, it is confirmed that no primary key conflict exception has been received.

[0125] If a primary key conflict exception is received from the database, it indicates that a task instance with the same name already exists within the current scheduling time window, and step S402 is executed; if no primary key conflict exception is received from the database, it indicates that a task instance with the same name does not exist within the current scheduling time window, and step S403 is executed.

[0126] S402. Confirm that the attempt to insert the task instance into the database failed, terminate the task processing, and output a prompt message.

[0127] The specific implementation details of this step can be found in step S103 above, and will not be repeated here.

[0128] S403. Confirm that the task instance was successfully inserted into the database and trigger the execution engine to execute the task instance.

[0129] The specific implementation details of this step can be found in step S104 above, and will not be repeated here.

[0130] The method provided in this application accurately determines whether a task instance insertion is successful by monitoring primary key conflict exceptions in the database. If the insertion fails, the task is terminated and a prompt message is output; if the insertion is successful, the execution engine is triggered to execute the task, thus achieving automated and closed-loop management of the task scheduling process. This approach utilizes the database's native exception mechanism to replace the logic of verification through query operations, reducing the execution scheduling process, improving response speed, and also reducing data consistency issues caused by new conflict instances added between query verification and insertion operations. This reduces system resource consumption and improves data consistency.

[0131] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 5 As shown, the method may further include:

[0132] S501. Determine the granularity of the scheduling time window based on the task type of the target task.

[0133] The granularity of the scheduling time window refers to the length of the time interval within the scheduling time window, such as hours, minutes, seconds, milliseconds, etc. Different task types correspond to different time window granularities. The choice of granularity directly determines the time precision and format of the scheduling time window, thereby affecting the execution frequency of tasks and the timeliness of data updates.

[0134] Specifically, a set of mapping rules between task types and scheduling time window granularity can be pre-established. This set can be formulated based on dimensions such as the business attributes of the task, the timeliness requirements of data processing, and the system resource consumption. The core logic of the mapping rules is to match the business requirements of the task type with the appropriate time precision.

[0135] For example, task types categorized by time period can be directly matched with granularity of consistent duration, such as hourly tasks corresponding to hourly granularity, minute-based tasks corresponding to minute granularity, and second-based tasks corresponding to second granularity, ensuring that the task execution cycle and granularity are perfectly aligned, and avoiding situations where the time window and task execution cycle do not match.

[0136] Alternatively, task types categorized by business function can be matched with granularity based on the business's requirements for data timeliness. For example, system monitoring tasks require real-time monitoring of system status, corresponding to millisecond or second granularity; real-time transaction data synchronization tasks require ensuring no data delay, corresponding to second or minute granularity; daily report generation tasks only need to generate data once a day, corresponding to hourly or daily granularity; and system log archiving tasks have low timeliness requirements, corresponding to daily or weekly granularity.

[0137] Alternatively, task types categorized by data processing objects can be matched with granularity based on data update frequency. For example, user real-time behavior data processing tasks have a data update frequency of milliseconds, corresponding to millisecond or second granularity; product inventory data synchronization tasks have a data update frequency of minutes, corresponding to minute or hour granularity; and user monthly bill generation tasks have a data update frequency of days, corresponding to day or month granularity.

[0138] This set of mapping rules can be stored in a distributed configuration center, supporting dynamic adjustment and hot updates, and can adapt to changes in business scenarios without restarting the system.

[0139] In this step, when determining the scheduling time window granularity of the target task, the task type to which the target task belongs is first identified. This can be achieved by reading the type tag in the task metadata, parsing the business description in the task configuration file, and matching the characteristic keywords of the task execution logic. Then, the corresponding granularity is matched from the mapping rule set. If the target task belongs to multiple overlapping categories, the granularity with the highest priority among the mapping rules is selected. The priority can be preset according to business requirements. If there is no matching task type in the mapping rule set, a default granularity, such as hourly granularity, can be used, and an alarm message is triggered to notify operations personnel to supplement the mapping rules.

[0140] S502. Determine the scheduling time window for the target task based on the granularity of the scheduling time window.

[0141] The format of the scheduling time window is determined by the granularity of the scheduling time window. Different granularities correspond to different time precision formats: hourly granularity corresponds to a time interval in the format "YYYY-MM-DDHH:00:00", for example, [2025-11-27 14:00:00, 2025-11-27 14:00:00]; minute granularity corresponds to a time interval in the format "YYYY-MM-DDHH:MM:00", for example, [2025-11-27 14:30:00, 2025- [11-27 14:30:00]; the second granularity corresponds to the time interval in the format "YYYY-MM-DDHH:MM:SS", for example [2025-11-27 14:30:15,2025-11-27 14:30:15]; the millisecond granularity corresponds to the time interval in the format "YYYY-MM-DDHH:MM:SS.fff", for example [2025-11-27 14:30:15.123,2025-11-27 14:30:15.123].

[0142] In this step, the time precision format of the time window can be determined. Based on the time unit corresponding to the granularity, the time string format of the scheduling time window can be determined. For example, the hourly granularity corresponds to the format that is retained to the hour, the minute granularity corresponds to the format that is retained to the minute, and so on.

[0143] Then, calculate the base time for the current scheduling round. The base time can be set to the current system time, the scheduled task execution time, the preset start time, etc. If it is a periodic task, the end time of the last execution time window is used as the base time.

[0144] Finally, the base time is formatted according to the time precision format to obtain the start and end times of the scheduling time window for the current scheduling round.

[0145] The method provided in this application dynamically matches the corresponding scheduling time window granularity according to the task type, and then generates a scheduling time window with a corresponding time precision format based on the granularity, thereby achieving adaptive adjustment of task scheduling frequency and time precision. This approach can accurately adapt to the business needs of different tasks, satisfying the high-frequency scheduling requirements of tasks with high real-time requirements while avoiding the waste of system resources caused by over-scheduling of tasks with low timeliness requirements. At the same time, through a unified time window format, it improves the compatibility and maintainability of task scheduling in distributed systems, providing a reliable implementation solution for the unified scheduling of large-scale, multi-type tasks.

[0146] Figure 6This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 6 As shown, the method may further include:

[0147] S601, cache historical task scheduling records within a preset time period.

[0148] The historical task scheduling record includes the historical task type and the historical task scheduling time window.

[0149] The preset duration refers to the time range used to cache historical task scheduling records, such as 24 hours, 48 ​​hours, etc. The cached records are the core attributes of the task instances that were successfully inserted into the database within the most recent preset duration, including historical task type, historical task scheduling time window, historical task instance identifier, etc.

[0150] In this step, you can choose a suitable caching component, such as a distributed caching system like Redis or Memcached, or a local memory cache. Set the cache expiration time to a preset duration, such as 24 hours, to ensure that only the most recent records are retained in the cache, avoiding excessive memory consumption by cached data. Finally, after each task instance is successfully inserted into the database, write information such as the task's historical task type, historical task scheduling time window, and historical task instance identifier into the cache.

[0151] To ensure consistency between cached data and database data, a dual-write mechanism or subscribing to the database change log can be used for cache updates. The dual-write mechanism means that after a successful insertion into the database, data is simultaneously written to the cache. Subscribing to the database change log involves using tools such as Canal or Debezium to monitor the database's binlog; when a task instance successfully inserts a record, the cache is automatically updated.

[0152] S602. After generating the task instance identifier, match it with the historical task scheduling record based on the task instance identifier.

[0153] The core of matching task instance identifiers with historical task scheduling records is to check if the same task instance identifier exists in the cache. If it exists, it means that the current task instance already exists; if it does not exist, it means that the task instance is being scheduled for the first time.

[0154] Specifically, using the generated task instance identifier as the key, the system queries the cache to see if a corresponding record exists, and determines whether there is a conflict based on the query result. If the query result is not empty, it means that the same task instance identifier exists in the cache, and the current task instance already exists. If the query result is empty, it means that the identifier does not exist in the cache, and the task instance is being scheduled for the first time.

[0155] Optionally, if an exception occurs during cache query, such as when the cache system is unavailable, the cache matching step can be skipped and subsequent database insertion operations can be performed directly to avoid task scheduling being blocked due to cache exception.

[0156] Optionally, the cached keys can also be sharded, for example, sharded according to the prefix of the task instance identifier, and the records of hourly reporting tasks can be stored in the specified cache shard nodes, reducing the load on individual nodes and improving query speed.

[0157] S603. If the historical task scheduling record does not contain a target record that conflicts with the task instance identifier, then attempt to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key.

[0158] If the cached matching results show that there are no conflicting target records, the database insertion operation continues. If the matching results show that there are conflicting target records, the task processing is terminated directly and a prompt message is output. There is no need to perform the database insertion operation, thus reducing invalid database requests.

[0159] Specifically, if the cache query result is empty, it means that there is no instance of the same task within the same scheduling time window in the last 24 hours, and the database insertion operation is performed according to the logic of step S102. If the cache query result is not empty, it means that there is an instance of the same task within the same scheduling time window in the last 24 hours, and the logic of step S103 is executed directly to terminate task processing and output a prompt message.

[0160] If a cache query fails, a database insert operation is performed, and the database primary key uniqueness constraint is used for final verification to prevent legitimate tasks from being misjudged as duplicate tasks due to cache exceptions.

[0161] The method provided in this application implements a two-layer verification of task instances by caching historical task scheduling records within the most recent preset time period and performing cache matching after generating task instance identifiers. This approach leverages the high-speed read / write characteristics of caching to quickly filter duplicate tasks before database insertion, reducing invalid database operations. Simultaneously, a cache expiration mechanism ensures the timeliness of cached data, avoiding misjudgments caused by expired cached data and improving the overall performance and reliability of the system.

[0162] Figure 7 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application is shown below. Figure 7 As shown, the data processing device provided in this embodiment includes: a processing module 11, a first control module 12, a second control module 13, and a third control module 14.

[0163] Processing module 11 is used to generate a task instance identifier for the target task based on the task type and scheduling time window of the target task.

[0164] The first control module 12 is used to attempt to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key.

[0165] The second control module 13 is used to terminate task processing and output a prompt message if the attempt to insert a task instance into the database fails. The prompt message indicates that a task instance already exists within the scheduling time window.

[0166] The third control module 14 is used to trigger the execution engine to execute the task instance if the attempt to insert the task instance into the database is successful.

[0167] Optionally, processing module 11 is specifically used to determine the task type identifier based on the task type of the target task. It also determines the scheduling time window identifier based on the scheduling time window. Finally, it concatenates the task type identifier and the scheduling time window identifier to generate a task instance identifier.

[0168] Optionally, the first control module 12 is specifically used to obtain the globally unique identifier of the target task in the distributed system. Using the task instance identifier as the database primary key, it inserts a record of the task instance into the task instance table in the database to attempt to insert the task instance corresponding to the target task into the database. The task instance record includes the task instance identifier, scheduling time window, and globally unique identifier. The task instance table includes a uniqueness constraint on the task instance identifier.

[0169] Optionally, the second control module 13 is specifically used to determine that the attempt to insert the task instance into the database failed if a primary key conflict exception is received from the database. It then terminates the task processing and outputs a prompt message.

[0170] Optionally, the third control module 14 is specifically used to determine that the attempt to insert the task instance into the database was successful if no primary key conflict exception is received from the database. It then triggers the execution engine to execute the task instance.

[0171] Optionally, the processing module 11 is further configured to determine the granularity of the scheduling time window based on the task type of the target task. The scheduling time window for the target task is determined based on the granularity of the scheduling time window.

[0172] Optionally, processing module 11 is also used to cache historical task scheduling records within a preset time period. After generating a task instance identifier, it matches the task instance identifier with the historical task scheduling records. If the historical task scheduling records do not contain a target record that conflicts with the task instance identifier, it attempts to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key. The historical task scheduling records include the historical task type and the historical task scheduling time window.

[0173] The data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0174] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device is used to execute the aforementioned data processing method. Figure 8 As shown, the electronic device 800 may include at least one processor 801, a memory 802, and a communication interface 803.

[0175] The memory 802 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions.

[0176] The memory 802 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0177] The processor 801 is used to execute computer execution instructions stored in the memory 802 to implement the method described in the foregoing method embodiments. The processor 801 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0178] The processor 801 can communicate and interact with external devices through the communication interface 803. In specific implementations, if the communication interface 803, memory 802, and processor 801 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0179] Optionally, in a specific implementation, if the communication interface 803, memory 802, and processor 801 are integrated on a single chip, then the communication interface 803, memory 802, and processor 801 can communicate through an internal interface.

[0180] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0181] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0182] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0183] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0184] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0187] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0189] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: Based on the task type and scheduling time window of the target task, generate the task instance identifier of the target task; An attempt was made to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key; If the attempt to insert the task instance into the database fails, the task processing is terminated and a prompt message is output, which indicates that the task instance already exists within the scheduling time window. If the attempt to insert the task instance into the database is successful, the execution engine will be triggered to execute the task instance.

2. The method according to claim 1, characterized in that, The step of generating a task instance identifier for the target task based on the task type and scheduling time window of the target task includes: Determine the task type identifier based on the task type of the target task; The scheduling time window identifier is determined based on the aforementioned scheduling time window; The task type identifier and the scheduling time window identifier are combined to generate the task instance identifier.

3. The method according to claim 1, characterized in that, The step of attempting to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key includes: Obtain the globally unique identifier of the target task in the distributed system; Using the task instance identifier as the database primary key, a record of the task instance is inserted into the task instance table of the database to attempt to insert the task instance corresponding to the target task into the database. The record of the task instance includes the task instance identifier, the scheduling time window, and the globally unique identifier. The task instance table includes a uniqueness constraint on the task instance identifier.

4. The method according to claim 1, characterized in that, If the attempt to insert the task instance into the database fails, the task processing is terminated and a prompt message is output, including: If a primary key conflict exception is received from the database, it is determined that the attempt to insert the task instance into the database has failed. Terminate the task processing and output the prompt message.

5. The method according to claim 1, characterized in that, If the attempt to insert the task instance into the database is successful, the execution engine is triggered to execute the task instance, including: If no primary key conflict exception is received from the database, it is determined that the attempt to insert the task instance into the database was successful. The execution engine is triggered to execute the task instance.

6. The method according to any one of claims 1-5, characterized in that, Also includes: The scheduling time window granularity is determined based on the task type of the target task; The scheduling time window for the target task is determined based on the granularity of the scheduling time window.

7. The method according to any one of claims 1-5, characterized in that, Also includes: The historical task scheduling records within a preset time period are cached, and the historical task scheduling records include the historical task type and the historical task scheduling time window; After generating the task instance identifier, the task instance identifier is matched with the historical task scheduling record; If the historical task scheduling record does not contain a target record that conflicts with the task instance identifier, then the task instance corresponding to the target task will be inserted into the database using the task instance identifier as the database primary key.

8. A data processing apparatus, characterized in that, include: The processing module is used to generate a task instance identifier for the target task based on the task type and scheduling time window of the target task. The first control module is used to attempt to insert the task instance corresponding to the target task into the database using the task instance identifier as the database primary key; The second control module is used to terminate task processing and output a prompt message if the attempt to insert the task instance into the database fails. The prompt message is used to indicate that the task instance already exists within the scheduling time window. The third control module is used to trigger the execution engine to execute the task instance if the attempt to insert the task instance into the database is successful.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images