Data processing method, system, processor and storage medium

Abstract

The application discloses a data processing method and system, a processor and a storage medium. The method comprises the following steps: obtaining a task message, wherein the task message is used for executing task data; determining a node device corresponding to the task message in a scheduling system; detecting a serial number of the node device to obtain a detection result; determining an original shard in which the task message is located based on the detection result; and executing the task data corresponding to the task message based on a first message state of the task message in the original shard, wherein the first message state is used for indicating a state of the task message to be executed. The application solves the technical problem of low processing efficiency of the task message.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a data processing method, system, processor, and storage medium. Background Technology

[0002] Currently, when processing task messages, enterprise applications often encounter the need for task scheduling. However, due to the inability to track and query the execution status of these asynchronous tasks, it is difficult to quickly locate and troubleshoot problems, and task messages are easily lost. In addition, when a service node in the cluster fails, it will affect the processing of task messages that the node was originally responsible for, resulting in low efficiency in processing task messages.

[0003] There is currently no effective solution to the problem of low processing efficiency of task messages mentioned above. Summary of the Invention

[0004] This invention provides a data processing method, system, processor, and storage medium to at least solve the technical problem of low processing efficiency for task messages.

[0005] According to one aspect of the present invention, a data processing method is provided, comprising: acquiring a task message, wherein the task message is used to execute task data; determining a node device corresponding to the task message in a scheduling system; detecting the sequence number of the node device to obtain a detection result; determining the original fragment in which the task message is located based on the detection result; and executing the task data corresponding to the task message based on a first message state of the task message in the original fragment, wherein the first message state is used to indicate the state of the task message to be executed.

[0006] Optionally, the method further includes: in response to the detection result indicating a change in the node device's sequence number, storing the task message from the original shard to the target shard in the database; and executing the task data corresponding to the task message based on the first message state of the task message in the target shard.

[0007] Optionally, before executing the task data corresponding to the task message based on the first message state of the task message in the original fragment, the method further includes: verifying the task message to obtain a first verification result; updating the original message state of the task message to the first message state in response to the first verification result indicating that the task message is valid; and updating the original message state of the task message to a second message state in response to the first verification result indicating that the task message is invalid, wherein the second message state indicates that the task message execution failed.

[0008] Optionally, the method further includes: storing the task message in the target message queue in response to the original message state of the task message being updated to the first message state; and discarding the task message in response to the original message state of the task message not being updated to the first message state.

[0009] Optionally, based on the first message state of the task message in the original fragment, the task data corresponding to the task message is executed, including: in response to the message state of the task message being updated from the first message state to the third message state, the task data corresponding to the task message is executed to obtain the execution result, wherein the third message state is used to indicate the state in which the task message is being executed.

[0010] Optionally, the method further includes: in response to the execution result returning success to the client, updating the message status of the task message from the third message status to the fourth message status, wherein the fourth message status is used to indicate the status of successful execution of the task message; in response to the execution result returning failure to the client, verifying the task message to obtain a second verification result.

[0011] Optionally, after verifying the task message and obtaining a second verification result, the method includes: in response to the second verification result indicating that the task message is valid, updating the message status of the task message from a third message status to a fifth message status, wherein the fifth message status indicates that the task message is in the process of retrying; and in response to the second verification result indicating that the task message is invalid, updating the message status of the task message from a third message status to a second message status.

[0012] Optionally, obtaining task messages includes: if all task messages in the original slice have the same message state, then update the message state of all task messages to the fifth message state, where the fifth message state is used to indicate the status of the task message execution retry; and obtaining the task message with the fifth message state.

[0013] According to another aspect of the present invention, a data processing apparatus is also provided, comprising: an acquisition unit for acquiring a task message, wherein the task message is used to execute task data; a first determination unit for determining a node device corresponding to the task message in a scheduling system; a detection unit for detecting the sequence number of the node device and obtaining a detection result; a second determination unit for determining the original fragment in which the task message is located based on the detection result; and an execution unit for executing the task data corresponding to the task message based on a first message state of the task message in the original fragment, wherein the first message state is used to indicate the state of the task message to be executed.

[0014] According to another aspect of the present invention, a data processing system is also provided, comprising: a message receiving module for acquiring a task message and performing a hash operation on the message key of the task message to obtain a hash table, wherein the task message is used to execute task data; a database module for storing the task message; an execution module for executing the task data corresponding to the task message; and a scheduling service module for scheduling the execution module.

[0015] This invention also provides a computer-readable storage medium. The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the data processing method of this disclosure.

[0016] This invention also provides a processor. The processor is used to run a program, wherein the program, when run by the processor, executes the data processing method of this disclosure embodiment.

[0017] In this embodiment of the invention, a task message is obtained, wherein the task message is used to execute task data; in the scheduling system, the node device corresponding to the task message is determined; the sequence number of the node device is detected to obtain a detection result; based on the detection result, the original shard in which the task message is located is determined; based on the first message state of the task message in the original shard, the task data corresponding to the task message is executed, wherein the first message state is used to indicate the pending execution state of the task message. In other words, by employing a distributed storage system, this application can detect single-machine failures in the cluster and allocate the tasks of that single machine to normal nodes for execution, thereby solving the technical problem of low processing efficiency for task messages and achieving the technical effect of improving the processing efficiency of task messages. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of a data processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a distributed message processing system based on a distributed self-coordination algorithm according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of a database sharding rule according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of a message structure according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of a message flow process according to an embodiment of the present disclosure; Figure 6This is a schematic diagram illustrating an automatic adjustment of database sharding corresponding to an execution service according to an embodiment of this disclosure; Figure 7 This is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure; Figure 8 This is a schematic diagram of a data processing system according to an embodiment of the present disclosure. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] Example 1 According to an embodiment of the present invention, a data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0022] Figure 1 This is a schematic diagram of a data processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps: Step S102: Obtain task messages, where task messages are used to execute task data.

[0023] In the technical solution provided in step S102 of this disclosure, the task message can be a request message from an enterprise application to execute a specific task at a specific time when there is a task scheduling requirement, such as a daily scheduled task request to deduct funds.

[0024] In this embodiment, a task message is obtained, such as a request message to execute a specific task at a specific time.

[0025] In this embodiment, after obtaining the task message, the message key of the task message can be hashed according to the configured rules to calculate the storage location of the task message.

[0026] In this embodiment, the database system can be used to store task messages. The database can be sharded according to the configured database sharding rules to obtain multiple shards. Each shard is used to store a portion of the data in the database system. Then, the data stored on each shard is converted into multiple tables.

[0027] It should be noted that when storing task messages, the database tables can be partitioned using a sharding approach, such as 100 databases and 10 tables. No specific restrictions are placed on the partitioning method for the database tables.

[0028] In this embodiment, after determining the database sharding and table partitioning rules, these rules can be configured in the configuration files of the message receiving module and the execution service module. This way, when a task message is obtained, the message key of the task message can be hashed according to the configured database sharding and table partitioning rules to determine the storage location of the task message, and then it can be stored.

[0029] Step S104: In the scheduling system, determine the node device corresponding to the task message.

[0030] In the technical solution provided by step S104 of this disclosure, when the execution service module is started, the execution service module will be registered to a certain node in the central control and scheduling system.

[0031] In this embodiment, the node device corresponding to the task message can be determined in the scheduling system. For example, in the central control scheduling system, the sequence number corresponding to the node device can be calculated, thereby calculating the database shard corresponding to the node device, including the task message in the database shard.

[0032] In this embodiment, the database shard corresponding to the node device can be calculated by calculating the node device's serial number, thereby avoiding the technical effect of multiple execution service modules competing for the same database shard.

[0033] For example, when both Execution Service Module 1 and Execution Service Module 2 start up, they are registered to the scheduling system and store task messages in a database sharding method of 100 databases and 10 tables. Execution Service Module 1 is responsible for processing task messages for database sharding tables with indexes 0 to 499, and Execution Service Module 2 is responsible for processing task messages for database sharding tables with indexes 500 to 999.

[0034] Step S106: Detect the serial number of the node device and obtain the detection result.

[0035] In this embodiment, the serial number of the node device can be detected to obtain the detection result. For example, each execution service module will listen to the changes of the nodes corresponding to other execution service modules in the scheduling system.

[0036] Step S108: Based on the detection results, determine the original fragment where the task message is located.

[0037] In the technical solution provided by step S108 of this disclosure, the original fragment where the task message is located can be determined based on the detection result. For example, if it is detected that the node in the scheduling system has not changed, then the original fragment where the task message is located can be determined.

[0038] Optionally, if a change in a node is detected in the scheduling system, the task message is stored from the original shard to the target shard in the database.

[0039] Step S110: Based on the first message state of the task message in the original fragment, execute the task data corresponding to the task message, wherein the first message state is used to indicate the state of the task message to be executed.

[0040] In the technical solution provided in step S110 of this disclosure, the first message state can be a state indicating that the task message is to be executed. The execution module adopts a producer-consumer model. The producer thread in the execution module can determine the start execution time (begin_time) of the task message in the database shard by scanning the corresponding database shard.

[0041] Optionally, the message status of the task message is updated from the original message status to the first message status, and the task data corresponding to the task message is executed.

[0042] Through steps S102 to S110, a task message is obtained, wherein the task message is used to execute task data. In the scheduling system, the node device corresponding to the task message is determined; the sequence number of the node device is detected to obtain a detection result; based on the detection result, the original shard in which the task message is located is determined; based on the first message state of the task message in the original shard, the task data corresponding to the task message is executed, wherein the first message state is used to indicate the pending execution state of the task message. In other words, by adopting a distributed storage system, this application can detect single-machine failures in the cluster and allocate the tasks of the single machine to normal nodes for execution, thereby solving the technical problem of low processing efficiency of task messages and achieving the technical effect of improving the processing efficiency of task messages.

[0043] The method described in this embodiment will now be described in further detail.

[0044] As an optional implementation, the method further includes: in response to a detection result indicating a change in the node device's sequence number, storing the task message from the original fragment to a target fragment in the database; and executing the task data corresponding to the task message based on the first message state of the task message in the target fragment.

[0045] In this embodiment, in response to a detection result indicating a change in the node device's serial number, the task message can be stored from the original shard to the target shard in the database. For example, when a detection result indicating a change in the node device's serial number is detected, a signal indicating this information is generated, and in response to this signal, the task message is stored from the original shard to the target shard in the database.

[0046] In this embodiment, whenever a new execution module service joins or leaves, other execution module services will listen for the node change and update their own database shards. This ensures that the tasks stored in each database will be executed. Furthermore, when the message volume increases, the execution module service can be deployed independently for horizontal scaling without needing to worry about other existing services.

[0047] For example, using a database sharding system with 10 tables per database, "Execution Service 1" handles database table messages with indices 0-499, while "Execution Service 2" handles messages with indices 500-999. When "Execution Service 3" starts, "Execution Service 1" and "Execution Service 2" detect new service additions in the scheduling system and automatically adjust their respective database shards to 0-332 and 333-665, respectively. The shards 666-999 are then handled by "Execution Service 3." Similarly, if "Execution Service 2" crashes, "Execution Service 1" and "Execution Service 3" will detect the change in service availability and automatically adjust their services to take over the work previously handled by "Execution Service 2."

[0048] As an optional implementation, before executing the task data corresponding to the task message based on the first message state of the task message in the original fragment, the method further includes: verifying the task message to obtain a first verification result; updating the original message state of the task message to the first message state in response to the first verification result indicating that the task message is valid; and updating the original message state of the task message to a second message state in response to the first verification result indicating that the task message is invalid, wherein the second message state indicates that the task message execution has failed.

[0049] In this embodiment, in response to the first verification result indicating that the task message is valid, the original message state of the task message can be updated to the first message state. For example, when the first verification result is detected to indicate that the task message is valid, a signal is generated to indicate that the information is valid, and in response to the signal, the original message state of the task message is updated to the first message state.

[0050] In this embodiment, in response to a first verification result indicating that the task message is invalid, the original message state of the task message can be updated to a second message state, wherein the second message state is used to indicate that the task message has failed to execute. For example, when the first verification result is detected to indicate that the task message is invalid, a signal is generated to indicate this information, and in response to the signal, the original message state of the task message is updated to a second message state, wherein the second message state is used to indicate that the task message has failed to execute.

[0051] In this embodiment, before executing the task data corresponding to the task message based on the first message state of the task message in the original shard, the task message can be checked to obtain a first check result. For example, the producer thread scans the corresponding database shard, retrieves the message body up to the start execution time (begin_time<=now), and checks the validity of the message. For example, it checks whether the message expiration time (expire_time) is within the validity period and / or whether the message retry count (retry_times) has reached the maximum retry count, etc. No specific restrictions are made here.

[0052] As an optional implementation, the method further includes: storing the task message in a target message queue in response to the original message state of the task message being updated to a first message state; and discarding the task message in response to the original message state of the task message not being updated to the first message state.

[0053] In this embodiment, in response to the original message state of the task message being updated to the first message state, the task message can be stored in the target message queue. For example, when it is detected that the original message state of the task message has been updated to the first message state, a signal is generated to represent the information, and in response to the signal, the task message is stored in the target message queue.

[0054] In this embodiment, the task message can be discarded in response to the original message state of the task message not being updated to the first message state. For example, when it is detected that the original message state of the task message has not been updated to the first message state, a signal is generated to indicate this information, and the task message is discarded in response to the signal.

[0055] In this embodiment, the target message queue can be a topic queue in a producer-consumer model, where task messages can be stored in the corresponding topic queue.

[0056] In this embodiment, if the original message state of the task message is not updated to the first message state, it is assumed that the task message may have been processed by other execution modules, and therefore the task is abandoned.

[0057] As an optional implementation, based on the first message state of the task message in the original or target shard, the task data corresponding to the task message is executed, including: in response to the message state of the task message being updated from the first message state to the third message state, the task data corresponding to the task message is executed to obtain the execution result, wherein the third message state is used to indicate the state in which the task message is being executed.

[0058] In this embodiment, in response to the message state of a task message being updated from the first message state to the third message state, the task data corresponding to the task message can be executed to obtain the execution result. For example, when it is detected that the message state of a task message has been updated from the first message state to the third message state, a signal is generated to represent the information. In response to the signal, the task data corresponding to the task message is executed to obtain the execution result.

[0059] In this embodiment, the consumer thread listens to changes in the corresponding topic queue. When a message is written to the topic queue, the message is retrieved and the status of the message in the database is updated from pending execution (first message status) to executing (third message status).

[0060] As an optional implementation, the method further includes: in response to the execution result returning success to the client, updating the message status of the task message from a third message status to a fourth message status, wherein the fourth message status is used to indicate the status of successful execution of the task message; in response to the execution result returning failure to the client, verifying the task message to obtain a second verification result.

[0061] In this embodiment, in response to the execution result returning success to the client, the message status of the task message can be updated from the third message status to the fourth message status, where the fourth message status is used to indicate that the task message has been executed successfully. For example, when it is detected that the execution result has returned success to the client, a signal is generated to indicate this information, and in response to the signal, the message status of the task message is updated from the third message status to the fourth message status, where the fourth message status is used to indicate that the task message has been executed successfully.

[0062] In this embodiment, in response to the failure returned to the client by the execution result, the task message can be checked to obtain a second check result. For example, when the failure returned to the client by the execution result is detected, a signal is generated to indicate the information, and in response to the signal, the task message can be checked to obtain a second check result.

[0063] In this embodiment, if the task data corresponding to the task message in the execution state is executed successfully, a success message is returned to the client, and the status of the message in the database is updated to success; if the task data corresponding to the task message in the execution state is not executed successfully, a failure message is returned to the client, and it is checked whether the task message can be retried, for example, checking whether the message expiration time (expire_time) is within the validity period and / or whether the message retry count (retry_times) has reached the maximum retry count.

[0064] As an optional implementation, after verifying the task message and obtaining a second verification result, the method further includes: in response to the second verification result indicating that the task message is valid, updating the message status of the task message from a third message status to a fifth message status, wherein the fifth message status indicates that the task message is in the process of retrying; and in response to the second verification result indicating that the task message is invalid, updating the message status of the task message from a third message status to a second message status.

[0065] In this embodiment, in response to the second verification result indicating that the task message is valid, the message state of the task message can be updated from the third message state to the fifth message state, wherein the fifth message state is used to indicate the state of the task message execution retry. For example, when the second verification result is detected to indicate that the task message is valid, a signal is generated to indicate the information, and in response to the signal, the message state of the task message is updated from the third message state to the fifth message state, wherein the fifth message state is used to indicate the state of the task message execution retry.

[0066] In this embodiment, in response to the second verification result indicating that the task message is invalid, the message state of the task message can be updated from the third message state to the second message state. For example, when the second verification result is detected to indicate that the task message is invalid, a signal is generated to indicate the information, and in response to the signal, the message state of the task message is updated from the third message state to the second message state.

[0067] In this embodiment, if the task message can be retried, the retry time can be calculated according to a certain strategy, and the next execution time and status in the database of the message task can be updated. The status in the database can be updated from executing (third message status) to retrying (fifth message status). If the message task cannot be retried, the task is discarded, and the status of the task message in the database is updated from executing (third message status) to failed (second message status).

[0068] As an optional implementation, obtaining task messages includes: determining that all task messages in the original segment have the same message state, then updating the message state of all task messages to the fifth message state, wherein the fifth message state is used to indicate the status of the task message execution retry; and obtaining the task message with the fifth message state.

[0069] In this embodiment, if it is determined that all task messages in the original fragment have the same message state, then the message state of all task messages is updated to the fifth message state, and the task message in the fifth message state is obtained. For example, in this embodiment, the execution of task messages is based on the message state of the task messages. When all message states have flowed to the same (e.g., all are pending execution or in execution) message state, it is determined that the message state is abnormal. Then, the task message in the abnormal state can be obtained, and its message state is updated to retrying. The task message in the retrying state is obtained again, and the task data corresponding to the task message is re-executed.

[0070] Optionally, the message status of a task message in an abnormal state can be restored to a normal state through a recovery process. For example, its message status can be updated to "retrying". The timing of the re-execution of a task message in a "retrying" state can be determined, and its validity can be checked by the producer thread.

[0071] In this embodiment of the disclosure, by employing a distributed storage system, a single-machine failure in the cluster can be detected, and the tasks of that single machine can be assigned to normal nodes for execution. The execution of task messages is advanced based on the message status of the task messages. When the sequence number of a node device changes, the task message is stored from the original shard to the target shard, thus ensuring that the task data stored in each shard can be executed. When the message status of all task messages is in an abnormal state, the message status can be restored to a normal state through a recovery process, thereby improving the efficiency of task message processing and solving the technical problem of low processing efficiency of task messages, achieving the technical effect of improving the processing efficiency of task messages.

[0072] Example 2 The data processing method of this disclosure will be further described below with reference to preferred embodiments.

[0073] Currently, enterprise applications often encounter the need for task scheduling, executing specific tasks at specific times, such as daily scheduled fund deductions. However, downstream bank channels usually have traffic limitations. When the query per second (QPS) exceeds a certain threshold, requests will be rejected. In this case, it is desirable for rejected requests to be automatically retried. Therefore, it is particularly important to be able to track and query the execution status of these asynchronous tasks.

[0074] In related technologies, due to the large volume of messages, file storage is used to store task messages. However, after the task is completed, there is no efficient retrieval mechanism or fast query solution to quickly locate, troubleshoot, or implement post-compensation measures. For tasks involving changes in user funds, users are sensitive to timeliness and need tasks such as deductions and withdrawals to start within seconds. However, existing messaging systems cannot support scheduled task startup at any time. When a task is received by a node, if that node fails, the task cannot be lost, affecting its execution. Even if some systems can solve the loss problem, it comes at the cost of sacrificing overall system performance. When a service node in the cluster fails, existing systems will affect the processing of messages that node was originally responsible for to varying degrees.

[0075] In this application, a distributed storage system is used to persist task messages, providing storage for large amounts of data and supporting the retrieval of historical task messages. The system supports setting arbitrary start times and allows retries at specific time intervals when a task fails. A database is used as the message storage medium, ensuring that messages are never lost after being received. A central control scheduling service module is provided, which automatically detects system pressure and adjusts the configuration to quickly adjust the throughput of the entire cluster. The system also supports manual intervention. When a single machine in the cluster fails and cannot execute a task, the scheduling system can detect this and allocate the task of that single machine to other normal nodes for execution.

[0076] Figure 2 This is a schematic diagram of a distributed message processing system based on a distributed self-coordination algorithm according to an embodiment of this disclosure. Figure 2 As shown, the system can be composed of four parts: message acceptance module (Process) 1, storage module 2, execution module (Producer-consumer) 3, and central control scheduling service module 4.

[0077] The message acceptance module (Process 1) supports access via protocols such as Hypertext Transfer Protocol (HTTP) and Hypertext Transfer Protocol over Secure Socket Layer (HTTPS), and supports management using Business Networking Services (BNS) and Internet Protocol (IP) authorized services. It also supports the simultaneous deployment of multiple acceptance service processes and supports idempotency of message keys. When the message acceptance module 1 receives a message, it performs certain rule operations (such as hashing) on ​​the message key to obtain a hash table, which determines the message's storage location. The storage module 2 stores task messages and can support scaling from a single database to multiple databases to improve system concurrency and storage capacity. The execution module 3 is distributed and uses a producer-consumer model. Each execution module is independent and uses a central control scheduling system for global coordination. The central scheduling module is responsible for sharding and executing the storage system. Each execution module is responsible for an independent storage shard. Within a single execution module, the number of producers and consumers can be adjusted through the central scheduling system, and the responsible storage shards can be further sharded to improve the overall system throughput. The central scheduling service module 4 can be a distributed cluster self-coordinating system built on a message-passing-based distributed consensus algorithm (Paxos algorithm). It can scale horizontally and quickly self-heal when the system fails. It is mainly responsible for scheduling the execution modules. Optionally, this module can also be replaced by other similar systems, such as ZooKeeper, etc. No specific restrictions are made here.

[0078] The operation process of the above system is described below.

[0079] In this embodiment of the disclosure, a distributed message processing method based on a distributed self-coordination algorithm is provided, which may include the following steps: Step 1: The business unit sends the task message to the message receiving module and stores the received message in the database module.

[0080] Step two: The producer in the execution module retrieves the task message from the database module, updates the message status of the task message in the database module to the pending execution status, and then puts the task message into the corresponding topic queue.

[0081] Step 3: The producer in the execution module retrieves the task message from the topic queue, updates the status of the task message in the database module to "in execution", then executes message sending, and updates the status of the task message in the database module to "success / pending retry / execution failure" based on the sending result.

[0082] The message processing flow of the message acceptance module in this embodiment is described below.

[0083] The message receiving system is deployed independently as a resident process. It can be deployed as a single instance or as a cluster of multiple instances. After receiving a message, this module performs a hash operation on the message key according to the configured rules to calculate the storage location of the message, and then stores it.

[0084] The message processing flow of the database module in this embodiment is described below.

[0085] The database module is built using a database, mainly due to the strong requirements for reliability and traceability in business operations. In terms of speed: file system > distributed key-value (persistent) > distributed file system > database, but the reliability is the opposite. Databases can use distributed storage systems (DDBS), and can choose a single-instance service or a multi-sharded cluster system.

[0086] Figure 3 This is a schematic diagram of a database sharding rule according to an embodiment of this disclosure. For example... Figure 3 As shown, database tables can be partitioned using a sharding approach, such as designing a 100-database-10-table configuration. After determining the sharding rules, these rules are configured in the configuration files of the message receiving and execution modules. When the database system encounters storage bottlenecks, simply expanding the database shards is sufficient, and this is transparent to the execution and message receiving modules.

[0087] The message processing method of the execution module in this embodiment is described below.

[0088] Figure 4 This is a schematic diagram of a message structure according to an embodiment of the present disclosure. Figure 4 As shown, some key attribute information of the message body may include: message unique key (id_ext), message expiration time (expire_time), maximum number of message retries (max_retry_times), number of message retries (retry_times), message start execution time (begin_time), and message state (state).

[0089] Figure 5This is a schematic diagram of a message flow process according to an embodiment of this disclosure. Figure 5 As shown, when the message receiving (process) module receives a message, it uses id_ext as the primary key, calculates database sharding and table partitioning according to the specified rules, and then stores it in the database. At this time, the message's state is the initial state (state=0).

[0090] Then, the producer thread scans the corresponding database shards, retrieves messages whose execution start time (begin_time <= now), and checks their validity. For example, it checks whether expire_time is within the validity period and whether max_retry_times has reached the maximum number of retries. If invalid, the message status is updated to failure (f_state = 5). If valid, the message status is updated to state = 1. If the message status update fails, it is assumed that the message may have been processed by other execution modules, and the task is abandoned. If the update is successful, the message is put into the corresponding topic queue to wait for execution.

[0091] The consumer thread listens for changes in the corresponding message queue. When a message is written to the queue, it updates the message status in the database to "in progress" (state=2). If the message status update fails, the message task is abandoned; if the update succeeds, the message task is executed. If execution succeeds, the message status in the database is updated to "success" (state=3). If execution fails, it checks whether the message can be retried, for example, whether `expire_time` is within its validity period and / or whether `max_retry_times` has reached the maximum number of retries. If retry is possible, the retry time is calculated according to a certain strategy, and the message's next start execution time and status are updated (state=4). If retry is not possible, the message status is updated to "failure" (state=5).

[0092] It should be noted that in this embodiment, for messages in the retry (state=4) state, the above steps can be re-executed after the start execution time has elapsed.

[0093] The following describes the method for message anomaly recovery according to embodiments of this disclosure.

[0094] In this embodiment, message execution is driven by message status. When a message is in an abnormal state, such as when all messages transition to the pending (state=1) or executing (state=2) state (i.e., the service has crashed), a recovery process is needed to restore the message status to normal. Specifically, the recovery process continuously scans all database shards, retrieves messages with abnormal status (state=1 or 2), and updates the status of these abnormal messages to retrying (state=4).

[0095] It is important to note that the recover process here does not need to concern itself with whether the abnormal status message is valid. The producer process will check its validity when the abnormal status message is being processed.

[0096] The rapid scaling and fault self-healing functions of the embodiments of this disclosure are described below.

[0097] In this embodiment, when an execution module service starts, it registers with a node in the central control and scheduling system and listens for changes to that node. Based on the node in the central control system, it calculates its own sequence number and thus determines the database partition it needs to process. This avoids multiple execution module services competing for the same database. Whenever a new execution module service joins or leaves, other execution module services will detect the node change and update the database shards corresponding to each execution module. This ensures that tasks stored in each database are executed. When the message volume increases, execution module services can be deployed independently for horizontal scaling without needing to consider other existing services.

[0098] Figure 6 This is a schematic diagram illustrating an automatic adjustment of database sharding corresponding to an execution service according to an embodiment of this disclosure. For example... Figure 6 As shown, initially there were only two execution services registered with the scheduling system. Taking a database sharding system with 10 tables per 100 databases as an example, "Execution Service 1" was responsible for processing database table messages with indices 0-499, and "Execution Service 2" was responsible for processing database table messages with indices 500-999. When "Execution Service 3" started, "Execution Service 1" and "Execution Service 2" detected the addition of a service in the scheduling system and automatically adjusted their respective database shards to 0-332 and 333-665, respectively. The database shards 666-999 were then handled by "Execution Service 3". Similarly, if "Execution Service 2" crashed at this time, "Execution Service 1" and "Execution Service 3" would also detect the change in the number of services and automatically adjust the service to take over the work originally done by "Execution Service 2".

[0099] The central control and scheduling service module of this disclosure embodiment is described below.

[0100] The central control scheduling service module can be implemented by building a distributed coordination service (ZooKeeper) cluster. ZooKeeper is a distributed service with no single point of failure. Because ZooKeeper has sequential access characteristics—for example, for each update request from a client, ZooKeeper assigns a globally unique, incrementing number—it can sort all execution service module instances based on this incrementing number, thereby calculating the database shard corresponding to each execution module.

[0101] In this embodiment, a distributed database is used as the storage medium, providing the necessary message status query capability for the business, ensuring that messages are not lost or duplicated, and employing a distributed storage scheme that can be physically sharded. The system has no single point of failure and can scale system throughput as needed. Through an innovative distributed self-coordination algorithm, it automatically detects the addition or removal of services, can scale horizontally freely, can detect system faults and self-heal, and can also complete the second-level cluster service intervention and adjustment through the central control platform. This solves the technical problem of low processing efficiency for task messages and achieves the technical effect of improving the processing efficiency of task messages.

[0102] Example 3 According to embodiments of the present invention, a method for performing Figure 1 The data processing apparatus of the data processing method of the illustrated embodiment.

[0103] Figure 7 This is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure. Figure 7 As shown, the data processing device 70 may include: an acquisition unit 71, a first determination unit 72, a detection unit 73, a second determination unit 74, and an execution unit 75.

[0104] Acquisition unit 71 is used to acquire task messages, wherein the task messages are used to execute task data; The first determining unit 72 is used to determine the node device corresponding to the task message in the scheduling system; The detection unit 73 is used to detect the serial number of the node device and obtain the detection result; The second determining unit 74 is used to determine the original fragment where the task message is located based on the detection results; The execution unit 75 is used to execute the task data corresponding to the task message based on the first message state of the task message in the original fragment, wherein the first message state is used to indicate the state of the task message to be executed.

[0105] Optionally, the apparatus further includes: a storage unit, configured to store the task message from the original shard to a target shard in the database in response to a detection result indicating a change in the serial number of the node device; and a first execution unit, configured to execute the task data corresponding to the task message based on the first message state of the task message in the target shard.

[0106] Optionally, the apparatus further includes: a verification unit for verifying the task message to obtain a first verification result; a first update unit for updating the original message state of the task message to a first message state in response to the first verification result indicating that the task message is valid; and a second update unit for updating the original message state of the task message to a second message state in response to the first verification result indicating that the task message is invalid, wherein the second message state indicates that the task message execution failed.

[0107] Optionally, the first update unit includes: a storage module, used to store the task message in the target message queue in response to the original message state of the task message being updated to the first message state; and a discard module, used to discard the task message in response to the original message state of the task message not being updated to the first message state.

[0108] Optionally, the execution unit 75 includes: an execution module, configured to execute the task data corresponding to the task message in response to the message state of the task message being updated from a first message state to a third message state, and obtain an execution result, wherein the third message state is used to indicate the state in which the task message is being executed.

[0109] Optionally, the execution module includes: a first update submodule, used to update the message status of the task message from the third message status to the fourth message status in response to the execution result returning success to the client, wherein the fourth message status is used to indicate the status of successful execution of the task message; and a verification submodule, used to verify the task message in response to the execution result returning failure to the client, and obtain a second verification result.

[0110] Optionally, the execution module further includes: a second update submodule, used to update the message status of the task message from the third message status to the fifth message status in response to the second verification result indicating that the task message is valid, wherein the fifth message status is used to indicate the status of the task message execution retry; and a third update submodule, used to update the message status of the task message from the third message status to the second message status in response to the second verification result indicating that the task message is invalid.

[0111] Optionally, the acquisition unit 71 includes: a determining module, configured to determine that if all task messages in the original fragment have the same message state, then update the message state of all task messages to the fifth message state, wherein the fifth message state is used to indicate the status of the task message execution retry; and an acquisition module, configured to acquire the task message in the fifth message state.

[0112] It should be noted that the above-mentioned acquisition unit 71, first determination unit 72, detection unit 73, second determination unit 74 and execution unit 75 correspond to steps S102 and S110 in Embodiment 1, respectively. The five units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1.

[0113] In the above embodiments of this disclosure, an acquisition unit acquires a task message, wherein the task message is used to execute task data; a first determination unit determines the node device corresponding to the task message in the scheduling system; a detection unit detects the sequence number of the node device and obtains a detection result; a second determination unit determines the original fragment in which the task message is located based on the detection result; and an execution unit executes the task data corresponding to the task message based on the first message state of the task message in the original fragment, wherein the first message state is used to indicate the state of the task message to be executed, thereby solving the technical problem of low processing efficiency of task messages and achieving the technical effect of improving the processing efficiency of task messages.

[0114] Figure 8 This is a schematic diagram of a data processing system according to an embodiment of the present disclosure. Figure 8 As shown, the data processing system 80 may include: a message receiving module 81, a scheduling service module 82, a database module 83, and an execution module 84.

[0115] The message receiving module 81 is used to obtain task messages, where the task messages are used to execute task data; The scheduling service module 82 is used to determine the node device corresponding to the task message in the scheduling system, and to detect the sequence number of the node device to obtain the detection result. Database module 83 is used to determine the original fragment where the task message is located based on the detection results; The execution module 84 is used to execute the task data corresponding to the task message based on the first message state of the task message in the original fragment, wherein the first message state is used to indicate the state of the task message to be executed.

[0116] In the data processing system described in this embodiment, a distributed database is used as the storage medium, providing the necessary message status query capability for the business, ensuring that messages are not lost or duplicated, and employing a distributed storage scheme that can be physically sharded. The system has no single point of failure and can scale system throughput as needed. Through an innovative distributed self-coordination algorithm, it automatically detects the addition or removal of services, can freely scale horizontally, can detect system faults and self-heal, and can also complete the second-level cluster service intervention and adjustment through the central control platform. This solves the technical problem of low processing efficiency for task messages and achieves the technical effect of improving the processing efficiency of task messages.

[0117] Example 4 According to embodiments of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute the data processing method of the present disclosure embodiments.

[0118] Example 5 According to an embodiment of the present invention, a processor is also provided for running a program, wherein the program is executed by the processor to perform the data processing method of the present disclosure embodiments.

[0119] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0120] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0125] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: Obtain task messages, wherein the task messages are used to execute task data; In the scheduling system, the node device corresponding to the task message is determined; The serial number of the node device is detected to determine whether the node device has changed, and the detection result is obtained. In response to the detection result indicating that the node device has not changed, the original fragment in which the task message is located is determined; Based on the first message state of the task message in the original fragment, the task data corresponding to the task message is executed, wherein the first message state is used to indicate the state of the task message to be executed; Specifically, based on the first message state of the task message in the original segment, executing the task data corresponding to the task message includes: in response to the message state of the task message being updated from the first message state to the third message state, executing the task data corresponding to the task message to obtain an execution result, wherein the third message state is used to indicate the state in which the task message is being executed; The method further includes: monitoring changes in a target message queue via a thread, wherein the target message queue is used to store the task message; and updating the message status of the task message from the first message status to the third message status in response to the task message being written to the target message queue.

2. The method according to claim 1, characterized in that, The method further includes: In response to the detection result indicating a change in the node device's serial number, the task message is stored from the original shard to the target shard in the database; Based on the first message state of the task message in the target segment, the task data corresponding to the task message is executed.

3. The method according to claim 1, characterized in that, Before executing the task data corresponding to the task message based on the first message state of the task message in the original fragment, the method further includes: The task message is examined to obtain a first examination result; In response to the first verification result indicating that the task message is valid, the original message status of the task message is updated to the first message status; In response to the first verification result indicating that the task message is invalid, the original message status of the task message is updated to a second message status, wherein the second message status indicates that the task message has failed to execute.

4. The method according to claim 3, characterized in that, The method further includes: In response to the original message state of the task message being updated to the first message state, the task message is stored in the target message queue; If the original message state of the task message is not updated to the first message state, the task message is discarded.

5. The method according to claim 1, characterized in that, The method further includes: In response to the execution result returning success to the client, the message status of the task message is updated from the third message status to the fourth message status, wherein the fourth message status is used to indicate that the task message has been executed successfully; In response to the execution result returning a failure to the client, the task message is examined to obtain a second examination result.

6. The method according to claim 5, characterized in that, After verifying the task message and obtaining a second verification result, the method further includes: In response to the second verification result indicating that the task message is valid, the message status of the task message is updated from the third message status to the fifth message status, wherein the fifth message status indicates that the task message is in the process of retrying. In response to the second verification result indicating that the task message is invalid, the message status of the task message is updated from the third message status to the second message status, wherein the second message status indicates that the task message has failed to execute.

7. The method according to claim 1, characterized in that, Get task messages, including: If it is determined that the message status of all task messages in the original fragment is the same, then the message status of all task messages is updated to the fifth message status, where the fifth message status is used to indicate the status of task message execution retry. Retrieve the task message for the fifth message status.

8. A data processing apparatus, characterized in that, include: An acquisition unit is used to acquire task messages, wherein the task messages are used to execute task data; The determining unit is used to determine the node device corresponding to the task message in the scheduling system; The detection unit is used to detect the serial number of the node device, determine whether the node device has changed, and obtain the detection result; Storage unit, used to determine the original fragment where the task message is located in response to the detection result indicating that the node device has not changed; An execution unit is configured to execute task data corresponding to the task message based on the first message state of the task message in the original segment, wherein the first message state is used to indicate the state of the task message to be executed; The execution unit is further configured to respond to the message status of the task message being updated from the first message status to the third message status, execute the task data corresponding to the task message, and obtain the execution result, wherein the third message status is used to indicate the status of the task message being executed; The execution unit is further configured to monitor changes in the target message queue via a thread, wherein the target message queue is used to store the task message; in response to the task message being written to the target message queue, the message status of the task message is updated from the first message status to the third message status.

9. A data processing system, characterized in that, include: The message receiving module is used to obtain task messages, wherein the task messages are used to execute task data; The scheduling service module is used to determine the node device corresponding to the task message in the scheduling system, detect the sequence number of the node device, determine whether the node device has changed, and obtain the detection result. The database module is used to determine the original shard where the task message is located in response to the detection result indicating that the node device has not changed; An execution module is used to execute task data corresponding to the task message based on the first message state of the task message in the original segment, wherein the first message state is used to indicate the state of the task message to be executed; The execution module responds to the task message's message status being updated from the first message status to the third message status, executes the task data corresponding to the task message, and obtains the execution result. The third message status indicates that the task message is being executed. The execution module monitors changes in the target message queue via a thread, wherein the target message queue is used to store the task message; in response to the task message being written to the target message queue, the message status of the task message is updated from the first message status to the third message status.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the data processing method according to any one of claims 1 to 7.

11. A processor, characterized in that, The processor is used to run a program, wherein the program, when run by the processor, performs the data processing method according to any one of claims 1 to 7.

Images