Data processing method and device, computer device and storage medium

Abstract

The application relates to a data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to a data processing request, a data processing flow is created; each data processing flow comprises a plurality of task jobs; based on a hierarchical aggregation processing strategy, data processing and aggregation are performed on the plurality of task jobs in the cache queues of preset levels of transactions to obtain an aggregated cache; when the aggregated cache meets preset transaction cache submission conditions, the aggregated cache is submitted to a log device, and the log device is instructed to perform log solidification processing on the data in the aggregated cache. The method improves the efficiency of data processing.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] In a local object management system, data is stored in the form of data objects. Each data object contains data with the same attributes and metadata describing that same attribute.

[0003] When processing data in a local object management system, such as when a user requests to write data, the local object management system needs to execute multiple data processing flows, including opening (or creating) data objects, writing data objects, and closing data objects. Each data processing flow contains multiple tasks (jobs). For each data processing flow, the local object management system can protect the data processing flow through a transaction. Under transaction protection, the tasks within the transaction are processed, and then the processed jobs are logged and applied.

[0004] However, in each data processing flow, each data processing job requires log solidification and log application caching. The log solidification process requires data to be written to disk, and multiple data processing jobs require multiple data writes to disk, which consumes a lot of time and makes the data processing efficiency low. Summary of the Invention

[0005] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] Firstly, this application provides a data processing method. The method includes:

[0007] In response to a data processing request, a data processing flow is created; each data processing flow contains multiple task jobs.

[0008] Based on a hierarchical aggregation processing strategy, the multiple task jobs are processed and aggregated in the preset cache queues of each level of transactions to obtain an aggregate cache.

[0009] When the aggregated cache meets the preset transaction cache commit conditions, the aggregated cache is committed to the log device, instructing the log device to perform log persistence processing on the data in the aggregated cache.

[0010] This method uses a pre-defined transaction cache queue to cache multiple tasks. Then, based on a hierarchical aggregation processing strategy, data processing and hierarchical aggregation of multiple tasks can be performed in the cache queue. Finally, log solidification is performed on the aggregated cache only when the aggregated cache meets the pre-defined transaction cache commit conditions, reducing the number of log solidifications, saving the data persistence time required for log solidification, and improving data processing efficiency.

[0011] In one embodiment, the hierarchical aggregation processing strategy involves processing and aggregating the data of the multiple task jobs in preset cache queues for each level of transaction to obtain an aggregated cache, including:

[0012] For each of the data processing flows, the multiple task jobs are stored in the second cache queue of the secondary transaction;

[0013] By executing the secondary transaction, data processing is performed on multiple tasks in the second cache queue, and the multiple tasks after the data processing is completed are aggregated to obtain the secondary transaction cache.

[0014] The secondary transaction cache is stored in the first cache queue of the primary transaction, and the secondary transaction cache is aggregated in the first cache queue to obtain the primary transaction cache, which is then used as the aggregated cache.

[0015] In this embodiment, the data processing of multiple tasks in each level of transaction is protected by a preset hierarchical transaction, so that the data processing process is not disturbed, thereby ensuring the consistency of data during the data processing process. Furthermore, the hierarchical aggregation of multiple tasks in the transaction solves the problem of multiple log solidifications required for multiple tasks, saving data processing time and improving data processing efficiency.

[0016] In one embodiment, after storing the secondary transaction cache into the first cache queue of the primary transaction, the method further includes:

[0017] Add a transaction cache sequence number to the secondary transaction cache stored in the first cache queue. The transaction cache sequence number represents the processing order of each secondary transaction cache in the first cache queue.

[0018] The transaction cache sequence is updated according to the transaction cache number corresponding to the secondary transaction cache; the transaction cache sequence is used to reflect the secondary transaction cache information contained in the current primary transaction cache;

[0019] The updated transaction cache sequence number in the transaction cache sequence is fed back to the data processing requester.

[0020] In this embodiment, a transaction cache sequence is recorded in the first-level transaction. The updated transaction cache sequence number in the transaction cache sequence reflects the current transaction cache progress. At the same time, the updated transaction cache sequence number is fed back to each data processing requester to notify each data processing requester of the current transaction cache progress.

[0021] In one embodiment, the preset transaction cache commit conditions include at least one of the following: the number of transactions meets a preset transaction volume threshold, the transaction cache time is greater than or equal to a preset transaction cache time threshold, there is a waiting data processing requester, or the log device is abnormal.

[0022] In this embodiment, corresponding conditions for commit log solidification are set for different situations that may occur during the first-level transaction cache process, so that the first-level transaction cache can be committed to solidify logs under different conditions, thereby improving the stability of log solidification of the first-level transaction cache.

[0023] In one embodiment, after instructing the log device to perform log persistence processing on the data in the aggregate cache, the method further includes:

[0024] Use the target transaction cache sequence number corresponding to the first-level transaction cache after log solidification as the transaction completion sequence number;

[0025] Based on the transaction completion sequence number and the preset data processing requester wake-up conditions, the target data processing requester is determined from among the data processing requesters.

[0026] A wake-up message is sent to the target data processing requester, the wake-up message being used to notify the target data processing requester that data processing has ended.

[0027] In this embodiment, based on transaction aggregation caching, each time a log is committed and solidified in a first-level transaction, the local object management system updates the transaction completion sequence number. Then, based on the transaction completion sequence number and the preset data processing requester wake-up conditions, the system wakes up each target data processing requester waiting for data processing to finish, so as to notify each data processing requester that data processing has ended.

[0028] In one embodiment, the data processing requester corresponds to an end-of-transaction cache sequence number, which is an end identifier for the data processing request triggered by the data processing requester. The step of determining the target data processing requester among the data processing requesters based on the transaction completion sequence number and preset data processing requester wake-up conditions includes:

[0029] Among the end transaction cache sequence numbers corresponding to each of the data processing requesters, a target end transaction cache sequence number that is less than or equal to the transaction completion sequence number is determined.

[0030] Based on the correspondence between the end-of-transaction cache sequence number and the data processing requester, the target data processing requester corresponding to the target end-of-transaction cache sequence number is determined.

[0031] In this embodiment, the target data processing requester is determined among the data processing requesters by using the transaction completion sequence number and the end transaction cache sequence number corresponding to each data processing requester, so as to realize the wake-up of the target data processing requester.

[0032] In one embodiment, before determining the target data processing requester from among the data processing requesters based on the transaction completion sequence number and preset data processing requester wake-up conditions, the method further includes:

[0033] The target transaction cache sequence number corresponding to the first-level transaction cache that failed the log solidification process is used as the transaction failure sequence number; the transaction failure sequence number is used to identify the transaction cache that was not successfully committed in the current log solidification process.

[0034] Obtain the end-of-transaction cache sequence number corresponding to each data processing requester;

[0035] If the transaction end cache sequence number is less than or equal to the transaction failure sequence number, it is determined that the data processing request triggered by the data processing requester was executed successfully.

[0036] If the transaction end cache sequence number is greater than the transaction failure sequence number, and the transaction end cache sequence number is less than or equal to the transaction completion sequence number, then the data processing request triggered by the data processing request is determined to have failed.

[0037] In this embodiment, by determining the transaction failure sequence number, the transaction cache that fails during the log solidification process is marked. This not only determines the execution result of the data processing request, but also locates the specific failed transaction cache in the case of execution failure. Furthermore, the local object management system can notify the corresponding target data processing requester of the execution result, thereby realizing the monitoring and feedback of the data processing process.

[0038] Secondly, this application also provides a data processing apparatus. The apparatus includes:

[0039] A creation module is used to create a data processing flow in response to a data processing request; each data processing flow contains multiple task jobs.

[0040] The aggregation module is used to process and aggregate the data of the multiple tasks in the preset cache queues of each level of transactions based on a hierarchical aggregation processing strategy, so as to obtain an aggregate cache.

[0041] The submission module is used to submit the aggregated cache to the log device when the aggregated cache meets the preset transaction cache submission conditions, and to instruct the log device to perform log persistence processing on the data in the aggregated cache.

[0042] Using this device, multiple tasks can be cached through a preset transaction cache queue. Then, based on a hierarchical aggregation processing strategy, data processing and hierarchical aggregation of multiple tasks can be performed in the cache queue. Then, log solidification is performed on the aggregated cache only when the aggregated cache meets the preset transaction cache commit conditions, reducing the number of log solidifications, saving the data persistence time required for log solidification, and improving data processing efficiency.

[0043] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0044] In response to a data processing request, a data processing flow is created; each data processing flow contains multiple task jobs.

[0045] Based on a hierarchical aggregation processing strategy, the multiple task jobs are processed and aggregated in the preset cache queues of each level of transactions to obtain an aggregate cache.

[0046] When the aggregated cache meets the preset transaction cache commit conditions, the aggregated cache is committed to the log device, instructing the log device to perform log persistence processing on the data in the aggregated cache.

[0047] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0048] In response to a data processing request, a data processing flow is created; each data processing flow contains multiple task jobs.

[0049] Based on a hierarchical aggregation processing strategy, the multiple task jobs are processed and aggregated in the preset cache queues of each level of transactions to obtain an aggregate cache.

[0050] When the aggregated cache meets the preset transaction cache commit conditions, the aggregated cache is committed to the log device, instructing the log device to perform log persistence processing on the data in the aggregated cache.

[0051] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0052] In response to a data processing request, a data processing flow is created; each data processing flow contains multiple task jobs.

[0053] Based on a hierarchical aggregation processing strategy, the multiple task jobs are processed and aggregated in the preset cache queues of each level of transactions to obtain an aggregate cache.

[0054] When the aggregated cache meets the preset transaction cache commit conditions, the aggregated cache is committed to the log device, instructing the log device to perform log persistence processing on the data in the aggregated cache.

[0055] The aforementioned data processing methods, apparatus, computer equipment, storage media, and computer program products enable the local object management system to create data processing flows in response to data processing requests. Each data processing flow contains multiple tasks. Then, based on a hierarchical aggregation processing strategy, the local object management system processes and aggregates the multiple tasks within each data processing flow in a preset transaction cache queue, resulting in an aggregate cache. When the aggregate cache meets preset transaction cache commit conditions, it is committed to the log device, instructing the log device to perform log persistence processing on the data in the aggregate cache. This method achieves caching of multiple tasks through a preset transaction cache queue. Furthermore, based on a hierarchical aggregation processing strategy, data processing and hierarchical aggregation of multiple tasks can be implemented in the cache queue. Then, log persistence processing is only performed on the aggregate cache when it meets preset transaction cache commit conditions, reducing the number of log persistence operations, saving the data persistence time required for log persistence, and improving data processing efficiency. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the interaction process of a data processing method in one embodiment;

[0057] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0058] Figure 3 This is a flowchart illustrating the hierarchical aggregation process in one embodiment;

[0059] Figure 4 This is a flowchart illustrating a data processing method in one embodiment;

[0060] Figure 5This is a flowchart illustrating a method for updating the transaction cache sequence number in one embodiment;

[0061] Figure 6 This is a flowchart illustrating the steps of waking up the target data processing requester in one embodiment;

[0062] Figure 7 This is a flowchart illustrating the steps for determining the target data processing requester in one embodiment;

[0063] Figure 8 This is a flowchart illustrating the steps for determining the execution result of a data processing request in one embodiment.

[0064] Figure 9 This is a schematic diagram of an example flow for determining the execution result of a target data processing request in one embodiment;

[0065] Figure 10 This is a flowchart illustrating an example of a data processing method in one embodiment;

[0066] Figure 11 This is a structural block diagram of a data processing device in one embodiment;

[0067] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] In one embodiment, when processing data objects in a local object management system, for the same data processing requester, a single data processing request includes multiple data processing flows. Only after all these data processing flows have been executed can the completion of the data processing task be confirmed. Therefore, when facing data processing flows triggered by the same data processing requester, this application provides a data processing method. This method uses a hierarchical aggregation processing strategy to perform data processing and hierarchical aggregation on multiple tasks included in each data processing flow in preset cache queues for each level of transaction, thus aggregating the cache. Furthermore, when the aggregated cache meets preset transaction cache commit conditions, log persistence processing is performed on the aggregated cache, such as... Figure 1As shown, for write data requests (or data modification requests) triggered by the same user, log persistence and log application can be omitted during the data processing flow of opening and writing objects (i.e., no transaction application is required). Log persistence and log application (i.e., transaction application is required) can only be performed based on the overall aggregated cache after the task in the data processing flow of closing the object is completed. This reduces the number of log persistence and log application operations during data processing, saving the data persistence time consumed by multiple log persistence processes and improving data processing efficiency.

[0070] In one embodiment, the data processing method described above is illustrated using an example of its application to a local object management system. The local object management system is integrated on a server, specifically on the server's disk device. Figure 2 As shown, the data processing method includes the following steps:

[0071] Step 202: In response to the data processing request, create a data processing flow.

[0072] Each data processing flow contains multiple tasks.

[0073] In implementation, users can send data processing requests to the local object management system via a client. These requests instruct upper-level callers to perform data processing operations on the local object management system; the upper-level callers are the direct callers of the local object management system. The local object management system then responds to the upper-level caller's data processing operation by creating a corresponding data processing flow, such as... Figure 1 In this context, when a user triggers a data modification operation on the metadata of a target data object in the local object management system through an upper-level caller, this data modification operation includes at least three data processing flows: opening the target data object, writing to the target data object, and closing the target data object. Each data processing flow contains multiple tasks for data processing, where each task can be represented as a job.

[0074] Step 204: Based on the hierarchical aggregation processing strategy, multiple task jobs are processed and aggregated in the preset cache queues of each level of transactions to obtain an aggregate cache.

[0075] In implementation, when processing data objects in the local object management system (e.g., data modification), to ensure data consistency (i.e., restoring all programs or data to their previous correct state when errors occur), the local object management system protects the data processing process through transactions and controls the concurrency of multiple tasks through a locking mechanism. Specifically, the local object management system pre-defines hierarchical transactions and uses a pre-defined locking mechanism to ensure that only tasks that acquire the lock can enter a transaction. Tasks entering each level of transaction are stored in a cache queue. Then, based on a hierarchical aggregation processing strategy, the local object management system processes and aggregates multiple tasks in the pre-defined cache queues of each transaction level, thereby obtaining an aggregated cache.

[0076] Optionally, the locks set in the local object management system can be, but are not limited to, asynchronous read-write locks. This application embodiment does not limit the type of lock.

[0077] Step 206: When the aggregate cache meets the preset transaction cache commit conditions, the aggregate cache is committed to the log device, instructing the log device to perform log persistence processing on the data in the aggregate cache.

[0078] In implementation, the local object management system adopts a shuttle mechanism to periodically check whether the aggregate cache meets the preset transaction cache commit conditions. If the aggregate cache meets the preset transaction cache commit conditions, the local object management system will submit the aggregate cache to the log device, instructing the log device to perform log persistence processing on the data in the aggregate cache, that is, to write the data in the aggregate cache to the log device.

[0079] Optionally, after performing log persistence processing on the aggregate cache, the data written to the log device can be further processed by log caching, that is, the data written to the log device can be written to the cache of the log device.

[0080] Optionally, the log device may be, but is not limited to, a non-volatile device such as a disk, and this application embodiment does not limit this.

[0081] In the above data processing method, the local object management system responds to data processing requests and creates data processing flows. Each data processing flow contains multiple tasks. Then, based on a hierarchical aggregation processing strategy, the local object management system processes and aggregates the multiple tasks contained in each data processing flow in a preset transaction cache queue, resulting in an aggregate cache. When the aggregate cache meets the preset transaction cache commit conditions, it is committed to the log device, instructing the log device to perform log persistence processing on the data in the aggregate cache. This method uses a preset transaction cache queue to cache multiple tasks. Therefore, based on a hierarchical aggregation processing strategy, data processing and hierarchical aggregation of multiple tasks can be performed in the cache queue. Then, log persistence processing is only performed on the aggregate cache when it meets the preset transaction cache commit conditions, reducing the number of log persistence operations, saving the data persistence time required for log persistence, and improving data processing efficiency.

[0082] In one embodiment, the local object management system pre-defines transactions. A transaction is a program unit and possesses ACID properties (atomicity, consistency, isolation, and durability). Based on the ACID properties of transactions, it can be guaranteed that all operations within a transaction are either performed or not performed at all, and that they are not interfered with by other transactions, thus ensuring data consistency during data processing. Therefore, the local object management system uses transactions to protect the data processing process, enabling the completion of data processing and aggregation for multiple tasks within a transaction. Specifically, for example... Figure 3 As shown, in step 204, based on the hierarchical aggregation processing strategy, data processing and hierarchical aggregation are performed on multiple tasks included in each data processing flow in the preset transaction cache queue to obtain the specific processing process of the aggregation cache, including:

[0083] Step 302: For each data processing flow, store multiple task jobs into the second cache queue in the secondary transaction.

[0084] In implementation, user-triggered data processing requests correspond to multiple data processing flows (e.g., opening a data object, writing a data object, and closing a data object). For each data processing flow, the local object management system uses a locking mechanism to control the sequential entry of multiple tasks (jobs) within that flow into a secondary transaction. Specifically, as follows... Figure 4 As shown, after multiple tasks enter the secondary transaction, the secondary transaction has a pre-set second buffer queue. Therefore, the multiple tasks entering the secondary transaction are first cached through the second buffer queue to realize the serial processing of multiple tasks.

[0085] Step 304: By executing a secondary transaction, data processing is performed on multiple tasks in the second cache queue, and the multiple tasks after data processing are aggregated to obtain a secondary transaction cache.

[0086] In implementation, the local object management system executes the operations within the second-level transaction, that is, it processes the tasks in the second cache queue serially. First, it checks if there are any tasks being processed in the second-level transaction. If so, the task at the current position in the second cache queue remains in an incomplete state. If there are no tasks being processed, it performs data processing (e.g., metadata modification) on the task at the current position. Then, after all tasks in the second cache queue have been processed, the local object management system can treat all tasks in the second cache queue within the second-level transaction as an aggregation unit and perform second-level aggregation processing on all tasks in the second cache queue to obtain a second-level transaction cache.

[0087] Step 306: Store the secondary transaction cache into the first cache queue of the primary transaction, and aggregate the secondary transaction cache in the first cache queue to obtain the primary transaction cache, and use the primary transaction cache as the aggregated cache.

[0088] The transaction structure of the local object management system also includes a first-level transaction. The transaction processing order of the first-level transaction is after the second-level transaction, and the amount of aggregated data cached in the first-level transaction is higher than the amount of aggregated data in the second-level transaction.

[0089] In implementation, the local object management system includes a transaction structure of first-level transactions in addition to the second-level transaction structure, forming a multi-level transaction structure. This multi-level transaction structure enables hierarchical aggregation of tasks. Specifically, after the tasks have been aggregated through second-level transactions to obtain a second-level transaction cache, the local object management system can use a locking mechanism to control each second-level transaction cache to enter the first-level transaction. Then, the second-level transactions are cached through a first cache queue in the first-level transaction. Subsequently, the second-level transaction caches stored in the first cache queue are aggregated. That is, for the second-level transaction caches that have entered the first-level transaction, when the amount of second-level transaction caches in the first cache queue reaches a certain amount, the local object management system aggregates each second-level transaction cache in the first cache queue to obtain the first-level transaction cache.

[0090] In this embodiment, the data processing of multiple tasks in each level of transaction is protected by a preset hierarchical transaction, so that the data processing process is not disturbed, thereby ensuring the consistency of data during the data processing process. Furthermore, the hierarchical aggregation of multiple tasks in the transaction solves the problem of multiple log solidifications required for multiple tasks, saving data processing time and improving data processing efficiency.

[0091] In one embodiment, the first-level transaction also stores a transaction cache sequence to characterize the transaction cache status. This transaction cache sequence is constructed based on the transaction cache sequence number corresponding to each second-level transaction cache entering the first-level transaction. The transaction cache sequence number corresponding to the second-level transaction cache is determined according to a preset transaction cache sequence number determination strategy. The specific processing procedure for applying this transaction cache sequence number determination strategy will be described in detail in the following embodiments of this application, and will not be repeated here. Figure 5 As shown, after storing the secondary transaction cache into the first cache queue in the primary transaction in step S306, the method further includes:

[0092] Step S502: Add a transaction cache sequence number to the secondary transaction cache stored in the first cache queue.

[0093] The transaction cache sequence number represents the processing order of each secondary transaction cache in the first cache queue.

[0094] In implementation, the local object management system uses a locking mechanism to control the entry of second-level transaction caches into first-level transactions. When a second-level transaction cache enters a first-level transaction, the local object management system adds a transaction cache sequence number to that second-level transaction cache based on a preset cache sequence number determination strategy. Specifically, this transaction cache sequence number determination strategy is an incrementing strategy. For each second-level transaction cache entering a first-level transaction, based on its position in the first cache queue, the system obtains the transaction cache sequence number of the second-level transaction cache in the preceding adjacent position, and increments that sequence number to obtain the transaction cache sequence number of the newly entered second-level transaction cache. For example... Figure 3 The first cache queue of a first-level transaction already contains n3 second-level transaction caches, and the transaction cache sequence number corresponding to the n3rd second-level transaction cache is Trs_sn=n3. Therefore, when a new second-level transaction cache enters the first-level transaction again, the transaction cache sequence number corresponding to the second-level transaction cache can be determined as Trs_sn=n3+1 based on the preset transaction cache sequence number determination strategy. The transaction cache sequence numbers of the subsequent second-level transaction caches can be Trs_sn=n3+2, Trs_sn=n3+3, etc.

[0095] Step S504: Update the transaction cache sequence according to the transaction cache sequence number corresponding to the secondary transaction cache.

[0096] The transaction cache sequence is used to reflect the second-level transaction cache information contained in the current first-level transaction cache.

[0097] In implementation, when a new second-level transaction cache enters a first-level transaction, the local object management system updates the transaction cache sequence according to the transaction cache sequence number corresponding to the newly entered second-level transaction cache. Specifically, the local object management system records a transaction cache sequence in the first-level transaction. This transaction cache sequence is an increasing sequence, and the transaction cache sequence numbers contained in this sequence are used to reflect the second-level transaction cache information contained in the current first-level transaction.

[0098] Step S506: Feed back the updated transaction cache sequence number in the transaction cache sequence to the data processing requester.

[0099] In implementation, the local object management system can implement concurrent control of data processing requests triggered by each data processing requester. Therefore, the first cache queue can contain the second-level transaction cache of the data processing process triggered by each data processing requester. When a new second-level transaction cache is stored in the first cache queue, the transaction cache sequence will be updated accordingly with the transaction cache sequence number. The local object management system will feed back the updated transaction cache sequence number to the data processing requester, thereby notifying each data processing requester of the current first-level transaction cache progress.

[0100] In this embodiment, a transaction cache sequence is recorded in the first-level transaction. The updated transaction cache sequence number in the transaction cache sequence reflects the current transaction cache progress. At the same time, the updated transaction cache sequence number is fed back to each data processing requester to notify each data processing requester of the current transaction cache progress.

[0101] In one embodiment, the preset transaction cache commit conditions include at least one of the following: the number of transactions meets a preset transaction volume threshold, the transaction cache time is greater than or equal to a preset transaction cache time threshold, and there is a waiting data processing requester.

[0102] In implementation, after aggregating the caches of each secondary transaction through the primary transaction, the current primary transaction can be log-stabilized. Specifically, the conditions for committing log-stabilization may include the number of transactions meeting a preset transaction volume threshold, the transaction cache time being greater than or equal to a preset transaction cache time threshold, the existence of waiting data processing requesters, disk device anomalies, etc. When the transaction cache in the primary transaction meets at least one of the preset conditions for committing log-stabilization, the local object management system performs batch commit of the primary transaction cache in the primary transaction, that is, batch writes the primary transaction cache to the log device.

[0103] Specifically, for the same data processing requester, a single data processing request includes multiple data processing flows. Only after all these data processing flows have been executed can the completion of that data processing flow be confirmed. Therefore, the first-level transaction cache only needs to ensure that a log commit is performed after all multiple data processing flows corresponding to the same data processing requester have been executed (i.e., after data processing and hierarchical aggregation). Thus, a transaction volume threshold can be preset. When the transaction volume stored in the first cache queue meets the preset threshold, a log commit can be performed on the first-level transaction cache in the first cache queue, achieving batch processing of the first-level transaction cache. Furthermore, if the concurrency of current data processing requests is low, causing the transaction volume in the first cache queue to never reach the preset threshold, in order to ensure that the first-level transaction cache within the first-level transaction can complete the log commit, if there are data processing requesters (i.e., there are data processing request waiters) in the current local object management system, the first-level transaction cache within the first-level transaction can be committed to complete the data processing process for that data processing requester. To further ensure the processing of the first-level transaction cache, a transaction cache time threshold can be set. This threshold ensures that the waiting time after the data processing requester submits the data processing request will not be too long.

[0104] In this embodiment, corresponding conditions for commit log solidification are set for different situations that may occur during the first-level transaction cache process, so that the first-level transaction cache can be committed to solidify logs under different conditions, thereby improving the stability of log solidification of the first-level transaction cache.

[0105] In one embodiment, such as Figure 6 As shown, after instructing the log device to perform log persistence processing on the data in the aggregate cache in step S206, the method further includes:

[0106] Step S602: Use the sequence numbers of multiple transaction caches corresponding to the first-level transaction cache after log solidification as the transaction completion sequence numbers.

[0107] In implementation, after the local object management system writes the first-level transaction cache in batches to the log device, it uses the sequence numbers of multiple transaction caches corresponding to the first-level transaction caches committed to the log device as transaction completion numbers representing the completed transaction caches, and updates the transaction completion sequence to the transaction completion sequence. This transaction completion sequence is used to reflect the completion status of the transaction caches.

[0108] Step S604: Determine the target data processing requester from among the data processing requesters based on the transaction completion sequence number and the preset data processing requester wake-up conditions.

[0109] In implementation, a data processing request queue is pre-established in the local object management system. Data processing requesters that trigger data processing requests are stored in this queue according to the order in which they were triggered. Each time the local object management system performs a commit operation on the first-level transaction cache, it updates the transaction completion sequence number. Then, based on the updated transaction completion sequence number and the pre-defined data processing requester wake-up conditions, the system queries the data processing requester queue for the target data processing requester that meets those conditions.

[0110] Step S606: Send a wake-up message to the target data processing requester.

[0111] The wake-up message is used to notify the target data processing requester that data processing has ended.

[0112] In implementation, the local object management system sends a wake-up message to the target data processing requester, notifying the target data processing requester that the current data processing request has been completed.

[0113] In this embodiment, based on transaction aggregation caching, each time a log is committed and solidified in a first-level transaction, the local object management system updates the transaction completion sequence number. Then, based on the transaction completion sequence number and the preset data processing requester wake-up conditions, the system wakes up each target data processing requester waiting for data processing to finish, so as to notify each data processing requester that data processing has ended.

[0114] In one embodiment, such as Figure 7 As shown, step S604, which involves determining the specific processing procedure of the target data processing requester among the various data processing requesters based on the transaction completion sequence number and the preset data processing requester wake-up conditions, includes:

[0115] Step S702: Determine the target end transaction cache sequence number that is less than or equal to the transaction completion sequence number from the end transaction cache sequence numbers corresponding to each data processing requester.

[0116] The data processing requester has a corresponding end transaction cache sequence number, which serves as the end identifier for the data processing request triggered by the data processing requester.

[0117] Each data processing request corresponds to multiple data processing flows, which are executed in a specific order, such as opening a data object, writing a data object, and closing a data object. When processing and aggregating data within each flow, multiple tasks within each flow are aggregated into a second-level transaction cache in a second-level transaction. This cache is then further aggregated in a first-level transaction. Within the first-level transaction, a transaction cache identifier is added to each second-level transaction cache. Specifically, the transaction cache sequence number corresponding to the aggregated second-level transaction cache of the data processing flows at their end (e.g., the data processing flow for closing a data object) can be used as the end transaction cache number for the data processing request, thus identifying the end of the data processing request.

[0118] In practice, the local object management system determines a target end transaction cache sequence number that is less than or equal to the transaction completion sequence number from the end transaction cache sequence number corresponding to each data processing requester.

[0119] Step S704: Based on the correspondence between the end transaction cache sequence number and the data processing requester, determine the target data processing requester corresponding to the target end transaction cache sequence number.

[0120] In implementation, the local object management system determines the target data processing requester corresponding to the target end-of-transaction cache sequence number based on the correspondence between the end-of-transaction cache sequence number and the data processing requester.

[0121] In this embodiment, the target data processing requester is determined among the data processing requesters by using the transaction completion sequence number and the end transaction cache sequence number corresponding to each data processing requester, so as to realize the wake-up of the target data processing requester.

[0122] In one embodiment, for each data processing requester triggering multiple data processing flows, only after all data processing flows (i.e., sequentially executed to the last data processing flow) have completed their data processing and hierarchical aggregation operations, and only after ensuring that all transaction caches of all data processing flows (i.e., the last data processing flow detected in the execution order) have undergone log solidification processing, can it be determined that the data processing tasks triggered by the current data processing request have been completed sequentially. Therefore, before determining the target data processing requester among the data processing requesters in step S604 based on the transaction completion sequence number and the preset data processing requester wake-up conditions, this application uses the log solidification result of the last data processing flow (i.e., the final stage data processing flow) of each data processing requester to verify the execution result of this data processing request. Specifically, as follows... Figure 8 As shown, the method also includes:

[0123] Step S802: Use the target transaction cache sequence number corresponding to the first-level transaction cache that failed the log solidification process as the transaction failure sequence number.

[0124] The transaction failure sequence number is used to identify the cached transactions that were not successfully committed during the current log persistence process.

[0125] In implementation, when batch committing the first-level transaction cache from the first-level transaction to the log device, log persistence processing may fail. Therefore, to address log persistence failures, the local object management system records the target transaction cache sequence number corresponding to the failed first-level transaction cache as the transaction failure sequence number. The first-level transaction cache is aggregated from multiple second-level transaction caches; therefore, the target transaction cache sequence number corresponding to the first-level transaction cache is the transaction cache sequence number of the target second-level transaction cache within the first-level transaction cache where log persistence failed when the first-level transaction cache was committed.

[0126] Step S804: Obtain the end transaction cache sequence number corresponding to the target data processing requester.

[0127] In implementation, the local object management system obtains the end-of-transaction cache sequence number corresponding to the target data processing requester. This end-of-transaction cache sequence number serves as the end identifier of the data processing request triggered by the target data processing requester, and also as the log-based end identifier of the data processing flow at the end stage corresponding to the data processing request. The method for determining the end-of-transaction cache sequence number has already been given in the foregoing embodiments of this application, and will not be repeated here.

[0128] Step S806: If the transaction cache sequence number is less than or equal to the transaction failure sequence number, it is determined that the data processing request triggered by the target data processing requester was successfully executed.

[0129] In practice, if the transaction end cache sequence number is less than or equal to the transaction failure sequence number, it indicates that the log solidification failure only occurs in the first-level transaction cache after the data processing task triggered by the target data processing requester has ended and the data processing process log solidification is completed. This means that the data processing task triggered by the target data processing requester has not failed. Therefore, the local object management system determines that the data processing request triggered by the target data processing requester was executed successfully.

[0130] Step S808: If the transaction end cache sequence number is greater than the transaction failure sequence number, and the transaction end cache sequence number is less than or equal to the transaction completion sequence number, it is determined that the data processing request triggered by the target data processing request has failed.

[0131] In practice, if the end transaction cache sequence number corresponding to the target data processing requester is greater than the transaction failure sequence number, it means that the data processing request triggered by the target data processing requester failed to log solidification before execution was completed. In this case, if the end transaction cache sequence number is less than or equal to the transaction completion sequence number, it indicates that the data processing request triggered by the target data processing requester has also been completed. Therefore, the local object management system can determine that the data processing request triggered by the target data processing requester failed to execute.

[0132] like Figure 9 As shown, an example is given of verifying the execution result of a data processing request corresponding to a target data requester. This data processing request is a data modification request for a data object. The final stage of the data processing request is closing the data object. The verification of the execution result of the data processing request is the same as the specific processing steps S806 to S808 in the above embodiments, and will not be repeated in this application embodiment. However, if it is not one of the two situations given in steps S806 to S808, such as... Figure 9 As shown, if the end transaction cache sequence number corresponding to the target data processing requester is greater than the transaction failure sequence number and greater than the transaction completion sequence number, it indicates that the data processing request has not yet been completed. Therefore, the local object management system stores the target data processing requester corresponding to the data processing request in the data processing requester queue, waits for the data processing request to be completed, wakes up the target data processing requester, and notifies the target data processing requester of the execution result of the data processing request.

[0133] In this embodiment, by determining the transaction failure sequence number, the transaction cache that fails during the log solidification process is marked. This not only determines the execution result of the data processing request, but also locates the specific failed transaction cache in the case of execution failure. Furthermore, the local object management system can notify the corresponding target data processing requester of the execution result, thereby realizing the monitoring and feedback of the data processing process.

[0134] In one embodiment, such as Figure 10 As shown, an example method for a data processing request is provided, which includes:

[0135] Step S1001: In a secondary transaction, data processing is performed on multiple tasks (jobs) in the second cache queue, and the multiple tasks (jobs) after data processing are completed are aggregated.

[0136] In step S1002, if multiple task jobs successfully process and aggregate data in the secondary transaction, then proceed to step S1003; if multiple task jobs fail to process all data successfully, then roll back all data of multiple task jobs and return to step S1001.

[0137] Step S1003: Store the secondary transaction cache into the first cache queue of the primary transaction, and aggregate the secondary transaction cache in the first cache queue to obtain the primary transaction cache, and use the primary transaction cache as the aggregated cache.

[0138] Step S1004: Determine whether the current first-level transaction cache meets the preset transaction cache commit conditions. If it does, proceed to step S1006; otherwise, proceed to step S1005.

[0139] Step S1005: Return the transaction cache sequence number corresponding to the current first-level transaction cache, and return to step S1001.

[0140] Step S1006: Batch commit the first-level transaction cache and record the transaction cache amount in each commit log (i.e., mark the transaction completion sequence number).

[0141] Step S1007: Log submission and solidification successful.

[0142] Step S1008: Execute the completion callbacks for all first-level transaction caches in the current batch.

[0143] Step S1009: Verify the execution result of the currently completed first-level transaction cache, and wake up the target data processing requester based on the execution result obtained from the verification.

[0144] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0145] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0146] In one embodiment, such as Figure 11As shown, a data processing apparatus 1100 is provided, including: a creation module 1101, an aggregation module 1102, and a submission module 1103, wherein:

[0147] Create module 1101 to create a data processing flow in response to a data processing request; each data processing flow contains multiple tasks.

[0148] The aggregation module 1102 is used to process and aggregate data from multiple tasks in the cache queues of pre-defined transactions at each level based on a hierarchical aggregation processing strategy, and obtain an aggregate cache.

[0149] The submission module 1103 is used to submit the aggregate cache to the log device when the aggregate cache meets the preset transaction cache submission conditions, and instruct the log device to perform log solidification processing on the data in the aggregate cache.

[0150] Using this device, multiple tasks can be cached through a preset transaction cache queue. Then, based on a hierarchical aggregation processing strategy, data processing and hierarchical aggregation of multiple tasks can be performed in the cache queue. Then, log solidification is performed on the aggregated cache only when the aggregated cache meets the preset transaction cache commit conditions, reducing the number of log solidifications, saving the data persistence time required for log solidification, and improving data processing efficiency.

[0151] In one embodiment, the aggregation module 1101 is specifically used to store multiple task jobs into a second cache queue in a secondary transaction for each data processing flow;

[0152] By executing secondary transactions, data processing is performed on multiple tasks in the second cache queue, and the processed tasks are aggregated to obtain the secondary transaction cache.

[0153] The second-level transaction cache is stored in the first cache queue of the first-level transaction, and the second-level transaction cache is aggregated in the first cache queue to obtain the first-level transaction cache. The first-level transaction cache is used as the aggregated cache.

[0154] In one embodiment, the data processing apparatus 1100 further includes:

[0155] The module adds transaction cache numbers to the secondary transaction caches stored in the first cache queue. The transaction cache number represents the processing order of each secondary transaction cache in the first cache queue.

[0156] The update module is used to update the transaction cache sequence according to the transaction cache sequence number corresponding to the second-level transaction cache; the transaction cache sequence is used to reflect the second-level transaction cache information contained in the current first-level transaction cache;

[0157] The feedback module is used to send the updated transaction cache sequence number in the transaction cache sequence back to the data processing requester.

[0158] In one embodiment, the preset transaction cache commit conditions include at least one of the following: the number of transactions meets a preset transaction volume threshold, the transaction cache time is greater than or equal to a preset transaction cache time threshold, there is a waiting data processing requester, or the log device is abnormal.

[0159] In one embodiment, the data processing apparatus 1100 further includes:

[0160] The first processing module is used to take the target transaction cache sequence number corresponding to the first-level transaction cache after log solidification as the transaction completion sequence number;

[0161] The determination module is used to determine the target data processing requester from among the data processing requesters based on the transaction completion sequence number and the preset data processing requester wake-up conditions.

[0162] The sending module is used to send a wake-up message to the target data processing requester. The wake-up message is used to notify the target data processing requester that data processing has ended.

[0163] In one embodiment, the data processing requester corresponds to an end transaction cache sequence number, which is the end identifier of the data processing request triggered by the data processing requester. The determining module is specifically used to determine a target end transaction cache sequence number that is less than or equal to the transaction completion sequence number from the end transaction cache sequence numbers corresponding to each data processing requester.

[0164] Based on the correspondence between the end-of-transaction cache sequence number and the data processing requester, the target data processing requester corresponding to the target end-of-transaction cache sequence number is determined.

[0165] In one embodiment, the data processing apparatus further includes:

[0166] The second processing module is used to take the target transaction cache sequence number corresponding to the first-level transaction cache that failed the log solidification process as the transaction failure sequence number; the transaction failure sequence number is used to identify the transaction cache that was not successfully committed in the current log solidification process.

[0167] The acquisition module is used to obtain the end transaction cache sequence number corresponding to each data processing requester;

[0168] The third determination module is used to determine that the data processing request triggered by the data processing requester was successfully executed if the end transaction cache sequence number is less than or equal to the transaction failure sequence number.

[0169] The fourth determination module is used to determine that the data processing request triggered by the data processing request has failed if the transaction completion cache number is less than or equal to the transaction completion number, provided that the transaction completion cache number is greater than the transaction failure number.

[0170] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0171] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0172] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0173] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0174] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0175] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0176] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: In response to a data processing request, a data processing flow is created; each data processing flow contains multiple task jobs. For each of the data processing flows, the multiple task jobs are stored in the second cache queue of the secondary transaction; By executing the secondary transaction, data processing is performed on multiple tasks in the second cache queue, and the multiple tasks after the data processing is completed are aggregated to obtain the secondary transaction cache. The secondary transaction cache is stored in the first cache queue of the primary transaction, and the secondary transaction cache is aggregated in the first cache queue to obtain the primary transaction cache, which is then used as the aggregated cache. When the aggregated cache meets the preset transaction cache commit conditions, the aggregated cache is committed to the log device, instructing the log device to perform log persistence processing on the data in the aggregated cache.

2. The method according to claim 1, characterized in that, After storing the secondary transaction cache into the first cache queue of the primary transaction, the method further includes: Add a transaction cache sequence number to the secondary transaction cache stored in the first cache queue. The transaction cache sequence number represents the processing order of each secondary transaction cache in the first cache queue. The transaction cache sequence is updated according to the transaction cache number corresponding to the secondary transaction cache; the transaction cache sequence is used to reflect the secondary transaction cache information contained in the current primary transaction cache; The updated transaction cache sequence number in the transaction cache sequence is fed back to the data processing requester.

3. The method according to claim 1, characterized in that, The preset transaction cache commit conditions include at least one of the following: the number of transactions meets a preset transaction volume threshold, the transaction cache time is greater than or equal to a preset transaction cache time threshold, there is a waiting data processing requester, or the log device is abnormal.

4. The method according to claim 2, characterized in that, After instructing the log device to perform log persistence processing on the data in the aggregate cache, the method further includes: Use the target transaction cache sequence number corresponding to the first-level transaction cache after log solidification as the transaction completion sequence number; Based on the transaction completion sequence number and the preset data processing requester wake-up conditions, the target data processing requester is determined from among the data processing requesters. A wake-up message is sent to the target data processing requester, the wake-up message being used to notify the target data processing requester that data processing has ended.

5. The method according to claim 4, characterized in that, The data processing requester corresponds to an end-of-transaction cache sequence number, which is the end identifier of the data processing request triggered by the data processing requester. The step of determining the target data processing requester among the data processing requesters based on the transaction completion sequence number and preset data processing requester wake-up conditions includes: Among the end transaction cache sequence numbers corresponding to each of the data processing requesters, a target end transaction cache sequence number that is less than or equal to the transaction completion sequence number is determined. Based on the correspondence between the end-of-transaction cache sequence number and the data processing requester, the target data processing requester corresponding to the target end-of-transaction cache sequence number is determined.

6. The method according to claim 4, characterized in that, Before determining the target data processing requester from among the data processing requesters based on the transaction completion sequence number and the preset data processing requester wake-up conditions, the method further includes: The target transaction cache sequence number corresponding to the first-level transaction cache that failed the log solidification process is used as the transaction failure sequence number; the transaction failure sequence number is used to identify the transaction cache that was not successfully committed in the current log solidification process. Obtain the end-of-transaction cache sequence number corresponding to each data processing requester; If the transaction end cache sequence number is less than or equal to the transaction failure sequence number, it is determined that the data processing request triggered by the data processing requester was executed successfully. If the transaction end cache sequence number is greater than the transaction failure sequence number, and the transaction end cache sequence number is less than or equal to the transaction completion sequence number, then the data processing request triggered by the data processing request is determined to have failed.

7. A data processing apparatus, characterized in that, The device includes: A creation module is used to create a data processing flow in response to a data processing request; each data processing flow contains multiple task jobs. An aggregation module is used to store the multiple task jobs into a second cache queue in a second-level transaction for each data processing flow; to process the multiple task jobs in the second cache queue by executing the second-level transaction, and to aggregate the multiple task jobs after the data processing is completed to obtain a second-level transaction cache; to store the second-level transaction cache into a first cache queue of a first-level transaction, and to aggregate the second-level transaction cache in the first cache queue to obtain a first-level transaction cache, and to use the first-level transaction cache as an aggregation cache; The submission module is used to submit the aggregated cache to the log device when the aggregated cache meets the preset transaction cache submission conditions, and to instruct the log device to perform log persistence processing on the data in the aggregated cache.

8. The apparatus according to claim 7, characterized in that, The device further includes: An add module is used to add a transaction cache sequence number to the secondary transaction cache stored in the first cache queue. The transaction cache sequence number represents the processing order of each secondary transaction cache in the first cache queue. The update module is used to update the transaction cache sequence according to the transaction cache sequence number corresponding to the secondary transaction cache; the transaction cache sequence is used to reflect the secondary transaction cache information contained in the current primary transaction cache; The feedback module is used to send the updated transaction cache sequence number in the transaction cache sequence back to the data processing requester.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images