Log playback method, apparatus, device, and medium

Abstract

The application provides a log playback method, device and equipment and medium, wherein the method comprises: generating a to-be-playback log and storing the to-be-playback log in a log storage module according to a user request; converting the to-be-playback log in the log storage module into a to-be-playback event by calling an event center thread; selecting a target playback queue from a plurality of playback queues; the plurality of playback queues and a plurality of playback threads correspond to each other; adding the to-be-playback event into the target playback queue, so that the to-be-playback event in the target playback queue is sequentially taken out by a target playback thread corresponding to the target playback queue for processing, and playback processing of the to-be-playback event is completed. The scheme of the application can improve the log playback speed of the client and accelerate the synchronization speed of the client to the storage backend.

Description

Technical Field

[0001] This application relates to the field of communications, and more particularly to a log playback method, apparatus, device, and medium. Background Technology

[0002] As the number of concurrent client requests increases, the client will ensure the atomicity of the requests through log replay. Log replay also updates metadata and synchronizes user data to the storage backend, so the speed of log replay is crucial for updating data and metadata.

[0003] If the latency of data synchronization to the S3 storage backend is high, and the playback speed is lower than the write speed for a long time, it will cause too much data to accumulate on the cache disk. In severe cases, it may even cause the upper-layer IO to slow down or even malfunction. Therefore, how to improve the log playback speed of the client becomes a problem that needs to be solved. Summary of the Invention

[0004] This application provides a log playback method, apparatus, device, and medium to improve the log playback speed of a client.

[0005] On the one hand, this application provides a log playback method, the method comprising:

[0006] Based on the user request, generate a log to be replayed and store the log to be replayed in the log storage module; by calling the event center thread, convert the log to be replayed in the log storage module into a replay event;

[0007] Select a target replay queue from multiple replay queues; wherein, each of the multiple replay queues corresponds to a multiple replay thread;

[0008] The event to be replayed is added to the target replay queue, so that the target replay thread corresponding to the target replay queue can sequentially retrieve the event to be replayed in the target replay queue for processing, thereby completing the replay processing of the event to be replayed.

[0009] In one embodiment, selecting a target replay queue from a plurality of replay queues includes:

[0010] Perform a modulo operation between the hash value of the event to be replayed and the number of queues in the plurality of replay queues;

[0011] The replay queue corresponding to the result of the modulo operation is determined as the target replay queue.

[0012] In one embodiment, adding the event to be replayed to the target replay queue includes:

[0013] Determine the operation type of the event to be replayed;

[0014] If the operation type of the event to be replayed is a rename operation, then check whether the current multiple replay queues are empty; if they are empty, then add the event to be replayed to the target replay queue; otherwise, wait for all events in the multiple replay queues to be replayed before adding the event to the target replay queue.

[0015] If the operation type of the event to be replayed is a write operation or an update metadata operation, then obtain the resource amount of the first event belonging to the write operation and update metadata operation in the multiple replay queues; if the resource amount of the first event exceeds a preset first threshold, then wait for all events in the multiple replay queues to be replayed before adding the event to be replayed to the target replay queue; otherwise, add the event to be replayed to the target replay queue.

[0016] If the operation type of the event to be replayed is a write operation or an update metadata operation, then it is detected whether the amount of worker thread resources in the working state among the multiple replay threads exceeds a preset second threshold; if it does not exceed the threshold, then the event to be replayed is added to the target replay queue; otherwise, it waits until the amount of worker thread resources does not exceed the second threshold.

[0017] In one embodiment, generating the log to be replayed according to the user request includes:

[0018] Based on the user's request, a log to be replayed is generated and an identifier is added to the log to be replayed. The identifier represents the log file where the log to be replayed is located and the offset of the log to be replayed in the log file.

[0019] In one embodiment, the method further includes:

[0020] Add the logs corresponding to the events that have completed playback processing to the completed event queue set;

[0021] If multiple events with the same identifier and consecutive offsets exist in the event queue set, then that log file is deleted from the log storage module.

[0022] In one embodiment, the method further includes:

[0023] For each event being processed by the playback thread, calculate the difference between the amount of thread resources allocated to the event and the amount of thread resources required by the event. If the difference is not zero, release a portion of the thread resources allocated to the event until the current difference is zero.

[0024] On the other hand, this application provides a log playback device, the device comprising:

[0025] The generation module is used to generate logs to be replayed according to user requests and store the logs to be replayed in the log storage module; and to convert the logs to be replayed in the log storage module into events to be replayed by calling the event center thread.

[0026] The selection module is used to select a target replay queue from multiple replay queues; wherein, the multiple replay queues correspond one-to-one with multiple replay threads;

[0027] The processing module is used to add the event to be replayed to the target replay queue, so that the target replay thread corresponding to the target replay queue can sequentially retrieve the event to be replayed in the target replay queue for processing, thereby completing the replay processing of the event to be replayed.

[0028] In one embodiment, the selection module is specifically used to perform a modulo operation on the hash value of the event to be replayed and the number of queues of the plurality of replay queues;

[0029] The selection module is further configured to determine the replay queue corresponding to the result of the modulo operation as the target replay queue.

[0030] In one embodiment, the processing module is specifically used to determine the operation type of the event to be replayed;

[0031] The processing module is further configured to, if the operation type of the event to be replayed is a renaming operation, detect whether the current multiple replay queues are empty; if empty, add the event to be replayed to the target replay queue; otherwise, wait for all events in the multiple replay queues to be replayed to be completed before adding the event to be replayed to the target replay queue.

[0032] The processing module is further configured to, if the operation type of the event to be replayed is a write operation or an update metadata operation, obtain the resource amount of the first event belonging to the write operation and update metadata operation in the plurality of replay queues; if the resource amount of the first event exceeds a preset first threshold, wait for all events in the plurality of replay queues to be replayed before adding the event to be replayed to the target replay queue; otherwise, add the event to be replayed to the target replay queue.

[0033] The processing module is further configured to, if the operation type of the event to be replayed is a write operation or an update metadata operation, detect whether the amount of worker thread resources in the working state among the multiple replay threads exceeds a preset second threshold; if it does not exceed the threshold, add the event to be replayed to the target replay queue; otherwise, wait until the amount of worker thread resources does not exceed the second threshold.

[0034] In one embodiment, the generation module is specifically used to generate a log to be replayed according to a user request and add an identifier to the log to be replayed, wherein the identifier represents the log file where the log to be replayed is located and the offset of the log to be replayed in the log file.

[0035] In one embodiment, the processing module is further configured to add the log corresponding to the completed playback process to the completed event queue set;

[0036] The processing module is further configured to delete the log file in the log storage module if there are multiple events in the event queue set that have the same identifier and consecutive offsets.

[0037] In one embodiment, the processing module is further configured to calculate, for each event being processed by the playback thread, the difference between the amount of thread resources allocated to the event and the amount of thread resources required by the event; if the difference is not zero, release a portion of the thread resources allocated to the event until the current difference is zero.

[0038] In another aspect, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0039] The memory stores computer-executed instructions;

[0040] The processor executes the computer execution instructions stored in the memory to implement the log playback method as described above.

[0041] In another aspect, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the log playback method described above.

[0042] The log replay method, apparatus, device, and medium provided in this application generate a log to be replayed based on a user request, and store the log to be replayed in a log file of a log storage module. The log to be replayed is then converted into a replay event by calling an event center thread. A target replay queue is selected from multiple replay queues, and the replay event is added to the target replay queue. The target replay thread corresponding to the target replay queue then sequentially retrieves the replay events from the target replay queue for processing, completing the replay process. This application improves the log replay speed of the client and accelerates the synchronization speed between the client and the storage backend by using the multiple replay queues and their corresponding replay threads to complete the parallel replay of the replay events when a large number of users concurrently request the client. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 This is a schematic diagram illustrating an application scenario for this application;

[0045] Figure 2 A flowchart illustrating a log playback method provided in Embodiment 1 of this application;

[0046] Figure 3 This is a schematic diagram of a log playback processing system;

[0047] Figure 4 This is a schematic diagram illustrating the interaction of the replay thread;

[0048] Figure 5 A flowchart illustrating another log playback method provided in Embodiment 1 of this application;

[0049] Figure 6 This is a schematic diagram of the structure of a log playback device provided in Embodiment 2 of this application;

[0050] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0051] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] Figure 1 This is a schematic diagram of an application scenario for this application. The diagram illustrates the communication between the playback processing module and the storage backend. The playback processing module processes the logs to be played back and stores the data corresponding to the logs to be played back in the storage backend. This application uses the S3 storage backend as an example to complete the playback processing of the logs to be played back.

[0054] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0055] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In the description of this application, unless otherwise expressly specified and limited, the terms should be broadly understood within the art. The embodiments of this application will now be described with reference to the accompanying drawings.

[0056] Example 1

[0057] Figure 2 This is a flowchart illustrating a log playback method provided in Embodiment 1 of this application, as shown below. Figure 2 As shown, the method includes:

[0058] Step 201: Based on the user request, generate a log to be replayed and store the log to be replayed in the log storage module; by calling the event center thread, convert the log to be replayed in the log storage module into a replay event;

[0059] Step 202: Select a target replay queue from multiple replay queues; wherein, each of the multiple replay queues corresponds to a different replay thread.

[0060] Step 203: Add the event to be replayed to the target replay queue, so that the target replay thread corresponding to the target replay queue can sequentially retrieve the events to be replayed in the target replay queue for processing, thereby completing the replay processing of the events to be replayed.

[0061] The execution entity in this embodiment is a log playback processing system, which can be implemented by a computer program, such as application software; or it can be implemented by a medium storing the relevant computer program, such as a USB flash drive or cloud drive; or it can be implemented by a physical device that integrates or installs the relevant computer program, such as a chip.

[0062] Figure 3 This is a schematic diagram of a log replay processing system. The system primarily facilitates information exchange between the client and the storage backend, which is an S3 storage backend. The S3 storage backend provides disk storage for log replay. The client includes a request processing module, a log storage module, and a replay processing module. The replay module includes an EventCenter thread, a replay queue, and a replay thread. The request processing module converts user requests received by the client into logs to be replayed and writes these logs to log files in the log storage module. The log storage module includes multiple log files, which primarily record and store the logs to be replayed. The log storage module can be selected as a Journaler module. The request processing module stores the logs to be replayed in one of the log files. When the logs stored in one log file reach the maximum storage capacity of that log file, the request processing module then stores the remaining logs to be replayed in other log files. The EventCenter thread in the playback module deserializes the logs to be played back in the log file into specific events to be played back, and puts the events to be played back into the playback queue. The playback thread corresponding to the playback queue plays back the events to be played back and writes the data to disk to the core module of the S3 storage backend.

[0063] like Figure 3 As shown, in the log replay processing system, there are multiple replay queues, each with a corresponding replay thread. Each replay thread is responsible for processing the replay of the events to be replayed. The EventCenter thread selects a target replay queue for each event from the multiple replay queues, places the event into the selected target replay queue, and the replay thread corresponding to the target replay queue completes the replay processing of the event.

[0064] Figure 4This is a diagram illustrating the interaction of the replay thread, such as... Figure 4 As shown, each playback thread contains two modules: a Metadata Service Client (MDSC) module and an Object Storage Service Client (S3C) module. During the playback process of the playback thread, the MDSC is responsible for interacting with the metadata cluster, which is a collection of attributes of each log to be played back. The MDSC interacts with the metadata cluster to obtain attributes such as the log name and size corresponding to the event to be played back. The S3C module interacts with the S3 storage backend to persist the data of the event to be played back to disk and store it on the S3 storage backend, thus completing the playback process of the event to be played back.

[0065] In this example, the request processing module converts user requests into logs to be replayed and stores them in the log file of the log storage module. Then, the EventCenter thread converts the logs to be replayed into specific events to be replayed and selects a target replay queue from multiple replay queues for the logs to be replayed. The replay thread corresponding to the target replay queue completes the replay processing of the events to be replayed. In this example, there are multiple replay queues in the log replay processing system, and each replay queue corresponds to a replay thread. Each replay thread completes the replay processing of the events to be replayed in its corresponding replay queue. The parallel completion of the replay processing of events to be replayed in multiple replay queues by multiple replay threads can improve the log replay speed of the client and speed up the synchronization speed from the client to the storage backend.

[0066] Optional, Figure 5 This is a flowchart illustrating another log playback method provided in Embodiment 1 of this application, as shown below. Figure 5 As shown, step 202 includes:

[0067] Step 501: Perform a modulo operation between the hash value of the event to be replayed and the number of queues in the multiple replay queues;

[0068] Step 502: Determine the replay queue corresponding to the result of the modulo operation as the target replay queue.

[0069] In the example scenario, each event to be replayed includes its corresponding hash value in its attributes. The EventCenter thread performs a modulo operation between the hash value of the event to be replayed and the number of replay queues. Based on the result of the modulo operation, it determines the replay queue corresponding to the event to be replayed and places the event to be replayed into the corresponding replay queue. It's worth noting that a user's request may contain multiple steps, each of which is converted into an event to be replayed. Multiple events to be replayed from the same user request belong to the same key event. Each event to be replayed within the same key event has the same hash value. Because each event to be replayed within the same key event has the same hash value, they are all placed into the same replay queue.

[0070] This example determines the replay queue corresponding to the event to be replayed by performing a modulo operation on the hash value corresponding to the event to be replayed and the number of the multiple replay queues, which can ensure the balance of the number of events to be replayed in each replay queue.

[0071] Optionally, adding the event to be replayed to the target replay queue includes:

[0072] Determine the operation type of the event to be replayed;

[0073] If the operation type of the event to be replayed is a rename operation, then check whether the current multiple replay queues are empty; if they are empty, then add the event to be replayed to the target replay queue; otherwise, wait for all events in the multiple replay queues to be replayed before adding the event to the target replay queue.

[0074] If the operation type of the event to be replayed is a write operation or an update metadata operation, then obtain the resource amount of the first event belonging to the write operation and update metadata operation in the multiple replay queues; if the resource amount of the first event exceeds a preset first threshold, then wait for all events in the multiple replay queues to be replayed before adding the event to be replayed to the target replay queue; otherwise, add the event to be replayed to the target replay queue.

[0075] If the operation type of the event to be replayed is a write operation or an update metadata operation, then it is detected whether the amount of worker thread resources in the working state among the multiple replay threads exceeds a preset second threshold; if it does not exceed the threshold, then the event to be replayed is added to the target replay queue; otherwise, it waits until the amount of worker thread resources does not exceed the second threshold.

[0076] Based on the scenario example, after determining the playback queue corresponding to the event to be played back, before putting the event to be played back into the corresponding playback queue, the operation type of the event to be played back should be determined first. The event to be played back has multiple operation types, and different playback processing operations correspond to different operation types.

[0077] If the operation type of the event to be replayed is a rename operation, the number of events to be replayed in each replay queue must be zero before the event to be replayed can be added to the corresponding replay queue for subsequent replay processing. Therefore, when the operation type of the event to be replayed is a rename operation, the number of events to be replayed in each replay queue should be checked first. If there are still events to be replayed in any replay queue, the event to be replayed can only be added to the corresponding replay queue after all the events to be replayed in the replay queue have completed their replay processing.

[0078] If the operation type of the event to be replayed is a write operation or a metadata update operation, the resource amount occupied by all events to be replayed in each replay queue should be detected first. A first threshold should be set for the resource amount occupied by the events to be replayed, which is the maximum capacity of the total resource amount occupied by the events to be replayed in each queue. Before putting the events to be replayed for write operations or metadata update operations into the replay queue, if it is detected that the resource amount occupied by all events to be replayed for write operations or metadata update operations in all replay queues has exceeded the first threshold, the EventCenter thread needs to manage the resource amount in each replay queue to avoid congestion of events to be replayed. The management method is to wait until all events to be replayed in all replay queues have completed the replay processing before putting the events to be replayed for write operations or metadata update operations into the corresponding replay queue for replay processing. A second threshold is set for the amount of worker thread resources in the working state. The second threshold is the same as the first threshold. If the operation type of the event to be replayed is a write operation or a metadata update operation, and the amount of worker thread resources in the working state exceeds the second threshold, the excess part is set to a waiting state. When the amount of worker thread resources in the working state is less than the second threshold, the event to be replayed in the waiting state is then replayed.

[0079] This example categorizes the operation types of the events to be replayed and takes different replay processing operations according to different categories to avoid errors in the replay processing of the events to be replayed.

[0080] Optionally, generating the replay log according to the user request includes:

[0081] Based on the user's request, a log to be replayed is generated and an identifier is added to the log to be replayed. The identifier represents the log file where the log to be replayed is located and the offset of the log to be replayed in the log file.

[0082] In this scenario example, after the request processing module converts the user request into a log to be replayed and stores it in a log file in the log storage module, the Header module in the log storage module marks the log to be replayed. This mark indicates the log file containing the log to be replayed and its offset within that file. This example demonstrates how marking the log to be replayed determines its location.

[0083] Optionally, the method further includes:

[0084] Add the logs corresponding to the events that have completed playback processing to the completed event queue set;

[0085] If multiple events with the same identifier and consecutive offsets exist in the event queue set, then that log file is deleted from the log storage module.

[0086] In a scenario example, after the playback thread selects an event to be played back from the corresponding playback queue and completes the playback processing, the completed playback events are added to a completed event queue set, which is a collection of all events that have completed playback processing. Based on the tag information in the completed playback events, the log file containing the completed playback events and their offsets within that log file are obtained. According to the tag information, if the offsets corresponding to completed playback events in the same log file are consecutive, it indicates that all the log files stored in the log file have completed playback processing. At this time, the log file can be updated by deleting the completed playback log files to facilitate the restoring of new log files.

[0087] Optionally, the method further includes:

[0088] For each event being processed by the playback thread, calculate the difference between the amount of thread resources allocated to the event and the amount of thread resources required by the event. If the difference is not zero, release a portion of the thread resources allocated to the event until the current difference is zero.

[0089] In this scenario example, the process of the playback thread selecting an event to be played back from the corresponding playback queue requires the allocation of thread resources. The difference between the number of thread resources allocated to the playback queue for completing the playback of the event and the number of thread resources required to complete the playback of the event is used to determine whether excess processing threads have been allocated for the playback of the event. If the difference is not zero, it can be confirmed that excess processing threads have been allocated for the playback of the event, and some processing threads need to be released to make the difference zero. This example determines whether excess processing threads have been allocated for the playback of the event by calculating the difference between the two, and releases some threads after allocating excess threads, avoiding the waste of available threads and maximizing resource utilization.

[0090] In this embodiment, the request processing module converts user requests into logs to be replayed and stores them in the log file of the log storage module. Then, the EventCenter thread converts the logs to be replayed into specific events to be replayed and selects a target replay queue from multiple replay queues for the logs to be replayed. The replay thread corresponding to the target replay queue completes the replay processing of the events to be replayed. In this embodiment, multiple replay threads complete the replay processing of events to be replayed in multiple replay queues in parallel, which can improve the log replay speed of the client and speed up the synchronization speed from the client to the storage backend.

[0091] Example 2

[0092] Figure 6 This is a schematic diagram of a log playback device provided in Embodiment 2 of this application. The device includes:

[0093] The generation module 61 is used to generate logs to be replayed according to user requests and store the logs to be replayed in the log storage module; and to convert the logs to be replayed in the log storage module into events to be replayed by calling the event center thread.

[0094] The selection module 62 is used to select a target replay queue from multiple replay queues; wherein, the multiple replay queues correspond one-to-one with multiple replay threads;

[0095] The processing module 63 is used to add the event to be replayed to the target replay queue, so that the target replay thread corresponding to the target replay queue can sequentially retrieve the event to be replayed in the target replay queue for processing, thereby completing the replay processing of the event to be replayed.

[0096] In a scenario example, generation module 61 converts user requests received by the client into logs to be replayed and writes these logs to log files in the log storage module. The log storage module includes multiple log files. The request processing module stores the logs to be replayed in one of these log files. When the logs stored in one log file reach the maximum storage capacity of that file, the request processing module stores the remaining logs to be replayed in other log files. Generation module 61 deserializes the logs to be replayed in the log files into specific events to be replayed. In the log replay processing system, there are multiple replay queues, each with a corresponding replay thread. Selection module 62 selects a target replay queue for each event to be replayed from the multiple replay queues and places the event into the selected target replay queue. The replay thread in processing module 63 then replays the event and writes the data to disk to the core module of the S3 storage backend.

[0097] In this example, the generation module 61 converts user requests into logs to be replayed and stores them in a log file. Then, the logs to be replayed are converted into specific events to be replayed. The selection module 62 selects a target replay queue from multiple replay queues for the logs to be replayed. The replay thread in the processing module 63 completes the replay processing of the events to be replayed. In this example, there are multiple replay queues, each corresponding to a replay thread. Each replay thread completes the replay processing of the events to be replayed in its corresponding replay queue. The parallel completion of the replay processing of the events to be replayed in multiple replay queues by multiple replay threads can improve the log replay speed of the client and speed up the synchronization speed from the client to the storage backend.

[0098] Optionally, module 62 is selected, specifically for performing a modulo operation on the hash value of the event to be replayed and the number of queues in the plurality of replay queues;

[0099] The selection module 62 is further used to determine the replay queue corresponding to the result of the modulo operation as the target replay queue.

[0100] In the example scenario, each event to be replayed includes its corresponding hash value in its attributes. The selection module 62 performs a modulo operation between the hash value of the event to be replayed and the number of replay queues. Based on the result of the modulo operation, it determines the replay queue corresponding to the event to be replayed and places the event to be replayed into the corresponding replay queue. It is worth noting that a user's request may contain multiple steps, each of which is converted into a single event to be replayed. Multiple events to be replayed from the same user request belong to the same key event. Each event to be replayed within the same key event has the same hash value. Because each event to be replayed within the same key event has the same hash value, each event to be replayed within the same key event will be placed into the same replay queue.

[0101] In this example, module 62 determines the replay queue corresponding to the event to be replayed by performing a modulo operation based on the hash value corresponding to the event to be replayed and the number of the multiple replay queues, which can ensure the balance of the number of events to be replayed in each replay queue.

[0102] Optionally, the processing module 63 is specifically used to determine the operation type of the event to be replayed;

[0103] The processing module 63 is further configured to detect whether the current multiple playback queues are empty if the operation type of the event to be replayed is a renaming operation; if it is empty, the event to be replayed is added to the target playback queue; otherwise, the event to be replayed is added to the target playback queue after all events in the multiple playback queues have been replayed.

[0104] The processing module 63 is further configured to, if the operation type of the event to be replayed is a write operation or an update metadata operation, obtain the resource amount of the first event belonging to the write operation and update metadata operation in the plurality of replay queues; if the resource amount of the first event exceeds a preset first threshold, wait for all events in the plurality of replay queues to be replayed before adding the event to be replayed to the target replay queue; otherwise, add the event to be replayed to the target replay queue.

[0105] The processing module 63 is further configured to, if the operation type of the event to be replayed is a write operation or an update metadata operation, detect whether the amount of worker thread resources in the working state among the multiple replay threads exceeds a preset second threshold; if it does not exceed the threshold, add the event to be replayed to the target replay queue; otherwise, wait until the amount of worker thread resources does not exceed the second threshold.

[0106] Based on the scenario example, after determining the playback queue corresponding to the event to be played back, before putting the event to be played back into the corresponding playback queue, the processing module 63 should first determine the operation type of the event to be played back. The event to be played back has multiple operation types, and different playback processing operations correspond to different operation types.

[0107] If the operation type of the event to be replayed is a rename operation, the number of events to be replayed in each replay queue must be zero before the event to be replayed can be added to the corresponding replay queue for subsequent replay processing. Therefore, when the operation type of the event to be replayed is a rename operation, the processing module 63 should first check the number of events to be replayed in each replay queue. If there are still events to be replayed in any replay queue, the event to be replayed can only be added to the corresponding replay queue after all the events to be replayed in the replay queue have completed their replay processing.

[0108] If the operation type of the event to be replayed is a write operation or a metadata update operation, the resource amount occupied by all events to be replayed in each replay queue should be detected first. The processing module 63 sets a first threshold for the resource amount occupied by the events to be replayed, which is the maximum capacity of the total resource amount occupied by the events to be replayed in each queue. Before putting the events to be replayed (write or metadata update) into the replay queue, if it is detected that the resource amount occupied by all events to be replayed (write or metadata update) in all replay queues has exceeded the first threshold, the processing module 63 needs to control the resource amount in each replay queue to avoid congestion of events to be replayed. The control method is to wait until all events to be replayed in all replay queues have completed the replay processing before putting the events to be replayed (write or metadata update) into the corresponding replay queue for replay processing. The processing module 63 sets a second threshold for the amount of worker thread resources in the working state. The second threshold is the same as the first threshold. If the operation type of the event to be replayed is a write operation or a metadata update operation, and the amount of worker thread resources in the working state exceeds the second threshold, the processing module 63 sets the excess part to a waiting state. When the amount of worker thread resources in the working state is less than the second threshold, the processing module 63 then replays the event in the waiting state.

[0109] In this example, the processing module 63 classifies the operation type of the event to be replayed and takes different replay processing operations according to different classifications to avoid errors in the event to be replayed during the replay processing operation.

[0110] Optionally, the generation module 61 is specifically used to generate a log to be replayed according to a user request and add an identifier to the log to be replayed, wherein the identifier represents the log file where the log to be replayed is located and the offset of the log to be replayed in the log file.

[0111] In this scenario example, after generation module 61 converts the user request into a log file to be replayed and stores it in the log file, generation module 61 marks the log file to be replayed. The mark indicates the log file containing the log file and its offset within that file. In this example, by marking the log file to be replayed using generation module 61, the location of the log file to be replayed can be determined.

[0112] Optionally, the processing module 63 is further used to add the logs corresponding to the completed playback processing events to the completed event queue set;

[0113] The processing module 63 is further configured to delete the log file if there are multiple events in the event queue set that have the same identifier and consecutive offsets.

[0114] In a scenario example, after the playback thread selects an event to be played back from the corresponding playback queue and completes the playback processing, the processing module 63 adds the completed playback events to the completed event queue set, which is a collection of all events that have completed playback processing. Based on the tag information in the completed playback events, the log file where the completed playback events are located and their offsets in the log file are obtained. According to the tag information, if the offsets corresponding to completed playback events in the same log file are consecutive, it indicates that all the log files to be played back stored in the log file have completed playback processing. At this time, the processing module 63 can update the log file by deleting the completed playback log files to facilitate the restoring of new log files to be played back.

[0115] Optionally, the processing module 63 is further configured to calculate, for each event being processed by the playback thread, the difference between the amount of thread resources allocated to the event and the amount of thread resources required by the event; if the difference is not zero, release a portion of the thread resources allocated to the event until the current difference is zero.

[0116] In this scenario example, the process of the playback thread selecting an event to be played back from the corresponding playback queue requires the allocation of thread resources. The difference between the number of thread resources allocated to the playback queue for completing the playback of the event and the number of thread resources required to complete the playback of the event is used to determine whether excess processing threads have been allocated for the playback of the event. If the difference is not zero, it can be confirmed that excess processing threads have been allocated for the playback of the event. In this case, the processing module 63 needs to release some processing threads to make the difference zero. This example determines whether excess processing threads have been allocated for the playback of the event by calculating the difference between the two, and releases some threads after allocating excess threads, avoiding waste of available threads and maximizing resource utilization.

[0117] In this embodiment, the generation module converts user requests into logs to be replayed and stores them in a log file. Then, the logs to be replayed are converted into specific events to be replayed. The selection module selects a target replay queue from multiple replay queues for the logs to be replayed. The replay thread corresponding to the target replay queue in the processing module completes the replay processing of the events to be replayed. In this embodiment, multiple replay threads complete the replay processing of events to be replayed in multiple replay queues in parallel, which can improve the log replay speed of the client and speed up the synchronization speed from the client to the storage backend.

[0118] Example 3

[0119] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, the electronic device includes:

[0120] The electronic device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can invoke logical instructions stored in the memory 292 to execute the methods of the above embodiments.

[0121] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0122] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, thereby implementing the methods in the above-described method embodiments.

[0123] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0124] This application provides a non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in the foregoing embodiments.

Claims

1. A log replay method, characterized by, The method comprises: generating a log to be played back and storing the log to be played back into a log storage module according to a user request; converting the log to be played back in the log storage module into an event to be played back by calling an event center thread; selecting a target playback queue from a plurality of playback queues, wherein the plurality of playback queues and a plurality of playback threads correspond to each other in a one-to-one manner; adding the event to be played back into the target playback queue, so as to sequentially take out the event to be played back in the target playback queue for processing by a target playback thread corresponding to the target playback queue, and complete playback processing of the event to be played back; the adding of the event to be played back into the target playback queue comprises: determining an operation type of the event to be played back; if the operation type of the event to be played back is a rename operation, detecting whether the plurality of playback queues are empty; if yes, adding the event to be played back into the target playback queue, otherwise, waiting until all events in the plurality of playback queues complete playback, and then adding the event to be played back into the target playback queue; if the operation type of the event to be played back is a write operation or an update metadata operation, obtaining a resource amount of a first event belonging to the write operation and the update metadata operation in the plurality of playback queues; if the resource amount of the first event exceeds a first threshold value set in advance, waiting until all events in the plurality of playback queues complete playback, and then adding the event to be played back into the target playback queue, otherwise, adding the event to be played back into the target playback queue; if the operation type of the event to be played back is the write operation or the update metadata operation, detecting whether a working thread resource amount of working threads in the plurality of playback threads in a working state exceeds a second threshold value set in advance; if no, adding the event to be played back into the target playback queue, otherwise, waiting until the working thread resource amount does not exceed the second threshold value.

2. The method of claim 1, wherein, the selecting of the target playback queue from the plurality of playback queues comprises: performing a modulo operation on a hash value of the event to be played back and a queue number of the plurality of playback queues; determining a playback queue corresponding to a result of the modulo operation as the target playback queue.

3. The method of claim 1, wherein, the generating of the log to be played back according to the user request comprises: generating a log to be played back and adding an identifier to the log to be played back according to a user request, the identifier representing a log file where the log to be played back is located and an offset amount of the log to be played back in the log file.

4. The method of claim 3, wherein, The method further comprises: adding a log corresponding to an event completing playback processing into a completed event queue set; if a plurality of events in the event queue set have the same log file and continuous offset amounts, deleting the log file in the log storage module.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: for each event being processed by a playback thread, calculating a difference value between a thread resource amount allocated to the event and a thread resource amount required by the event, and if the difference value is not zero, releasing part of the thread resource amount allocated to the event until the current difference value is zero.

6. A log replay apparatus characterized by comprising: The device comprises: generating a log to be played back according to a user request and storing the log to be played back to a log storage module; and converting the log to be played back in the log storage module into an event to be played back by invoking an event center thread; selecting a target playback queue from a plurality of playback queues, wherein the plurality of playback queues and a plurality of playback threads are in one-to-one correspondence; adding the event to be played back to the target playback queue, and sequentially taking out the event to be played back in the target playback queue by a target playback thread corresponding to the target playback queue to process the event to be played back, thereby completing playback processing of the event to be played back; adding the event to be played back to the target playback queue in the processing module includes determining an operation type of the event to be played back; if the operation type of the event to be played back is a rename operation, detecting whether the plurality of playback queues are empty; if the plurality of playback queues are empty, adding the event to be played back to the target playback queue, otherwise, waiting until all events in the plurality of playback queues are completed to add the event to be played back to the target playback queue; if the operation type of the event to be played back is a write operation or an update metadata operation, obtaining a resource amount of a first event belonging to the write operation and the update metadata operation in the plurality of playback queues; if the resource amount of the first event exceeds a first threshold value set in advance, waiting until all events in the plurality of playback queues are completed to add the event to be played back to the target playback queue, otherwise, adding the event to be played back to the target playback queue; if the operation type of the event to be played back is the write operation or the update metadata operation, detecting whether a working thread resource amount of working threads in the plurality of playback threads in a working state exceeds a second threshold value set in advance; if the working thread resource amount does not exceed the second threshold value, adding the event to be played back to the target playback queue, otherwise, waiting until the working thread resource amount does not exceed the second threshold value.

7. The apparatus of claim 6, wherein: the selecting module is specifically configured to perform a modulo operation on a hash value of the event to be played back and a queue number of the plurality of playback queues; and the selecting module is specifically configured to determine a playback queue corresponding to a result of the modulo operation as the target playback queue.

8. The apparatus of claim 6, wherein: the generating module is specifically configured to generate the log to be played back according to the user request and add an identifier to the log to be played back, the identifier representing a log file in which the log to be played back is located and an offset of the log to be played back in the log file.

9. The apparatus of claim 8, wherein: the processing module is specifically configured to add a log corresponding to an event that is completed playback processing to a completed event queue set; and the processing module is specifically configured to delete a log file in the log storage module if a plurality of events in the event queue set have the same log file and continuous offsets.

10. The apparatus of any one of claims 6-9, wherein: The processing module is specifically further configured to, for each event being processed by the playback thread, calculate a difference between an amount of thread resources allocated to the event and an amount of thread resources required by the event, and if the difference is not zero, release a part of the amount of thread resources allocated to the event until the current difference is zero.

11. An electronic device, comprising: The method comprises: a processor, and a memory connected to the processor in communication; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the log playback method according to any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by the processor to implement the log playback method according to any one of claims 1-5.

Images