Data processing methods, apparatus, electronic devices and storage media

Abstract

This disclosure relates to a data processing method, apparatus, electronic device, and storage medium. The method includes: upon detecting a restart of a data storage system, acquiring log indication information; if the log indication information indicates the presence of log data in a preset storage, sequentially acquiring log data to be replayed from the preset storage based on the operation time of the log data; performing log parsing processing on the log data to be replayed to obtain operation parsing information; placing the operation parsing information into a target queue based on operation object identification information; and sequentially acquiring target operation parsing information from each queue based on the data modification thread corresponding to each queue, and performing data modification processing on the stored data in the data storage system based on the target operation parsing information to achieve the replay of the log data to be replayed. Using embodiments of this disclosure, the replay speed after a metadata server restart can be improved while ensuring data consistency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] CephFS (Ceph File System) is a log-based file system that uses logs to ensure the atomicity of file operations. Modifications to file system metadata are first written to the log, and then deleted after the changes are persisted to RADOS (Reliable Autonomic Distributed Object Storage). This ensures that if a power outage occurs before file system metadata modifications are persisted to RADOS, the modified data remains in the log. When the MDS (Metadata Server) service restarts, log replay ensures that no file system metadata is lost, thus guaranteeing the consistency of CephFS file system data. However, when the metadata server experiences heavy requests and pressure, logs can accumulate. If the metadata server service crashes and restarts, the log replay process, being single-threaded, takes a long time from startup to being ready to provide services. This replay time affects when the metadata server is ready to serve clients, potentially increasing the pressure on the distributed file system's metadata access. In addition, distributed file system clusters typically provide services from multiple metadata servers to share the load of client metadata requests. When one metadata server restarts, client requests will be redirected to other metadata servers. The slow restart replay of the metadata server may cause other metadata servers to crash when bearing the extra load for a long time, resulting in instability of the distributed file system cluster. Summary of the Invention

[0003] In view of the aforementioned technical problems, this disclosure proposes a data processing method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of the embodiments of this disclosure, a data processing method is provided, including:

[0005] If a data storage system restart is detected, obtain log indication information;

[0006] When the log indication information indicates that log data exists in the preset memory, log data to be replayed is sequentially retrieved from the preset memory based on the operation time corresponding to the log data. The log data includes multiple log data to be replayed.

[0007] The log data to be replayed is parsed to obtain operation parsing information;

[0008] Based on the operation object identification information in the operation parsing information, the operation parsing information is placed into the target queue of multiple pending queues;

[0009] Based on the data modification thread corresponding to each pending queue, target operation parsing information is sequentially obtained from each pending queue. Based on the target operation parsing information, the stored data corresponding to the target operation parsing information in the data storage system is modified to realize the playback of the log data to be replayed corresponding to the target operation parsing information. The target operation parsing information is the operation parsing information currently obtained from each pending queue.

[0010] According to another aspect of the embodiments of this disclosure, a data processing apparatus is provided, comprising:

[0011] The log indication information acquisition module is used to acquire log indication information when a data storage system restart is detected.

[0012] The log data to be replayed acquisition module is used to acquire log data to be replayed sequentially from the preset memory based on the operation time corresponding to the log data when the log indication information indicates that there is log data in the preset memory. The log data includes multiple log data to be replayed.

[0013] The log parsing and processing module is used to perform log parsing and processing on the log data to be replayed to obtain operation parsing information;

[0014] The operation parsing information placement module is used to place the operation parsing information into a target queue among multiple pending queues based on the operation object identification information in the operation parsing information.

[0015] The data modification processing module is used to sequentially obtain target operation parsing information from each queue based on the data modification thread corresponding to each queue, and modify the stored data corresponding to the target operation parsing information in the data storage system based on the target operation parsing information, so as to realize the playback of the log data to be replayed corresponding to the target operation parsing information. The target operation parsing information is the operation parsing information currently obtained from each queue.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the data processing method described above.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-described data processing method.

[0018] According to another aspect of the present disclosure, a computer program product containing instructions is provided that, when run on a computer, causes the computer to perform the above-described data processing method.

[0019] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0020] Upon detecting a data storage system restart, log indication information is retrieved. If the log indication information indicates the presence of log data in a preset storage location, the log data to be replayed is retrieved sequentially from the preset storage location based on the operation time corresponding to multiple log data items to be replayed. This ensures the sequential reading of the log data to be replayed, avoiding disruption of the log data's order during the reading process and preventing data inconsistency. The log data to be replayed is then parsed to obtain operation parsing information. Based on the operation object identifier information in the operation parsing information, the operation parsing information is placed into the target queue of multiple pending processing queues. This ensures the independence of the log data reading and parsing process from the data modification process, improving log efficiency. To improve the efficiency of data replay, the system first obtains target operation parsing information sequentially from each queue based on the data modification thread corresponding to each queue. Then, based on the target operation parsing information, it modifies the stored data corresponding to the target operation parsing information in the data storage system to achieve the replay of the log data to be replayed corresponding to the target operation parsing information. By using multiple queues, the data modification process can be processed in parallel. While ensuring data consistency, this improves the replay speed, efficiency, and accuracy of the replay data after the metadata server restarts, reduces the time for load sharing to other metadata servers, and thus improves the availability and stability of the distributed file system and the stability of the distributed file system cluster.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0023] Figure 1 This is a schematic diagram illustrating an application system according to an exemplary embodiment;

[0024] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment;

[0025] Figure 3 This is a schematic diagram illustrating the process of generating historical modification instruction information according to an exemplary embodiment;

[0026] Figure 4 This is a schematic diagram illustrating a data processing procedure according to an exemplary embodiment;

[0027] Figure 5 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment;

[0028] Figure 6 This is a block diagram illustrating an electronic device for replaying log data according to an exemplary embodiment;

[0029] Figure 7 This is a block diagram illustrating another electronic device for replaying log data according to an exemplary embodiment. Detailed Implementation

[0030] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0031] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0032] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0033] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application system according to an exemplary embodiment. The application system can be used in the data processing method of this application. Figure 1 As shown, the application system may include at least server 01 and terminal 02.

[0034] In this embodiment, server 01 can be used to replay log data. Specifically, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0035] In this embodiment, terminal 02 can be used to generate a file information display request and display the file page corresponding to the file information display request. The terminal may include physical devices such as smartphones, desktop computers, tablets, laptops, smart speakers, in-vehicle terminals, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, and may also include software running on the physical device, such as applications. The operating system running on the terminal in this embodiment may include, but is not limited to, Android, iOS, Linux, and Windows.

[0036] In practical applications, users can trigger a file information display request through the terminal. Upon receiving the file information display request, the server reads the corresponding stored data from the data storage system and sends the stored data to the terminal. Accordingly, the terminal can display the file page corresponding to the file information display request based on the stored data.

[0037] In addition, it should be noted that, Figure 1 The example shown is only one application environment provided by this disclosure. In practical applications, other application environments may also be included. For example, the process of replaying log data can also be implemented on terminal 02.

[0038] In the embodiments described in this specification, the terminal 02 and the server 01 can be directly or indirectly connected through wired or wireless communication, and this application does not limit this connection.

[0039] It should be noted that the following diagram shows one possible sequence of steps, and it is not strictly necessary to follow this order. Some steps can be executed in parallel without interdependence.

[0040] Specifically, Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment. For example... Figure 2 As shown, this data processing method can be used in electronic devices such as terminals or servers, and specifically includes the following steps:

[0041] In step S201, if a data storage system restart is detected, log indication information is obtained.

[0042] In one specific embodiment, the data storage system may be a system for storing file data. The data storage system may include a distributed file system.

[0043] In one specific embodiment, log indication information can be used to indicate whether log data exists in a preset memory. The log indication information may include first indication information and second indication information. The first indication information can be used to indicate the presence of log data in the preset memory; the second indication information can be used to indicate the absence of log data in the preset memory. Specifically, the preset memory can be used to store log data. The preset memory can be memory. The log data can be used to record metadata operation information.

[0044] In one specific embodiment, when a data storage system restart is detected, log data can be searched in a preset memory. If the log data is found, the log indication information can be determined as the first indication information; if the log data is not found, the log indication information can be determined as the second indication information.

[0045] In step S203, if the log indication information indicates that there is log data in the preset memory, the log data to be replayed is sequentially retrieved from the preset memory based on the operation time corresponding to the log data.

[0046] In one specific embodiment, the log data may include multiple log data items to be replayed. Specifically, the log data to be replayed may refer to operation information for a single operation on an operation object. A log data item to be replayed may be a log entry; the log data to be replayed may include operation object identification information and operation content, etc. Here, the operation object may refer to the object of metadata operation; the operation object may include a file or directory; the operation object identification information may represent the identifier of the operation object.

[0047] In one specific embodiment, step S203 above may include:

[0048] Based on the operation time corresponding to the log data, the distribution thread sequentially retrieves the log data to be replayed from the preset storage.

[0049] In one specific embodiment, the operation time corresponding to the log data may include the operation time corresponding to each of the multiple log data to be replayed. The operation time corresponding to each log data to be replayed can characterize the time taken for the metadata operation corresponding to that log data to be replayed to be executed.

[0050] In one specific embodiment, the distribution thread can be used to retrieve log data to be replayed from a preset storage and to perform log parsing processing on the log data to be replayed.

[0051] In one specific embodiment, the distribution thread can retrieve the log data to be replayed from a preset storage in order of operation time from earliest to latest, based on the operation time corresponding to the log data.

[0052] In the above embodiments, by having the distribution thread retrieve the log data to be replayed from the preset memory in the order of operation time from earliest to latest according to the operation time corresponding to the log data, the log data to be replayed can be read in sequence, avoiding the disruption of the order of log data during the reading process, which would lead to the loss of data consistency.

[0053] In step S205, log parsing processing is performed on the log data to be replayed to obtain operation parsing information.

[0054] In one specific embodiment, operation parsing information can characterize the specific event information of the metadata operation corresponding to the log data to be replayed before parsing. Operation parsing information may include event information of modifying file extension attributes or event information of creating a new file.

[0055] In one specific embodiment, step S203 above may include:

[0056] Based on the operation time corresponding to the log data, the distribution thread sequentially retrieves the log data to be replayed from the preset storage.

[0057] Accordingly, step S205 above may include:

[0058] The distribution thread performs log parsing processing on the log data to be replayed to obtain operation parsing information.

[0059] In one specific embodiment, the distribution thread can retrieve the log data to be replayed from a preset storage in order of operation time from earliest to latest, based on the operation time corresponding to the log data.

[0060] In a specific embodiment, after obtaining the log data to be replayed, log parsing processing of the log data to be replayed can be achieved by serializing and deserializing the log data to be replayed according to preset rules, so as to obtain the operation parsing information corresponding to the log data to be replayed.

[0061] In step S207, based on the operation object identification information in the operation parsing information, the operation parsing information is placed into the target queue of multiple pending queues.

[0062] In one specific embodiment, the operation object identification information may include the path name of the operation object.

[0063] In one specific embodiment, step S207 above may include:

[0064] The operation object identification information is hashed to obtain the operation object hash information corresponding to the operation parsing information;

[0065] The target queue is determined based on the hash information of the operation object and the number of queues in multiple pending queues;

[0066] Place the operation parsing information into the target queue.

[0067] In one specific embodiment, the hash information of the operation object may include the hash value corresponding to the operation object identification information.

[0068] In one specific embodiment, any queue to be processed can be used to store at least one operation parsing information. Optionally, the queue to be processed can be a shard. The number of queues mentioned above can refer to the number of multiple queues to be processed. The target queue can be one of the multiple queues to be processed. Specifically, the number of queues can be set according to the actual application needs; optionally, the number of queues can be 3-7, and for example, the number of queues can be 5.

[0069] In a specific embodiment, determining the target queue based on the hash information of the operation object and the number of queues in multiple pending queues may include:

[0070] The target queue identifier information is obtained by performing a modulo operation on the hash information of the operation object and the number of queues.

[0071] The target queue is determined based on the target queue identifier information.

[0072] In one specific embodiment, the target queue identification information can be used to indicate the target queue corresponding to the operation parsing information.

[0073] In a specific embodiment, the target queue identification information can be obtained by the following formula:

[0074] Shard index=hash(path name)%shard num

[0075] Wherein, Shard index is the target queue identifier information corresponding to the operation parsing information; hash(path name) is the operation object hash information in the operation parsing information; and shard num is the number of queues.

[0076] In one specific embodiment, the queue to be processed indicated by the target queue identification information can be used as the target queue.

[0077] In the above embodiments, by performing a modulo operation on the hash information of the operation object and the number of queues to obtain the target queue identifier, it can be ensured that the operation parsing information of the same operation object is processed by the same data modification thread. This ensures that the order of operation parsing information for the same operation object is not disrupted due to parallel processing of threads, thereby guaranteeing data consistency. It can be understood that by performing a modulo operation on the hash information of the operation object and the number of queues to obtain the target queue identifier, the sequentiality of metadata operation requests for the same operation object can be guaranteed. That is, there will be no situation where a later operation runs first, avoiding the situation where a later operation is overwritten by a previous operation, thus ensuring data consistency.

[0078] In one specific embodiment, after step S205, the method may further include:

[0079] If the operation type in the operation parsing information is an associated operation type, the thread processing state of the dispatch thread will be switched from the running state to the waiting state to stop the acquisition and parsing of log data.

[0080] If the historical operation parsing information in multiple pending queues has been modified and processed, the operation parsing information will be placed into a random queue.

[0081] Based on the data modification thread corresponding to the random queue, operation parsing information is obtained from the random queue, and based on the operation parsing information, the stored data corresponding to the operation parsing information in the data storage system is modified to realize the playback of the log data to be replayed corresponding to the operation parsing information.

[0082] Upon receiving a modification completion indication, the thread processing state is switched from the waiting state to the running state to resume the process of acquiring and parsing log data.

[0083] In one specific embodiment, the log data to be replayed may further include operation types. The operation type in the log data to be replayed can characterize whether the metadata operation corresponding to the log data involves multiple operation objects. Operation types can include associated operation types and non-associated operation types. Associated operation types can characterize that the metadata operation corresponding to the log data to be replayed involves multiple operation objects; non-associated operation types can characterize that the metadata operation corresponding to the log data to be replayed involves only one operation object.

[0084] In a specific embodiment, the thread processing state of the distribution thread can include a running state and a waiting state. Switching the thread processing state of the distribution thread can be used to control the operation of the distribution thread. Specifically, when the thread processing state of the distribution thread is running, the distribution thread can normally execute the operations of acquiring and parsing the log data to be replayed; when the thread processing state of the distribution thread is waiting, the distribution thread can stop executing the operations of acquiring and parsing the log data to be replayed. It can be understood that the distribution thread remains in a stopped state of acquiring and parsing log data while in the waiting state until it switches from the waiting state to the running state, at which point it resumes executing the operations of acquiring and parsing the log data to be replayed.

[0085] In a specific embodiment, after the log data to be replayed is parsed, operation parsing information including the operation type can be obtained. Accordingly, it can be determined whether the operation type corresponding to the operation parsing information is an associated operation type or a non-associated operation type. Then, if the operation type in the operation parsing information is an associated operation type, the thread processing state of the dispatch thread is switched from the running state to the waiting state to stop the log data acquisition and parsing process.

[0086] In a specific embodiment, historical operation parsing information can refer to operation parsing information whose operation time is earlier than the current operation parsing information, that is, the operation time corresponding to the historical operation parsing information is earlier than the operation time corresponding to the current operation parsing information.

[0087] In one specific embodiment, a random queue can refer to any one of a plurality of queues to be processed. A random queue can be a queue randomly selected from a plurality of queues to be processed.

[0088] In a specific embodiment, after switching the thread processing state of the dispatch thread to the waiting state, it can wait for historical modification indication information; wherein, the historical modification indication information can be used to indicate that the modification processing of historical operation parsing information in multiple pending queues has been completed. The historical modification indication information can include modification indication information corresponding to each of the multiple pending queues, and each modification indication information can be used to indicate that the modification processing of historical operation parsing information in the pending queue corresponding to the modification indication information has been completed. Figure 3 This is a schematic diagram illustrating the process of generating historical modification instruction information according to an exemplary embodiment. Specifically, as shown below... Figure 3As shown, after all the historical operation parsing information in any pending queue has been modified and processed, the data modification thread corresponding to that pending queue can generate modification indication information. Accordingly, if historical modification indication information generated by multiple data modification threads is detected, it can be determined that all the historical operation parsing information in multiple pending queues has been modified and processed, and the operation parsing information can be put into a random queue.

[0089] In one specific embodiment, the stored data in the data storage system may include metadata stored in the data storage system.

[0090] In one specific embodiment, the data modification thread corresponding to the random queue containing operation parsing information can retrieve the operation parsing information from the random queue and, based on the operation parsing information, modify the stored data corresponding to the operation parsing information in the data storage system to achieve the playback of the log data to be replayed corresponding to the operation parsing information. Specifically, the data modification thread corresponding to the random queue can, based on the operation parsing information, modify and persist metadata to RADOS and update relevant metadata in MDS memory to achieve the playback of the log data to be replayed corresponding to the operation parsing information.

[0091] In one specific embodiment, the modification completion indication information can be used to indicate that the data modification processing of the log data to be replayed corresponding to the operation parsing information has been completed.

[0092] In a specific embodiment, after the data modification thread corresponding to the random queue completes the data modification processing of the operation parsing information, it can generate modification completion indication information to switch the thread processing state of the distribution thread from the waiting state to the running state. Correspondingly, after switching the thread processing state of the distribution thread to the running state, the acquisition and parsing processing of log data of the distribution thread can be resumed.

[0093] In the above embodiments, when the operation type in the operation parsing information is an associated operation type, the thread processing state of the dispatch thread is switched from the running state to the waiting state. When the historical operation parsing information in multiple pending queues has been modified and processed, the operation parsing information is placed into a random queue. Based on the data modification thread corresponding to the random queue, the operation parsing information is obtained from the random queue, and based on the operation parsing information, the stored data corresponding to the operation parsing information in the data storage system is modified. This can realize the serial processing of operation parsing information with associated operation types, avoid the inability to guarantee the order of metadata operations due to parallel processing of operation parsing information with associated operation types, and further ensure data consistency, while improving the playback speed and efficiency of log data.

[0094] In one specific embodiment, step S207 above may include:

[0095] If the operation type in the operation parsing information is a non-associated operation type, the operation parsing information is placed into the target queue based on the operation object identification information.

[0096] In a specific embodiment, after performing log parsing processing on the log data to be replayed, operation parsing information including the operation type can be obtained. Accordingly, it can be determined whether the operation type corresponding to the operation parsing information is an associated operation type or a non-associated operation type. Then, if the operation type in the operation parsing information is a non-associated operation type, the operation parsing information can be placed into a target queue based on the operation object identification information, so as to wait for the data modification thread corresponding to the target queue to retrieve the operation parsing information from the target queue, and perform data modification processing on the stored data corresponding to the operation parsing information in the data storage system based on the operation parsing information.

[0097] In the above embodiments, by combining the parallel modification processing of operation parsing information for non-associated operation types with the serial modification processing of operation parsing information for associated operation types, parallel and serial coexistence can be achieved, thereby further ensuring data consistency and improving the playback speed and efficiency of log data.

[0098] In step S209, based on the data modification thread corresponding to each queue to be processed, the target operation parsing information is obtained sequentially from each queue to be processed, and based on the target operation parsing information, the stored data corresponding to the target operation parsing information in the data storage system is modified to achieve the playback of the log data to be replayed corresponding to the target operation parsing information.

[0099] In one specific embodiment, the target operation parsing information can be the operation parsing information currently obtained from each pending queue.

[0100] In one specific embodiment, the data modification thread corresponding to each queue can sequentially obtain target operation parsing information from the queue. Specifically, the target operation parsing information can be obtained sequentially from earliest to latest based on the time when each operation parsing information in the queue was written into the queue, or it can be obtained sequentially from earliest to latest based on the operation time corresponding to each operation parsing information in the queue.

[0101] In one specific embodiment, the data modification thread can modify the stored data corresponding to the target operation parsing information in the data storage system based on the target operation parsing information, so as to realize the replay of the log data to be replayed corresponding to the target operation parsing information. Specifically, the data modification thread can modify and persist metadata to RADOS and update relevant metadata in MDS memory based on the operation parsing information, so as to realize the replay of the log data to be replayed corresponding to the operation parsing information.

[0102] In the above embodiments, multiple queues can be used to achieve parallel processing of data modification, which can improve the playback speed, playback efficiency and playback data accuracy after the metadata server restarts, while ensuring data consistency.

[0103] In one specific embodiment, while the data modification thread is processing data modification, the distribution thread can simultaneously acquire and parse the log data to be replayed, which can further improve the replay speed and efficiency of the log data.

[0104] In one specific embodiment, the above method may further include:

[0105] Retrieve the thread identifier information of each of the multiple data modification threads;

[0106] The thread identifier information of any data modification thread and the number of queues in multiple pending queues are moduloed to obtain the queue identifier information of the pending queue corresponding to any data modification thread.

[0107] Based on the queue identifier information of the queue to be processed corresponding to any data modification thread, bind any data modification thread to the corresponding queue to be processed.

[0108] Accordingly, step S209 above may also include:

[0109] Based on the data modification thread bound to each pending queue, target operation parsing information is sequentially obtained from each pending queue, and based on the target operation parsing information, the stored data corresponding to the target operation parsing information in the data storage system is modified.

[0110] In one specific embodiment, any thread identification information can be used to indicate the data modification thread corresponding to that thread identification information. Optionally, the thread identification information can be a value greater than or equal to 0.

[0111] In one specific embodiment, when a data modification thread starts, the identifiers of multiple data modification threads can be assigned sequentially from 0 according to the thread order through initialization. Accordingly, the thread identifier information of each of the multiple data modification threads can be obtained.

[0112] In one specific embodiment, by performing a modulo operation on the thread identifier information of any data modification thread and the number of queues in multiple pending queues, the remainder when the thread identifier information is divided by the number of queues can be obtained. Accordingly, the remainder can be used as the queue identifier information of the pending queue corresponding to the data modification thread.

[0113] In a specific embodiment, after obtaining the queue identification information through the modulo operation, the queue to be processed corresponding to the queue identification information can be bound to the data modification thread corresponding to the thread identification information used for the modulo operation, so that each data modification thread can obtain operation parsing information from the queue to be processed bound to it.

[0114] In one specific embodiment, each data modification thread can sequentially obtain target operation parsing information from the pending queue bound to it, and modify the stored data corresponding to the target operation parsing information in the data storage system based on the obtained target operation parsing information, so as to realize the playback of the log data to be replayed corresponding to the target operation parsing information.

[0115] In one specific embodiment, after step S209, the method may further include:

[0116] If there is no log identifier for unreplayed log data between the end log identifier and the log deletion location identifier corresponding to the end log data, update the log deletion location identifier based on the end log identifier to obtain the updated log deletion location identifier;

[0117] Delete the log data corresponding to the log identifier located between the end log identifier and the updated log deletion position identifier.

[0118] In one specific embodiment, the end log data can refer to the log data with the latest operation time among the currently replayed log data. The end log identifier can be a log position identifier for the end log data. The end log identifier can be used to indicate the log position of the end log data within the log data.

[0119] In one specific embodiment, the log deletion location identifier is used to indicate log data to be deleted in a preset storage. It is understood that by updating the log deletion location identifier, the log data to be deleted in the preset storage can be identified, and the log data to be deleted can be deleted, thereby freeing up the space corresponding to the log data to be deleted.

[0120] In one specific embodiment, after the log data to be replayed has been replayed, it can be marked as replayed. Accordingly, the replay mark can be used to determine whether each log data to be replayed in the log data has been replayed. Based on this, the log data with the latest operation time in the current log data can be determined based on the above replay mark, and this log data can be used as the end log data.

[0121] In one specific embodiment, if there is no log identifier for unreplayed log data between the end log identifier corresponding to the end log data and the log deletion location identifier, the log deletion location identifier can be updated by using the end log identifier as the updated log deletion location identifier.

[0122] In one specific embodiment, after the log deletion location identifier is updated, the log data to be deleted can be determined based on the end log identifier and the updated log deletion location identifier, and accordingly, the log data to be deleted can be deleted. Specifically, the log data corresponding to the log identifier located between the end log identifier and the updated log deletion location identifier can be used as the log data to be deleted.

[0123] In a specific embodiment, if there are log identifiers with unplayed log data between the end log identifier and the log deletion position identifier, the log deletion position identifier can be updated based on the end log identifier to obtain the updated log deletion position identifier. This can be achieved by waiting for the log data to be played back and continuing to monitor in real time whether the log data corresponding to the end log data and the log deletion position identifier has been played back.

[0124] In the above embodiments, by updating the log deletion location identifier based on the end log identifier when there is no log identifier for unreplayed log data between the end log identifier corresponding to the end log data and the log deletion location identifier, the updated log deletion location identifier can be obtained. This can avoid the deletion of log data that has not been replayed during parallel processing and realize the timely release of storage space in the preset memory, thereby improving the accuracy of data replay.

[0125] In the above embodiments, when a data storage system restart is detected, log indication information is obtained. If the log indication information indicates that log data exists in a preset memory, the log data to be replayed is sequentially retrieved from the preset memory based on the operation time corresponding to multiple log data to be replayed. This enables sequential reading of the log data to be replayed, avoiding disruption of the log data's order during the reading process and preventing data inconsistency. Then, log parsing processing is performed on the log data to be replayed to obtain operation parsing information. Based on the operation object identifier information in the operation parsing information, the operation parsing information is placed into the target queue of multiple pending queues. This ensures that the log data reading and parsing process is independent of the data modification process. To improve the efficiency of log data replay, the system uses a data modification thread corresponding to each pending queue to sequentially retrieve target operation parsing information from each queue. Based on this parsing information, it modifies the stored data corresponding to the target operation parsing information in the data storage system, thereby enabling the replay of the log data to be replayed. Multiple pending queues allow for parallel processing of the data modification process. While ensuring data consistency, this improves the replay speed, efficiency, and accuracy of the replay data after the metadata server restarts, reduces the time spent distributing the load to other metadata servers, and ultimately enhances the availability and stability of the distributed file system and the stability of the distributed file system cluster.

[0126] Figure 4 This is a schematic diagram illustrating a data processing procedure according to an exemplary embodiment. For example... Figure 4 As shown, the distribution thread can sequentially retrieve log data to be replayed from the log data and perform log parsing processing on the log data to obtain operation parsing information. After obtaining the operation parsing information, the distribution thread can determine the target queue based on the operation object identifier information in the operation parsing information and put the operation parsing information into the target queue. The data modification thread corresponding to each queue can retrieve the target operation parsing information from its respective queue and modify the stored data corresponding to the target operation parsing information in the data storage system based on the target operation parsing information to realize the replay of the log data to be replayed corresponding to the target operation parsing information.

[0127] Figure 5 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment. Figure 5 As shown, the device may include:

[0128] The log indication information acquisition module 510 can be used to acquire log indication information when a data storage system restart is detected.

[0129] The log data acquisition module 520 can be used to acquire log data to be replayed sequentially from the preset memory based on the operation time corresponding to the log data when the log indication information indicates that there is log data in the preset memory. The log data includes multiple log data to be replayed.

[0130] The log parsing and processing module 530 can be used to parse and process log data to be replayed, and obtain operation parsing information.

[0131] The operation parsing information is placed into module 540, which can be used to place the operation parsing information into the target queue of multiple pending queues based on the operation object identification information in the operation parsing information.

[0132] The data modification processing module 550 can be used to sequentially obtain target operation parsing information from each queue based on the data modification thread corresponding to each queue, and modify the stored data corresponding to the target operation parsing information in the data storage system based on the target operation parsing information, so as to realize the playback of the log data to be replayed corresponding to the target operation parsing information. The target operation parsing information is the operation parsing information currently obtained from each queue.

[0133] In one specific embodiment, the above-mentioned log data acquisition module 520 to be replayed may include:

[0134] The first distribution thread execution module can be used to retrieve log data to be replayed from a preset storage according to the operation time corresponding to the log data;

[0135] Accordingly, the log parsing and processing module 530 mentioned above may include:

[0136] The second dispatch thread execution module can be used to perform log parsing processing on the log data to be replayed based on the dispatch thread, and obtain operation parsing information.

[0137] In one specific embodiment, the above-described apparatus may further include:

[0138] The first thread processing state switching module can be used to switch the thread processing state of the dispatch thread from the running state to the waiting state when the operation type in the operation parsing information is an associated operation type, so as to stop the process of acquiring and parsing log data.

[0139] The first parsing information placement module can be used to place the operation parsing information into a random queue when the historical operation parsing information in multiple pending queues has been modified and processed. The operation time corresponding to the historical operation parsing information is earlier than the operation time corresponding to the operation parsing information. The random queue is a queue randomly selected from multiple pending queues.

[0140] The first modification processing module can be used to obtain operation parsing information from the random queue based on the data modification thread corresponding to the random queue, and modify the stored data corresponding to the operation parsing information in the data storage system based on the operation parsing information, so as to realize the playback of the log data to be replayed corresponding to the operation parsing information.

[0141] The second thread processing state switching module can be used to switch the thread processing state from the waiting state to the running state when a modification completion indication is received, so as to resume the process of acquiring and parsing log data. The modification completion indication is used to indicate that the data modification processing of the log data to be replayed corresponding to the operation parsing information has been completed.

[0142] In one specific embodiment, the above-mentioned operation parsing information is placed into module 540, which may include:

[0143] The second parsing information placement module can be used to place the operation parsing information into the target queue based on the operation object identification information when the operation type in the operation parsing information is a non-associated operation type.

[0144] In one specific embodiment, the above-mentioned operation parsing information is placed into module 540, which may include:

[0145] The hash processing module can be used to perform hash processing on the operation object identification information to obtain the operation object hash information corresponding to the operation parsing information;

[0146] The first target queue determination module can be used to determine the target queue based on the hash information of the operation object and the number of queues in multiple pending queues;

[0147] The third parsing information putting module can be used to put operation parsing information into the target queue.

[0148] In one specific embodiment, the first target queue determination module described above may include:

[0149] The first modulo processing module can be used to perform modulo processing on the hash information of the operation object and the number of queues to obtain the target queue identification information;

[0150] The second target queue determination module can be used to determine the target queue based on the target queue identification information.

[0151] In one specific embodiment, the above-described apparatus may further include:

[0152] The log deletion location identifier update module can be used to update the log deletion location identifier based on the end log identifier when there is no log identifier of unplayed log data between the end log identifier corresponding to the end log data and the log deletion location identifier. The end log data is the log data with the latest operation time among the currently played log data. The log deletion location identifier is used to indicate the log data to be deleted in the preset storage.

[0153] The log data deletion module can be used to delete log data corresponding to log identifiers located between the end log identifier and the updated log deletion position identifier.

[0154] In one specific embodiment, the above-described apparatus may further include:

[0155] The thread identification information acquisition module can be used to obtain the thread identification information of multiple data modification threads.

[0156] The second modulo processing module can be used to perform modulo processing on the thread identifier information of any data modification thread and the number of multiple pending queues to obtain the queue identifier information of the pending queue corresponding to any data modification thread.

[0157] The thread queue binding module can be used to bind any data-modifying thread to the corresponding pending queue based on the queue identifier information of the pending queue.

[0158] Accordingly, the aforementioned data modification processing module 550 may include:

[0159] The second modification processing module can be used to sequentially obtain target operation parsing information from each queue based on the data modification thread bound to each queue, and perform data modification processing on the stored data corresponding to the target operation parsing information in the data storage system based on the target operation parsing information.

[0160] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0161] Figure 6 This is a block diagram illustrating an electronic device for replaying log data according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 6As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0162] Figure 7 This is a block diagram illustrating another electronic device for playing back log data according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0163] Those skilled in the art will understand that Figure 6 or Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0164] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the data processing method as described in the embodiments of this disclosure.

[0165] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the data processing method of the present disclosure embodiments.

[0166] In an exemplary embodiment, a computer program product including instructions is also provided, which, when run on a computer, causes the computer to perform the data processing method of the present disclosure embodiments.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0168] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0169] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0170] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized by, The method comprises: In the case of detecting the data storage system restart, obtaining log indication information; In the case where the log indication information indicates that there is log data in the preset storage, based on the operation time corresponding to the log data, the distribution thread sequentially obtains the to-be-replayed log data from the preset storage based on the distribution thread, the log data comprising a plurality of to-be-replayed log data; the to-be-replayed log data is operation information for a single operation on an operation object; Based on the operation analysis information in the operation analysis information, the operation analysis information is put into a target queue in a plurality of to-be-processed queues; any to-be-processed queue is used to store at least one operation analysis information; Based on the data modification thread corresponding to each to-be-processed queue, the target operation analysis information is sequentially obtained from each to-be-processed queue, and the target operation analysis information is used to modify the storage data in the data storage system corresponding to the target operation analysis information, so as to realize the playback of the to-be-replayed log data corresponding to the target operation analysis information; the target operation analysis information is the operation analysis information currently obtained from each to-be-processed queue; Wherein, in the process of data modification processing of the data modification thread, the distribution thread synchronously carries out the acquisition and log analysis processing of the to-be-replayed log data. After the log analysis processing of the to-be-replayed log data is carried out, the method further comprises:

2. The method of claim 1, wherein, In the case where the operation type in the operation analysis information is an association operation type, the thread processing state of the distribution thread is switched from the running state to the waiting state to stop the acquisition and analysis processing of the log data; In the case where the historical operation analysis information in the plurality of to-be-processed queues is all modified and processed, the operation analysis information is put into a random queue; wherein the operation time corresponding to the historical operation analysis information is earlier than the operation time corresponding to the operation analysis information, and the random queue is a queue randomly selected from the plurality of to-be-processed queues; Based on the data modification thread corresponding to the random queue, the operation analysis information is obtained from the random queue, and the storage data in the data storage system corresponding to the operation analysis information is modified based on the operation analysis information, so as to realize the playback of the to-be-replayed log data corresponding to the operation analysis information; In the case where the modification completion indication information is received, the thread processing state is switched from the waiting state to the running state to restore the acquisition and analysis processing of the log data, and the modification completion indication information is used to indicate that the data modification processing of the to-be-replayed log data corresponding to the operation analysis information is completed. ​ 3. The method of claim 2, wherein, The method further comprises: In a case where the operation type in the operation analysis information is a non-association operation type, the operation analysis information is put into the target queue based on the operation object identification information.

4. The method of claim 1, wherein, The method further comprises: The operation object identification information is subjected to hash processing to obtain operation object hash information corresponding to the operation analysis information; The target queue is determined based on the operation object hash information and a queue number of the plurality of to-be-processed queues; The operation analysis information is put into the target queue.

5. The method of claim 4, wherein, The target queue is determined based on the operation object hash information and the queue number, and the method further comprises: The operation object hash information and the queue number are subjected to remainder processing to obtain target queue identification information; The target queue is determined based on the target queue identification information.

6. The method of claim 1, wherein, The method further comprises: In a case where there is no log identification of unplayed log data between an end log identification corresponding to end log data and a log deletion position identification, the log deletion position identification is updated based on the end log identification to obtain an updated log deletion position identification; the end log data is log data with the latest operation time among currently played log data; the log deletion position identification is used to indicate log data to be deleted in the preset storage; Log data corresponding to log identification between the end log identification and the updated log deletion position identification is deleted.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: Thread identification information of the plurality of data modification threads is obtained; The thread identification information of any data modification thread and a queue number of the plurality of to-be-processed queues are subjected to remainder processing to obtain queue identification information of a to-be-processed queue corresponding to any data modification thread; Any data modification thread and the corresponding to-be-processed queue are bound based on the queue identification information of the to-be-processed queue corresponding to any data modification thread. The method further comprises: The target operation analysis information is obtained from each to-be-processed queue based on the data modification thread bound with the to-be-processed queue, and the storage data corresponding to the target operation analysis information in the data storage system is subjected to data modification processing based on the target operation analysis information.

8. A data processing apparatus, characterized by, The apparatus comprises: The log indication information acquisition module is configured to acquire log indication information when it is detected that the data storage system is restarted. The log data to be played back acquisition module is configured to acquire, based on a distribution thread, log data to be played back from the preset storage in sequence based on operation time corresponding to the log data, when the log indication information indicates that there is log data in the preset storage, the log data including a plurality of log data to be played back; the log data to be played back is operation information of a single operation on an operation object. The log analysis processing module is configured to perform log analysis processing on the log data to be played back based on the distribution thread to obtain operation analysis information corresponding to the log data to be played back; the operation analysis information represents specific event information of a metadata operation corresponding to the log data to be played back before analysis. The operation analysis information placement module is configured to place the operation analysis information in a target queue in a plurality of to-be-processed queues based on operation object identification information in the operation analysis information; any to-be-processed queue is configured to store at least one operation analysis information. The data modification processing module is configured to acquire target operation analysis information from each to-be-processed queue in sequence based on a data modification thread corresponding to each to-be-processed queue, and perform data modification processing on storage data corresponding to the target operation analysis information in the data storage system based on the target operation analysis information, to realize playback of log data to be played back corresponding to the target operation analysis information; the target operation analysis information is operation analysis information currently acquired from each to-be-processed queue; wherein, in the process of data modification processing by the data modification thread, the distribution thread synchronously performs acquisition and log analysis processing of the log data to be played back.

9. The apparatus of claim 8, wherein, The device further comprises: The first thread processing state switching module is configured to switch a thread processing state of the distribution thread from a running state to a waiting state to stop the acquisition and analysis processing of log data when an operation type in the operation analysis information is an association operation type. The first analysis information placement module is configured to place the operation analysis information in a random queue when historical operation analysis information in the plurality of to-be-processed queues is all modified and processed; the operation time corresponding to the historical operation analysis information is earlier than the operation time corresponding to the operation analysis information; the random queue is a queue randomly selected from the plurality of to-be-processed queues. The first modification processing module is configured to acquire the operation analysis information from the random queue based on a data modification thread corresponding to the random queue, and perform data modification processing on storage data corresponding to the operation analysis information in the data storage system based on the operation analysis information, to realize playback of log data to be played back corresponding to the operation analysis information. The second thread processing state switching module is configured to switch the thread processing state from the waiting state to the running state to resume the log data acquisition and analysis process in response to receiving the modification completion indication information, which indicates that the data modification process of the log data corresponding to the operation analysis information is completed.

10. The apparatus of claim 9, wherein, The operation analysis information placing module comprises: The second analysis information placing module is configured to place the operation analysis information into the target queue based on the operation object identification information in response to the operation type in the operation analysis information being a non-association operation type.

11. The apparatus of claim 8, wherein, The operation analysis information placing module comprises: The hash processing module is configured to perform hash processing on the operation object identification information to obtain operation object hash information corresponding to the operation analysis information. The first target queue determining module is configured to determine the target queue based on the operation object hash information and the number of queues of the plurality of to-be-processed queues. The third analysis information placing module is configured to place the operation analysis information into the target queue.

12. The apparatus of claim 11, wherein, The first target queue determining module comprises: The first remainder processing module is configured to perform remainder processing on the operation object hash information and the number of queues to obtain target queue identification information. The second target queue determining module is configured to determine the target queue based on the target queue identification information.

13. The apparatus of claim 8, wherein, The device further comprises: The log deletion position identification updating module is configured to update the log deletion position identification based on the end log identification to obtain an updated log deletion position identification in response to there being no log identification of unplayed log data between the end log identification and the log deletion position identification, the end log data being the log data with the latest operation time among the currently played log data, and the log deletion position identification indicating the log data to be deleted in the preset storage. The log data deletion module is configured to delete the log data corresponding to the log identification between the end log identification and the updated log deletion position identification.

14. The apparatus of any of claims 8-13, wherein, The device further comprises: The thread identification information obtaining module is configured to obtain thread identification information of a plurality of data modification threads. The second remainder processing module is configured to perform remainder processing on the thread identification information of any data modification thread and the number of queues of the plurality of to-be-processed queues to obtain queue identification information of the to-be-processed queue corresponding to any data modification thread. The thread queue binding module is configured to bind any data modification thread with the corresponding to-be-processed queue based on the queue identification information of the to-be-processed queue corresponding to any data modification thread. The data modification processing module comprises: The second modification processing module is configured to sequentially obtain the target operation analysis information from each to-be-processed queue based on the data modification thread bound with the to-be-processed queue, and perform data modification processing on the storage data corresponding to the target operation analysis information in the data storage system based on the target operation analysis information.

15. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions to implement the data processing method in any one of claims 1 to 7.

16. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the data processing method in any one of claims 1 to 7.

17. A computer program product comprising computer instructions, characterized in that, The computer program instructions, when executed by the processor, implement the data processing method in any one of claims 1 to 7.

Images