Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: receiving a data migration event sent by a metadata service coordinator, wherein the data migration event comprises a plurality of sub-events; carrying out serialization processing on the data migration event to obtain a serialized event; storing the serialized event to a local log file; processing a data migration event corresponding to the serialization event from a playback position recorded by the local log file; in the processing process, if it is monitored that processing of any sub-event is completed, the playback position in the local log file is updated according to the completed sub-event; and if it is monitored that the proxy service equipment is restarted, searching a playback position from the local log file, and continuing to process the uncompleted sub-events from the playback position. The method is used for achieving the effect of improving the stability and availability of the file system in a high-performance computing scene.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In high-performance computing scenarios, ultra-low latency is a core requirement for ensuring efficient system operation. During large-scale parallel computing, the rapid reading and writing of data directly affects computing efficiency. Any tiny delay can accumulate and slow down the overall computing progress, or even cause the computing task to fail to complete on time. Therefore, ultra-low latency has become a key indicator for measuring the performance of high-performance computing storage systems.

[0003] Distributed file systems are designed for high-performance computing scenarios, achieving hierarchical data management through tiered storage management and data replication mechanisms. When infrequently used, cold data is accessed, it is restored from low-speed storage to the distributed file system by a proxy service device.

[0004] However, when the agent service of the agent service device unexpectedly exits, the hierarchical storage management events being processed will be lost, causing the recovery operation terminal to be interrupted, thereby affecting the stability and availability of the file system in high-performance computing scenarios. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, storage medium, and program product to improve the stability and availability of the file system.

[0006] In a first aspect, embodiments of this application provide a data processing method applied to a proxy service device, comprising: receiving a data migration event sent by a metadata service coordinator, wherein the data migration event includes multiple sub-events; performing serialization processing on the data migration event to obtain a serialized event; saving the serialized event to a local log file; processing the data migration event corresponding to the serialized event starting from the replay position recorded in the local log file; during the processing, if it is detected that any sub-event has been processed, updating the replay position in the local log file according to the completed sub-event; if it is detected that the proxy service device has restarted, searching for the replay position in the local log file, and continuing to process the unfinished sub-events starting from the replay position.

[0007] In one possible implementation, processing the data migration event corresponding to the serialization event includes: reading the serialization event from the local log file and performing a deserialization operation on the serialization event to obtain the data migration event; parsing each sub-event in the data migration event to determine the event type and storage path of each sub-event; if the event type is an archive event, locating the file to be archived in the file system based on the storage path; generating the target file corresponding to the file to be archived in the low-speed storage according to the storage path and preset mapping rules; opening the file to be archived in the file system and the target file in the low-speed storage; sequentially reading data from the file to be archived and writing it to the target file according to the preset data block size, and sending the current archiving progress to the metadata service coordinator after completing the writing of each block of data; and determining that the data migration event processing is complete when the current archiving progress reaches the state of full data archiving completion.

[0008] In one possible implementation, after determining the event type and storage path of each sub-event, the process further includes: if the event type is a recovery event, locating the corresponding file to be recovered in the low-speed storage according to the storage path; determining the storage location of the target file in the file system according to the storage path and preset mapping rules; creating a temporary file in the file system and opening the file to be recovered; sequentially reading data from the file to be recovered according to a preset data block size and writing it to the temporary file, and sending the current recovery progress to the metadata service coordinator after completing the writing of each block of data; when the current recovery progress is detected to have reached the state of full data recovery completion, obtaining the layout information of the temporary file and the layout information of the target file; replacing the layout information of the target file with the layout information of the temporary file; performing access verification on the target file, deleting the temporary file after successful verification, and determining that the data migration event processing is complete.

[0009] In one possible implementation, the process of locating the replay position in the local log file and continuing to process unfinished sub-events from that position includes: loading the header information of the local log file, wherein the header information includes a replay position field; extracting the corresponding byte offset from the replay position field, wherein the byte offset indicates the end position of the previously processed event in the local log file; reading subsequent serialized events from the local log file starting from the byte offset; performing deserialization operations on the subsequent serialized events to obtain unfinished sub-events; and adding the unfinished sub-events to the event processing queue in the order they appear in the local log file for continued processing.

[0010] In one possible implementation, updating the replay position in the local log file based on the completed sub-events includes: when any sub-event is processed, obtaining the byte offset endpoint of the serialized event corresponding to the sub-event in the local log file; writing the byte offset endpoint as the new replay position into the header information of the local log file, and overwriting the original replay position.

[0011] Secondly, embodiments of this application provide a data processing apparatus applied to a proxy service device, comprising:

[0012] The event receiving module is used to receive data migration events sent by the metadata service coordinator, where each data migration event includes multiple sub-events.

[0013] The serialization processing module is used to serialize data migration events to obtain serialized events.

[0014] The local log saving module is used to save serialized events to local log files.

[0015] The event handling module is used to process data migration events corresponding to serialized events, starting from the playback position recorded in the local log file.

[0016] The replay position update module is used to update the replay position in the local log file based on the completed sub-event if any sub-event is detected to have been processed during the processing.

[0017] The restart follow-up processing module is used to find the replay position in the local log file and continue processing the unfinished sub-events from the replay position if a restart of the agent service device is detected.

[0018] Thirdly, embodiments of this application provide a proxy service device, including: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0020] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0021] The data processing method, apparatus, device, storage medium, and program products provided in this application effectively solve the problem of event loss caused by the unexpected exit of the proxy service in high-performance computing scenarios by serializing and persisting data migration events to local log files, combined with dynamic updates of replay positions and a mechanism for resuming interrupted transmission after restart. On the one hand, the local log storage of events ensures that all sub-events can be completely retained even if the service is interrupted, avoiding the situation where the recovery operation is terminated due to event loss in the traditional mechanism. On the other hand, the continuation processing logic based on the replay position enables the proxy service to accurately pick up the unfinished sub-events after restarting, without having to execute the process from the beginning. This reduces the latency accumulation caused by repeated operations, ensures the continuity of data access at the application layer, and prevents timeouts due to event loss, thereby significantly improving the stability and availability of Lustre hierarchical storage management in high-performance computing scenarios. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 A flowchart illustrating the hierarchical storage management mechanism architecture provided in this application embodiment;

[0024] Figure 2 Internal architecture diagram of the agent service device provided in the embodiments of this application;

[0025] Figure 3 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0027] Figure 5 A schematic diagram of the structure of the agent service device provided in the embodiments of this application.

[0028] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0030] To clearly understand the technical solution of this application, we will first provide a detailed introduction to the solutions of existing technologies. In high-performance computing scenarios, ultra-low latency is a core requirement for ensuring the efficient operation of the system. During large-scale parallel computing, the rapid reading and writing of data directly affects computing efficiency. Any tiny delay can accumulate and slow down the overall computing progress, or even cause the computing task to fail to complete on time. Therefore, ultra-low latency has become a key indicator for measuring the performance of high-performance computing storage systems. As a file system designed for ultra-low latency in high-performance computing scenarios, the Lustre distributed file system typically requires NVMe disks and RDMA network hardware to guarantee ultra-low latency. This hardware requirement leads to extremely high storage costs. To reduce storage costs, the Lustre distributed file system provides a mechanism called HSM & Copytool (Hierarchical Storage Management & Copytool), which can archive infrequently accessed cold data in the Lustre distributed file system to low-speed storage; when the file is accessed, it is restored from the low-speed storage to the Lustre file system for application access. In Lustre's HSM (Hardware Modeling System) data tiering mechanism, Copytool handles file archiving and recovery by registering an Agent service. However, this mechanism is vulnerable to loss of ongoing archiving and recovery processes if the Agent service exits, causing related application I / O to freeze and become unrecoverable, eventually requiring a timeout. This reduces the availability and stability of Lustre's HSM mechanism.

[0031] To address the aforementioned technical challenges, the inventors devised a method that utilizes local log persistence and a breakpoint resume mechanism to ensure the continuity of data migration event processing. Upon receiving a data migration event, the proxy service device serializes and saves it to a local log file. Processing then begins from the replay position recorded in the log. The replay position is updated after each sub-event is completed. If the device restarts, processing of unfinished sub-events can resume from the replay position. This avoids data migration event conflicts caused by unexpected proxy service exits, ensuring the application layer remains unaware and improving the availability and stability of the Lustre distributed file system's hierarchical storage mechanism in high-performance computing scenarios.

[0032] Figure 1A flowchart illustrating the hierarchical storage management mechanism architecture provided in this application embodiment. Figure 1 As shown, the core components of the hierarchical storage management mechanism include the client, metadata service coordinator, object storage, and proxy service devices.

[0033] In this context, the client acts as a bridge connecting the application layer and the Lustre distributed file system. On the one hand, the application layer accesses the Lustre distributed file system through the client to initiate data read, write, and access operations. On the other hand, the proxy service device uses the client to interact with the metadata service coordinator and object storage to carry out tasks such as data migration related to hierarchical storage management.

[0034] Specifically, when the application layer accesses a file, if the client discovers that the file is cold data that has not been accessed for a long time, it sends an archiving request for the file to the metadata service coordinator. The metadata service coordinator generates an HSM archiving task for the file and writes it to its own log to ensure process traceability. Based on the status of each proxy service device, the metadata service coordinator selects a suitable proxy service device and sends the HSM archiving task to the proxy service device through the client. After receiving the task, the proxy service device serializes the archiving task and writes it to its local log file for persistent storage. It also reads the file data from object storage, migrates it to low-speed storage, releases storage space in object storage, and updates the replay position in its local log file.

[0035] Specifically, when the application layer accesses an archived file, the client sends an access request for the file to the metadata service coordinator. The metadata service coordinator queries the file's metadata and finds that it is in an archived state. It then generates an HSM recovery task for that file and writes it to its own log to ensure process traceability. Based on the status of each proxy service device, the metadata service coordinator selects a suitable proxy service device and sends the HSM recovery task to the proxy service device through the client. After receiving the task, the proxy service device serializes the recovery task and writes it to its local log file for persistent storage. It also reads the file data from low-speed storage, migrates it to object storage, and updates the replay position in its local log file. Once the data is completely written to object storage, it reports recovery completion to the metadata service coordinator, which updates the file metadata, allowing the application layer to access the file normally.

[0036] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0037] Figure 2This is an internal architecture diagram of the proxy service device provided in an embodiment of this application. Figure 2 As shown, the agent service receiver acts as the entry point for data migration events. It receives archive or restore task instructions from the Lustre distributed file system via both kernel-level ioctl and pipe communication. After unified parsing, these instructions are passed to the logger. The logger serializes the received data migration events and writes them to a local log file for persistent storage. The agent service processor performs specific data operations, such as migrating and releasing data during archiving, and recalling and writing data during restore. It interacts with the Lustre distributed file system via ioctl to complete data read and write operations. The agent service device also includes agent service triggers, which periodically scan and identify cold data, initiating task instructions to the Lustre distributed file system via the ioctl interface.

[0038] Figure 3 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The execution entity of this embodiment is... Figure 1 The agent service device shown can also be other devices with similar functions; there are no restrictions here. For example... Figure 3 As shown, the method is described in detail below:

[0039] S301: Receive data migration events sent by the metadata service coordinator, where each data migration event includes multiple sub-events.

[0040] Specifically, the proxy service device continuously listens for and receives data migration events distributed by the metadata service coordinator through communication channels with the client, such as ioctl interfaces or pipes. These events may be archiving or recovery instructions and contain multiple sub-events, such as migration tasks for multiple data blocks. After receiving the events, the proxy service device encapsulates the raw events into a unified JournalEvent structure, integrating information such as event type, storage path, and parameters.

[0041] S302: Serialize the data migration event to obtain a serialized event.

[0042] Specifically, the proxy service device calls a serialization tool to convert the JournalEvent structure into a binary stream or structured text, such as JSON.

[0043] S303: Save the serialized event to a local log file.

[0044] Specifically, the proxy service device opens a pre-created local log file in append mode, writes the serialized events to the end of the file, and updates the current byte offset of the Journal in real time. Simultaneously, it wakes up the Reader thread responsible for reading the logs, notifying it that new events are pending processing.

[0045] S304: Starting from the playback position recorded in the local log file, process the data migration event corresponding to the serialization event.

[0046] Specifically, after the Reader thread is awakened, it checks the write and replay positions of the local log file to confirm the existence of unprocessed events. Then, it reads serialized events starting from the replay position, calls the deserialization tool to restore them to a JournalEvent structure, and places it in a thread-safe processing queue. The event processing thread is then awakened to perform specific operations. If it is an archive event, the Sa3-Sa7 process is followed: the file to be archived is read from the Lustre file system, migrated to low-speed storage according to a preset block size, and the progress is reported to the metadata service coordinator after each block of data transfer is completed, until the full data migration is complete. If it is a recovery event, the Sb1-Sb7 process is followed: the file to be recovered is retrieved from low-speed storage, written to a temporary file in the Lustre distributed file system, the target file layout information is replaced, access is verified, and finally the temporary file is deleted.

[0047] Sa1: Reads serialized events from the local log file and performs deserialization operations on the serialized events to obtain data migration events.

[0048] Specifically, the agent service device reads serialized archived events from local log files, calls a deserialization tool to restore them to the original archived event objects, which contain detailed information about the files to be archived.

[0049] Sa2: Analyze each sub-event in the data migration event to determine the event type and storage path of each sub-event.

[0050] Specifically, each sub-event in the archived event is parsed to determine the type of each sub-event and the storage path of the file to be archived in the Lustre distributed file system.

[0051] Sa3: If the event type is an archive event, the archived file in the file system is located based on the storage path.

[0052] Specifically, based on the parsed storage path, the corresponding file to be archived in the object storage is located through the client's file access interface, confirming that the file exists and can be read.

[0053] Sa4: Generates the target file corresponding to the file to be archived in the low-speed storage according to the storage path and preset mapping rules.

[0054] Specifically, based on preset path mapping rules, the target file path corresponding to the file to be archived is generated in the low-speed storage. The path mapping rules are a mapping table of "Lustre path → low-speed storage path". The low-speed storage is a magnetic tape library.

[0055] Sa5: Opens archived files in the file system and target files in slow storage.

[0056] Specifically, by opening the read mode of the archived files in the object storage of the Lustre distributed file system and the write mode of the target files in the low-speed storage through the file system interface, a data transmission channel is established.

[0057] Sa6: Based on the preset data block size, read data sequentially from the file to be archived and write it to the target file in turn. After completing the writing of each block of data, send the current archiving progress to the metadata service coordinator.

[0058] Specifically, according to the preset data block size, such as 4MB, data is read from the file to be archived block by block and written to the target file; after each block of data is written, a progress report is sent to the metadata service coordinator through the client.

[0059] Sa7: When the current archiving progress is detected to have reached the state of full data archiving completion, the data migration event is determined to be completed.

[0060] Specifically, when the last piece of data is written and the progress reaches 100%, the archiving event is considered complete, and the storage space of the original file in the object storage of the Lustre distributed file system is released.

[0061] Accordingly, if the event type is a recovery event, the process also includes:

[0062] Sb1: If the event type is a recovery event, then locate the corresponding file to be recovered in the low-speed storage according to the storage path.

[0063] Specifically, if the sub-event type is determined to be recovery, the corresponding file to be recovered in the low-speed storage is located according to the storage path, and the file is confirmed to exist and be readable.

[0064] Sb2: Determines the storage location of the target file in the file system based on the storage path and preset mapping rules.

[0065] Specifically, based on preset mapping rules, the storage path of the target file and the corresponding object storage location are determined in the Lustre distributed file system.

[0066] Sb3: Creates a temporary file in the file system and opens the file to be recovered.

[0067] Specifically, temporary files are created in the Lustre distributed file system to temporarily store the data to be recovered; at the same time, read mode is enabled for the files to be recovered in the low-speed storage.

[0068] Sb4: Reads data sequentially from the file to be recovered according to the preset data block size and writes it to a temporary file. After completing the writing of each block of data, it sends the current recovery progress to the metadata service coordinator.

[0069] Specifically, data is read from the file to be recovered block by block according to the preset data block size and written to a temporary file; after each block is written, the recovery progress is sent to the metadata service coordinator.

[0070] Sb5: When the current recovery progress is detected to have reached the state of full data recovery completion, obtain the layout information of temporary files and the layout information of target files.

[0071] Specifically, after the full data recovery is completed, the layout information of the temporary files is obtained, such as the distribution of data blocks in object storage, verification information, and the original layout information of the target files.

[0072] Sb6: Replaces the layout information of the target file with the layout information of the temporary file.

[0073] Specifically, the layout information of the target file is replaced with the layout information of the temporary file through the file system interface, so that the target file points to the restored data block.

[0074] Sb7: Performs access verification on the target file, deletes temporary files after successful verification, and determines that the data migration event processing is complete.

[0075] Specifically, an access verification is performed on the target file. For example, a portion of the data is read to verify its integrity. Once the verification is successful, the temporary file is deleted, and the recovery event is considered complete.

[0076] S305: During processing, if any sub-event is detected to have been completed, the replay position in the local log file is updated according to the completed sub-event.

[0077] Specifically, whenever a sub-event is completed, the byte offset of the serialized content corresponding to that sub-event in the local log file is obtained. This offset is then written to the beginning of the local log file as the new replay position, overwriting the original record. This ensures that subsequent processing can accurately locate the starting point of the incomplete task and avoids duplicate processing of completed events. The update process specifically includes Sd1-Sd2:

[0078] Sd1: After any sub-event is processed, obtain the byte offset endpoint of the serialized event corresponding to the sub-event in the local log file.

[0079] Specifically, when the proxy service device writes serialized events to the local log file, it records in real time the starting byte offset and the length of bytes occupied by the serialized content corresponding to each sub-event in the file. When it detects that a sub-event has been processed, it looks up the starting offset and byte length corresponding to the sub-event in the locally maintained "event-offset mapping table", and determines the byte offset endpoint of the serialized content of the sub-event in the log file by calculating "starting offset + byte length".

[0080] Sd2: The end of the byte offset is used as the new replay position. This is written to the header of the local log file and overwrites the original replay position.

[0081] Specifically, the proxy service device opens the local log file and locates the file header, reading the reserved "playback position field" in the header. Then, it converts the calculated byte offset endpoint into binary format, writes it to the field, and overwrites the original value. After writing is complete, it calls the file system's synchronous refresh interface to ensure that the modified header information is physically written to the disk, preventing the new playback position from being lost due to a sudden power outage.

[0082] S306: If a restart of the agent service device is detected, the replay location is found in the local log file, and the unfinished sub-events are processed from the replay location.

[0083] Specifically, when the device restarts, it first loads the header information of the local log file, extracts the latest playback position (byte offset), and uses this offset as the starting point to read subsequent serialized events and deserialize them into a JournalEvent structure. These events are then added to the processing queue in chronological order from the log. The event processing thread retrieves unfinished sub-events from the queue and continues archiving or recovery operations, enabling breakpoint resumption. This process includes Sc1-Sc5:

[0084] Sc1: Loads the header information of the local log file, which includes the replay position field.

[0085] Specifically, the header information of the local log file typically contains a fixed-format metadata area, which reserves a field specifically for storing the replay position, such as a 64-bit integer area occupying 8 bytes of space. After the agent service device restarts, it first opens the local log file through the file system interface, locates the beginning of the file in read-only mode, reads the preset length of byte data in the header information, and thus loads the complete header information containing the replay position field.

[0086] Sc2: Extract the corresponding byte offset from the playback position field, where the byte offset indicates the end position of the last processed event in the local log file.

[0087] Specifically, the field storing the replay position is parsed from the loaded header information. This field records a one-byte offset value in binary form. Through data type conversion, the specific offset value is extracted. This value precisely identifies the end position of the last processed sub-event in the local log file, that is, the offset of the last byte of the serialized content of that sub-event.

[0088] Sc3: Starting from a byte offset, read subsequent serialization events from the local log file.

[0089] Specifically, the agent service device moves the read / write pointer of the local log file to the extracted byte offset position through file pointer positioning operations. Then, starting from that position, it reads subsequent serialized event data sequentially according to the preset event storage format.

[0090] Sc4: Performs deserialization on subsequent serialized events to obtain incomplete sub-events.

[0091] Specifically, for each segment of serialized event data, the deserialization tool corresponding to the serialization operation is invoked to restore the serialized data in binary or structured text format to the original JournalEvent structure.

[0092] Sc5: Adds unfinished sub-events to the event handling queue in the order they appear in the local log file, so that they can continue to be processed.

[0093] Specifically, the proxy service device maintains a thread-safe event processing queue, such as a FIFO queue based on mutexes and condition variables. Unfinished sub-events are added to the queue sequentially according to their storage order in the local log file. Each time a sub-event is added, the waiting event processing thread is woken up via a condition variable, allowing it to retrieve the sub-event from the queue and continue archiving or restoring operations, thus enabling the task to resume from where it was interrupted.

[0094] In summary, by serializing and persisting data migration events to local log files, combined with dynamic updates of replay locations and a restart-and-resume mechanism, the problem of event loss caused by unexpected agent service exits in high-performance computing scenarios is effectively solved by the Lustre distributed file system. On the one hand, local log storage of events ensures that all sub-events are completely preserved even if the service is interrupted, avoiding the situation where recovery operations are terminated due to event loss in traditional mechanisms. On the other hand, the continuation processing logic based on replay locations allows the agent service to accurately pick up unfinished sub-events after restarting, without having to execute the process from the beginning. This reduces the latency accumulation caused by repeated operations, ensures the continuity of data access at the application layer, and prevents timeouts due to event loss. As a result, the stability and availability of Lustre tiered storage management in high-performance computing scenarios are significantly improved.

[0095] Figure 4 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application, such as... Figure 4 As shown, the data processing device includes: an event receiving module 401, a serialization processing module 402, a local log saving module 403, an event processing module 404, a playback position update module 405, and a restart follow-up processing module 406.

[0096] The event receiving module 401 is used to receive data migration events sent by the metadata service coordinator, wherein the data migration event includes multiple sub-events.

[0097] The serialization processing module 402 is used to serialize the data migration event to obtain a serialized event.

[0098] The local log saving module 403 is used to save serialized events to a local log file.

[0099] The event handling module 404 is used to process the data migration events corresponding to the serialization events, starting from the playback position recorded in the local log file.

[0100] The replay position update module 405 is used to update the replay position in the local log file according to the completed sub-event if it is detected that any sub-event has been processed during the processing.

[0101] The restart follow-up processing module 406 is used to find the replay position in the local log file and continue processing the unfinished sub-events from the replay position if a restart of the agent service device is detected.

[0102] In one possible implementation, the event handling module is specifically used to read serialized events from the local log file and perform deserialization operations on the serialized events to obtain data migration events; parse each sub-event in the data migration event to determine the event type and storage path of each sub-event; if the event type is an archive event, locate the file to be archived in the file system based on the storage path; generate the target file corresponding to the file to be archived in the low-speed storage according to the storage path and preset mapping rules; open the file to be archived in the file system and the target file in the low-speed storage; read data sequentially from the file to be archived according to the preset data block size and write it to the target file in turn, and send the current archiving progress to the metadata service coordinator after completing the writing of each block of data; when the current archiving progress is detected to have reached the state of full data archiving completion, it is determined that the data migration event processing is complete.

[0103] In one possible implementation, the device further includes a recovery event processing module. This module, if the event type is a recovery event, locates the corresponding file to be recovered in the low-speed storage based on the storage path; determines the storage location of the target file in the file system according to the storage path and preset mapping rules; creates a temporary file in the file system and opens the file to be recovered; sequentially reads data from the file to be recovered according to a preset data block size and writes it to the temporary file; after completing the writing of each block of data, sends the current recovery progress to the metadata service coordinator; when the current recovery progress is detected to have reached the state of full data recovery completion, obtains the layout information of the temporary file and the target file; replaces the layout information of the target file with the layout information of the temporary file; performs access verification on the target file; deletes the temporary file after successful verification; and determines that the data migration event processing is complete.

[0104] In one possible implementation, the restart follow-up processing module 406 is specifically used to load the header information of the local log file, which includes a replay position field; extract the corresponding byte offset from the replay position field, where the byte offset indicates the end position of the previously processed event in the local log file; read subsequent serialized events from the local log file starting from the byte offset; perform deserialization operations on the subsequent serialized events to obtain unfinished sub-events; and add the unfinished sub-events to the event processing queue in the order of the local log file for continued processing.

[0105] In one possible implementation, the replay position update module 405 is specifically used to obtain the byte offset endpoint of the serialized event corresponding to the sub-event in the local log file after any sub-event is processed; write the byte offset endpoint as the new replay position into the header information of the local log file, and overwrite the original replay position.

[0106] The data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0107] Figure 5 A schematic diagram of the structure of the agent service device provided in the embodiments of this application. Figure 5 As shown, the proxy service device provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the proxy service device further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0108] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0109] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0110] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0111] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0112] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0113] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0114] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0115] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0116] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0117] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0120] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0122] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to agency service equipment, including: Receive data migration events sent by the metadata service coordinator, wherein the data migration events include multiple sub-events; The data migration event is serialized to obtain a serialized event; Save the serialized event to a local log file; Starting from the playback position recorded in the local log file, the data migration event corresponding to the serialization event is processed; During the processing, if the completion of any sub-event is detected, the replay position in the local log file is updated according to the completed sub-event. If a restart of the proxy service device is detected, the replay position is located in the local log file, and the unfinished sub-events are processed starting from the replay position.

2. The method according to claim 1, characterized in that, The processing of the data migration event corresponding to the serialization event includes: The serialized event is read from the local log file, and the serialized event is deserialized to obtain the data migration event; Each sub-event in the data migration event is parsed to determine the event type and storage path of each sub-event; If the event type is an archive event, then the file to be archived in the file system is located based on the storage path; Based on the storage path and preset mapping rules, generate the target file corresponding to the file to be archived in the low-speed storage. Open the archived file in the file system and the target file in the low-speed storage; According to the preset data block size, data is sequentially read from the file to be archived and written to the target file in turn. After the writing of each block of data is completed, the current archiving progress is sent to the metadata service coordinator. When the current archiving progress is detected to have reached the state of full data archiving completion, the data migration event is determined to be completed.

3. The method according to claim 2, characterized in that, After determining the event type and storage path of each sub-event, the process further includes: If the event type is a recovery event, then the corresponding file to be recovered in the low-speed storage is located according to the storage path; Based on the storage path and the preset mapping rules, determine the storage location of the target file in the file system; Create a temporary file in the file system and open the file to be recovered; According to the preset data block size, data is read sequentially from the file to be recovered and written to the temporary file. After each block of data is written, the current recovery progress is sent to the metadata service coordinator. When the current recovery progress is detected to have reached the state of full data recovery completion, the layout information of the temporary file and the layout information of the target file are obtained; Replace the layout information of the target file with the layout information of the temporary file; Access verification is performed on the target file. If the verification is successful, the temporary file is deleted, and the data migration event is determined to be completed.

4. The method according to claim 1, characterized in that, The step of locating the replay position from the local log file and continuing to process the unfinished sub-events from the replay position includes: Load the header information of the local log file, wherein the header information includes the playback position field; Extract the corresponding byte offset from the playback position field, where the byte offset indicates the end position of the last processed event in the local log file; Starting from the byte offset, read subsequent serialization events from the local log file; Perform a deserialization operation on the subsequent serialized events to obtain the incomplete sub-events; The unfinished sub-events are added to the event processing queue in the order they appear in the local log file, so that processing can continue.

5. The method according to any one of claims 1 to 4, characterized in that, Updating the replay position in the local log file based on completed sub-events includes: After any of the sub-events is processed, obtain the byte offset endpoint of the serialized event corresponding to the sub-event in the local log file; The endpoint of the byte offset is used as the new replay position and written into the header information of the local log file, overwriting the original replay position.

6. A data processing apparatus, characterized in that, Applied to agency service equipment, including: The event receiving module is used to receive data migration events sent by the metadata service coordinator, wherein the data migration event includes multiple sub-events; A serialization processing module is used to serialize the data migration event to obtain a serialized event; The local log saving module is used to save the serialized events to a local log file; The event handling module is used to process the data migration event corresponding to the serialized event, starting from the playback position recorded in the local log file; The replay position update module is used to update the replay position in the local log file according to the completed sub-event if it is detected that any sub-event has been processed during the processing. The restart follow-up processing module is used to find the replay position in the local log file and continue processing the unfinished sub-events from the replay position if the restart of the proxy service device is detected.

7. The apparatus according to claim 6, characterized in that, The event handling module is specifically used for, The serialized event is read from the local log file, and the serialized event is deserialized to obtain the data migration event; Each sub-event in the data migration event is parsed to determine the event type and storage path of each sub-event; if the event type is an archive event, the file to be archived in the file system is located based on the storage path; Based on the storage path and preset mapping rules, generate the target file corresponding to the file to be archived in the low-speed storage. Open the archived file in the file system and the target file in the low-speed storage; According to the preset data block size, data is sequentially read from the file to be archived and written to the target file in turn. After each block of data is written, the current archiving progress is sent to the metadata service coordinator. When the current archiving progress is detected to have reached the state of full data archiving completion, the data migration event is determined to be completed.

8. An agency service device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 5.

Images