Data processing method and device, server, storage medium and program product

Abstract

The application relates to a data processing method and device, a server, a storage medium and a program product. The method comprises the following steps: when it is monitored that a master log system in a distributed log system fails, shadow information of to-be-written data is acquired from a target slave log system, and the shadow information is sent to a back-end back flushing component of the distributed log system, so as to instruct the back-end back flushing component to acquire the to-be-written data from a temporary disk of the target slave log system according to the shadow information, and write the to-be-written data into a back-end disk of the distributed log system. By using the above method, when any log system in the distributed log system fails, the back-end back flushing component can read the to-be-written data of the failed log system from the temporary disk of the non-failed log system according to the shadow information of the to-be-written data, and the process can shorten the replay time of the non-failed log system when the data in the temporary disk of the failed log system is replayed to the back-end disk.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, server, storage medium, and program product. Background Technology

[0002] With the continuous development of big data technology, distributed systems have gradually become the mainstream systems for data processing. Distributed systems often contain special types of data, such as metadata. This type of special data needs to satisfy order preservation and transactional properties, and distributed systems typically employ multiple log systems to process it.

[0003] To prevent the loss of special data in the log system during abnormal situations in a distributed system, related technologies typically read the special data from a high-speed disk and then replay it to the backend disk of the distributed system for permanent storage. However, when using these technologies to replay data to the backend disk, the replay time of the log system is relatively long. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, device, server, storage medium, and program product that can shorten the replay time of the log system when replaying data to the back-end disk, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a data processing method, which includes:

[0006] If a failure is detected in the master log system of the distributed log system, the target obtains the shadow information of the data to be written from the log system; after the master log system fails, the target selects any one of the normal slave log systems from among the multiple slave log systems.

[0007] The target sends shadow information from the log system to the backend flushing component of the distributed log system, instructing the backend flushing component to retrieve the data to be written from the temporary disk of the log system based on the shadow information, and write the data to be written to the backend disk of the distributed log system.

[0008] In the technical solution of this application embodiment, when a failure is detected in the main log system of the distributed log system, the target obtains the shadow information of the data to be written from the log system and sends the shadow information to the back-end flushing component of the distributed log system, instructing the back-end flushing component to obtain the data to be written from the temporary disk of the log system according to the shadow information, and write the data to be written to the back-end disk of the distributed log system. The above method allows the backend flushing component to read the data to be written from the temporary disk of the faulty log system using the shadow information of the data to be written when any log system in the distributed log system fails. This process avoids the faulty log system reading the data to be written from the temporary disk of the faulty log system and writing it to the backend disk. This shortens the replay time of the faulty log system when replaying the data from the temporary disk of the faulty log system to the backend disk, improving the replay efficiency of the faulty log system. In addition, by improving the replay efficiency of the faulty log system, the temporary disk of the faulty log system can provide storage services faster, thereby shortening the outage time of the temporary disk in the faulty log system and increasing the frequency of use of the temporary disk in the faulty log system. At the same time, it also avoids the faulty log system reading the data to be written from the temporary disk from affecting the normal operation of other subsequent business operations.

[0009] In one embodiment, the method further includes:

[0010] The goal is to respond to data write requests sent by the main log system from the log system, write the data to be written carried in the data write request to a temporary disk, and construct shadow information of the data to be written.

[0011] In the technical solution of this application embodiment, the target log system can respond to the data write request sent by the main log system, write the data to be written carried in the data write request to a temporary disk and construct the shadow information of the data to be written, so that the backend flushing component can read the data to be written from the temporary disk of the fault-free log system through the shadow information of the data to be written, thereby avoiding the process of the fault-free log system reading the data to be written from the faulty log system in the temporary disk and writing it to the backend disk, saving the time of the fault-free log system reading the data to be written.

[0012] In one embodiment, the objective is to construct shadow information of the data to be written from the log system, including:

[0013] The goal is to extract disk header data from the data to be written in the log system and obtain the attribute information of the disk header data;

[0014] The goal is to determine the shadow information of the data to be written from the log system based on the attribute information of the disk header data.

[0015] The goal is to store shadow information of the data to be written from the log system into its own memory.

[0016] In the technical solution of this application embodiment, the target log system can extract disk header data from the data to be written and obtain the attribute information of the disk header data. Based on the attribute information of the disk header data, it determines the shadow information of the data to be written and then stores the shadow information of the data to be written in its own memory. This method can temporarily store the pre-built shadow information of the data to be written in the memory of the log system itself, so that when the main log system fails, the shadow information of the data to be written can be sent to the backend flushing component in a timely manner, further improving the reading efficiency of the backend flushing component and the replay efficiency of the fault-free log system.

[0017] In one embodiment, obtaining the attribute information of the disk header data includes:

[0018] Retrieve the storage location of the disk header data in the temporary disk, as well as the size and type of the disk header data;

[0019] The attribute information of the disk header data is determined based on its storage location, size, and type.

[0020] In the technical solution of this application embodiment, the target can obtain the storage location, size and type of disk header data in the temporary disk from the log system, and determine the attribute information of the disk header data based on the storage location, size and type of the disk header data. This method can obtain the attribute information of disk header data that is more similar to the attribute information of the data to be written, thereby preparing for further construction of more accurate shadow information.

[0021] In one embodiment, before storing the shadow information of the data to be written into its own memory, the method further includes:

[0022] Check whether the remaining memory storage is greater than or equal to the required storage for storing shadow information;

[0023] If the remaining memory storage is greater than or equal to the required storage of the shadow information, then the step of storing the shadow information of the data to be written into its own memory is executed.

[0024] In the technical solution of this application embodiment, before storing the shadow information, it can detect whether the remaining storage capacity of the memory is sufficient to store the shadow information of the data to be written. Only when it is determined that the remaining storage capacity of the memory is sufficient to store the shadow information of the data to be written, will the shadow information of the data to be written be stored in its own memory. This method performs a storage verification process when storing the current shadow information into memory to ensure that the shadow information can be successfully stored into memory, and further improves the success rate of the backend flushing component in successfully reading the data to be written based on the shadow information.

[0025] In one embodiment, the method further includes:

[0026] If the remaining storage capacity in memory is less than the required storage capacity of shadow information, then historical shadow information that meets the preset conditions will be selected from all the shadow information already stored in memory.

[0027] After deleting the historical shadow information in memory, the step of storing the shadow information of the data to be written into its own memory is executed.

[0028] This application embodiment can, when it is determined that the remaining storage capacity of the memory is less than the required storage capacity of the shadow information, select historical shadow information that meets preset conditions from all the shadow information already stored in the memory, delete the historical shadow information in the memory, and then perform the step of storing the shadow information of the data to be written into its own memory; when it is determined that the remaining storage capacity of the memory cannot meet the conditions for storing the shadow information of the data to be written, this method can pre-eliminate the historical shadow information in the memory to ensure that the remaining storage capacity of the memory after elimination can meet the conditions for storing the shadow information of the data to be written, thereby improving the success rate of shadow information storage.

[0029] In one embodiment, after the backend flush component obtains the data to be written, it preprocesses the data to be written to obtain preprocessed data, and then writes the preprocessed data to the backend disk; the preprocessing includes format conversion and encapsulation processing.

[0030] Secondly, this application provides a data processing apparatus, which includes:

[0031] If a failure is detected in the master log system of the distributed log system, the target obtains the shadow information of the data to be written from the log system; after the master log system fails, the target selects any one of the normal slave log systems from among the multiple slave log systems.

[0032] The target sends shadow information from the log system to the backend flushing component of the distributed log system, instructing the backend flushing component to retrieve the data to be written from the temporary disk of the log system based on the shadow information, and write the data to be written to the backend disk of the distributed log system.

[0033] Thirdly, this application provides a server including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method in any embodiment of the first aspect.

[0034] Fourthly, this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the embodiments of the first aspect described above.

[0035] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method in any of the embodiments of the first aspect described above. Attached Figure Description

[0036] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0037] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0038] Figure 3 This is a schematic diagram illustrating data transmission between a distributed log system, a backend flushing component, and a backend disk within a distributed system in one embodiment.

[0039] Figure 4 This is a flowchart illustrating the data processing method in another embodiment;

[0040] Figure 5 This is a flowchart illustrating the data processing method in another embodiment;

[0041] Figure 6 This is a flowchart illustrating the data processing method in another embodiment;

[0042] Figure 7 This is a flowchart illustrating the data processing method in another embodiment;

[0043] Figure 8 This is a structural block diagram of a data processing device in one embodiment;

[0044] Figure 9 This is a diagram of the internal structure of a server in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] In distributed systems, there are often special types of data that need to satisfy order preservation and transactional properties. To ensure that this data satisfies these properties, distributed systems typically use distributed log (JNL) systems from multiple distributed systems to process such special data.

[0047] In practice, distributed logging systems initially store special data on their own high-speed disks. Since these high-speed disks only serve as temporary storage, and to meet the actual application requirements of this special data, it's necessary to permanently store it. Traditionally, after storing the special data on the high-speed disk, the distributed logging system needs to read this data from the high-speed disk and store it on its backend disk for permanent storage. However, in the traditional process of replaying the special data from the high-speed disk to the backend disk, the entire special data needs to be read from the high-speed disk first, and then stored on the backend disk. This results in a significant time commitment for the logging system during the data replay process.

[0048] Based on this, this application provides a data processing method to shorten the time required for distributed log systems to replay data to the backend disk. This data processing method is applicable to... Figure 1 The application environment diagram shown includes a distributed logging system, which comprises multiple logging systems. Each logging system can be a system server, and may include one master logging system and at least two slave logging systems. The types of the master and slave logging systems in the distributed logging system are not limited to... Figure 1 The type shown; Figure 1 The diagram illustrates a distributed logging system comprising one master logging system and three slave logging systems (i.e., slave logging system 1, slave logging system 2, and slave logging system 3). The master logging system and each slave logging system can communicate via a connection, such as Wi-Fi, mobile network, or Bluetooth. The master and slave logging systems can be various personal computers, laptops, smartphones, tablets, and portable wearable devices, but are not limited to these.

[0049] In one embodiment, such as Figure 2As shown, a data processing method is provided. Taking the application of this method to a distributed log system as an example, the method includes the following steps:

[0050] S100. If a failure is detected in the primary log system of the distributed log system, the target obtains shadow information of the data to be written from the log system. Specifically, the target obtains any one of the normal secondary log systems selected from among the multiple secondary log systems after the primary log system fails.

[0051] Specifically, the aforementioned distributed log system can be understood as a subsystem within a distributed system. In this embodiment, the master log system and all slave log systems in the distributed log system can monitor for failures based on a preset monitoring mechanism. To prevent data loss in the temporary disks of any failed log system or during a power outage, the data stored in the temporary disks of the failed log system can be sent to the backend disks of the distributed log system for permanent storage. Therefore, in practice, the data stored in the temporary disks of each log system can be backed up and stored on the temporary disks of other log systems within the distributed log system, ensuring that if any log system in the distributed log system fails, the data stored in the temporary disks of the failed log system can be completely stored and transferred to the backend disks. Optionally, the aforementioned temporary disk is a disk capable of temporarily storing data. In the embodiments of this application, the temporary disk is also a disk of high-speed media, namely a high-speed disk, such as a hard disk based on the Non-Volatile Memory Express (NVMe) protocol, a solid-state disk (SSD) or a non-volatile dual in-line memory module (NVDIMM) for computers, etc.

[0052] It should be noted that a distributed logging system includes one master logging system and multiple slave logging systems. Any one of the logging systems in a distributed logging system can be the master, and the others are slaves. The target slave logging system can be any one of the slaves elected by the multiple fault-free slaves based on a preset election strategy after the master logging system fails.

[0053] Optionally, the preset election strategy can be a distributed system election algorithm, a distributed coordination service (such as ZooKeeper), or any user-defined election strategy. Optionally, the distributed system election algorithm can be the Bully algorithm, Raft algorithm, and / or ZAB algorithm, etc.

[0054] If a failure is detected in the main log system of the distributed log system, the target can obtain shadow information of the data to be written from the log system. The data to be written is the data stored in the temporary disk of the main log system.

[0055] Optionally, the shadow information of the data to be written can be understood as the identification information of the data to be written, which is used by the backend flushing component to read the data to be written based on the identification information of the data to be written. The shadow information of the data to be written can be represented by a portion of the data to be written, or by the location of the data to be written in the temporary disk of the main log system. Of course, it can also be represented by characters that can distinguish different data to be written, which is not limited in this embodiment.

[0056] S200: The target sends shadow information from the log system to the backend flushing component of the distributed log system, instructing the backend flushing component to retrieve the data to be written from the temporary disk of the log system based on the shadow information, and write the data to be written to the backend disk of the distributed log system.

[0057] To reduce the number of times the target log system reads data from its temporary disk and improves the replay efficiency of the data to be written, the target log system can send shadow information of the data to be written to the backend flushing component of the distributed log system. This allows the backend flushing component to read the data from the target log system's temporary disk based on the shadow information and then write it to the backend disk of the distributed log system. Simultaneously, after the backend flushing component receives the shadow information from the target log system, the target log system can output a response to the distributed system, indicating that it can continue to be used for other business operations. Figure 3 The diagram illustrates data transmission between the distributed log system, the backend flushing component, and the backend disk in a distributed system. Figure 3 The diagram illustrates a distributed log system consisting of a master log system and two slave log systems (i.e., slave log system 1 and slave log system 2).

[0058] In this embodiment, the aforementioned backend flushing component is a page cache component, i.e., a PageCache component. The aforementioned backend disk can be understood as the backend storage pool of the distributed log system, i.e., a storpool. In this embodiment, the aforementioned backend disk is a hard disk drive (HDD) of the distributed log system.

[0059] In one embodiment, after obtaining the data to be written, the backend flush component preprocesses the data to be written to obtain preprocessed data, and then writes the preprocessed data to the backend disk; the preprocessing includes format conversion and encapsulation processing.

[0060] In practical applications, there can be multiple pieces of data to be written. Therefore, after the backend flushing component reads the data to be written from the temporary disk in the target log system, it needs to preprocess the data to be written to obtain a complete preprocessed data, and then write the preprocessed data to the backend disk.

[0061] The aforementioned preprocessing may include invalid data removal and duplicate data merging, etc. However, in this embodiment, the preprocessing includes format conversion and encapsulation. The format conversion process can be understood as converting multiple data to be written into a unified format; the encapsulation process can be understood as packaging or binding multiple format-converted data to be written.

[0062] This application embodiment can process the data to be written into a complete data storage and store it on the backend disk, so that the backend disk can distinguish the data to be written corresponding to different log systems, and can also avoid the problem that the data to be written corresponding to the same log system is not completely stored on the backend disk.

[0063] In the technical solution of this application embodiment, when a failure is detected in the main log system of the distributed log system, the target obtains the shadow information of the data to be written from the log system and sends the shadow information to the back-end flushing component of the distributed log system, instructing the back-end flushing component to obtain the data to be written from the temporary disk of the log system according to the shadow information, and write the data to be written to the back-end disk of the distributed log system. The above method allows the backend flushing component to read the data to be written from the temporary disk of the faulty log system when any log system in the distributed log system fails, using the shadow information of the data to be written. This process avoids the faulty log system reading the data to be written from the temporary disk of the faulty log system and writing it to the backend disk. This shortens the replay time of the faulty log system when replaying the data from the temporary disk of the faulty log system to the backend disk, improving the replay efficiency of the faulty log system. In addition, by improving the replay efficiency of the faulty log system, the temporary disk of the faulty log system can provide storage services faster, thereby shortening the outage time of the temporary disk in the faulty log system and increasing the frequency of use of the temporary disk in the faulty log system. At the same time, it also avoids the faulty log system reading the data to be written from the temporary disk from affecting the normal operation of other subsequent business operations.

[0064] When the primary log system in a distributed log system fails, the target system needs to obtain shadow information of the data to be written from the log system. However, before obtaining the shadow information, the target system must first obtain the data to be written and construct its shadow information. The process of how the target system obtains the data to be written and constructs its shadow information is described below. In one embodiment, before the target system obtains the shadow information of the data to be written from the log system, the data processing method may further include: the target system responding to a data write request sent by the primary log system, writing the data to be written carried in the data write request to a temporary disk, and constructing the shadow information of the data to be written.

[0065] In actual processing, after the master log system receives the data to be written from the distributed system, it stores the data to be written in its own temporary disk and synchronously sends the data to be written to each slave log system in the distributed log system so that the data to be written can be backed up in the temporary disk of each slave log system. When synchronously backing up the data to be written, each slave log system can also synchronously build shadow information of the data to be written.

[0066] In this system, the master log system can send data write requests to each slave log system in the distributed log system. After receiving and responding to the data write request, each slave log system writes the data to be written carried in the data write request to its own temporary disk. Naturally, the target slave log system receives and responds to the data write request sent by the master log system and writes the data to be written carried in the data write request to its temporary disk.

[0067] Meanwhile, the data write request also serves to instruct the log system to construct shadow information of the data to be written. After each log system receives and responds to the data write request, it can also synchronously construct shadow information of the data to be written.

[0068] In one embodiment, the method for constructing the shadow information of the data to be written can be to obtain a pre-trained algorithm model, and then input the obtained data to be written into the algorithm model to obtain the shadow information of the data to be written.

[0069] In another embodiment, the method for constructing the shadow information of the data to be written can also be to search for successfully matched data in the mapping relationship based on the data to be written, and determine the shadow information corresponding to the successfully matched data as the shadow information of the data to be written. Optionally, the mapping relationship can include data and the corresponding shadow information, and the data and the corresponding shadow information in the mapping relationship have a one-to-one correspondence.

[0070] In the technical solution of this application embodiment, the target log system can respond to the data write request sent by the main log system, write the data to be written carried in the data write request to a temporary disk and construct the shadow information of the data to be written, so that the backend flushing component can read the data to be written from the temporary disk of the fault-free log system through the shadow information of the data to be written, thereby avoiding the process of the fault-free log system reading the data to be written from the faulty log system in the temporary disk and writing it to the backend disk, saving the time of the fault-free log system reading the data to be written.

[0071] The process of constructing shadow information of the data to be written from the log system is described below. In one embodiment, as... Figure 4 As shown, the process by which the above objective constructs shadow information of the data to be written from the log system may include:

[0072] S300: The target extracts disk header data from the data to be written from the log system and obtains the attribute information of the disk header data.

[0073] The data to be written can include three parts: disk header data, middle data, and disk tail data. For example, if the data to be written is 12 bits, the disk header data can be the first 4 bits, the middle data can be the middle 4 bits, and the disk tail data can be the last 4 bits. As another example, if the data to be written is 11 bits, the disk header data can be the first 3 bits, the middle data can be the middle 5 bits, and the disk tail data can be the last 3 bits. In this embodiment, the number of bits and positions of the disk header data, middle data, and disk tail data within the data to be written are not limited to the examples; however, the specific number of bits and positions of the disk header data, middle data, and disk tail data are predetermined.

[0074] Specifically, the target can extract disk header data from the data to be written based on the specified number of bits and position of the disk header data within the data to be written. Alternatively, the target can obtain a pre-trained algorithm model from the log system, and then input the obtained data to be written into the algorithm model, which will then extract the disk header data from the data to be written.

[0075] Based on the disk header data extracted in the above steps, the attribute information of the disk header data can be obtained. Optionally, the attribute information of the disk header data can be the size of the disk header data, the position of the disk header data in the data to be written (i.e., before, in, or after), the storage location of the data to be written corresponding to the disk header data in the temporary disk, and the checksum data of the disk header data, etc.

[0076] S400: The target log system determines the shadow information of the data to be written based on the attribute information of the disk header data.

[0077] It should be noted that the target log system can directly determine the attribute information of the disk header data as the shadow information of the data to be written, and can also perform information transformation on the attribute information of the disk header data to obtain the shadow information of the data to be written.

[0078] S500: The target stores the shadow information of the data to be written from the log system into its own memory.

[0079] In this process, after the target receives and responds to a data write request sent by the main log system, it can construct a memory space within its own storage space and then store the shadow information of the data to be written in that memory. Optionally, this memory can be the target's own storage space within the log system, or it can be a portion of that storage space.

[0080] In this embodiment of the application, in order not to affect the normal operating speed of the target log system, the memory storage space is small, only large enough to store the shadow information to be stored once.

[0081] In the technical solution of this application embodiment, the target log system can extract disk header data from the data to be written and obtain the attribute information of the disk header data. Based on the attribute information of the disk header data, it determines the shadow information of the data to be written and then stores the shadow information of the data to be written in its own memory. This method can temporarily store the pre-built shadow information of the data to be written in the memory of the log system itself, so that when the main log system fails, the shadow information of the data to be written can be sent to the backend flushing component in a timely manner, further improving the reading efficiency of the backend flushing component and the replay efficiency of the fault-free log system.

[0082] In some scenarios, it is necessary to ensure a higher similarity between the attribute information of the disk header data required to determine the shadow information and the attribute information of the data to be written, in order to further determine the shadow information of the data to be written with higher accuracy and efficiently read the correct data to be written from the temporary disk. Based on this, the process of obtaining the attribute information of the disk header data in the above steps is described below. In one embodiment, as... Figure 5 As shown, the steps in S300 above can be implemented in the following ways:

[0083] S310. Obtain the storage location of the disk header data in the temporary disk, the size and type of the disk header data.

[0084] In this embodiment, the storage location of the disk header data in the temporary disk is also the storage location of the data to be written in the temporary disk. Optionally, the size of the disk header data can be understood as the length of the numerical value or the total number of bits in the disk header data; the type of the disk header data can be understood as floating-point, integer, character, Boolean, etc., wherein the type of disk header data in different data to be written can be different.

[0085] Specifically, when the target log system stores data to be written to its own temporary disk, it can determine the storage location of the data to be written, thereby obtaining the storage location of the disk header data on the temporary disk. Furthermore, the target log system can directly determine the size and type of the disk header data based on the disk header data.

[0086] S320. Determine the attribute information of the disk header data based on the storage location, size, and type of the disk header data.

[0087] In this embodiment, the target log system can combine the storage location, size, and type of disk header data in a preset order to obtain the attribute information of the disk header data. Optionally, the preset combination order can be any order among the storage location, size, and type of disk header data, or it can be a specific order among the storage location, size, and type of disk header data that is user-defined.

[0088] In the technical solution of this application embodiment, the target can obtain the storage location, size and type of disk header data in the temporary disk from the log system, and determine the attribute information of the disk header data based on the storage location, size and type of the disk header data. This method can obtain the attribute information of disk header data that is more similar to the attribute information of the data to be written, thereby preparing for further construction of more accurate shadow information.

[0089] In some scenarios, the storage capacity of the memory created by the target from the log system's storage space is also limited. When the stored memory reaches a certain value, the memory can no longer store shadow information normally. To solve this problem, when the stored memory reaches a certain value, it is necessary to evict the historical shadow information already stored in memory to periodically and quantitatively clean up unnecessary memory and leave more remaining storage capacity in memory so that the currently acquired shadow information can be stored normally. The process of whether to evict the historical shadow information already stored in memory is described below. In one embodiment, before the step of storing the shadow information of the data to be written into its own memory in S500 above, such as Figure 6 As shown, the above data processing method may further include:

[0090] S510: Detect whether the remaining memory storage is greater than or equal to the required storage for storing shadow information.

[0091] Optionally, the remaining storage capacity in memory can be equal to the difference between the total storage space in memory and the occupied storage space of the historical shadow information already stored in memory, or it can be equal to the total storage space in memory and the size of the historical shadow information already stored in memory. Optionally, the required storage capacity of the shadow information can be equal to the storage space required to store the shadow information in memory, or it can be equal to the size of the shadow information required to store it in memory.

[0092] In actual processing, the target can determine from the log system whether the remaining storage in memory is greater than or equal to the required storage for storing shadow information, in order to complete the detection process.

[0093] S520. If the remaining storage capacity of the memory is greater than or equal to the required storage capacity of the shadow information, then execute the step of storing the shadow information of the data to be written into its own memory.

[0094] In this embodiment, if the remaining storage capacity of the memory is determined to be greater than or equal to the required storage capacity of the shadow information, it indicates that the current remaining storage capacity in the memory can meet the conditions for storing the shadow information of the data to be written. At this time, the step of storing the shadow information of the data to be written into its own memory can be directly executed, without needing to eliminate the historical shadow information already stored in the memory to store the current shadow information. It should be noted here that eliminating the historical shadow information already stored in the memory is equivalent to deleting the historical shadow information already stored in the memory.

[0095] In another embodiment, after the steps in S510 described above, as follows: Figure 7 As shown, the above data processing method may further include the following steps:

[0096] S530. If the remaining storage capacity of memory is less than the required storage capacity of shadow information, then select historical shadow information that meets the preset conditions from all the shadow information already stored in memory.

[0097] In this embodiment of the application, if it is determined that the remaining storage capacity of the memory is less than the required storage capacity of the shadow information, it indicates that the current remaining storage capacity in the memory cannot meet the conditions for storing the shadow information of the data to be written. At this time, historical shadow information that meets the preset conditions can be selected from all the shadow information already stored in the memory.

[0098] Optionally, the historical shadow information with preset conditions can be all historical shadow information before the current moment, or it can be all historical shadow information before a certain moment in history, and the storage amount of all historical shadow information before that certain moment in history is greater than or equal to a preset storage amount threshold, so as to ensure that after deleting historical shadow information that meets the preset conditions from memory, the remaining storage amount in memory can meet the conditions for storing shadow information of data to be written.

[0099] S540. After deleting the historical shadow information in memory, execute the step of storing the shadow information of the data to be written into its own memory.

[0100] In practical applications, after deleting historical shadow information that meets preset conditions from memory, the step of storing the shadow information of the data to be written into its own memory is performed to ensure that the current shadow information can be successfully stored into memory.

[0101] This application embodiment can, when it is determined that the remaining storage capacity of the memory is less than the required storage capacity of the shadow information, select historical shadow information that meets preset conditions from all the shadow information already stored in the memory, delete the historical shadow information in the memory, and then perform the step of storing the shadow information of the data to be written into its own memory; when it is determined that the remaining storage capacity of the memory cannot meet the conditions for storing the shadow information of the data to be written, this method can pre-eliminate the historical shadow information in the memory to ensure that the remaining storage capacity of the memory after elimination can meet the conditions for storing the shadow information of the data to be written, thereby improving the success rate of shadow information storage.

[0102] In the technical solution of this application embodiment, before storing the shadow information, it can detect whether the remaining storage capacity of the memory is sufficient to store the shadow information of the data to be written. Only when it is determined that the remaining storage capacity of the memory is sufficient to store the shadow information of the data to be written, will the shadow information of the data to be written be stored in its own memory. This method performs a storage verification process when storing the current shadow information into memory to ensure that the shadow information can be successfully stored into memory, and further improves the success rate of the backend flushing component in successfully reading the data to be written based on the shadow information.

[0103] In one embodiment, this application also provides a data processing method, which includes the following steps:

[0104] (1) The target responds to the data write request sent by the main log system from the log system and writes the data to be written carried in the data write request to the temporary disk.

[0105] (2) While writing the data to be written to the temporary disk, the target extracts the disk header data from the data to be written from the log system and obtains the storage location of the disk header data in the temporary disk, the size and type of the disk header data.

[0106] (3) The goal is to determine the attribute information of the disk header data from the log system based on the storage location, size and type of the disk header data.

[0107] (4) The target system determines the shadow information of the data to be written from the attribute information of the disk header data.

[0108] (5) The objective is to detect from the log system whether the remaining memory storage is greater than or equal to the required storage for storing shadow information.

[0109] (6) If the remaining storage capacity of the memory is greater than or equal to the required storage capacity of the shadow information, then the step of storing the shadow information of the data to be written into its own memory is executed; if the remaining storage capacity of the memory is less than the required storage capacity of the shadow information, then the historical shadow information that meets the preset conditions is selected from all the shadow information already stored in the memory, the historical shadow information in the memory is deleted, and then the step of storing the shadow information of the data to be written into its own memory is executed.

[0110] (7) If the main log system in the distributed log system is detected to be faulty, the target obtains the shadow information of the data to be written from the log system; the target selects any normal slave log system from multiple slave log systems after the main log system fails.

[0111] (8) The target sends shadow information from the log system to the backend flushing component of the distributed log system, instructing the backend flushing component to obtain the data to be written from the temporary disk of the log system based on the shadow information, and preprocesses the data to be written to obtain preprocessed data, and then writes the preprocessed data to the backend disk; the preprocessing includes format conversion and encapsulation processing.

[0112] For details of the execution process of (1) to (8) above, please refer to the description of the above embodiments. The implementation principle and technical effect are similar, and will not be repeated here.

[0113] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0114] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0115] In one embodiment, Figure 8This is a schematic diagram of the structure of a data processing device in one embodiment of this application. The data processing device provided in this embodiment can be applied to a server. Figure 8 As shown, the data processing apparatus of this application embodiment may include: a shadow acquisition module 11 and a sending module 12, wherein:

[0116] The shadow acquisition module 11 is used to acquire shadow information of the data to be written when the main log system in the distributed log system is detected to have failed.

[0117] The sending module 12 is used to send shadow information to the backend flushing component of the distributed log system, instructing the backend flushing component to obtain the data to be written from the temporary disk of the log system based on the shadow information, and write the data to be written to the backend disk of the distributed log system.

[0118] The backend flushing component, after obtaining the data to be written, preprocesses the data to obtain preprocessed data, and then writes the preprocessed data to the backend disk; the preprocessing includes format conversion and encapsulation processing.

[0119] The data processing apparatus provided in this application embodiment can be used to execute the technical solutions in the above-described data processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0120] In one embodiment, the data processing apparatus further includes a request-response module, wherein:

[0121] The request and response module is used to respond to data write requests sent by the main log system, write the data to be written carried in the data write request to a temporary disk, and construct shadow information of the data to be written.

[0122] The data processing apparatus provided in this application embodiment can be used to execute the technical solutions in the above-described data processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0123] In one embodiment, the request-response module includes: an information acquisition unit, a shadow information determination unit, and a shadow information storage unit, wherein:

[0124] The information acquisition unit is used to extract disk header data from the data to be written and to obtain the attribute information of the disk header data;

[0125] The shadow information determination unit is used to determine the shadow information of the data to be written based on the attribute information of the disk header data.

[0126] The shadow information storage unit is used to store the shadow information of the data to be written into its own memory.

[0127] The data processing apparatus provided in this application embodiment can be used to execute the technical solutions in the above-described data processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0128] In one embodiment, the information acquisition unit is specifically used for:

[0129] Retrieve the storage location of the disk header data in the temporary disk, as well as the size and type of the disk header data;

[0130] The attribute information of the disk header data is determined based on its storage location, size, and type.

[0131] The data processing apparatus provided in this application embodiment can be used to execute the technical solutions in the above-described data processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0132] In one embodiment, the request-response module further includes: a detection unit and a first execution unit, wherein:

[0133] The detection unit is used to detect whether the remaining storage capacity of the memory is greater than or equal to the required storage capacity for storing the shadow information.

[0134] The first execution unit is used to execute the step of storing the shadow information of the data to be written into its own memory when the detection result of the detection unit is that the remaining storage amount of memory is greater than or equal to the required storage amount of the shadow information.

[0135] The data processing apparatus provided in this application embodiment can be used to execute the technical solutions in the above-described data processing method embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0136] In one embodiment, the request response module further includes: a shadow information filtering unit and a second execution unit, wherein:

[0137] The shadow information filtering unit is used to filter historical shadow information that meets preset conditions from all the shadow information already stored in memory when the detection result of the detection unit is that the remaining storage capacity of memory is less than the required storage capacity of shadow information.

[0138] The second execution unit is used to delete the historical shadow information in memory and then execute the step of storing the shadow information of the data to be written into its own memory.

[0139] Specific limitations regarding the data processing device can be found in the limitations regarding the data processing method described above, and will not be repeated here. Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the server's processor in hardware form or independent of it, or stored in the server's memory in software form, so that the processor can call and execute the operations corresponding to each module.

[0140] In one embodiment, a server is provided, the server including memory and a processor, and the internal structure diagram of the server can be as follows: Figure 9 As shown, the server includes a processor, memory, and a network interface connected via a system bus. The processor provides processing power. The server's memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The server's database stores shadow information of data to be written. The server's network interface communicates with external endpoints via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0141] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the server to which the solution of this application is applied. A specific server may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one embodiment, a server is also provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the technical solutions in the above-described data processing method embodiments of this application. The implementation principle and technical effects are similar, and will not be repeated here.

[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the technical solution of the data processing method described above in this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0144] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the technical solution of the data processing method described above in this application. The implementation principle and technical effects are similar and will not be repeated here.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: If a failure is detected in the primary log system of the distributed log system, the target obtains shadow information of the data to be written from the log system; the target log system is any normal log system selected from multiple log systems after the primary log system fails. The target sends the shadow information from the log system to the backend flushing component of the distributed log system, instructing the backend flushing component to retrieve the data to be written from the temporary disk of the log system based on the shadow information, and write the data to be written to the backend disk of the distributed log system; The target responds to the data write request sent by the main log system from the log system and writes the data to be written carried in the data write request to the temporary disk; The objective is to extract disk header data from the data to be written from the log system and obtain the attribute information of the disk header data; The target determines the shadow information of the data to be written from the log system based on the attribute information of the disk header data; The target stores the shadow information of the data to be written from the log system into its own memory.

2. The method according to claim 1, characterized in that, The attribute information for obtaining the disk header data includes: Obtain the storage location of the disk header data in the temporary disk, the size of the disk header data, and the type of the disk header data; The attribute information of the disk header data is determined based on the storage location, size, and type of the disk header data.

3. The method according to claim 1, characterized in that, Before storing the shadow information of the data to be written into its own memory, the method further includes: Detect whether the remaining storage capacity of the memory is greater than or equal to the required storage capacity for storing the shadow information; If the remaining storage capacity of the memory is greater than or equal to the required storage capacity of the shadow information, then the step of storing the shadow information of the data to be written into its own memory is executed.

4. The method according to claim 3, characterized in that, The method further includes: If the remaining storage capacity of the memory is less than the required storage capacity of the shadow information, then historical shadow information that meets the preset conditions is selected from all the shadow information already stored in the memory. After deleting the historical shadow information in the memory, the step of storing the shadow information of the data to be written into its own memory is performed.

5. The method according to claim 4, characterized in that, The historical shadow information of the preset conditions includes any one of the following: All historical shadow information prior to the current moment; All historical shadow information prior to a preset historical time, and the storage amount of all historical shadow information prior to the preset historical time is greater than or equal to a preset storage amount threshold.

6. The method according to any one of claims 1-5, characterized in that, After obtaining the data to be written, the backend flushing component preprocesses the data to obtain preprocessed data, and then writes the preprocessed data to the backend disk; the preprocessing includes format conversion and encapsulation processing.

7. The method according to claim 1, characterized in that, After the primary log system fails, a healthy slave log system is elected from among multiple slave log systems, including: Multiple fault-free slave log systems elect one slave log system based on a preset election strategy; wherein the preset election strategy includes at least one of a distributed system election algorithm and a distributed coordination service.

8. A data processing apparatus, characterized in that, The device includes: The shadow acquisition module is used to acquire shadow information of data to be written when a failure is detected in the main log system of the distributed log system. The sending module is used to send the shadow information to the backend flushing component of the distributed log system, instructing the backend flushing component to obtain the data to be written from the temporary disk of the log system based on the shadow information, and to write the data to be written to the backend disk of the distributed log system. The request-response module is used to respond to the data write request sent by the main log system, write the data to be written carried in the data write request to the temporary disk; extract disk header data from the data to be written, and obtain the attribute information of the disk header data; determine the shadow information of the data to be written based on the attribute information of the disk header data; and store the shadow information of the data to be written in its own memory.

9. A server comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images