Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a data processing method and device, electronic equipment and storage medium, relating to the field of databases, which can be used in the field of distributed databases, for example. The method is applied to a storage node, which is provided with at least two disks; the method comprises: obtaining a Raft log write request; writing the Raft log in the Raft log write request into each disk in the storage node; and returning a write completion notification to a computing node if a write success notification returned by any disk is received. Embodiments of the present disclosure determine that the writing is completed when it is detected that any disk in the multiple disks completes the writing, so that even if the writing delay is significantly increased due to the lag of a certain disk in the multiple disks, the overall response time of the system is not affected, the writing delay jitter caused by the lag of a single disk is avoided, the stability of the system is improved, and the application requirements are better met.

Description

Technical Field

[0001] This disclosure relates to the field of database technology, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In distributed database systems, such as the Raft consensus algorithm used by TiKV, log append operations (Raftlog append) require frequent synchronization and disk writing to ensure data consistency and system fault tolerance. However, in cloud computing environments, the I / O latency of the underlying storage medium (such as cloud disks) is often affected by uncertain factors, resulting in write latency jitter, which can impact system stability. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, electronic device, and storage medium, which can solve the problem of system instability caused by write latency jitter in the prior art. The technical solution provided by this disclosure is as follows: According to one aspect of the present disclosure, a data processing method is provided, applied to a storage node, the storage node being provided with at least two disks; The method includes: Retrieve Raft log write requests; Write the Raft log in the Raft log write request to each disk of the storage node; If a write success notification is received from any disk, a write completion notification is returned to the compute node.

[0004] Optionally, each disk corresponds to an asynchronous task queue; The step of writing the Raft log in the Raft log write request to each disk in the storage node includes: For each disk, the Raft log write request is sent to the disk through the asynchronous task queue corresponding to the disk, and the Raft log is written to the disk.

[0005] Optionally, the method further includes: Receive Raft log read requests; The primary disk is determined from the at least two disks, and the read data requested by the read request is obtained from the primary disk; the primary disk is the disk where the latest Raft log is located among the data stored on each disk.

[0006] Optionally, each disk corresponds to an asynchronous task queue with a corresponding task processing identifier. If the disk completes processing a Raft log write request, the task processing identifier of the asynchronous task queue corresponding to the disk is incremented by 1. Determining the primary disk from the at least two disks includes: The disk with the largest task processing identifier in the asynchronous task queue corresponding to each disk is designated as the master disk.

[0007] Optionally, the method further includes: If an abnormal disk is detected among the at least two disks, then stop sending any requests to the abnormal disk; If the abnormal disk is detected to have recovered, the latest Raft logs will be synchronized to the recovered abnormal disk.

[0008] Optionally, the method further includes: If the storage node restarts, and in the event that inconsistencies are detected in the data stored on each disk, the latest Raft log will be determined from the data stored on each disk. The disk containing the latest Raft log is designated as the primary disk, and the latest Raft log is synchronized to each of the at least two disks other than the primary disk.

[0009] Optionally, the latest Raft log is determined based on the following method: Retrieve the latest log files stored on each disk; Based on the file number of each latest log file, at least one target log file is determined from each latest log file; the target log file is the latest log file with the largest file number among the latest log files. If the number of target log files is 1, then the target log file is considered the latest Raft log. If the number of target log files is greater than 1, then based on the offset of the target log files, the target log file with the largest offset is taken as the latest Raft log.

[0010] According to another aspect of the present disclosure, a data processing apparatus is provided for a storage node, the storage node being provided with at least two disks; The device includes: The write request retrieval module is used to retrieve Raft log write requests; The log writing module is used to write the Raft log in the Raft log write request to each disk in the storage node; The response module is used to return a write completion notification to the compute node if a write success notification is received from any disk.

[0011] According to another aspect of the present disclosure, an electronic device is provided, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described data processing methods.

[0012] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps of any of the data processing methods described above.

[0013] According to one aspect of the present disclosure, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps of any of the data processing methods described above.

[0014] The beneficial effects of the technical solutions provided in this disclosure are: By setting up multiple disks in the storage node and writing Raft logs to multiple disks, the write is considered complete when any one of the multiple disks completes the write operation. Even if one of the multiple disks experiences a significant write delay, it will not affect the overall system response time. This avoids write delay jitter caused by a single disk's delay, improves system stability, and better meets application requirements. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0016] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this disclosure; Figure 2 A flowchart illustrating a latest method for determining Raft logs provided in this disclosure embodiment; Figure 3 A schematic diagram illustrating a dual-disk Raft log writing method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0017] The embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions of the embodiments of this disclosure.

[0018] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this disclosure mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element are connected through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term, for example, “A and / or B” or “A, B” indicates implementation as “A,” or implementation as “B,” or implementation as “A and B.”

[0019] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0020] The following description of several exemplary embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0021] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present disclosure, as shown below. Figure 1 As shown, the method includes: Step S110: Obtain the Raft log write request; Step S120: Write the Raft log in the Raft log write request to each disk in the storage node; Step S130: If a write success notification is received from any disk, a write completion notification is returned to the compute node.

[0022] Specifically, a distributed database system may include multiple nodes, and the method provided in this disclosure embodiment can be applied to storage nodes, wherein the storage nodes can be used to store data.

[0023] A storage node can be configured with at least two disks. The number of disks configured on a storage node can be determined by at least one of the database resources and actual business needs. When the database resources are sufficient, more disks can be configured; when the timeliness requirements of the business are high, more disks can be configured. This disclosure does not specifically limit the number of disks.

[0024] A Raft log is an append-only, ordered array of entries used to ensure data consistency among multiple replicas in a distributed system. A Raft log write request is triggered under certain conditions, such as when a data change operation (e.g., update or delete) is received from a client or when the role of a node in the database system changes (e.g., a new Leader node comes online).

[0025] Raft log write requests can include the Raft logs to be written. After receiving the Raft log write request, the Raft logs in the Raft log write request can be written to each disk in the storage node.

[0026] When any disk successfully writes the Raft log, it can return a corresponding write success notification. If a write success notification is received from any disk, a write completion notification is returned to the compute node (i.e., the application above the storage node).

[0027] In this embodiment of the disclosure, multiple disks are set in the storage node and Raft logs are written to multiple disks. When any disk among the multiple disks is detected to have completed writing, the writing is determined to be complete. Even if one of the multiple disks experiences a significant write delay due to a lag, it will not affect the overall response time of the system. This avoids write delay jitter caused by a lag on a single disk, improves the stability of the system, and better meets application requirements.

[0028] As an optional implementation, each disk corresponds to an asynchronous task queue; Write the Raft log from the Raft log write request to each disk on the storage node, including: For each disk, Raft log write requests are sent to the disk through the corresponding asynchronous task queue, and Raft logs are written to the disk.

[0029] Specifically, for each disk in the storage node, a corresponding asynchronous task queue can be allocated to each disk, and the asynchronous task queue can be used to transmit requests sent to that disk.

[0030] For each disk, after receiving the Raft log write request, the Raft log write request can be treated as a write task and assigned to the asynchronous task queue of each disk. The write task is then sent to the disk corresponding to the asynchronous task queue through each asynchronous task queue, and the corresponding Raft log is written on that disk.

[0031] In this embodiment of the disclosure, by setting up corresponding asynchronous task queues for each disk, these multiple asynchronous task queues are independent and do not interfere with each other. The processing of each queue is parallel and does not block each other. When any disk is completed writing, a write success notification can be returned through the asynchronous task queue corresponding to that disk, thus shortening the response time of the request.

[0032] As an optional embodiment, the method further includes: Get Raft log read requests; The primary disk is determined from at least two disks, and the read data requested by the read request is obtained from the primary disk; the primary disk is the disk where the latest Raft log is located among the data stored on each disk.

[0033] Specifically, each disk stores corresponding data. Since the log writes of each disk are not synchronized, there may be inconsistencies in the data stored on each disk.

[0034] When Follower log replication or Leader election occurs in the database system, Raft log read requests can be triggered. When a storage node receives a Raft log read request, it can compare the Raft logs stored on each disk, and use the disk containing the latest Raft log as the primary disk. The data requested by the read request is then retrieved from the primary disk to ensure that the data read is the latest data.

[0035] As an optional embodiment, each disk corresponds to an asynchronous task queue with a corresponding task processing identifier. If a disk completes processing a Raft log write request, the task processing identifier of the asynchronous task queue corresponding to the disk is incremented by 1. The primary disk is determined from at least two disks, including: The disk with the largest task processing identifier in the asynchronous task queue corresponding to each disk is designated as the primary disk.

[0036] Specifically, for each asynchronous task queue corresponding to a disk, a corresponding task processing identifier can be set for each asynchronous task queue. The task processing identifier is used to indicate the progress of the Raft log write request processed by the corresponding disk. When the asynchronous task queue sends a Raft log write request to the corresponding disk, the task processing identifier of the asynchronous task queue is automatically incremented by 1. That is to say, the larger the task processing identifier of the asynchronous task queue, the more updated the version of the data stored on the disk is.

[0037] Therefore, based on the task processing identifiers of each asynchronous task queue, the disk with the largest task processing identifier can be used as the primary disk.

[0038] In this embodiment of the disclosure, by setting task processing identifiers for each asynchronous task queue and setting the task processing identifiers to be automatically updated as write requests are processed, the main disk can be located quickly and accurately from multiple disks, thereby improving read efficiency.

[0039] As an optional embodiment, the method further includes: If an abnormal disk is detected in at least two disks, stop sending any requests to the abnormal disk; If the abnormal disk is detected and then recovers, the latest Raft logs will be synchronized to the recovered abnormal disk.

[0040] Specifically, for multiple disks in a storage node, if a disk is unavailable for a long time or the channel backlog exceeds the threshold, the disk can be considered abnormal and the abnormal disk is designated as an abnormal disk. If an abnormal disk is detected among multiple disks, any requests sent to the abnormal disk can be stopped, that is, reading and writing to the abnormal disk can be stopped.

[0041] If the abnormal disk is detected and recovers to normal, the Raft log may have changed due to new Raft log write requests received by the storage node during the abnormal disk's abnormal state. Therefore, the latest Raft log can be obtained and synchronized to the recovered abnormal disk to ensure its availability.

[0042] As an optional embodiment, the method further includes: If the storage node restarts, and in the event that the data stored on each disk is inconsistent, the latest Raft log is determined from the data stored on each disk, and the disk containing the latest Raft log is designated as the primary disk. The latest Raft log is then synchronized to at least two disks other than the primary disk.

[0043] Specifically, storage nodes may restart during operation. After a storage node restarts, it can first compare the data stored on each disk. If the data stored on each disk is consistent, there is no need to synchronize the data on each disk. If the data stored on each disk is inconsistent, it is necessary to determine the latest Raft log from the data stored on each disk, designate the disk storing the latest Raft log as the master disk, and synchronize the latest Raft log to all other disks except the master disk to ensure the consistency of the data stored on each disk.

[0044] As an optional embodiment, Figure 2 A flowchart illustrating a latest method for determining Raft logs provided in this disclosure is shown below. Figure 2 The latest Raft log shown is determined based on the following method: Step S210: Obtain the latest log files stored on each disk; Step S220: Based on the file number of each latest log file, determine at least one target log file from each latest log file; the target log file is the latest log file with the largest file number among the latest log files; Step S230: If the number of target log files is 1, then the target log file is the latest Raft log. Step S240: If the number of target log files is greater than 1, then based on the offset of the target log files, the target log file with the largest offset is taken as the latest Raft log.

[0045] Specifically, when a storage node restarts, the asynchronous task queues corresponding to each disk need to be recreated, which may cause the information stored in the asynchronous task queues corresponding to each disk before the restart to become invalid. Therefore, the latest Raft logs can be determined by comparing the log files stored on each disk.

[0046] For each disk, the Raft log can be stored in the form of log files, and each log file corresponds to a file number. Understandably, for each disk, the file numbers of the multiple log files stored on that disk are sequentially incremented. The log file with the largest file number on that disk can be considered the latest log file on that disk.

[0047] Compare the file numbers of the latest log files on each disk, and select the latest log file with the largest file number as the target log file.

[0048] When the number of target log files is 1, the target log file is used as the latest Raft log.

[0049] When the number of identified target log files is greater than one, the offsets of each target log file can be compared, and the target log file with the largest offset can be taken as the latest Raft log. The offset of a log file refers to the byte distance from the beginning of the file (byte 0) to a specific data location.

[0050] In this embodiment of the disclosure, the latest Raft log is determined by comparing the file number and effective offset of the latest log file on each disk, and the latest Raft log is synchronized to each disk, thereby ensuring the consistency of data on each disk.

[0051] As an optional embodiment, the method further includes: Obtain disk operating mode adjustment information; If the disk working mode adjustment information is a switch from single disk working mode to multi disk working mode, then the disk working in single disk working mode will be used as the master disk, and the data stored on the master disk will be synchronized to the other disks. If the data on each disk is consistent, and a new Raft log write request is obtained, then the Raft log in the Raft log write request will be written to each disk. If the disk operating mode adjustment information indicates a switch from multi-disk operating mode to single-disk operating mode, then the primary disk is determined from the disks operating in multi-disk operating mode. If a new Raft log write request is obtained, the Raft log in the Raft log write request is written to the primary disk, and writing to other disks is stopped.

[0052] Specifically, during database system operation, the disk operating mode of storage nodes can be adjusted according to actual needs.

[0053] Disk operating mode adjustment information can be used to adjust the disk operating mode, including switching from single-disk operating mode to multi-disk operating mode or from multi-disk operating mode to single-disk operating mode.

[0054] For database systems, an initial disk operating mode configuration can be set. When the disk operating mode needs to be adjusted, adjustment information can be set. For example, if the initial disk operating mode is a single-disk mode, it can be adjusted to a multi-disk mode; if the initial disk operating mode is a multi-disk mode, it can be adjusted to a single-disk mode.

[0055] When switching from single-disk working mode to multi-disk working mode, the disk working in single-disk working mode can be used as the master disk, and the data stored on the master disk can be synchronized to other disks. If the data on each disk is consistent, and a new Raft log write request is obtained, the Raft log in the Raft log write request will be written to each disk.

[0056] When switching from multi-disk working mode to single-disk working mode, the primary disk can be determined from the disks working in multi-disk working mode. If a new Raft log write request is obtained, the Raft log in the Raft log write request is written to the primary disk, and writing to other disks is stopped.

[0057] In this embodiment of the disclosure, by adjusting the disk working mode configuration information, the disk working mode can be set according to different needs, thereby improving the flexibility of the system.

[0058] Figure 3 This is a schematic diagram of a dual-disk Raft log writing method provided in an embodiment of this disclosure, as shown below. Figure 3 As shown, a HedgedFileSystem can be defined to implement the original FileSystem interface to support concurrent writes to two independent paths (disks).

[0059] Two disks, disk 1 and disk 2, can be set up in a storage node. Asynchronous task queues are allocated to the two disks respectively (i.e., queue 1 corresponds to disk 1 and queue 2 corresponds to disk 2). Each write task (i.e. Raft log write request) is broadcast. As long as one disk returns successfully, it is considered that the write is completed, which significantly reduces the worst-case path latency. By introducing the "first to complete returns" mechanism at the Raft engine layer, the elasticity of the write path and the concurrency fault tolerance are improved.

[0060] For read operations, the most recently written data to the disk can be identified, and the data required for the read operation can be read from the most recently written data to ensure data consistency and read efficiency.

[0061] When a disk becomes unavailable for an extended period or when the backlog of tasks on a channel exceeds a threshold, write operations to that disk are automatically disconnected. Data is then resynchronized once the disk recovers, ensuring overall system stability. This ensures automatic disconnection and recovery in the event of disk failure, guaranteeing continuous system availability.

[0062] After restarting, the system can automatically compare the latest log file number and valid offset between the two disks, select the primary disk, and perform data restoration. The primary and secondary disks are determined by comparing the file number and offset, enhancing reliability. The method provided in this disclosure significantly reduces write latency caused by single-disk I / O stuttering without affecting the original storage engine's non-intrusive logic. In addition, it provides controllable upgrade (i.e., switching from a single disk to a dual disk) and downgrade path (i.e., switching from a dual disk to a single disk), enhancing flexibility.

[0063] Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, the device is applied to a storage node, which is equipped with at least two disks; The apparatus in this embodiment may include: The write request acquisition module 310 is used to acquire Raft log write requests; Log writing module 320 is used to write the Raft log in the Raft log write request to each disk in the storage node; The response module 330 is used to return a write completion notification to the compute node if a write success notification is received from any disk.

[0064] As an optional implementation, each disk corresponds to an asynchronous task queue; When the log writing module writes the Raft log in the Raft log write request to each disk in the storage node, it is used to: For each disk, the Raft log write request is sent to the disk through the asynchronous task queue corresponding to the disk, and the Raft log is written to the disk.

[0065] As an optional embodiment, the device further includes a read request processing module for: Receive Raft log read requests; The primary disk is determined from the at least two disks, and the read data requested by the read request is obtained from the primary disk; the primary disk is the disk where the latest Raft log is located among the data stored on each disk.

[0066] As an optional embodiment, each disk corresponds to an asynchronous task queue with a corresponding task processing identifier. If the disk completes processing a Raft log write request, the task processing identifier of the asynchronous task queue corresponding to the disk is incremented by 1. When determining the primary disk from the at least two disks, the read request processing module is used to: The disk with the largest task processing identifier in the asynchronous task queue corresponding to each disk is designated as the master disk.

[0067] As an optional embodiment, the device further includes an abnormal disk processing module for: If an abnormal disk is detected among the at least two disks, then stop sending any requests to the abnormal disk; If the abnormal disk is detected to have recovered, the latest Raft logs will be synchronized to the recovered abnormal disk.

[0068] As an optional embodiment, the device further includes a restart processing module for: If the storage node restarts, and in the event that inconsistencies are detected in the data stored on each disk, the latest Raft log will be determined from the data stored on each disk. The disk containing the latest Raft log is designated as the primary disk, and the latest Raft log is synchronized to each of the at least two disks other than the primary disk.

[0069] As an optional embodiment, the latest Raft log is determined based on the following method: Retrieve the latest log files stored on each disk; Based on the file number of each latest log file, at least one target log file is determined from each latest log file; the target log file is the latest log file with the largest file number among the latest log files. If the number of target log files is 1, then the target log file is considered the latest Raft log. If the number of target log files is greater than 1, then based on the offset of the target log files, the target log file with the largest offset is taken as the latest Raft log.

[0070] The apparatus of this disclosure embodiment can execute the method provided in this disclosure embodiment, and its implementation principle is similar, and it has corresponding technical effects. The actions performed by each module in the apparatus of each embodiment of this disclosure correspond to the steps in the method of each embodiment of this disclosure. For a detailed functional description of each module of the apparatus, please refer to the description in the corresponding method shown above, and it will not be repeated here.

[0071] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0072] This disclosure provides an electronic device including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the method provided in any optional embodiment of this disclosure. Compared with the prior art, this achieves the following: by setting up multiple disks in the storage node and writing Raft logs to the multiple disks, the write is determined to be complete when any disk among the multiple disks is detected to have completed writing. Even if one of the multiple disks experiences a significant write latency increase due to a single disk's stutter, it will not affect the overall system response time. This avoids write latency jitter caused by a single disk's stutter, improves system stability, and better meets application requirements.

[0073] In one alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure.

[0074] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0075] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0076] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0077] The memory 4003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0078] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), wearable devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc.

[0079] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0080] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0081] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0082] The above description is only an optional implementation method for some implementation scenarios of this disclosure. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this disclosure without departing from the technical concept of this disclosure also fall within the protection scope of the embodiments of this disclosure.

Claims

1. A data processing method, characterized in that, Applied to storage nodes, wherein the storage nodes are provided with at least two disks; The method includes: Retrieve Raft log write requests; Write the Raft log in the Raft log write request to each disk of the storage node; If a write success notification is received from any disk, a write completion notification is returned to the compute node.

2. The method according to claim 1, characterized in that, Each disk corresponds to an asynchronous task queue; The step of writing the Raft log in the Raft log write request to each disk in the storage node includes: For each disk, the Raft log write request is sent to the disk through the asynchronous task queue corresponding to the disk, and the Raft log is written to the disk.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Receive Raft log read requests; The primary disk is determined from the at least two disks, and the read data requested by the read request is obtained from the primary disk; the primary disk is the disk where the latest Raft log is located among the data stored on each disk.

4. The method according to claim 3, characterized in that, Each disk corresponds to an asynchronous task queue with a corresponding task processing identifier. If the disk completes processing a Raft log write request, the task processing identifier of the asynchronous task queue corresponding to the disk is incremented by 1. Determining the primary disk from the at least two disks includes: The disk with the largest task processing identifier in the asynchronous task queue corresponding to each disk is designated as the master disk.

5. The method according to claim 1, characterized in that, The method further includes: If an abnormal disk is detected among the at least two disks, then stop sending any requests to the abnormal disk; If the abnormal disk is detected to have recovered, the latest Raft logs will be synchronized to the recovered abnormal disk.

6. The method according to claim 1, characterized in that, The method further includes: If the storage node restarts, and in the event that inconsistencies are detected in the data stored on each disk, the latest Raft log will be determined from the data stored on each disk. The disk containing the latest Raft log is designated as the primary disk, and the latest Raft log is synchronized to each of the at least two disks other than the primary disk.

7. The method according to claim 5 or 6, characterized in that, The latest Raft log is determined based on the following method: Retrieve the latest log files stored on each disk; Based on the file number of each latest log file, at least one target log file is determined from each latest log file; the target log file is the latest log file with the largest file number among the latest log files. If the number of target log files is 1, then the target log file is considered the latest Raft log. If the number of target log files is greater than 1, then based on the offset of the target log files, the target log file with the largest offset is taken as the latest Raft log.

8. A data processing apparatus, characterized in that, Applied to storage nodes, wherein the storage nodes are provided with at least two disks; The device includes: The write request retrieval module is used to retrieve Raft log write requests; The log writing module is used to write the Raft log in the Raft log write request to each disk in the storage node; The response module is used to return a write completion notification to the compute node if a write success notification is received from any disk.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images