Data processing method, device and system

Abstract

According to the data processing method, device and system, in the data processing method, only a first computing node needs to execute processing operation on an input and output request, meanwhile, only a processing result obtained by processing the request needs to be stored on the first computing node, and a second computing node serving as a backup generates a transaction log of the input and output request and stores the transaction log. However, processing operation does not need to be executed on the first request, a processing result obtained by processing the request does not need to be stored, and finally the first computing node writes the processing result into the storage node for storage. In this way, compared with the situation that in a distributed system, a plurality of computing nodes need to process one input and output request and store the processing result, consumption of computing resources and consumption of storage resources are reduced.

Description

Technical Field

[0001] This application relates to the field of computers, and specifically to data processing methods, devices, and systems. Background Art

[0002] Generally, in a distributed system, after a computing node receives an input / output (I / O) request, it stores the data after processing the request in a storage node or a computing node, and forwards the input / output request to other backup computing nodes. The backup computing nodes also perform processing operations on the request and store the data after processing the request in a storage node or a computing node. Since multiple computing nodes all need to perform processing operations on an input / output request, and multiple computing nodes all store the data after processing the request, it results in the consumption of more computing resources and storage resources. Summary of the Invention

[0003] This application provides a data processing method, device, and system, which can solve the problem that in a distributed system, a master node and backup nodes need to process an input / output request and store the processing result, resulting in the consumption of more computing resources and storage resources.

[0004] In a first aspect, a data processing method is provided. The data processing method is applied to a distributed system. The distributed system includes multiple computing nodes, and the multiple computing nodes include a first computing node and a second computing node, where the second computing node is a backup node of the first computing node. After the first computing node obtains a first request, it processes the data to be written included in the first request and saves the processing result of the first request; and, the first computing node forwards the first request to the second computing node. The second computing node generates a transaction log of the first request and saves the transaction log of the first request; and the first computing node receives a write success response fed back by the second computing node; finally, the first computing node writes the processing result of the first request into a storage node.

[0005] Compared with multiple computing nodes performing processing operations on an input / output request, for the data processing method provided in this application, only the first computing node needs to perform processing operations on the first request, and the second computing node as a backup does not need to perform processing operations on the first request, reducing the number of times of performing processing operations on the request and reducing the consumption of computing resources. Also, compared with multiple computing nodes all storing the processing results of processing requests, for the data processing method provided in this application, only the first computing node needs to store the processing result of the processing request, and the second computing node as a backup does not need to store the processing result of the processing request, reducing the number of data storage copies and reducing the consumption of storage resources.

[0006] In addition, the second computing node generates and saves the transaction log of the first request, enabling the restoration of the damaged data in the first request and the reprocessing of the request to obtain the processing result based on the transaction log saved by the second computing node in the event of a failure of the first computing node and the damage of the data and processing result of the saved first request, ensuring the security and reliability of the data.

[0007] In a possible implementation, the first request includes the data to be written and the index of the data, and the processing result obtained by the first computing node for processing the first request includes the inverted index of the data.

[0008] In another possible implementation, the first computing node also generates the transaction log of the above first request and saves the transaction log of the first request.

[0009] The transaction log may include information about the data in the first request. In the event of damage to the data and processing result of the first request saved by the first computing node, the damaged data of the first request can also be restored and the request can be reprocessed to obtain the processing result based on the transaction log saved by the first computing node, ensuring the security and reliability of the data.

[0010] In another possible implementation, the first computing node saves the processing result of the first request in the local memory of the first computing node; the first computing node saves the transaction log of the first request in a storage medium outside the local memory of the first computing node.

[0011] Compared with saving the transaction log in the memory and writing it to the storage node together with the processing result of the first request, separating the storage of the processing result and the transaction log of the first request enables the transaction log not to be written to the storage node when the first computing node writes the data in the memory to the storage node for storage, accelerating the writing speed.

[0012] In another possible implementation, the transaction log of the first request includes the above data and the index of the data.

[0013] In another possible implementation, after the first computing node writes the processing result of the first request to the storage node, the first computing node also deletes the transaction log of the first request saved by the first computing node according to the storage file sequence number when the processing result of the first request is written to the storage node.

[0014] Thus, the first computing node can delete the transaction log of the first request it saves to release more storage space and further save storage resources.

[0015] In another possible implementation, the first computing node also sends the storage file sequence number to the second computing node; the second computing node also deletes the transaction log of the first request saved by the second computing node according to the storage file sequence number.

[0016] The first computing node and the second computing node can synchronously clear the transaction logs of the first requests saved on their respective devices. The second computing node can also free up more storage space, further saving storage resources.

[0017] In another possible implementation, among the multiple computing nodes described above, there is also a third computing node. When the first computing node disconnects from the second computing node, the second computing node still receives a second request, and the second request contains data to be written. Then the second computing node processes the second request and saves the processing result of the second request. And the second computing node forwards the second request to the third computing node. Then, the second computing node receives a write success response feedback from the third computing node. Finally, the second computing node writes the processing result of the second request to the storage node, and the storage file serial number when writing to the storage node is incremented by M on the basis of N, where N is the storage file serial number corresponding to the previous write process, and M is greater than or equal to 2.

[0018] When the first computing node disconnects from the second computing node, the first computing node can no longer continue to receive and process new input / output requests and will automatically shut down after a waiting period. Also, the second computing node takes over the functions of the first computing node, that is, the second computing node receives new second requests and processes the second requests. During the automatic shutdown waiting period of the first computing node, if there is still data in the memory of the first computing node, it will continue to be written to the storage node, and the storage file serial number when the first computing node writes to the storage node is N + 1, where N is the storage file serial number N corresponding to the previous write process. At this time, if the storage file serial number when the processing result of the request is finally written to the storage node after the second computing node receives a new second request is also N + 1, then two computing nodes will write different data to the same file in the storage node, resulting in data errors. Therefore, by setting the storage file serial number when the second computing node writes the processing result of the request to the storage node to N + M, where M is greater than or equal to 2, the second computing node and the first computing node can write their respective data to different files in the storage node, avoiding the aforementioned data error problem.

[0019] In another possible implementation, the above M is related to the concurrency number of data write operations of the first computing node and the first time ratio; the first time ratio is determined based on the automatic shutdown waiting time after the first computing unit disconnects from the second computing node and the data write cycle of the first computing unit.

[0020] During the automatic shutdown waiting time of the first computing node, the first computing node can write data to the storage node multiple times according to the data writing period, and each write can include multiple concurrent write operations. Thus, the first computing node will write data to multiple files in the storage node during the automatic shutdown waiting time. By making M related to the ratio of the number of concurrent data write operations of the first computing node and the above first time, it is possible for the second computing node to accurately avoid multiple files to which the first computing node has written data when writing data to the storage node.

[0021] In another possible implementation, the second computing node also deletes files in the storage node whose stored file serial numbers are greater than N and less than N + M.

[0022] Thus, unnecessary files stored in the storage node can be cleared, saving storage resources of the storage node.

[0023] In a second aspect, a data processing method is provided. The data processing method is applied to a distributed system. The distributed system includes multiple computing nodes, and the multiple computing nodes include a first computing node and a second computing node. The second computing node is a backup node of the first computing node. After receiving a first request forwarded by the first computing node, a transaction log of the first request is generated and the transaction log of the first request is saved; then a write success response is fed back to the first computing node, and the write success response is used to instruct the first computing node to write the processing result of the first request to the storage node.

[0024] In a possible implementation, the first request includes data to be written and an index of the data.

[0025] In another possible implementation, the processing result obtained by processing the first request includes an inverted index of the data.

[0026] In another possible implementation, the transaction log of the first request includes data and an index of the data.

[0027] In another possible implementation, a stored file serial number sent by the first computing node is also received.

[0028] In another possible implementation, after receiving the stored file serial number sent by the first computing node, the saved transaction log of the first request is also deleted according to the stored file serial number.

[0029] In a third aspect, a data processing method is provided. The data processing method is applied to a distributed system, which includes multiple computing nodes. The multiple computing nodes include a first computing node and a second computing node, where the second computing node is a backup node of the first computing node. After obtaining a first request, process the first request and save the processing result of the first request. The first request includes data to be written; and forward the first request to the second computing node; then receive a write success response feedback from the second computing node; finally, write the processing result of the first request to a storage node.

[0030] In a possible implementation, the first request includes data to be written and an index of the data.

[0031] In another possible implementation, the processing result obtained by processing the first request includes an inverted index of the data.

[0032] In a possible implementation, a transaction log of the first request is also generated and the transaction log of the first request is saved.

[0033] In another possible implementation, the processing result of the first request is saved in the local memory of the first computing node; the transaction log of the first request is saved in a storage medium outside the local memory of the first computing node.

[0034] In another possible implementation, the transaction log of the first request includes data and an index of the data.

[0035] In another possible implementation, the saved transaction log of the first request is also deleted according to the storage file sequence number when the processing result of the first request is written to the storage node.

[0036] In another possible implementation, the storage file sequence number is also sent to the second computing node.

[0037] In a fourth aspect, a data processing device is provided. The data processing device includes a communication module and a processing module.

[0038] Specifically, the communication module is used to obtain a first request, and the first request includes data to be written; the processing module is used to process the first request and save the processing result of the first request; the communication module is also used to forward the first request to the second computing node; the processing module is also used to generate a transaction log of the first request and save the transaction log of the first request; the communication module is also used to receive a write success response feedback from the second computing node; the processing module is also used to write the processing result of the first request to a storage node.

[0039] In a possible implementation, the first request includes data to be written and an index of the data.

[0040] In another possible implementation, the processing result obtained by processing the first request includes the inverted index of the data.

[0041] In another possible implementation, the processing module is further configured to generate a transaction log of the first request and save the transaction log of the first request.

[0042] In another possible implementation, the processing module is further configured to save the processing result of the first request in the local memory of the first computing node, and save the transaction log of the first request in a storage medium outside the local memory of the first computing node.

[0043] In another possible implementation, the transaction log of the first request includes the data and the index of the data.

[0044] In another possible implementation, after writing the processing result of the first request to the storage node, the processing module is further configured to delete the transaction log of the first request saved by the first computing node according to the storage file sequence number when the processing result of the first request is written to the storage node.

[0045] In another possible implementation, the communication module is further configured to send the storage file sequence number to the second computing node.

[0046] In another possible implementation, the processing module is further configured to delete the transaction log of the first request saved by the second computing node according to the storage file sequence number.

[0047] In another possible implementation, among the multiple computing nodes, there is also a third computing node. When the first computing node disconnects from the second computing node, the communication module is further configured to receive a second request.

[0048] In another possible implementation, the processing module is further configured to process the second request and save the processing result of the second request.

[0049] In another possible implementation, the communication module is further configured to forward the second request to the third computing node.

[0050] In another possible implementation, the communication module is further configured to receive a write success response feedback from the third computing node.

[0051] In another possible implementation, after receiving the write success response feedback from the third computing node, the processing module is further configured to write the processing result of the second request to the storage node, and the storage file sequence number when written to the storage node is increased by M on the basis of N, where N is the storage file sequence number corresponding to the previous write process, and M is greater than or equal to 2.

[0052] In another possible implementation, M is related to the ratio of the number of data writing operations concurrent with the first computing node to the first time. The first time ratio is determined based on the automatic shutdown waiting time after the first computing unit disconnects from the second computing node and the data writing cycle of the first computing unit.

[0053] In another possible implementation, the processing module is further configured to delete files in the storage node whose stored file serial numbers are greater than N and less than N + M.

[0054] In a fifth aspect, a data processing system is provided. The data processing system includes a plurality of computing nodes, and the plurality of computing nodes execute the data processing method described in the first aspect, the second aspect, the third aspect, or any possible implementation.

[0055] In a sixth aspect, a computer program product including instructions is provided. When the instructions are run by a cluster of computing devices, the cluster of computing devices is caused to execute the data processing method described in the first aspect, the second aspect, the third aspect, or any possible implementation.

[0056] In a seventh aspect, a computer-readable storage medium is provided. The computer-readable storage medium includes computer program instructions. When the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices executes the data processing method described in the first aspect, the second aspect, the third aspect, or any possible implementation.

[0057] In this application, the names of the above data processing devices do not constitute a limitation on the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this application and fall within the scope of this application and its equivalent technologies.

[0058] For the specific descriptions of the fourth aspect to the seventh aspect and their various implementations in this application, reference may be made to the detailed descriptions in the first aspect, the second aspect, the third aspect, and their various implementations; and for the beneficial effects of the fourth aspect to the seventh aspect and their various implementations, reference may be made to the beneficial effect analysis in the first aspect, the second aspect, the third aspect, or their various implementations, and will not be elaborated here.

[0059] Based on the implementations provided in the above aspects of this application, further combinations can be made to provide more implementations.

[0060] In the following description, more specific content about the implementations provided in the above aspects is included. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the data processing process provided by the embodiment of this applicationFigure 1 ;

[0062] Figure 2 Schematic diagram of the data processing process provided by the embodiment of the present application Figure 2 ;

[0063] Figure 3 Schematic diagram of the data processing process provided by the embodiment of the present application Figure 3 ;

[0064] Figure 4 Schematic diagram of the data processing process provided by the embodiment of the present application Figure 4 ;

[0065] Figure 5 Schematic diagram of the architecture of the data processing system provided by the embodiment of the present application;

[0066] Figure 6 Schematic diagram of the process of the data processing method provided by the embodiment of the present application Figure 1 ;

[0067] Figure 7 Schematic diagram of the process of the data processing method provided by the embodiment of the present application Figure 2 ;

[0068] Figure 8 Schematic diagram of the structure of the data processing device provided by the embodiment of the present application;

[0069] Figure 9 Schematic diagram of the structure of the computing device provided by the embodiment of the present application;

[0070] Figure 10 Schematic diagram of the structure of the computing device cluster provided by the embodiment of the present application. Detailed implementation manners

[0071] For ease of understanding, the terms involved in the present application are introduced first.

[0072] Input / Output (I / O) refers to the communication between an information processing system (such as a calculator) and the external world (which may be a human or another information processing system). Input is the signal or data received by the system, and output is the signal or data sent from it.

[0073] An I / O request is a request transmitted through the communication between an information processing system and the external world, and the request may include information such as data and tasks. As an example, an I / O request may be a request for indicating a data writing task or a request for indicating a data reading task. The present application mainly describes the I / O request for indicating a data writing task.

[0074] A distributed system refers to a system composed of a cluster of devices that communicate through a network and coordinate to complete a common task. As an example, a distributed system may include multiple computing devices, or a distributed system may include control devices, computing devices, storage devices, etc.

[0075] In some embodiments, a computing device may include multiple computing units, and each computing unit can perform operations required to complete a task.

[0076] A node in this application refers to a device. For example, a computing node refers to a computing device, a control node refers to a control device, and a storage node refers to a storage device.

[0077] An index is a decentralized data structure used to store a collection of data directory entries.

[0078] An inverted index is a data structure used for full-text search, which is used to store the attribute values and the addresses of each record having the attribute value. Exemplarily, the inverted index may indicate a mapping of the storage locations of words, and the mapping of the storage locations includes a list of documents containing the word and the position of the word in a document. Through the inverted index, information such as the document list where the word is located, the position of the word in the document, and the word frequency can be quickly located.

[0079] A transaction log is a file used to save the execution records of operations. As an example, the transaction log may include records of various operations such as writes, deletes, and updates.

[0080] Figure 1 Schematic diagram of the data processing process provided for the embodiments of this application Figure 1 , which shows the data processing process of an existing distributed system. As Figure 1 shown, the distributed system 110 includes a control node and multiple computing nodes, and each computing node includes multiple computing units. Among them, the control node is used to manage the states and data information of the multiple computing nodes. Exemplarily, the distributed system 110 may be an ElasticSearch system.

[0081] The client can send an I / O request to the distributed system 110. The I / O request is used to indicate a data writing task, which includes the data to be written.

[0082] After receiving the I / O request, the computing unit a1 of the computing node in the distributed system performs the following operations: processing the request to obtain a processing result, and generating a transaction log of the operation, saving the processing result in local memory, and also saving the transaction log in local memory. The processing result may include the result obtained by calculating the inverted index or other analysis calculations on the data to be written. Then, the computing unit a1 forwards the I / O request to other backup computing nodes.

[0083] In this application, it is assumed that the first computing node receives an I / O request sent by a client, and the second computing node serves as a backup computing node.

[0084] After the second computing node receives the I / O request, it also performs the same operations on the I / O request as computing unit a1. Specifically, computing unit a2 of the second computing node and computing unit a3 of the second computing node both process the request to obtain a processing result, save the processing result in local memory, generate a transaction log of the operation, and also save the transaction log in local memory.

[0085] After the second computing node completes the foregoing operations, it feeds back a completion response to the first computing node. Subsequently, computing unit a1, computing unit a2, and computing unit a3 will write the data in memory to the disks of their respective computing nodes for storage. It can be understood that the disk of the second computing node stores backup data of the data stored on the disk of the first computing node. In some embodiments, the first computing node will also feed back a successful write to the client.

[0086] Figure 2 Schematic diagram of the data processing process provided by an embodiment of this application Figure 2 , which shows the data processing process of another existing distributed system. As Figure 2 shown, the distributed system includes multiple computing nodes 220.

[0087] In this data processing process, the client can send an I / O request to the distributed system, and the computing node among the multiple computing nodes 220 of the distributed system receives the I / O request and performs the processing of the I / O request. The process of these multiple computing nodes 220 processing the I / O request is the same as that of the multiple computing nodes in Figure 1 . Finally, the computing node among the multiple computing nodes 220 writes the data stored on its local disk to another storage node for transfer. The query node can read the data from the storage node. [Prior art embodiment, corresponding to Figure 3

[0088] Figure 3 Schematic diagram of the data processing process provided by an embodiment of this application Figure 3 , which shows the data processing process of another existing distributed system. As Figure 3 shown, the same distributed system 110 as in Figure 1 receives the I / O request and performs the processing of the I / O request.

[0089] In this data processing process, Figure 3 the process of the distributed system 110 in Figure 1 processing the I / O request is the same as that of the distributed system 110 in Figure 1In contrast, finally, the first computing node writes the data stored in its local disk to another storage node for storage, and then the first computing node sends a notice to the second computing node to instruct the second computing node to delete the backup data stored on the disk this time.

[0090] Figure 4 Schematic diagram of the data processing process provided by the embodiments of the present application Figure 4 , which shows the data processing process of another existing distributed system. As Figure 4 shown, two distributed systems identical to Figure 1 are used, namely the first distributed system 110-1 and the second distributed system 110-2, to jointly complete the processing of I / O requests.

[0091] In this data processing process, the first distributed system 110-1 receives the I / O request sent by the client and forwards the processing task to the second distributed system 110-2 for processing. The data processing process of the second distributed system 110-2 is the same as that of the distributed system 110 in Figure 1 . Then, the second distributed system 110-2 sends the processing result to the first distributed system 110-1 and stores the processing result on the computing node of the first distributed system 110-1. Finally, the computing node of the first distributed system 110-1 writes the stored data to another storage node for transfer.

[0092] It can be seen that in the data processing process of the above existing distributed system: Figure 1 , in the process shown, multiple computing nodes need to perform processing operations on an I / O request, and multiple computing nodes store the data after processing the request. Although the data security is guaranteed, it results in more consumption of computing resources and storage resources; Figure 2 in the process shown, although compared with Figure 1 , finally the data stored on multiple computing nodes is transferred to the remote storage, releasing the storage resources of the computing nodes, but the consumption of computing resources and storage resources of multiple computing nodes in the data processing process has not decreased; Figure 3 the process shown in Figure 2 has a similar effect to the process shown in Figure 4 the process shown in

[0093] To solve the problem that multiple computing nodes process an I / O request and store the results of the processed request on multiple computing nodes, consuming a large amount of computing resources and storage resources, this application provides a data processing method and system. Only the first computing node processes the I / O request, and only stores the processing result obtained by processing the request on the first computing node. The second computing node as a backup generates and saves the transaction log of the I / O request, but does not need to process the first request or store the processing result obtained by processing the request. Finally, the first computing node writes the processing result to another storage node for storage, achieving the security of data while reducing the consumption of computing resources and storage resources compared with the data processing process of existing distributed systems.

[0094] Figure 5 This is a schematic diagram of the architecture of the data processing system provided by the embodiments of this application. As Figure 5 shown, the data processing system 500 includes multiple computing nodes. Exemplarily, the multiple computing nodes include a first computing node 510-1 and a second computing node 510-2. The computing node includes a computing unit and a storage medium. Among them, the computing unit can be a processor such as a central processing unit (CPU), a microprocessor (MP), or a partial module of a processor, and can execute the operations required by the computing node. The storage medium can include one or more of various storage media such as the storage space of the CPU, memory, etc. Exemplarily, the storage medium can include memory and cache.

[0095] The data processing system 500 is connected to the client through a network for communication, and the client can send an I / O request to the data processing system 500.

[0096] The data processing system 500 is also connected to another storage node 520 through a network. The computing nodes in the data processing system 500 can write data to the storage node 520 for storage or read data from the storage node 520.

[0097] In some embodiments, the storage node 520 can be a storage device with a data security guarantee mechanism, and the storage device can be implemented through hardware or virtual devices, such as physical servers, virtual cloud servers, etc. In some embodiments, the storage node 520 can be a shared storage device that can be connected to multiple nodes through a network.

[0098] The data security guarantee mechanism can refer to the operating mode for guaranteeing data security. As an example, it can include data encryption, data verification, etc.

[0099] The embodiments of the data processing method provided by the present application will be described in detail below in conjunction with the accompanying drawings.

[0100] Figure 6 Flow diagram of the data processing method provided by the embodiments of the present application Figure 1 . Here, taking Figure 5 the architecture of the data processing system shown as an example, the data processing method will be described. As Figure 6 shown, the method includes the following steps.

[0101] Step 610: The first computing node obtains a first request.

[0102] When the client needs to instruct the data processing system 500 to process a task, it can send a first request to the data processing system 500.

[0103] The first request may be an I / O request. Exemplarily, the first request may be a request for instructing a data writing task, which contains the data to be written.

[0104] In some embodiments, after the data indicated by the data writing task is stored by the data processing system 500, the query node can read the data from the data storage location for data retrieval.

[0105] In some embodiments, the first request may further contain an index of the data to be written. Thus, the computing node can subsequently calculate the inverted index of the data based on the data and the index of the data, and store the inverted index of the data. When the query node performs data retrieval, it can quickly locate the data based on the inverted index of the data. It should be noted that the index of the data included in the first request may be other indexes different from the inverted index, such as a forward index.

[0106] In addition to calculating the inverted index, the computing node can also perform various analysis calculations on the data (such as calculating the word frequency of each word, text theme, etc.) based on the data, or based on the data and the index of the data, and store the analysis results. When the query node performs data retrieval, it can simultaneously query and obtain the analysis results of the data for a more in-depth understanding of the data information, or for using the analysis results in other data processing tasks.

[0107] Step 620: The first computing node processes the first request and saves the processing result of the first request.

[0108] The processing performed by the first computing node on the first request may be to process the data to be written according to a preset processing method, and the preset processing method may be various required calculations. Exemplarily, the preset processing method may be to calculate the inverted index of the data, or may be analysis calculations such as calculating the word frequency of each word of the data.

[0109] In some embodiments, when the first request includes the data to be written and the index of the data, and the preset processing method is to calculate the inverted index of the data, the first computing node can, based on the data and the index of the data, find the storage locations of each attribute value in the data, and can also obtain the word frequency information according to the word frequency calculation method, and then can obtain the inverted index of the data. Among them, the storage locations of each attribute value may include the document list containing the word and the position of the word in each document. The word frequency calculation method may include any one or more of methods such as dictionary-based word frequency calculation and MapReduce-based word frequency calculation. It can be understood that at this time, the processing result of the first request includes the inverted index of the data.

[0110] In some embodiments, when the first request includes the data to be written, and the preset processing method is to perform analysis calculations such as calculating the word frequency of each word in the data, the first computing node can calculate the data according to the corresponding analysis calculation method such as the word frequency calculation method, etc., to obtain the analysis result of the data. It can be understood that at this time, the processing result of the first request includes the analysis result of the data.

[0111] The first computing node can also generate a transaction log of the first request according to the received first request, and save the transaction log of the first request. The transaction log of the first request can record various data or information included in the first request.

[0112] In some embodiments, when the first request includes the data to be written and the index of the data, the transaction log generated by the first computing node may include the data to be written in the first request and the index of the data.

[0113] In some embodiments, the first computing node saves the processing result of the first request and the transaction log of the first request in different storage media. Specifically, the first computing node can save the processing result of the first request in the local memory, and save the transaction log of the first request in other storage media outside the local memory of the first computing node, such as the cache disk of the first computing node.

[0114] Step 630: The first computing node forwards the first request to the second computing node.

[0115] In some embodiments, there may be a second computing node as a backup, and the first computing node forwards the first request to the backup second computing node.

[0116] In some embodiments, there may be multiple second computing nodes as backups. At this time, the first computing node can forward the first request to each backup second computing node respectively.

[0117] Step 640: The second computing node generates a transaction log of the first request, and saves the transaction log of the first request.

[0118] The second computing node may generate a transaction log of the first request according to the received first request. The method by which the second computing node generates the transaction log of the first request may be the same as that of the aforementioned first computing node for generating the transaction log of the first request.

[0119] The second computing node may save the generated transaction log of the first request in a local storage medium.

[0120] In some embodiments, the second computing node saves the transaction log of the first request in a storage medium other than the local memory of the second computing node, such as the cache disk of the second computing node.

[0121] After the second computing node completes the generation and saving of the transaction log of the first request, it sends a write success response to the first computing node.

[0122] Step 650: The first computing node receives the write success response fed back by the second computing node.

[0123] The write success response may be in any message form that can be used to indicate that the second computing node has completed the generation and saving of the transaction log of the first request.

[0124] After the first computing node receives the write success response fed back by the second computing node, it can continue with subsequent operations.

[0125] Step 660: The first computing node writes the processing result of the first request to the storage node.

[0126] According to the foregoing embodiments, when the first request includes the data to be written and the index of the data, and the processing result of the first request obtained by the first computing node includes the inverted index of the data, the first computing node will write both the data and the inverted index of the data to the storage node.

[0127] In some embodiments, the first computing node may periodically write the processing result of the first request saved locally to the storage node at a preset time period, and the preset time period can be set according to requirements, such as 5s, 10s. Alternatively, the first computing node may write the processing result of the first request saved locally to the storage node in real time. This application does not limit the manner in which the first computing node writes data to the storage node.

[0128] In some embodiments, when the first computing node writes data to the storage node, it may obtain the storage file serial number when the data is written to the storage node. The storage file serial number is used to indicate the file location where the data is stored and can be represented in various data forms such as character-type data and integer-type data. As an example, if the storage file serial number when the data is written to the storage node is 20, it means that the data is stored in the 20th file on the storage node.

[0129] In some embodiments, the storage file sequence number may be determined by the first computing node. For example, the first computing node may determine the storage file sequence number according to a preset rule. The preset rule may include determining the storage file sequence number of the current write operation as increasing one or more file sequence numbers in sequence based on the storage file sequence number N of the previous write operation. The preset rule may also include, at the first write operation, determining the storage file sequence number of the current write operation as an initial value such as 0, etc.

[0130] In some embodiments, after writing the processing result of the first request to the storage node for storage, the first computing node may also delete the transaction log of the first request saved by itself. It can be understood that after the processing result is written to the storage node, the data security can be ensured by the storage node. At this time, the first computing node can delete the transaction log of the first request to release more storage space.

[0131] In some embodiments, the transaction log of the first computing node may include multiple transaction logs, including records of each operation executed by the first computing node. The multiple transaction logs may be recorded in chronological order or in the order of operation execution. When the first computing node writes the processing result to the storage node and obtains the storage file sequence number, it can obtain the record of the storage file sequence number. Since the transaction log of the first request is generated before the record of the storage file sequence number, the first computing node may delete the transaction logs generated before the record of the storage file sequence number according to the storage file sequence number, thereby realizing the deletion of the transaction log of the first request.

[0132] In some embodiments, the first computing node may immediately delete the transaction log of the first request after writing the processing result to the storage node and obtaining the storage file sequence number. Alternatively, the first computing node may also regularly delete the corresponding transaction log of the first request according to the storage file sequence number at a certain time period.

[0133] In some embodiments, the first computing node also sends the storage file sequence number of the processing result of the first request to the second computing node.

[0134] In some embodiments, the second computing node also deletes the transaction log of the first request saved by itself according to the received storage file sequence number. The method for the second computing node to delete the transaction log of the first request saved by itself is similar to that of the first computing node.

[0135] Figure 7 Schematic flow of the data processing method provided by the embodiments of the present application Figure 2 Here, continue with Figure 5 the architecture of the data processing system shown as an example to illustrate the data processing method.

[0136] The first computing node may be disconnected from other second computing nodes in the system due to network failures or equipment failures. In such a case, the first computing node cannot communicate with other second computing nodes, and the first computing node cannot continue to process new I / O requests. It will automatically shut down after a waiting period, which can be referred to as the automatic shutdown waiting time.

[0137] In some embodiments, within the automatic shutdown waiting time of the first computing node, if there is still data in the memory of the first computing node, it will continue to be written to the storage node for storage. And the storage file sequence number when the first computing node writes to the storage node is N + 1, where N is the storage file sequence number N corresponding to the previous write process of the first computing node.

[0138] As Figure 7 shown, in this method, when the first computing node disconnects from the second computing node, the function of the first computing node is taken over by one of the second computing nodes in the system. Other second computing nodes that do not take over the function of the first computing node act as the third computing nodes 510-3, and the third computing nodes can be used as backup nodes for the second computing node that takes over the function of the first computing node. This method includes the following steps.

[0139] Step 710: The second computing node obtains a second request.

[0140] Step 720: The second computing node processes the second request and saves the processing result of the second request.

[0141] Step 730: The second computing node forwards the second request to the third computing node.

[0142] Step 740: The second computing node receives the write success response feedback from the third computing node.

[0143] Step 750: The second computing node writes the processing result of the second request to the storage node.

[0144] In the above steps, the second request may be an I / O request sent by the client, which is similar to the first request in step 610 and will not be elaborated here.

[0145] And, in the above steps, the operations performed by the second computing node are similar to those performed by the Figure 6 first computing node, and the operations performed by the third computing node are similar to those performed by the Figure 6 second computing node. Reference can be made to the above steps 610 - step 660 and their related descriptions.

[0146] Compared with Figure 6Different from the operations performed by the method shown, in step 750, when the second computing node writes the processing result of the second request to the storage node, the storage file sequence number is incremented by M based on N, where M is greater than or equal to 2. Thus, the second computing node and the first computing node can write their respective data to different files in the storage node.

[0147] During the automatic shutdown waiting time of the first computing node, the first computing node can write data to the storage node multiple times according to a data writing period (such as 5 s), and each write can include multiple concurrent write operations. Thus, the first computing node will write data to multiple files in the storage node during the automatic shutdown waiting time.

[0148] In some embodiments, M may be related to the number of concurrent data writing operations of the first computing node and the first time ratio; the first time ratio is determined based on the automatic shutdown waiting time of the first computing unit and the data writing period of the first computing unit. Exemplarily, M can be determined according to the following method: M = c * (t1 / t2), where c represents the number of concurrent data writing operations of the first computing node, t1 represents the automatic shutdown waiting time of the first computing node, and t2 represents the data writing period of the first computing node.

[0149] By making M related to the number of concurrent data writing operations of the first computing node and the above-mentioned first time ratio, it can be achieved that when the second computing node writes data to the storage node, it can more accurately avoid multiple files to which the first computing node has written data.

[0150] In some embodiments, the second computing node also deletes files in the storage node whose storage file sequence numbers are greater than N and less than N + M. Thus, unnecessary files stored in the storage node can be cleared, saving the storage resources of the storage node.

[0151] As an example, according to Figure 7 the method described above, the first computing node disconnects the network connection with the second computing node at time T1. At this time, the second computing node takes over the function of the first computing node, receives the second request and performs the above operations at time T2, and the second computing node completes writing the processing result of the second request to the storage node at time T3. The storage file sequence number when the first computing node writes data to the storage node once before T1 is 20. Between T1 and T3, the first computing node also continues to write data to the 21st file in the storage node. If M determined according to the foregoing method is 2, then the second computing node writes the processing result of the second request to the 22nd file in the storage node.

[0152] This application also provides a data processing device, such as Figure 8As shown, the data processing device 800 includes a communication module 810 and a processing module 820. The communication module 810 and the processing module 820 can be used to execute Figure 6 and Figure 7 the method steps in.

[0153] The communication module 810 is used to obtain a first request, and the first request includes data to be written. For example, the communication module 810 is used to execute Figure 6 step 610 in.

[0154] The processing module 820 is used to process the first request and save the processing result of the first request. For example, the processing module 820 is used to execute Figure 6 step 620 in.

[0155] The communication module 810 is further used to forward the first request to a second computing node. For example, the communication module 810 is further used to execute Figure 6 step 630 in.

[0156] The processing module 820 is further used to generate a transaction log of the first request and save the transaction log of the first request. For example, the processing module 820 is further used to execute Figure 6 step 640 in.

[0157] The communication module 810 is further used to receive a write success response fed back by the second computing node. For example, the communication module 810 is further used to execute Figure 6 step 650 in.

[0158] The processing module 820 is further used to write the processing result of the first request into a storage node. For example, the processing module 820 is further used to execute Figure 6 step 660 in.

[0159] Optionally, the processing module 820 is further used to generate a transaction log of the first request and save the transaction log of the first request.

[0160] Optionally, the processing module 820 is further used to save the processing result of the first request in the local memory of the first computing node.

[0161] Optionally, the processing module 820 is further used to save the transaction log of the first request in a storage medium outside the local memory of the first computing node.

[0162] Optionally, the processing module 820 is further used to delete the transaction log of the first request saved by the first computing node according to the storage file serial number when the processing result of the first request is written into the storage node.

[0163] Optionally, the processing module 820 is further used to send the storage file serial number to the second computing node.

[0164] Optionally, the processing module 820 is further configured to delete the transaction log of the first request saved by the second computing node according to the stored file sequence number.

[0165] Optionally, the communication module 810 is further configured to receive a second request, where the second request includes data to be written. The processing module 820 is further configured to process the second request and save the processing result of the second request. The communication module 810 is further configured to forward the second request to a third computing node. The communication module 810 is further configured to receive a write success response fed back by the third computing node. The processing module 820 is further configured to write the processing result of the second request to the storage node, and the stored file sequence number when writing to the storage node is incremented by M based on N, where N is the stored file sequence number corresponding to the previous write process, and M is greater than or equal to 2.

[0166] Optionally, the processing module 820 is further configured to delete files in the storage node whose stored file sequence numbers are greater than N and less than N + M.

[0167] For more specific content of the method steps implemented by the communication module 810 and the processing module 820, reference can be made to Figure 6 、 Figure 7 and its related descriptions.

[0168] Optionally, the communication module 810 may include multiple sub - modules, and the multiple sub - modules may be separately deployed to respectively implement some functions of the communication module 810.

[0169] In some embodiments, the processing module 820 may include multiple sub - modules, and the multiple sub - modules may be separately deployed to respectively implement some functions of the processing module 820.

[0170] The device can be implemented by software or by hardware. Exemplarily, the implementation manner of the data processing device 800 is introduced next.

[0171] As an example of a software functional unit, the data processing device may include code running on a computing instance. Wherein, the computing instance may be at least one of computing devices such as a physical host (computing device), a virtual machine, a container, etc. Further, the above - mentioned computing devices may be one or more. For example, the data processing device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers for running the code may be distributed in the same AZ or in different AZs, and each AZ includes one data center or multiple geographically - proximate data centers. Usually, one region may include multiple AZs.

[0172] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed within the same VPC or across multiple VPCs. Usually, one VPC is set up within one region. For cross-region communication between two VPCs within the same region and between VPCs in different regions, communication gateways need to be set up within each VPC, and the interconnection between VPCs is achieved through the communication gateways.

[0173] As an example of a hardware functional unit, the data processing device may include at least one computing device, such as a server, etc. Alternatively, the data processing device may also be a device implemented using ASIC or PLD. Among them, the above PLD may be implemented by CPLD, FPGA, GAL, or any combination thereof.

[0174] The multiple computing devices included in the data processing device can be distributed in the same region or in different regions. The multiple computing devices included in the data processing device can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the YY device can be distributed within the same VPC or across multiple VPCs. Among them, the multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0175] This application also provides a computing device 900. As Figure 9 shown, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other through the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0176] The bus 902 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 9 only one line is used to represent it here, but it does not mean that there is only one bus or one type of bus. The bus 902 can include a path for transmitting information between various components (such as the memory 906, the processor 904, and the communication interface 908) of the computing device 900.

[0177] The processor 904 may include any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0178] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0179] The memory 906 stores executable program code, and the processor 904 executes the executable program code to implement the functions of the foregoing communication module 810 and processing module 820 respectively, so as to implement the data processing method provided by the embodiments of the present application. For example Figure 6 、 Figure 7 the provided data processing method. That is, the memory 906 stores instructions for executing the data processing method provided by the embodiments of the present application.

[0180] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0181] The embodiments of the present application further provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0182] As Figure 10 shown, the computing device cluster includes at least one computing device 900. The memory 906 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the data processing method provided by the embodiments of the present application, such as Figure 6 、 Figure 7 the instructions for the provided data processing method.

[0183] In some possible implementations, in the memories 906 of one or more computing devices 900 in the computing device cluster, there may also be stored some instructions for executing the data processing method provided in the embodiments of the present application, such as instructions for executing Figure 6 , Figure 7 the partial steps of the provided data processing method. In other words, the combination of one or more computing devices 900 can jointly execute the instructions for executing the data processing method provided in the embodiments of the present application.

[0184] It should be noted that the memories 906 in different computing devices 900 in the computing device cluster can store different instructions, respectively for executing partial functions of the data processing device. That is, the instructions stored in the memories 906 of different computing devices 900 can implement the functions of one or more modules in the communication module 810 and the processing module 820.

[0185] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. Among them, the network can be a wide area network or a local area network, etc.

[0186] In some possible implementations, in the memories 906 of one or more computing devices 900 in the computing device cluster, there may also be stored some instructions for executing the data processing method provided in the embodiments of the present application, such as instructions for executing Figure 6 , Figure 7 the partial steps of the provided data processing method. In other words, the combination of one or more computing devices 900 can jointly execute the instructions for executing the data processing method provided in the embodiments of the present application.

[0187] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it causes at least one computing device to execute the data processing method provided in the embodiments of the present application, such as instructions for executing Figure 6 , Figure 7 the provided data processing method.

[0188] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc. The computer-readable storage medium includes instructions that direct the computing device to execute the data processing method provided by the embodiments of the present application, for example, for executing Figure 6 , Figure 7 the instructions of the provided data processing method.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

[0190] The terms "first", "second", "third", and "fourth", etc. in the specification, claims, and the above drawings of the present application are used to distinguish different objects and are not used to define a specific order.

[0191] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

Claims

1. A data processing method, characterized in that, The described data processing method is applied to a distributed system, which includes multiple computing nodes. The multiple computing nodes include a first computing node and a second computing node, and the second computing node is a backup node of the first computing node. The method includes: The first computing node obtains a first request, and the first request contains data to be written; The first computing node processes the first request and saves the processing result of the first request; The first computing node forwards the first request to the second computing node; The second computing node generates a transaction log of the first request and saves the transaction log of the first request; The first computing node receives a write success response fed back by the second computing node; The first computing node writes the processing result of the first request into a storage node.

2. The method according to claim 1, wherein The first request contains the data and an index of the data, and the processing result obtained by the first computing node processing the first request contains an inverted index of the data.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The first computing node generates a transaction log of the first request and saves the transaction log of the first request.

4. The method according to claim 3, wherein The method further includes: The first computing node saves the processing result of the first request in the local memory of the first computing node; The first computing node saves the transaction log of the first request in a storage medium outside the local memory of the first computing node.

5. The method according to any one of claims 1 to 4, characterized in that, The transaction log of the first request includes the data and an index of the data.

6. The method according to any one of claims 1-5, characterized in that, After the first computing node writes the processing result of the first request into the storage node, the method further includes: The first computing node deletes the transaction log of the first request saved by the first computing node according to the storage file sequence number when the processing result of the first request is written into the storage node.

7. The method according to claim 6, wherein The method further includes: The first computing node sends the storage file sequence number to the second computing node; The second computing node deletes the transaction log of the first request saved by the second computing node according to the storage file sequence number.

8. The method according to any one of claims 1 to 7, characterized in that, The multiple computing nodes further include a third computing node. When the first computing node disconnects from the second computing node, the method further includes: The second computing node receives a second request, and the second request contains data to be written; The second computing node processes the second request and saves the processing result of the second request; The second computing node forwards the second request to the third computing node; The second computing node receives a write success response fed back by the third computing node; The second computing node writes the processing result of the second request into the storage node, and the storage file sequence number when written into the storage node is increased by M on the basis of N, where N is the storage file sequence number corresponding to the previous write process, and M is greater than or equal to 2.

9. The method according to claim 8, wherein The M is related to the concurrency number of data write operations of the first computing node and the ratio of the first time; The first time ratio is determined based on the automatic shutdown waiting time after the first calculation unit disconnects the connection relationship with the second calculation node and the data write cycle of the first calculation unit.

10. The method according to claim 8, wherein The method further includes: The second calculation node deletes the files in the storage node with file serial numbers greater than N and less than N + M.

11. A data processing method, characterized in that, The data processing method is applied to a distributed system, the distributed system includes a plurality of calculation nodes, the plurality of calculation nodes include a first calculation node and a second calculation node, the second calculation node is a backup node of the first calculation node, and the method includes: Receiving a first request forwarded by the first calculation node; Generating a transaction log of the first request and saving the transaction log of the first request; Feeding back a write success response to the first calculation node, and the write success response is used to instruct the first calculation node to write the processing result of the first request into the storage node.

12. A data processing device, characterized in that, The device includes a communication module and a processing module; The communication module is used to obtain a first request, and the first request includes data to be written; The processing module is used to process the first request and save the processing result of the first request; The communication module is further used to forward the first request to the second calculation node; The processing module is further used to generate a transaction log of the first request and save the transaction log of the first request; The communication module is further used to receive the write success response fed back by the second calculation node; The processing module is further used to write the processing result of the first request into the storage node.

13. A data processing system, characterized in that, The system includes a plurality of calculation nodes, and the plurality of calculation nodes are used to execute the method according to any one of claims 1-11.

14. A computer program product comprising instructions, characterized in that, When the instruction is run by a cluster of computing devices, the cluster of computing devices executes the method according to any one of claims 1-11.

15. A computer-readable storage medium, characterized in that, Including computer program instructions, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices executes the method according to any one of claims 1-11.

Images