Data processing method and device

Abstract

The embodiments of the present specification provide a data processing method and device, wherein the data processing method includes: a scheduling storage node receives a task processing request, and determines the execution status of each execution storage node based on the task processing request; the scheduling storage node determines a target execution storage node based on the execution status of each execution storage node, and forwards the task processing request to the target execution storage node; the target execution storage node determines data location information and a target processing function based on the task processing request, and obtains target business data according to the data location information and the target processing function.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, a data processing system, and a computer program. Background Art

[0002] With the continuous development of computer technology, many storage systems often adopt a computing and storage separation architecture in order to decouple computing and storage. The use of a computing and storage separation architecture can achieve elastic expansion of computing and storage and flexible allocation of data resources.

[0003] However, the computing end in the current computing and storage separation architecture will receive data processing requests for simple data processing. Such operations do not require complex operation calculations, but require obtaining data from the storage end, which consumes bandwidth resources on the computing end. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing device, a computing device, a data processing system, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, which is applied to a data storage cluster, wherein the data storage cluster includes a scheduling storage node and at least one execution storage node, including:

[0006] The scheduling storage node receives the task processing request and determines the execution state of each execution storage node based on the task processing request;

[0007] The scheduling storage node determines a target execution storage node based on the execution status of each execution storage node, and forwards the task processing request to the target execution storage node;

[0008] The target execution storage node determines data location information and a target processing function based on the task processing request, and obtains target business data according to the data location information and the target processing function.

[0009] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:

[0010] A receiving module, configured to receive a task processing request and determine an execution state of each execution storage node based on the task processing request;

[0011] a forwarding module, configured to determine a target execution storage node based on the execution status of each execution storage node, and forward the task processing request to the target execution storage node;

[0012] The determination module is configured to determine data location information and a target processing function based on the task processing request, and obtain target business data according to the data location information and the target processing function.

[0013] According to a third aspect of an embodiment of this specification, a data processing system is provided, wherein the data processing system includes a data storage cluster and a data computing cluster, wherein:

[0014] The data storage cluster is configured to receive a task processing request and determine a target processing function based on the task processing request;

[0015] The data storage cluster is further configured to process the task processing request when the target processing function is a preset processing function;

[0016] The data storage cluster is further configured to forward the task processing request to the data computing cluster when the target processing function is not a preset processing function;

[0017] The data computing cluster is configured to receive a task processing request and process the task processing request.

[0018] According to a fourth aspect of an embodiment of this specification, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method when executing the computer instructions.

[0019] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer instructions, and when the computer instructions are executed by a processor, the steps of the data processing method are implemented.

[0020] According to a sixth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0021] The data processing method provided in this specification schedules a storage node to receive a task processing request, and determines the execution status of each execution storage node based on the task processing request; the scheduling storage node determines a target execution storage node based on the execution status of each execution storage node, and forwards the task processing request to the target execution storage node; the target execution storage node determines data location information and a target processing function based on the task processing request, and obtains target business data according to the data location information and the target processing function.

[0022] An embodiment of the present specification implements scheduling of the execution storage node by the scheduling storage node, and processing of the task processing request by the target execution storage node, that is, improving data processing efficiency by assisting the processing of task processing requests by the scheduling storage node and the execution storage node. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a data processing method provided by an embodiment of this specification;

[0024] Figure 2 is a schematic diagram of a data processing method provided by an embodiment of this specification;

[0025] Figure 3 It is a schematic diagram of selecting a currently executing storage node provided by an embodiment of this specification;

[0026] Figure 4 It is a schematic diagram of selecting a currently scheduled storage node provided by an embodiment of this specification;

[0027] Figure 5 is a processing flow chart of a data processing method provided by an embodiment of this specification;

[0028] Figure 6 is a structural diagram of a data processing system provided by an embodiment of this specification;

[0029] Figure 7 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0030] Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0031] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0032] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to any or all possible combinations of one or more associated listed items.

[0033] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0034] First, the terms involved in one or more embodiments of this specification are explained.

[0035] Metadata: Data that describes data, mainly information that describes data attributes, used to support functions such as indicating storage location, historical data, resource search, and file records.

[0036] Distributed locks: In distributed systems, coordinated actions are often required. If different systems or different hosts in the same system share one or a group of resources, then when accessing these resources, mutual exclusion is often required to prevent interference with each other and ensure consistency. At this time, distributed locks are needed.

[0037] Execution storage node: A role that performs data processing tasks in a distributed storage system, usually deployed on a data node.

[0038] Scheduling storage node: A node in a distributed storage system that integrates resources, collects data processing requests, and schedules resources.

[0039] Distributed storage system: A distributed storage system writes data to multiple data nodes simultaneously. It is necessary to ensure that the status presented to the outside is consistent and that no status rollback occurs.

[0040] Metadata storage node: In a distributed system, a centralized metadata storage node is usually used to store file status information, location information, length information and other attribute information.

[0041] Client: A component in a distributed system that is provided to users to perform metadata operations and read and write data. It also refers to the component on the computing side that sends data processing requests.

[0042] At present, distributed storage systems are widely used in computing and storage separation architectures. This architecture decouples computing and storage so that they can use and scale resources independently without interference, and also prevents mutual influence and amplification problems from occurring in abnormal scenarios.

[0043] On the other hand, the computing side in the computing and storage separation architecture has the need to perform simple data processing, such as garbage collection by copying useful data to new files, performing simple data operations on a portion of the stored data and then rewriting it, etc.; these operations do not require complex computing operations, but require reading and writing a large amount of data from the storage side, so they will consume bandwidth resources on the computing side.

[0044] The solution of this specification introduces scheduling storage nodes and execution storage nodes on the storage side to assist in data processing, thereby sinking simple data processing to the storage side and avoiding bandwidth resource consumption on the computing side caused by simple data processing.

[0045] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, a computer-readable storage medium, a data processing system and a computer program, which are described in detail one by one in the following embodiments.

[0046] Figure 1 A flow chart of a data processing method applied to a data storage cluster provided according to an embodiment of the present specification is shown, wherein the data storage cluster includes a scheduling storage node and at least one execution storage node, including steps 102 to 106.

[0047] Step 102: The scheduling storage node receives the task processing request, and determines the execution status of each execution storage node based on the task processing request.

[0048] In order to avoid the problem of simple data processing occupying the bandwidth resources of computing nodes, the solution of this specification adds an execution process to an existing storage node to obtain an execution storage node, adds a scheduling process to the storage node to obtain a scheduling storage node; or creates a new storage node, adds an execution process or a scheduling process, so that the execution storage node can execute data processing tasks that can only be executed by computing nodes. In practical applications, since the execution storage node may fail and affect the normal execution of the task, multiple execution storage nodes can be set, that is, the execution storage nodes can be deployed in a distributed form in a data storage cluster so that different execution storage nodes can execute different task processing requests at the same time; the data storage cluster is composed of an execution storage node and at least one scheduling storage node.

[0049] Among them, an execution storage node refers to a storage node that can execute data processing tasks; a scheduling storage node refers to a storage node that can schedule data processing tasks to an execution storage node based on the execution status of each execution storage node; a task processing request refers to a request for processing data, that is, a task processing request sent by a device with data processing requirements, such as a task processing request sent by a client; it should be noted that the task processing requests that can be executed by the execution storage node in this specification are relatively simple task processing requests; the execution status of the execution storage node refers to the status information of the task processing status of the execution storage node, such as the load of the execution storage node, the current execution status, etc.

[0050] For example, the data storage cluster includes a scheduling storage node A, an execution storage node b, and an execution storage node c; the scheduling storage node A receives a task processing request Q, and determines the execution status of the execution storage node b and the execution status of the execution storage node c based on the task processing request Q.

[0051] Since the solution of this specification is to execute storage nodes to process simple data processing tasks, and for more complex data processing tasks, computing nodes are still required to execute them, after the scheduling storage node receives the task processing request, it is necessary to first determine whether the execution storage node can execute the received task processing request.

[0052] Specifically, after the scheduling storage node receives the task processing request, it needs to parse the task processing request, and determine whether the target processing function included in the task processing request is a preset processing function, that is, the preset execution storage node can execute the processing function, that is, the scheduling storage node receives the task processing request, and determines the execution status of each execution storage node based on the task processing request. The specific method may include:

[0053] Determining a target processing function based on the task processing request;

[0054] In the case where it is determined that the target processing function is a preset processing function, an execution state of each execution storage node in the data storage cluster is determined based on the task processing request.

[0055] Among them, the target processing function refers to the processing function parsed in the task processing request, such as the processing function for calculating the sum of the business data to be processed. It should be noted that the target processing function can be empty, that is, if the target processing function does not exist, then the task processing request can be determined as a task processing request for copying data; the preset processing function refers to a processing function that can be executed by the execution storage node.

[0056] In actual applications, in order to determine whether the task processing request can be executed by the execution storage node, a processing function table can be set in advance. If there is a preset processing function consistent with the target processing function in the processing function table, or the target processing function is empty, it can be determined that the execution storage node can execute the task processing request, and the execution status of each execution storage node in the data storage cluster can be further determined according to the task processing request.

[0057] For example, the computing node G sends a task processing request to the data storage cluster; the scheduling storage node H receives the task processing request, and parses the task processing request to obtain the processing function N; determines a preset processing function table, which includes processing function M, processing function N and processing function J; when it is determined that the processing function table contains processing function N, it is determined that the task processing request can be processed by the execution storage node, and then determines the execution status of each execution storage node in the data storage cluster based on the task processing request.

[0058] After determining that the execution storage node can execute the task processing request, it is possible to determine which execution storage node in the data storage cluster executes the task processing request by determining the execution status of the execution storage node.

[0059] Specifically, the method for the scheduling storage node to determine the execution status of each execution storage node based on the task processing request may include:

[0060] The scheduling storage node queries the current task load and node status information of each execution storage node in the data storage cluster based on the task processing request;

[0061] or

[0062] The scheduling storage node receives the current task load and node status information reported by each execution storage node in the data storage cluster.

[0063] Among them, the current task load refers to the number of tasks currently processed by the scheduling storage node, such as the execution storage node is currently executing 3 data processing tasks, that is, the load is 3; based on the current task load, the execution storage node with a smaller current load can be determined, so as to facilitate the subsequent determination of the execution storage node used to execute the task processing request; the node status information refers to the state of the execution storage node, such as a faulty state or a non-faulty state, etc. Based on the node status information, the faulty execution storage node can be determined to avoid scheduling the faulty node to execute the task processing request, thereby facilitating the subsequent scheduling of the execution storage node that can be executed normally.

[0064] Specifically, the scheduling storage node can detect the current load and node status information of each execution storage node in the data storage cluster based on the task processing request; and, the scheduling storage node can also receive the current load and node status information reported by each execution storage node at preset time intervals; any one of the above two methods, or a combination of the above two methods, can be used to obtain the current task load and node status information of each execution storage node.

[0065] By determining the execution status of each execution storage node in the data storage cluster, that is, determining the current task load and node status information of each execution storage node, subsequent scheduling storage nodes can screen execution storage nodes that can execute task processing requests in the data storage system based on the current task load and node status information.

[0066] Step 104: The scheduling storage node determines a target execution storage node based on the execution status of each execution storage node, and forwards the task processing request to the target execution storage node.

[0067] Among them, the target execution storage node refers to the storage node in the data storage system that can execute the task processing request; after determining the target execution storage node, the scheduling storage node can forward the received task processing request to the target execution storage node, so that the subsequent target execution storage node can execute the task processing request.

[0068] Specifically, determine the execution status of each execution storage node, that is, determine the current load and node status information of each execution storage node; based on the execution status, determine the target execution storage node that can process the task processing request in the data storage cluster; for example, determine that the execution storage node N and the execution storage node M in the data storage cluster are in a non-fault state according to the node status information of the execution storage node, and further determine the current load of the execution storage node N and the execution storage node M. Since the current load of the execution storage node N is less than the current load of the execution storage node M, the execution storage node N can be used as the target execution storage node; after determining the target execution storage node, send the task processing request to the target execution storage node.

[0069] For example, determine the current load and node status information of each execution storage node in the data storage cluster; determine that each execution storage node in the data storage cluster is in a non-faulty state based on the node status information, then determine the execution storage node with the lowest current load as the target execution storage node, and forward the task processing request received by the scheduling storage node to the target execution storage node.

[0070] By determining the target execution storage node in the data storage cluster based on the execution status of each execution storage node, the target execution storage node with better execution status can subsequently execute the task processing request, thereby improving the task processing efficiency and task processing quality.

[0071] Step 106: The target execution storage node determines data location information and a target processing function based on the task processing request, and obtains target business data according to the data location information and the target processing function.

[0072] After the target execution storage node receives the task processing request, it can determine the data location information and the target processing function according to the task processing request, and process the business data to be processed according to the data location information and the target processing function to obtain the target business data.

[0073] Among them, data location information refers to the location information of the business data to be processed; the target processing function refers to the code or program contained in the task processing request that can be used to calculate the target business data; the target business data refers to the data obtained by processing the business data to be processed based on the target processing function.

[0074] Specifically, the data storage cluster may include a metadata storage node, and the data location information of the business data to be processed may be determined based on the metadata; after determining the data location information, the business data to be processed may be obtained in the storage node storing the business data to be processed based on the data location information, for example, the data location information is determined to be "data with a starting offset of 1 and an offset length of 5 in file D", and then the business data to be processed "1, 2, 3, 4, 5" is obtained in file D based on the data location information; after obtaining the business data to be processed, the target business data may be calculated based on the target processing function.

[0075] In practical applications, the specific method for the target execution storage node to determine the data location information and the target processing function based on the task processing request may include:

[0076] The target execution storage node parses the task processing request;

[0077] The data location information and the target processing function in the task processing request are obtained.

[0078] Specifically, after receiving the task processing request, the target execution storage node parses the task processing request; by parsing the task processing request, the data location information and the target processing function carried in the task processing request are obtained.

[0079] For example, after receiving the task processing request Q, the execution storage node H parses the task processing request and determines the data location information J and the target processing function K carried in the task processing request.

[0080] By parsing the task processing request, the data location information of the business data to be processed and the target processing function, that is, the method used to process the business processing data to be processed, can be determined.

[0081] After acquiring the data location information and the target processing function, the target service data may be obtained based on the data location information and the target processing function. Specifically, a method for obtaining the target service data based on the data location information and the target processing function may include:

[0082] Acquire the business data to be processed according to the data location information;

[0083] The to-be-processed business data is processed based on the target processing function to obtain target business data.

[0084] The business data to be processed refers to the business data that needs to be processed based on the task processing request; for example, based on the data location information "data with a starting offset of 5 and an offset length of 4 in file G", the business data to be processed is "5, 6, 7, 8, 9" in file G, and the business data to be processed is processed according to the target processing function "increase each element by 1", and the target business data is "6, 7, 8, 9, 10". In actual applications, the task processing request received by the scheduling storage node can be an array <input file, initial offset + offset length, processing function, output file name>, wherein the input file refers to the file where the business data to be processed is located, and the output file name refers to the file name when the file storing the target business data is output.

[0085] Specifically, the data processed based on the target processing function can be stored in the output file, specifically: when writing the processed target business data to the file, the target business data is first written to a temporary file, and the temporary file is set to a read-only attribute. After the data processing is completed, the temporary file is renamed, and the file attribute is set to the default value to obtain an output file containing the complete target business data, so that the files seen by the user are all files containing the complete target business data. After detecting that the task processing is completed, the scheduling storage node marks the task processing request as completed.

[0086] For example, Figure 2 As shown, Figure 2 It is a schematic diagram of a data processing method provided by an embodiment of the present specification, wherein the scheduling storage node receives a task processing request sent by a client; and determines the execution status of the execution storage node 1, the execution storage node 2, and the execution storage node 3 based on the task processing request, selects the execution storage node 2 as the target execution storage node based on the execution status, and forwards the task processing request to the execution storage node 2. The execution storage node 2 determines the data location information and the target processing function based on the task processing request, determines the source file and the business data to be processed in the source file according to the data location information; processes the business data to be processed based on the target processing function to obtain the target business data; writes the business data to be processed into a temporary file; after all the business data to be processed are processed and the target business data is written into the temporary file, the temporary file is renamed to obtain the target file containing the complete target business data; after the execution storage node task is executed, the execution completion result will be fed back to the scheduling storage node, and the scheduling storage node will feed back the execution completion result to the client, in addition, the scheduling storage node can also detect the task execution status, and the client can detect the task execution status recorded in the scheduling storage node.

[0087] In actual applications, the target execution storage node may fail. In this case, the scheduling storage node needs to schedule other execution storage nodes to execute the task processing request.

[0088] Specifically, the data processing method further includes:

[0089] In the case where the scheduling storage node detects that the target execution storage node fails, determining a current execution storage node in the data storage cluster;

[0090] Sending a task processing request including the retry binary to the current execution storage node, wherein the task processing request includes the retry binary, and the retry binary consists of an execution identifier and a retry value.

[0091] Among them, the current execution storage node refers to the current execution storage node that can currently process the task processing request; the task processing request refers to the task processing request containing a retry tuple, wherein the retry tuple refers to a tuple containing an execution identifier and a retry value; the execution identifier refers to an identifier set for the task processing request, such as a character string that is randomly generated by the client that sends the task processing request and can uniquely identify the task processing request; the retry value is determined based on the number of client retries and the number of scheduling storage node retries; the number of scheduling storage node retries refers to the number of times the scheduling storage node re-obtains the execution status of the target execution storage node, such as, when the scheduling storage node detects that the execution storage node H has failed, it can re-obtain the execution result of the execution storage node H after a preset time, and at this time the number of scheduling storage node retries is increased by one; the number of client retries refers to the number of times the client retries to obtain the execution status of the scheduling storage node.

[0092] The client can generate a task processing request containing a retry tuple and send it to the scheduling storage node. If the client retries when determining the scheduling storage node, the retry tuple in the task processing request sent to the determined final scheduling storage node is a retry tuple with the client's retry number increased by the retry value; the scheduling storage node will send a task processing request containing a retry tuple to the determined target execution storage node. If the scheduling storage node retries when determining the target execution storage node, the retry tuple in the task processing request sent to the target execution storage node is a retry tuple with the scheduling storage node's retry number increased by the retry value.

[0093] Specifically, if the scheduling storage node detects that the target execution storage node has a failure, or the target execution storage node reports its heartbeat data of the failure to the scheduling storage node, the scheduling storage node will re-obtain the execution status of the target execution storage node once or multiple times. If it is determined again that the target execution storage node has a storage failure, the scheduling storage node will again obtain the execution status of each execution storage node in the data storage cluster, and determine the current execution storage node based on each execution status; and forward the task processing request containing the retry tuple to the current execution storage node.

[0094] In practical applications, due to various reasons such as network delay, a short-term failure of the target execution storage node may occur. At this time, the execution state of the target execution storage node may be re-determined, that is, before determining the current execution storage node in the data storage cluster, the following may also be included:

[0095] The scheduling storage node reacquires the execution status of the target execution storage node;

[0096] A retry value in the retry tuple is increased based on the number of times the execution state is re-obtained.

[0097] Specifically, when the scheduling storage node determines for the first time that the target execution storage node is in a faulty state, it can obtain the execution status of the target execution storage node again after a preset period of time; if the execution status is restored to a non-faulty state, the target execution storage node can continue to execute the task processing request; if the execution status is still in a faulty state, it can obtain the execution status of the target execution storage node again after a preset period of time until the upper limit of the number of acquisitions is reached; and in order to improve the success rate of re-acquiring the execution status, the preset period of time can also be increased with the increase in the number of acquisitions, that is, the time interval for re-acquiring the execution status each time increases; each time the execution status of the target execution storage node is re-acquired, it is necessary to increase the retry value in the retry binary tuple, that is, the retry value of the retry binary tuple can be increased based on the number of retries of the scheduling storage node.

[0098] If the number of retries reaches a preset number threshold, the re-acquisition of the execution status of the target execution storage node can be stopped, and the current execution storage node is determined in the data storage cluster to execute the task processing request.

[0099] After determining the current execution storage node, the current execution storage node receives the task processing request including the two-tuple, and executes the task processing request. Specifically, after sending the task processing request including the retry two-tuple to the current execution storage node, the method may further include:

[0100] The currently executing storage node determines a file to be written and a retry tuple based on the task processing request;

[0101] The file to be written is opened based on the retry tuple.

[0102] The file to be written refers to a file to which target business data needs to be written; and the retry tuple refers to a tuple included in a task processing request sent by a scheduling storage node to a currently executing storage node.

[0103] Specifically, after the current executing storage node determines the retry tuple, the file to be written is opened based on the retry tuple, that is, the data processing task of writing the target business data into the file to be written is executed.

[0104] Furthermore, since the target execution storage node may be unavailable for a short period of time, that is, after the current execution storage node is determined, the target execution storage node recovers to a non-faulty state and continues to execute the task processing request, resulting in a write conflict when writing the file to be written; in order to avoid the above situation, the task processing request in this specification includes a retry binary tuple. When the execution identifier of the retry binary tuple currently writing the data to be written is determined to be consistent with the execution identifier of the retry binary tuple corresponding to the current execution storage node, the execution storage node with a larger retry value can be determined as the execution storage node that can currently write the target business data by comparing the retry values.

[0105] Specifically, the specific method of opening the file to be written based on the retry binary includes:

[0106] Determine the current retry 2-tuple of the file to be written;

[0107] When the current execution identifier in the current retry binary is consistent with the execution identifier of the retry binary, comparing the current retry value of the current retry binary and the retry value of the retry binary;

[0108] When the retry value is greater than the current retry value, the file to be written is opened based on the retry 2-tuple.

[0109] Among them, the current retry tuple refers to the retry tuple corresponding to the execution storage node that is currently writing data in the file to be written; determine the current execution identifier and the current retry value in the current retry tuple, and compare the execution identifier of the current retry tuple with the execution identifier of the retry tuple corresponding to the current execution storage node. If they are inconsistent, the current execution storage node can execute the task processing request; if they are consistent, it is necessary to further compare the current retry value and the retry value. When the current retry value is greater than the retry value, the task execution of the execution storage node corresponding to the current reset tuple is terminated, and the current execution storage node executes the task of writing the target business data to the file to be written.

[0110] For example, Figure 3 As shown, Figure 3 is a schematic diagram of selecting a currently executing storage node provided by an embodiment of the present specification; scheduling the storage node to receive a retry tuple sent by the client<a1,0> The task processing request is executed; the scheduling storage node determines that the execution storage node 1 is the target execution storage node based on the execution status of the execution storage node 1, the execution storage node 2, and the execution storage node 3, and includes a retry tuple<a1,0*1024+0> The task processing request is sent to the execution storage node 1, and the execution storage node 1 can retry the task based on the tuple<a1,0*1024+0> Open the target file.

[0111] When the scheduling storage node detects that the execution status of the execution storage node 1 is in a fault state, it re-acquires the execution status of the execution storage node 1 and adds 1 to the retry value of the retry tuple to obtain the retry tuple<a1,0*1024+1> If the execution status of the execution storage node 1 is still a fault state, the current execution storage node is determined to be the execution storage node 3, and a retry tuple is sent to the execution storage node 3.<a1,0*1024+1> The task processing request is executed. At this time, the retry value of the retry tuple is 1; the execution storage node 3 is based on the retry tuple<a1,0*1024+1> Open the file to be written; since the retry value of the retry tuple corresponding to the execution storage node 3 is 1 at this time, even if the execution storage node 1 executes the task processing request again, its corresponding retry value is 0, and data cannot be continued to be written to the file to be written.

[0112] Furthermore, in order to avoid the inability to continue to schedule the execution storage node after a scheduling storage node fails, multiple scheduling storage nodes can be set in the data storage cluster, wherein the scheduling storage nodes can be centralized or semi-centralized, that is, there is a master-slave relationship between the scheduling storage nodes, and in the case of a master scheduling storage node failure, the slave scheduling storage node can determine a new master scheduling storage node by acquiring a distributed lock, that is, the method may include:

[0113] When it is detected that the scheduling storage node fails, determining a scheduling storage node set in the data storage cluster;

[0114] A task processing request including a retry tuple is sent to each scheduling storage node in the scheduling storage node set.

[0115] The scheduling storage node set refers to a set consisting of scheduling storage nodes in the data storage system.

[0116] Specifically, when a device (such as a client) that sends a task processing request detects that a scheduling storage node fails, a task processing request containing a retry tuple can be sent to each scheduling storage node set in the data storage cluster. The slave scheduling storage node that obtains the distributed lock serves as the scheduling storage node of the new master scheduling storage node; and the new master scheduling storage node determines the execution status of each execution storage node in the data storage cluster based on the task processing request containing the retry tuple, and determines the target execution storage node based on the execution status.

[0117] In addition, before determining the set of scheduling storage nodes in the data storage cluster, the execution status of the scheduling storage nodes may be re-acquired to avoid the problem of short-term unavailability of the scheduling storage nodes, which may specifically include:

[0118] Re-acquiring the scheduling execution status of the scheduling storage node;

[0119] The retry value of the retry tuple is increased based on the number of times the scheduling execution status is re-obtained.

[0120] Specifically, when the device (such as a client) that sends a task processing request detects that the scheduling storage node is in a faulty state for the first time, the execution status of the scheduling storage node can be obtained again after a preset time period; if the execution status is restored to a non-faulty state, the scheduling storage node can continue to accept the task processing request. If the execution status is still in a faulty state, the execution status of the scheduling storage node can be obtained again after a preset time period until the upper limit of the number of acquisitions is reached; and in order to improve the success rate of re-acquiring the execution status, the preset time period can also increase with the increase in the number of acquisitions, that is, the time interval for re-acquiring the execution status each time increases; each time the execution status of the scheduling storage node is re-acquired, it is necessary to increase the retry value in the retry binary tuple, that is, the retry value of the retry binary tuple can be increased based on the number of retries of the device that sends the task processing request; in addition, weight coefficients can be set for the number of client retries and the number of scheduling storage node retries in the retry binary tuple, for example, if the number of client retries has a greater impact on the retry value, the weight coefficient corresponding to the number of client retries can be set to 1024, and the weight coefficient corresponding to the number of scheduling storage node retries can be set to 1.

[0121] If the number of retries reaches the preset threshold, the re-acquisition of the execution status of the scheduling storage node can be stopped, a new scheduling storage node is determined in the data storage cluster to receive the task processing request, and a target execution storage node is determined in the execution storage node based on the task processing request.

[0122] For example, Figure 4 As shown, Figure 4is a schematic diagram of selecting a currently scheduled storage node provided by an embodiment of the present specification; the scheduling storage node receives a retry tuple sent by the client.<a1,0> The target execution storage node is determined to be the execution storage node 1 based on the execution status of the execution storage node 1, the execution storage node 2 and the execution storage node 3; the execution storage node 1 performs the task processing request based on the retry tuple in the task processing request.<a1,0*1024+0> Open the target file.

[0123] When the client detects that the execution status of the scheduling storage node is in a fault state, it re-acquires the execution status of the scheduling storage node and adds 1 to the retry value of the retry tuple to obtain the retry tuple<a1,1> ; If the execution status of the obtained scheduling storage node is still a fault state, a task processing request containing a retry tuple is sent to each slave scheduling storage node; if the distributed lock is obtained from the scheduling storage node 2, the scheduling storage node 2 is used as the current scheduling storage node, that is, the main scheduling storage node; the main scheduling storage node determines the execution status of each execution storage node, and determines the current target execution storage node as execution storage node 3 based on the execution status; sends a task processing request to execution storage node 3, and execution storage node 3 executes the task based on the retry tuple in the task processing request<a1,1*1024+0> Open the target file.

[0124] The data processing method provided in this specification is that the scheduling storage node receives the task processing request, and determines the execution status of each execution storage node based on the task processing request; the scheduling storage node determines the target execution storage node based on the execution status of each execution storage node, and forwards the task processing request to the target execution storage node; the target execution storage node determines the data location information and the target processing function based on the task processing request, and obtains the target business data according to the data location information and the target processing function. An embodiment of this specification implements that the scheduling storage node schedules the execution storage node, and the target execution storage node processes the task processing request, that is, the scheduling storage node and the execution storage node assist in processing the task processing request, thereby improving the data processing efficiency.

[0125] The following combination Figure 5 , taking the application of the data processing method provided in this specification in the data storage cluster K as an example, the data processing method is further described. Figure 5 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, and the specific steps include step 502 to step 518.

[0126] Step 502: the scheduling storage node e receives the task processing request sent by the client, wherein the task processing request includes a retry tuple <execution identifier 1, client retry count*1024+scheduling storage node retry count>.

[0127] Specifically, the retry tuple is currently<k1,0*1024+0> , where 1024 is the weight coefficient of the number of client retries.

[0128] Step 504: The scheduling storage node e queries the execution status of each execution storage node in the data storage cluster based on the task processing request.

[0129] Step 506: The scheduling storage node e selects the execution storage node M as the target execution storage node, and sends the task processing request to the execution storage node M.

[0130] Step 508: The execution storage node M parses the received task processing request to obtain data location information and target processing function.

[0131] Step 510: Execute the storage node M to obtain the business data to be processed based on the data location information, process the business data to be processed based on the target processing function, obtain the target business data, and open the target file based on the retry tuple.

[0132] Specifically, based on the retry tuple<k1,0*1024+0> Open the target file and write the target business data into the target file.

[0133] Step 512: When the scheduling storage node e detects that the executing storage node M is in a faulty state, it re-acquires the execution state of the executing storage node M and increases the number of scheduling storage node retries in the retry tuple by 1.

[0134] Specifically, the retry tuple at this time is<k1,0*1024+1> .

[0135] Step 514: When the execution status of the execution storage node M re-obtained by the scheduling storage node e is a faulty state, the execution status of each execution storage node in the data storage cluster K is obtained.

[0136] Step 516: Select execution storage node N as the target execution node based on the execution status of each execution storage node, and send a task processing request containing a retry tuple to the execution storage node N.

[0137] Specifically, the task processing request sent to the execution storage node N contains the retry tuple<k1,0*1024+1> .

[0138] Step 518: Execute storage node N to obtain target business data based on the task processing request, and open the target file based on the retry tuple.

[0139] Specifically, based on the retry tuple:<k1,0*1024+1> Open the target file and write the target business data into the target file. If the execution status of the storage node M is restored to a non-faulty state, but its corresponding retry tuple is<k1,0*1024+0> , where the retry value is 0, and the retry tuple corresponding to the execution storage node N is<k1,0*1024+1> , where the retry value is 1, which is greater than the retry value of the execution storage node M, so the execution storage node N still writes the target business data to the target file.

[0140] The data processing method provided in this specification adds a retry tuple in the task processing request to record the number of client retries and the number of scheduling storage node retries, so that when a write conflict occurs later, the execution storage node corresponding to the retry tuple with a higher retry value can perform the write operation to avoid data conflicts.

[0141] Figure 6 6 shows a schematic diagram of a data processing system provided by an embodiment of the present specification, the system includes a data storage cluster 602 and a data computing cluster 604, wherein:

[0142] The data storage cluster 602 is configured to receive a task processing request and determine a target processing function based on the task processing request;

[0143] The data storage cluster 602 is further configured to process the task processing request when the target processing function is a preset processing function;

[0144] The data storage cluster 602 is further configured to forward the task processing request to the data computing cluster when the target processing function is not a preset processing function;

[0145] The data computing cluster 604 is configured to receive a task processing request and process the task processing request.

[0146] Specifically, a data storage cluster refers to a cluster composed of scheduling storage nodes and execution storage nodes; a data computing cluster refers to a cluster composed of computing nodes.

[0147] The data processing system of the present specification, when it is determined that the target processing function in the task processing request is a preset processing function, the task processing request is processed by the data storage cluster; when the target processing function in the task processing request is not a preset processing function, the task processing request is forwarded to the data computing cluster, and the data computing cluster processes it, thereby achieving the situation that when the target processing function corresponds to a simple processing operation, the data storage cluster can directly execute the data processing task, thereby avoiding the consumption of computing resources caused by all data processing being completed by the data computing cluster.

[0148] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 7 FIG. 1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0149] The receiving module 702 is configured to receive a task processing request and determine the execution state of each execution storage node based on the task processing request;

[0150] The forwarding module 704 is configured to determine a target execution storage node based on the execution status of each execution storage node, and forward the task processing request to the target execution storage node;

[0151] The determination module 706 is configured to determine data location information and a target processing function based on the task processing request, and obtain target business data according to the data location information and the target processing function.

[0152] Optionally, the receiving module 702 is further configured to:

[0153] The scheduling storage node queries the current task load and node status information of each execution storage node in the data storage cluster based on the task processing request;

[0154] or

[0155] The scheduling storage node receives the current task load and node status information reported by each execution storage node in the data storage cluster.

[0156] Optionally, the receiving module 702 is further configured to:

[0157] Determining a target processing function based on the task processing request;

[0158] In the case where it is determined that the target processing function is a preset processing function, an execution state of each execution storage node in the data storage cluster is determined based on the task processing request.

[0159] Optionally, the determining module 706 is further configured to:

[0160] The target execution storage node parses the task processing request;

[0161] The data location information and the target processing function in the task processing request are obtained.

[0162] Optionally, the determining module 706 is further configured to:

[0163] Acquire the business data to be processed according to the data location information;

[0164] The to-be-processed business data is processed based on the target processing function to obtain target business data.

[0165] Optionally, the device further includes a first detection module configured to:

[0166] In the case where the scheduling storage node detects that the target execution storage node fails, determining a current execution storage node in the data storage cluster;

[0167] Sending a task processing request including the retry 2-tuple to the current execution storage node.

[0168] Optionally, the device further includes a first acquisition module configured to:

[0169] The scheduling storage node reacquires the execution status of the target execution storage node;

[0170] A retry value in the retry tuple is increased based on the number of times the execution state is re-obtained.

[0171] Optionally, the device further comprises an opening module configured to:

[0172] The currently executing storage node determines a file to be written and a retry tuple based on the task processing request;

[0173] The file to be written is opened based on the retry tuple.

[0174] Optionally, the device further comprises a comparison module configured to:

[0175] Determine the current retry 2-tuple of the file to be written;

[0176] When the current execution identifier in the current retry binary is consistent with the execution identifier of the retry binary, comparing the current retry value of the current retry binary and the retry value of the retry binary;

[0177] When the retry value is greater than the current retry value, the file to be written is opened based on the retry 2-tuple.

[0178] Optionally, the device further includes a second detection module configured to:

[0179] When it is detected that the scheduling storage node fails, determining a scheduling storage node set in the data storage cluster;

[0180] A task processing request including a retry tuple is sent to each scheduling storage node in the scheduling storage node set.

[0181] Optionally, the device further includes a second acquisition module configured to:

[0182] Re-acquiring the scheduling execution status of the scheduling storage node;

[0183] The retry value of the retry tuple is increased based on the number of times the scheduling execution status is re-obtained.

[0184] The data processing device provided in this specification is characterized in that the scheduling storage node receives the task processing request and determines the execution status of each execution storage node based on the task processing request; the scheduling storage node determines the target execution storage node based on the execution status of each execution storage node and forwards the task processing request to the target execution storage node; the target execution storage node determines the data location information and the target processing function based on the task processing request and obtains the target business data according to the data location information and the target processing function. The scheduling storage node schedules the execution storage node and the target execution storage node processes the task processing request, that is, the scheduling storage node and the execution storage node assist in processing the task processing request, thereby improving the data processing efficiency.

[0185] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0186] Figure 8 The structure block diagram of a computing device 800 provided according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0187] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0188] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0189] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 800 may also be a mobile or stationary server.

[0190] When the processor 820 executes the computer instructions, the steps of the data processing method are implemented.

[0191] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0192] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which implement the steps of the data processing method as described above when executed by a processor.

[0193] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0194] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0195] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above data processing method.

[0196] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0197] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0198] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0199] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0200] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a data storage cluster, wherein the data storage cluster includes a scheduling storage node and at least one execution storage node, wherein: The scheduling storage node receives the task processing request and determines the execution state of each execution storage node based on the task processing request; The scheduling storage node determines a target execution storage node based on the execution status of each execution storage node, and forwards the task processing request to the target execution storage node; The target execution storage node determines data location information and a target processing function based on the task processing request, and obtains target business data according to the data location information and the target processing function; Among them, the scheduling storage node receives a task processing request and determines the execution status of each execution storage node based on the task processing request, including: determining a target processing function based on the task processing request; when it is determined that the target processing function is a preset processing function, determining the execution status of each execution storage node in the data storage cluster based on the task processing request.

2. The method of claim 1, wherein the scheduling storage node determines the execution state of each execution storage node based on the task processing request, comprising: The scheduling storage node queries the current task load and node status information of each execution storage node in the data storage cluster based on the task processing request; or The scheduling storage node receives the current task load and node status information reported by each execution storage node in the data storage cluster.

3. The method according to claim 1, wherein the target execution storage node determines data location information and a target processing function based on the task processing request, comprising: The target execution storage node parses the task processing request; The data location information and the target processing function in the task processing request are obtained.

4. The method according to claim 1, wherein obtaining target service data according to the data location information and the target processing function comprises: Acquire the business data to be processed according to the data location information; The to-be-processed business data is processed based on the target processing function to obtain target business data.

5. The method according to claim 1, wherein the task processing request includes a retry tuple, wherein the retry tuple consists of an execution identifier and a retry value; The method further comprises: In the case where the scheduling storage node detects that the target execution storage node fails, determining a current execution storage node in the data storage cluster; Sending a task processing request including the retry 2-tuple to the current execution storage node.

6. The method according to claim 5, before determining the current executing storage node in the data storage cluster, further comprising: The scheduling storage node reacquires the execution status of the target execution storage node; A retry value in the retry tuple is increased based on the number of times the execution state is re-obtained.

7. The method according to claim 5, after sending the task processing request including the retry tuple to the current execution storage node, further comprising: The currently executing storage node determines a file to be written and a retry tuple based on the task processing request; The file to be written is opened based on the retry tuple.

8. The method according to claim 7, wherein opening the file to be written based on the retry tuple comprises: Determine the current retry 2-tuple of the file to be written; When the current execution identifier in the current retry binary is consistent with the execution identifier of the retry binary, comparing the current retry value of the current retry binary and the retry value of the retry binary; When the retry value is greater than the current retry value, the file to be written is opened based on the retry 2-tuple.

9. The method according to claim 1, wherein the task processing request includes a retry tuple, and the retry tuple consists of an execution identifier and a retry value; The method further comprises: When it is detected that the scheduling storage node fails, determining a scheduling storage node set in the data storage cluster; A task processing request including a retry tuple is sent to each scheduling storage node in the scheduling storage node set.

10. The method according to claim 9, before determining the set of scheduling storage nodes in the data storage cluster, further comprising: Re-acquiring the scheduling execution status of the scheduling storage node; The retry value of the retry tuple is increased based on the number of times the scheduling execution status is re-obtained.

11. A data processing system, comprising a data storage cluster and a data computing cluster, wherein: The data storage cluster is configured to receive a task processing request and determine a target processing function based on the task processing request; The data storage cluster is further configured to process the task processing request when the target processing function is a preset processing function; The data storage cluster is further configured to forward the task processing request to the data computing cluster when the target processing function is not a preset processing function; The data computing cluster is configured to receive a task processing request and process the task processing request.

12. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 10 or 11 when executing the computer instructions.

13. A computer-readable storage medium storing computer-executable instructions, wherein the computer instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10 or 11.

14. A computer program product, characterized in that The method comprises computer instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10 or 11.

Images