Data processing system, method, device and storage medium

Abstract

The embodiments of the present application provide a data processing system, method, device and storage medium. Among them, the storage cluster is deployed with a distributed file system for receiving the file handle code provided by the computing node and executing the distributed protocol based on the file handle code; the computing node is deployed with a forwarding system that works in conjunction with the distributed file system, for sending the file handle code determined based on the read and write request provided by the external device to the storage cluster. When performing distributed data processing on the target data, the computing node is only responsible for forwarding the data read and write request, and will not occupy the computing resources of the computing node, which can effectively solve the computing resource competition problem of the computing node and reduce the network bandwidth demand for data transmission between the computing node and the storage cluster.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to data processing systems, methods, devices, and storage media. Background Art

[0002] With the rapid development of cloud computing technology, distributed systems are experiencing increasing network speeds, and the bottleneck is no longer network speed. However, disk I / O performance has not seen significant growth, exposing the shortcomings of converged compute and storage architectures. Due to the increasing shortcomings of coupled compute and storage, coupled with ever-increasing network speeds, distributed system architectures are shifting back toward separate compute and storage technologies.

[0003] To reduce development costs, typical distributed file system solutions (such as HDFS) deploy native software development kits (SDKs) directly on compute nodes, providing services to users directly through the native SDK. However, because the distributed protocols contained in native SDKs are complex and store multiple redundant copies of data, running them on compute nodes not only creates CPU and memory competition between distributed protocol tasks and other compute tasks, but also consumes significant network bandwidth due to the generated duplicate data. Therefore, a solution is needed that can meet the requirements for efficient data read and write processing in compute and storage separation scenarios. Summary of the Invention

[0004] To solve or improve the problems existing in the prior art, the embodiments of the present application provide a data processing system, method, device and storage medium.

[0005] In one aspect, in one embodiment of the present application, a data processing system is provided. The system includes:

[0006] A storage cluster is deployed with input and output nodes and a distributed file system encapsulated in the input and output nodes, and is used to receive file handle codes provided by computing nodes and execute a distributed protocol based on the file handle codes;

[0007] The computing node has a forwarding system deployed thereon that cooperates with the distributed file system and is configured to send the file handle code determined based on the read and write request provided by the external device to the storage cluster.

[0008] In a second aspect, in one embodiment of the present application, a data processing method is provided. The method includes:

[0009] Receive read and write requests sent by users;

[0010] Determine a file handle encoding according to the read / write request;

[0011] The file handle code is sent to the storage cluster so that the storage cluster can search for target data corresponding to the file handle code through its own deployed distributed file system and perform read and write operations on the target data.

[0012] In a third aspect, in one embodiment of the present application, a data processing method is provided. The method includes:

[0013] Receiving a file handle code sent by a computing node; wherein the file handle code is determined by the computing node based on a read / write request sent by a user;

[0014] Determining a file handle according to the file handle code;

[0015] Searching for target data corresponding to the file handle through the self-deployed distributed file system;

[0016] Perform read and write operations on the target data.

[0017] In a fifth aspect, in one embodiment of the present application, an electronic device is provided, including a memory and a processor; wherein,

[0018] The memory is used to store programs;

[0019] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the data processing method described in the second aspect or the other data processing method described in the third aspect.

[0020] In the sixth aspect, in one embodiment of the present application, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes a data processing method as described in the second aspect or another data processing method as described in the third aspect.

[0021] The technical solution provided by the embodiment of the present application is to deploy lightweight tools in any computing node of the computing cluster, and deploy native tools in the storage cluster in a distributed system in which the storage cluster and the computing cluster are separated. After the computing cluster receives the user's read and write request for the target data in the storage cluster, it can forward the request according to the file handle encoding and the distributed file system, and then the storage cluster performs the data processing action corresponding to the read and write request on the target data. When performing distributed data processing on the target data, the computing node is only responsible for forwarding the data read and write request, and will not occupy the computing resources of the computing node. It can effectively solve the problem of computing resource competition of the computing node and reduce the demand for network bandwidth for data transmission between the computing node and the storage cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 The structural intention of the data processing system provided in the embodiment of the present application;

[0024] Figure 2 A schematic diagram of the structure of a data processing system for illustrating an embodiment of the present application;

[0025] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the file handle generation process provided in an embodiment of the present application;

[0027] Figure 5 A flowchart of another data processing method provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of a method for handling file handle failure provided in an embodiment of the present application;

[0029] Figure 7 A flowchart of another data processing method provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0031] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0032] Figure 10 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;

[0033] Figure 11 A schematic structural diagram of another electronic device provided in an embodiment of the present application;

[0034] Figure 12 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;

[0035] Figure 13 A schematic structural diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0037] In some processes described in the specification, claims and the above-mentioned figures of this application, multiple operations appearing in a specific order are included. These operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit "first" and "second" to different types. In addition, the embodiments described below are only some of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0038] With the rapid development of cloud computing technology, network speeds in distributed systems are increasing, and the bottleneck of the system is no longer network speed. However, disk I / O speed has not increased significantly, and the shortcomings of the integrated computing and storage architecture are gradually being exposed. For example, if computing node resources are insufficient, computing devices will be added, or if storage resources are insufficient, storage devices will be added. This can easily lead to an oversupply of computing or storage devices, resulting in resource waste. Moreover, compared to the architecture with separate computing and storage, it is not easy to expand. In existing architectures with separate computing and storage, to meet the needs of distributed file processing, distributed decomposition systems are usually deployed in a cluster of computing nodes, thereby meeting users' needs for data reading and writing to the storage cluster. However, distributed file systems consume a lot of computing power when performing distributed protocol processing. When computing nodes are simultaneously processing other user-related computing tasks, competition for CPU and memory resources is likely to occur, affecting the normal processing efficiency of user-related tasks. In addition, when the distributed file system in the computing node executes the distributed protocol, it also requires a high bandwidth for data transmission, which requires high network card performance and consumes bandwidth resources. Therefore, there is a need for a technical solution that can avoid the problem of computing node resource competition in the architecture with separate computing and storage. The specific working process of the technical solution of this application will be described in the following embodiments.

[0039] like Figure 1The data processing system provided in the embodiments of the present application is structurally intended. The data processing system includes a storage cluster 11, deployed with input / output nodes, and a distributed file system encapsulated within the input / output nodes, configured to receive file handle codes provided by computing nodes and execute a distributed protocol based on the file handle codes. Computing nodes 12 are deployed with a forwarding system 121 that cooperates with the distributed file system and transmits the file handle codes, determined based on read / write requests provided by external devices, to the storage cluster 11.

[0040] Specifically, the storage cluster 11 further includes: an input / output node 111 and a distributed storage node cluster 112; the input / output node 111 is communicatively connected to the distributed storage node cluster 112; the computing node 12 is communicatively connected to the input / output node 111. A distributed file system 111a is deployed on the input / output node 111.

[0041] In practical applications, there may be multiple input and output nodes 111. The distributed storage node cluster 112 also includes a central node 112a and multiple storage nodes 112b. The multiple input and output nodes 111 are communicatively connected to the central node 112a. Users can communicate with the computing nodes 12 through the client 13.

[0042] For ease of understanding, the following examples are given. Figure 2 This is a schematic diagram of the structure of the data processing system illustrated in the embodiment of this application. Figure 2 You can see:

[0043] There are multiple computing nodes 12 forming a computing node cluster, each computing node 12 mainly includes user logic information for providing services to users and a lightweight software development tool (Software Development Kit, SDK) for providing information forwarding function, which is the forwarding system mentioned above. Figure 2 As can be seen in the figure, the lightweight software development tool running in the computing node 12 can provide users with a distributed file system interface, so as to provide file handles for users to read and write the distributed file system, thereby realizing the forwarding of user read and write requests and target data.

[0044] The storage cluster 11 includes input / output nodes 111 and a distributed file system 111a. The input / output nodes 111 can be, for example, IO servers, and the corresponding distributed file system 111a can be a native software development kit (SDK). The IO server encapsulates the native software development kit and provides file handles for users to read and write to the distributed file system. The input / output nodes 111 mentioned here process data in a certain order. That is, it is necessary to ensure that the distributed storage node cluster can process data in the order in which users initiate data processing requests to avoid data processing errors.

[0045] In one or more embodiments of the present application, after the distributed file system receives the data write request, the distributed file system is also used to perform distributed file processing on the copy data of the target data through the distributed storage node cluster. In the distributed storage node cluster, there are multiple storage nodes and a central node for coordinating read and write tasks. When the native software development tool is packaged and run, it needs to implement the relevant processing logic of the complex distributed protocol inside. For example, when the user needs to write the target data, the distributed file system executes the redundant copy strategy, and the central node writes multiple copies of the data to other storage nodes in the distributed storage node cluster. These complex distributed protocol-related tasks are all completed by the storage cluster, and there is no need to occupy the processor and memory resources in the computing node, so that the computing node can better provide computing services to users. At the same time, there is no need for copy data to be transmitted between the computing node and the storage cluster to consume too much network bandwidth.

[0046] It should be noted that the computing node is specifically used when working in cooperation: after the user sends a data read and write request through the client, the computing node forwards the data write request sent by the user to the storage cluster, and after the storage cluster completes the processing of the target data pointed to by the data write request through the distributed file system (such as executing a redundant copy strategy and writing multiple copies of data to multiple storage nodes), it receives the file handle code corresponding to the target data sent by the storage cluster to generate a file handle corresponding to the target data for providing to the user.

[0047] In one or more embodiments of the present application, if the target data has not been stored in the storage cluster and is being written to the storage cluster for the first time, the input-output node is used to receive a data write request sent by the user through the computing node; and, through the distributed file system deployed by itself, the target data pointed to by the data write request is subjected to distributed file processing; the input-output node is used to generate a file handle encoding corresponding to the target data based on the processing result; and feedback information to the computing node regarding the file handle encoding so that the computing node generates a target file handle corresponding to the target data for providing to the user. Specifically,

[0048] If the target data and its related data are not stored in the storage cluster, the new target data needs to be written based on the data write request. Since the target data is newly written data, a target file handle for operating the target data needs to be allocated to the user first.

[0049] The user sends a data write request for new target data to the computing node through the client. The request does not contain relevant information such as file handles pointing to the target data. Then, the computing node forwards the data write request to the input and output node. After the input and output node finds that there is no information such as file handle encoding in the data write request, it sends a data write request to the distributed file system deployed by itself. Then, the distributed file system determines the storage information of the target data through the central node, and feeds back the storage information (such as storage address, data length, etc.) to the distributed file system (such as native software development tools). Then, based on the storage information, it generates the target file handle and file handle encoding, and feeds back the target file handle to the user's client. For details, please refer to Figure 3 The corresponding embodiments will not be repeated here.

[0050] It should be noted that the distributed file system mentioned here can be deployed locally or distributed based on a separate compute and storage architecture. The distributed file system on any input and output node in the storage cluster is called the distributed file system deployed by the storage cluster itself.

[0051] Based on the above embodiments, it can be seen that a distributed file system, input and output nodes, and a forwarding system are deployed in layers between the user and the distributed storage node cluster, and file handles are used to achieve isolation between the user and the distributed storage node cluster, thereby reducing the adverse effects of unstable factors and complex computing tasks in the storage cluster on the computing nodes and users. In one or more embodiments of the present application, the computing node is also used to receive read and write requests sent by the user; determine the file handle encoding based on the read and write requests; and send the file handle encoding to the storage cluster so that the storage cluster can search for the target data corresponding to the file handle encoding through its own deployed distributed file system and perform read and write operations on the target data. In addition, the computing node is also used to determine the file handle corresponding to the target data based on the read and write requests; and determine the file handle encoding based on the file handle.

[0052] The file handle code mentioned here is the code generated by the input and output nodes based on the second file handle. The computing node can forward the user's data read and write requests to the input and output nodes through the file handle code, so that the input and output nodes can find the corresponding target file based on the file handle code and perform the corresponding data processing (for example, read or write) operation. For details, please refer to Figure 5 The corresponding embodiments will not be repeated here.

[0053] The computing node is further configured to obtain a valid file handle reallocated for the target data from the storage cluster based on the computing node identifier carried in the read / write request if the file handle code fails; and determine a valid file handle code based on the valid file handle. Furthermore, the valid input / output node is further configured to generate a valid file handle code based on the valid file handle; and send the valid file handle code to the valid input / output node. The file handle code failure mentioned here may have multiple reasons, please refer to the following for details. Figure 6 The corresponding embodiments will not be repeated here.

[0054] In one or more embodiments of the present application, the input and output nodes are used to receive the file handle code sent by the computing node; wherein the file handle code is determined by the computing node based on the read and write request sent by the user; the file handle is determined according to the file handle code; and the target data corresponding to the file handle is searched through the distributed file system deployed by itself; and the target data is read and written. For details, please refer to Figure 7The corresponding embodiments will not be repeated here. Furthermore, the storage cluster is further configured to, if the file handle is not determined according to the file handle code, reallocate a valid file handle to the target data according to the computing node identifier corresponding to the file handle code; and send the valid file handle to a valid input and output node.

[0055] In actual application, if the file handle is not determined according to the file handle code, a valid file handle is reallocated to the target data according to the computing node identifier corresponding to the file handle code. When the file handle code is invalid, the specific processing method and file handle migration method can be found in Figure 6 The corresponding embodiments will not be repeated here.

[0056] Based on the same idea, the embodiment of the present application also provides a data processing method. Figure 3 This is a flow chart of a data processing method provided in an embodiment of the present application. The method can be applied to a storage cluster and specifically includes the following steps:

[0057] 301: Receive a data write request sent by a user through a computing node.

[0058] 302: Perform distributed file processing on the target data pointed to by the data write request through the distributed file system deployed by itself.

[0059] 303: Generate a file handle code corresponding to the target data according to the processing result.

[0060] 304: Feedback information regarding the file handle encoding to the computing node, so that the computing node generates a target file handle corresponding to the target data for providing to the user.

[0061] As described in step 303, generating the file handle code corresponding to the target data includes: 303a: generating the file handle corresponding to the target data according to the processing result; 303b: generating the file handle code based on the file handle; 303c: sending the file handle code to the computing node.

[0062] If the target data and target data related data are not stored in the storage cluster, it is necessary to implement the writing of new target data based on the data write request. Since the target data is newly written data, the target file handle for operating the target data needs to be allocated to the user first. Specifically,

[0063] In order to understand the process of generating the target file handle, the following example is given. Figure 4 Schematic diagram of the file handle generation process provided in the embodiment of the present application. Figure 4As you can see,

[0064] Distributed file systems (such as Figure 4 After receiving the storage information fed back by the central node in the distributed storage node cluster (the storage information includes the storage address of the target data and other related information), the native SDK generates a first file handle (such as Figure 4 The distributed file system calculates and generates a first handle identifier (eg, handle1 ID) using the first file handle, and sends the first handle identifier to the input and output nodes (eg, Figure 4 The IO node generates a second file handle (handle2) based on the received first handle identifier (corresponding to the file handle in step 303b above), calculates and generates a file handle code (e.g., handle12 ID) using the second file handle, and sends the file handle code to the forwarding system in the computing node (e.g., Figure 4 The forwarding system generates a third file handle (handle3) based on the received file handle code, calculates a third handle identifier (e.g., handle13 ID) using the third file handle, and then generates a fourth file handle (handle4) based on the third handle identifier (corresponding to the target file handle in step 304), and sends the fourth handle to the user. After obtaining the fourth file handle, the user can use it to read and write the target data.

[0065] Based on the same idea, the embodiment of the present application also provides a data processing method. Figure 5 This is a flow chart of another data processing method provided in an embodiment of the present application. The method can be applied to a computing node and a computing node cluster consisting of multiple computing nodes. The method includes the following steps:

[0066] 501: Receive read / write request from user.

[0067] 502: Determine the file handle encoding according to the read / write request.

[0068] 503: Send the file handle code to the storage cluster, so that the storage cluster can search for target data corresponding to the file handle code through its own distributed file system and perform read and write operations on the target data.

[0069] based on Figure 1 and Figure 2As shown in the system structure diagram, when a user needs to read or write data, they can send a read or write request through the client. In this embodiment, after the user has obtained the fourth file handle allocated by the compute node, the read or write request sent by the user will carry the fourth file handle indicating the target file, so that the storage cluster knows which target file is stored in which location and what operation to perform.

[0070] The step 502 of determining the file handle code according to the read / write request includes: 502a: determining the file handle corresponding to the target data according to the read / write request; 502b: determining the file handle code according to the file handle.

[0071] To facilitate understanding, the following examples are given:

[0072] based on Figure 3 In the corresponding embodiment, it is assumed that the user receives the fourth file handle handle4, the computing node maintains the third file handle handel3 (corresponding to the file handle described in steps 502a and 502b), the input and output node maintains the second file handle handel2, and the distributed file system maintains the first file handle handel1.

[0073] After the computing node receives the fourth file handle sent by the user, it will determine the third file handle based on the third file handle identifier corresponding to the fourth file handle. Then, the computing node obtains the file handle code contained in the third file handle. The forwarding system (lightweight software development tool, lightweight SDK) in the computing node sends the file handle code to the input and output node (IOserver). The corresponding second file handle is found through the file handle code. If the input and output node finds the second file handle, the first file handle identifier in the second file handle will be further sent to the distributed file system (native software development tool, native SDK). The distributed file system determines the corresponding target data through the central node, and the read and write request specifies the storage address and other information for performing read and write operations on the target data.

[0074] In actual applications, determining the file handle encoding based on the read and write requests includes: if the file handle encoding is invalid, obtaining a valid file handle reallocated for the target data from the storage cluster based on the computing node identifier carried in the read request; and determining the valid file handle encoding based on the valid file handle.

[0075] The file handle encoding failure mentioned here may have many reasons. Figure 6 A schematic diagram of a method for handling file handle failure provided by an embodiment of the present application. Figure 6The reasons for failure can include, for example, a failure or upgrade of the distributed file system, an I / O node, a failure in the transmission path between the forwarding system and the I / O node, or a failure in the transmission path between the I / O node and the distributed storage node cluster, which results in the original I / O node or distributed file system being unable to provide read and write services normally. The discovery of a file handle failure can occur when the compute node has already received a notification of a failure in the I / O node, distributed file system, or storage node in the storage cluster before the user issues a data read or write request, or when the file handle is discovered to be invalid after the user issues a data read or write request. Even if the user has already issued a data read or write request to the compute node, the file handle migration can still be completed without the user's awareness, meeting the user's data read and write needs.

[0076] For example, a distributed file system fails. In the forwarding system of the compute node, the third file handle handle3 is still in a normal state, and the user is unaware of the failure. At this point, the user sends a normal data read / write request, which carries the fourth file handle. After receiving the fourth file handle, the compute node determines the corresponding third file handle based on the fourth file handle and finds the file handle code contained in the third file handle. Assuming that the compute node and the forwarding system are unaware of the failure in the storage cluster at this time, the compute node uses the file handle code to search for the corresponding second file handle from the input / output node. During the search process, if the corresponding second file handle is not successfully found (for example, the input / output node does not respond or reports a search failure), the compute node knows that the input / output node cannot meet the data read / write requirements and cannot continue to provide data read / write services. The compute node then sends a request to process the target data (for example, read / write) to a new input / output node. The new input / output node reassigns a new valid file handle to the compute node based on the request and returns the valid file handle code to the compute node. On the compute node, the invalid file handle code in the original third file handle is replaced with a valid file handle code. This does not require changing the third file handle identifier or regenerating a fourth file handle, and the user's original fourth file handle remains valid. The compute node then continues to execute the user's data read and write requests. Even if a failure occurs after the user has issued a data read or write request, the data read and write operation can be completed without the user noticing, achieving file handle migration.

[0077] As an optional embodiment, the specific implementation method of the new input / output node re-assigning a new valid file handle to the computing node based on the request can be: when the computing node discovers that the original input / output node has failed, it sends a data read / write request to the new input / output node, carrying a computing node identifier or a third file handle identifier, etc., which is used to represent the uniqueness of the computing node. Then, the new distributed file system in the new input / output node sends the request to the central node based on the computing node identifier and the identity identifier of the new distributed file system. When the central node discovers that the same computing node identifier is associated with two input / output nodes at the same time, it will disassociate the identity identifier of the distributed file system in the failed input / output node with information such as the storage address of the target data, and establish an association relationship with information such as the storage address of the target data for the new distributed file system. Then, the distributed file system regenerates the first file handle and establishes an information forwarding path between the computing node and the storage cluster based on the new first file handle.

[0078] For another example, when the distributed file system (e.g., native SDK) is upgraded, the second file handle of the input and output node (e.g., IOserver) can no longer meet the data reading and writing service requirements of the forwarding system. The forwarding system in the computing node (e.g., lightweight SDK) will select another normal IOServer and call the native SDK to perform ReleaseFileLock and file Open operations. In this way, a new second file handle is established on the new IOServer and the file corresponding to the target data is reopened. In other words, the lightweight SDK establishes a new data information forwarding path on another IOServer.

[0079] In the above-mentioned embodiment, since the native SDK runs on the storage cluster, the file handle of the native SDK also only runs on the storage cluster. In the future evolution of the distributed file system, the update of the distributed protocol and the native SDK logic will cause changes in the storage node, but this will only cause the file handle handle on the native SDK and IOServer to be released, and the file handle handle on the lightweight SDK that directly serves the user will not be released due to the change; and through the above-mentioned file handle switching scheme, the third file handle handle on the lightweight SDK can find another storage node to perform the file handle switching operation, and establish a new data information forwarding path on another new IOServer to continue the service, realizing smooth upgrades and computing node switching without user perception.

[0080] Based on the same idea, the embodiment of the present application also provides another data processing method. Figure 7A flowchart of another data processing method provided in an embodiment of the present application is provided. The method can be applied to a storage cluster and specifically includes the following steps:

[0081] 701: Receive a file handle code sent by a computing node; wherein the file handle code is determined by the computing node based on a read / write request sent by a user.

[0082] 702: Determine the file handle according to the file handle code.

[0083] 703: Search for target data corresponding to the file handle through the self-deployed distributed file system.

[0084] 704: Perform read and write operations on the target data.

[0085] based on Figure 1 and Figure 2 In a corresponding embodiment, after receiving the file handle code, an input / output node (e.g., an IO server) in the storage cluster will query the corresponding second file handle (corresponding to the file handle described in step 702). The file handle code is determined by the computing node based on the fourth file handle carried in the received read / write request. Specifically, after obtaining the fourth file handle carried in the read / write request, the computing node will determine the third file handle maintained by the computing node based on the third file handle identifier contained in the fourth file handle, and determine the file handle code contained in the third file handle.

[0086] After the input and output nodes obtain the file handle code, they will search for a valid second file handle (that is, the file handle mentioned in step 702) based on the file handle code. If it can be found, the corresponding target data will be further searched through the distributed file system deployed by the storage cluster itself based on the second file handle. After determining the target data and the storage location of the target data in the storage cluster and other information, the read and write operations on the target data will be performed. For example, when performing a target data write operation, the distributed file system in the storage cluster executes a complex distributed protocol, such as writing copies of multiple target data to multiple storage nodes. This can avoid the consumption of computing resources (such as CPU, memory) of computing nodes by distributed protocols, and avoid competition for computing resources with other user computing tasks in the computing nodes, thereby better improving the user experience.

[0087] It should be noted that the read and write operations on the target data include: distributed file processing of the target data according to the encoding sequence of the file handle encoding. Because when actually performing data read and write operations, there may be many users and a lot of target data that need to be read and written, and since the present solution needs to implement the relevant operations on the target data by means of file handle forwarding, in order to avoid the read and write operations on the target data being out of order due to file handle forwarding, the distributed file system can perform the corresponding operations according to the encoding sequence of the file handle encoding.

[0088] Based on the same idea, the embodiment of the present application also provides a data processing device. Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device includes:

[0089] The receiving module 81 is configured to receive a data write request sent by a user through a computing node.

[0090] The processing module 82 is configured to perform distributed file processing on the target data pointed to by the data write request through the distributed file system deployed by itself.

[0091] The generating module 83 is used to generate the file handle code corresponding to the target data according to the processing result.

[0092] The feedback module 84 is configured to feed back information regarding the file handle encoding to the computing node so that the computing node generates a target file handle corresponding to the target data for providing to the user.

[0093] Optionally, the generating module 83 is further configured to generate a file handle corresponding to the target data according to the processing result; generate the file handle code based on the file handle; and send the file handle code to the computing node.

[0094] An embodiment of the present application also provides an electronic device. The electronic device is a master node electronic device in a computing unit. Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a memory 901, a processor 902 and a communication component 903; wherein,

[0095] The memory 901 is used to store programs;

[0096] The processor 902 is coupled to the memory and is configured to execute the program stored in the memory to:

[0097] Receive data write requests sent by users through computing nodes;

[0098] Perform distributed file processing on the target data pointed to by the data write request through the distributed file system deployed by itself;

[0099] Generate a file handle code corresponding to the target data according to the processing result;

[0100] Feedback information is sent to the computing node based on the file handle encoding so that the computing node generates a target file handle corresponding to the target data for providing to a user.

[0101] The processor 902 is further configured to generate a file handle corresponding to the target data according to the processing result; generate the file handle code based on the file handle; and send the file handle code to the computing node.

[0102] The memory 901 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0103] Furthermore, the processor 902 in this embodiment may specifically be: a programmable switching processing chip, which is configured with a data replication engine and can replicate received data.

[0104] When the processor 902 executes the program in the memory, in addition to the above functions, it can also realize other functions, for details, please refer to the description of the above embodiments. Figure 9 As shown, the electronic device further includes: a power supply component 904 and other components.

[0105] The embodiment of the present application further provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figure 3 The method described in the corresponding embodiment.

[0106] Based on the same idea, the embodiment of the present application also provides another data processing device. Figure 10 This is a schematic diagram of the structure of another data processing device provided in an embodiment of the present application. The data processing device includes:

[0107] The receiving module 101 is configured to receive a read / write request sent by a user.

[0108] The determination module 102 is configured to determine a file handle code according to the read / write request.

[0109] The search module 103 is configured to send the file handle code to the storage cluster so that the storage cluster can search for target data corresponding to the file handle code through its own distributed file system and perform read and write operations on the target data.

[0110] The determination module 102 is further configured to determine a file handle corresponding to the target data according to the read / write request; and determine the file handle code according to the file handle.

[0111] The determination module 102 is further configured to obtain a valid file handle reallocated for the target data from the storage cluster based on the computing node identifier carried in the read request if the file handle encoding is invalid;

[0112] A valid file handle code is determined according to the valid file handle.

[0113] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the processor to implement Figure 5 The method described in the corresponding embodiment.

[0114] An embodiment of the present application also provides an electronic device. The electronic device is a standby node electronic device in a computing unit. Figure 11 This is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. The electronic device includes a memory 1101, a processor 1102, and a communication component 1103; wherein,

[0115] The memory 1101 is used to store programs;

[0116] The processor 1102 is coupled to the memory and configured to execute the program stored in the memory to:

[0117] Receive read and write requests sent by users;

[0118] Determine a file handle encoding according to the read / write request;

[0119] The file handle code is sent to the storage cluster so that the storage cluster can search for target data corresponding to the file handle code through its own deployed distributed file system and perform read and write operations on the target data.

[0120] The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0121] Furthermore, the processor 1102 in this embodiment may specifically be: a programmable switching processing chip, which is configured with a data replication engine and can replicate received data.

[0122] When the processor 1102 executes the program in the memory, in addition to the above functions, it can also realize other functions, for details, please refer to the description of the above embodiments. Figure 11 As shown, the electronic device further includes: a power supply component 1104 and other components.

[0123] The embodiment of the present application further provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figure 5 The method described in the corresponding embodiment.

[0124] Based on the same idea, the embodiment of the present application also provides another data processing device. Figure 12 This is a schematic diagram of the structure of another data processing device provided in an embodiment of the present application. The data processing device includes:

[0125] The receiving module 1201 is configured to receive a file handle code sent by a computing node; wherein the file handle code is determined by the computing node based on a read / write request sent by a user.

[0126] The determination module 1202 is configured to determine the file handle according to the file handle code.

[0127] The search module 1203 is configured to search for target data corresponding to the file handle by deploying a distributed file system on its own.

[0128] The operation module 1204 is configured to perform read and write operations on the target data.

[0129] Optionally, the determination module 1202 is further configured to reallocate a valid file handle for the target data according to a computing node identifier corresponding to the file handle code if the file handle is not determined according to the file handle code.

[0130] Optionally, the operation module 1204 is further configured to perform distributed file processing on the target data according to the encoding sequence of the file handle encoding.

[0131] An embodiment of the present application also provides an electronic device. The electronic device is a standby node electronic device in a computing unit. Figure 13 This is a structural diagram of another electronic device provided in an embodiment of the present application. The electronic device includes a memory 1301, a processor 1302, and a communication component 1303; wherein,

[0132] The memory 1301 is used to store programs;

[0133] The processor 1302 is coupled to the memory and configured to execute the program stored in the memory to:

[0134] Receiving a file handle code sent by a computing node; wherein the file handle code is determined by the computing node based on a read / write request sent by a user;

[0135] Determining a file handle according to the file handle code;

[0136] Searching for target data corresponding to the file handle through the self-deployed distributed file system;

[0137] Perform read and write operations on the target data.

[0138] The memory 1301 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0139] Furthermore, the processor 1302 in this embodiment may specifically be: a programmable switching processing chip, which is configured with a data replication engine and can replicate received data.

[0140] When the processor 1302 executes the program in the memory, in addition to the above functions, it can also realize other functions, for details, please refer to the description of the above embodiments. Figure 13 As shown, the electronic device further includes: a power supply component 1304 and other components.

[0141] The embodiment of the present application further provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figure 7 The method described in the corresponding embodiment.

[0142] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the processor to implement Figure 7 The method described in the corresponding embodiment.

[0143] Based on the above embodiment, in a distributed system where the storage cluster and the computing cluster are separated, the lightweight tool is deployed in any computing node of the computing cluster, and the native tool is deployed in the storage cluster. After the computing cluster receives the user's read and write request for the target data in the storage cluster, it can forward the request according to the file handle encoding and the distributed file system, and then the storage cluster performs the data processing action corresponding to the read and write request on the target data. When performing distributed data processing on the target data, the computing node is only responsible for forwarding the data read and write request, and will not occupy the computing resources of the computing node. This can effectively solve the problem of computing resource competition of the computing node and reduce the demand for network bandwidth for data transmission between the computing node and the storage cluster.

[0144] In addition, since users are isolated from the storage cluster through the information forwarding function provided by the computing nodes, even if the input and output nodes or distributed file system in the storage system change, it will not affect the user's normal data reading and writing needs. This makes it possible to resolve failures in the input and output nodes, distributed file system or storage cluster without the user's awareness, and to achieve flexible and balanced upgrades of the distributed file system.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0146] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing system, wherein the data processing system is based on a computing and storage separation architecture, comprising: A storage cluster is deployed with input and output nodes, a distributed storage node cluster communicatively connected to the input and output nodes, and a distributed file system encapsulated in the input and output nodes; Used to receive a file handle code provided by a computing node and execute a distributed protocol based on the file handle code; wherein the distributed file system refers to a software development tool, and the distributed file system is used to execute a distributed protocol; wherein the distributed storage node cluster includes a central node and multiple storage nodes, and the central node is in communication with the input and output nodes; The computing node is deployed with a forwarding system that works in conjunction with the distributed file system, which is used to send the file handle code determined based on the read and write requests provided by the external device to the storage cluster. The file handle code is determined by the computing node based on the read and write requests.

2. The system according to claim 1, wherein the input and output nodes are configured to receive data write requests sent by users through the computing nodes; and Performing distributed file processing on the target data pointed to by the data write request through the distributed file system deployed by itself; The input and output nodes are used to generate a file handle code corresponding to the target data according to the processing results; and to feed back information about the file handle code to the computing node so that the computing node generates a target file handle corresponding to the target data for providing to the user.

3. The system according to claim 1, wherein the input and output nodes are used to receive the file handle code sent by the computing node; The file handle code is determined by the computing node based on the read and write request sent by the user; the file handle is determined according to the file handle code; and searching for target data corresponding to the file handle through the distributed file system deployed by itself; Perform read and write operations on the target data.

4. The system according to claim 2 or 3, wherein the storage cluster further comprises: Distributed storage node cluster; After the distributed file system receives the data write request, the distributed file system is further configured to perform distributed file processing on the replica data of the target data through the distributed storage node cluster.

5. According to the system of claim 4, if the input and output node fails to determine the file handle according to the file handle code, a valid file handle is reallocated for the target data through the distributed storage node cluster according to the computing node identifier corresponding to the file handle code; and the valid file handle is sent to a valid input and output node.

6. The system according to claim 5, wherein the valid input and output node is further used to generate a valid file handle code according to the valid file handle; and send the valid file handle code to the computing node.

7. The system according to claim 2, wherein the input and output nodes are configured to perform distributed file processing on the target data according to the encoding sequence of the file handle encoding.

8. The system according to claim 1, wherein the computing node is used to receive read and write requests sent by users; Determine a file handle encoding according to the read / write request; The file handle code is sent to the storage cluster so that the storage cluster can search for target data corresponding to the file handle code through its own deployed distributed file system and perform read and write operations on the target data.

9. According to the system of claim 8, the computing node is further used to determine the file handle corresponding to the target data according to the read and write request; and determine the file handle encoding according to the file handle.

10. The system according to claim 8, wherein the computing node is further configured to obtain a valid file handle reallocated for the target data from the storage cluster based on the computing node identifier carried in the read / write request if the file handle encoding is invalid; A valid file handle code is determined according to the valid file handle.

11. A data processing method, applied to a computing node in a data processing system, wherein the data processing system is based on a computing and storage separation architecture, the method comprising: Receive read and write requests sent by users; Determine a file handle encoding according to the read / write request; The file handle code is sent to the storage cluster so that the storage cluster can search for the target data corresponding to the file handle code through its own deployed distributed file system and perform read and write operations on the target data. The storage cluster includes input and output nodes and a distributed storage node cluster that is communicatively connected to the input and output nodes. The distributed file system is encapsulated on the input and output nodes. The distributed file system refers to a software development tool. The distributed file system is used to execute a distributed protocol. The distributed storage node cluster includes a central node and multiple storage nodes. The central node is communicatively connected to the input and output nodes. The file handle code is determined by the computing node based on the read and write request.

12. A data processing method, applied to a storage cluster in a data processing system, wherein the data processing system is based on a computing and storage separation architecture, the storage cluster comprising input / output nodes, a distributed storage node cluster communicatively connected to the input / output nodes, and a distributed file system encapsulated within the input / output nodes, wherein the distributed file system is a software development tool and is used to execute a distributed protocol, wherein: The distributed storage node cluster includes a central node and multiple storage nodes, and the central node is communicatively connected with the input and output nodes; the method includes: Receiving a file handle code sent by a computing node; wherein the file handle code is determined by the computing node based on a read / write request sent by a user; Determining a file handle according to the file handle code; Searching for target data corresponding to the file handle through the distributed file system deployed by itself; Perform read and write operations on the target data.

13. An electronic device comprising a memory and a processor; wherein: The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the method according to claim 11 or the method according to claim 12.

14. A non-transitory machine-readable storage medium having executable code stored thereon, wherein when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method according to claim 11; or the method according to claim 12 above.

Images