Data reading method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data reading method and device, electronic equipment and a storage medium, and relates to the technical field of distributed storage. The method comprises the steps that a Hadoop distributed file system HDFS determines position information corresponding to target data through a target first management node, and the position information is determined based on metadata stored in the target first management node; the position information is returned to the client through the target first management node, so that the client reads the target data from the target data node based on the position information; wherein the data stored in the target data node is obtained by executing a write operation by a second management node contained in the primary machine room, and metadata stored in the first management node contained in each machine room is consistent with metadata stored in the second management node contained in the primary machine room. Therefore, read-write separation is realized, the pressure of the main management node is reduced, the read performance of the HDFS is improved, and particularly in a high-frequency read scene, high-concurrency read access can be supported.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a data reading method, device, electronic device and storage medium. Background Art

[0002] Hadoop Distributed File System (HDFS) is a highly fault-tolerant system that provides high-throughput data access and is suitable for deploying big data services.

[0003] The HDFS system consists of a main server room and a standby server room. The main server room includes an active namenode and several data nodes, while the standby server room includes a standby namenode and several data nodes. Data nodes are responsible for storing files, which are divided into blocks. Each block is backed up as a replica and stored on the data nodes. The main server manages the metadata within the HDFS system, including file and directory names, block locations, and the number of replicas.

[0004] In the prior art, when performing a read operation or a write operation, a client usually sends a read request or a write request to a primary management node, and the primary management node performs the read operation or the write operation.

[0005] However, because the master management node must process both read and write requests, the read performance of HDFS is low. In particular, in high-frequency read scenarios, HDFS may not be able to process read requests normally. Summary of the Invention

[0006] The embodiments of the present application provide a data reading method, device, electronic device, and storage medium to implement read-write separation of HDFS, thereby improving the read performance of HDFS.

[0007] In a first aspect, an embodiment of the present application provides a data reading method, the method comprising:

[0008] In response to a read request for target data from a client, determining a target first management node based on identification information of the client and a working status of a first management node included in each computer room;

[0009] Determining, through the target first management node, location information corresponding to the target data, wherein the location information is determined based on metadata stored by the target first management node;

[0010] The location information is returned to the client through the target first management node, so that the client reads the target data from the target data node based on the location information; wherein, the data stored in the target data node is obtained by performing a write operation on the second management node contained in the main computer room, and the metadata stored in the first management node contained in each computer room is consistent with the metadata stored in the second management node contained in the main computer room.

[0011] In an optional embodiment, determining the target first management node based on the identification information of the client and the working status of the first management node included in each computer room includes:

[0012] Based on the identification information of the client, determine the target computer room corresponding to the client, the target computer room being the main computer room or the backup computer room;

[0013] When the working status of the first management node included in the target computer room is normal, determining the target first management node as the first management node included in the target computer room;

[0014] When the working status of the first management node included in the target computer room is abnormal, the target first management node is determined to be a first management node included in another computer room.

[0015] In an optional embodiment, when the working status of the data node contained in the target computer room is normal, the target data node is the data node contained in the target computer room;

[0016] When the working status of the data node included in the target computer room is abnormal, the target data node is a data node included in another computer room.

[0017] In an optional embodiment, determining, by the target first management node, location information corresponding to the target data includes:

[0018] When it is determined that the metadata corresponding to the target data is in the cache of the target first management node, the location information corresponding to the target data is determined based on the metadata corresponding to the target data contained in the cache, wherein the cache is used to store metadata whose access frequency meets the preset high-frequency access condition.

[0019] In an optional embodiment, determining, by the target first management node, location information corresponding to the target data includes:

[0020] Query the local metadata of the target first management node to determine the location information corresponding to the target data.

[0021] In an optional embodiment, the method further includes:

[0022] Responding to a write request from a client for data to be written;

[0023] Through the second management node contained in the computer room, write operations are performed on the data to be written, and metadata stored in the second management node is updated, and corresponding update information is generated;

[0024] Based on the update information, the metadata stored in the first management node included in each computer room is updated.

[0025] In an optional embodiment, the method further includes:

[0026] Obtain the amount of synchronized data between the first management node in each computer room and the second management node in the main computer room, and obtain the total number of remote procedure call (RPC) requests;

[0027] Display the amount of synchronized data and total requests through a visual interface;

[0028] When the amount of synchronized data exceeds the traffic threshold, or the total number of requests exceeds the RPC threshold, an alarm is triggered.

[0029] In a second aspect, an embodiment of the present application further provides a data reading device, the device comprising:

[0030] a response module, configured to respond to a read request from a client for target data and determine a target first management node based on identification information of the client and a working status of the first management node included in each computer room;

[0031] A first processing module, configured to determine, through a target first management node, location information corresponding to target data, wherein the location information is determined based on metadata stored by the target first management node;

[0032] The second processing module is used to return location information to the client through the target first management node, so that the client can read the target data from the target data node based on the location information; wherein, the data stored in the target data node is obtained by performing a write operation on the second management node included in the main computer room, and the metadata stored in the first management node included in each computer room is consistent with the metadata stored in the second management node included in the main computer room.

[0033] In an optional embodiment, when determining the target first management node based on the identification information of the client and the working status of the first management node included in each computer room, the response module is further configured to:

[0034] Based on the identification information of the client, determine the target computer room corresponding to the client, the target computer room being the main computer room or the backup computer room;

[0035] When the working status of the first management node included in the target computer room is normal, determining the target first management node as the first management node included in the target computer room;

[0036] When the working status of the first management node included in the target computer room is abnormal, the target first management node is determined to be a first management node included in another computer room.

[0037] In an optional embodiment, when the working status of the data node contained in the target computer room is normal, the target data node is the data node contained in the target computer room;

[0038] When the working status of the data node included in the target computer room is abnormal, the target data node is a data node included in another computer room.

[0039] In an optional embodiment, when determining the location information corresponding to the target data through the target first management node, the first processing module is further configured to:

[0040] When it is determined that the metadata corresponding to the target data is in the cache of the target first management node, the location information corresponding to the target data is determined based on the metadata corresponding to the target data contained in the cache, wherein the cache is used to store metadata whose access frequency meets the preset high-frequency access condition.

[0041] In an optional embodiment, when determining the location information corresponding to the target data through the target first management node, the first processing module is further configured to:

[0042] Query the local metadata of the target first management node to determine the location information corresponding to the target data.

[0043] In an optional embodiment, the device further includes a synchronization module, which is configured to:

[0044] Responding to a write request from a client for data to be written;

[0045] Through the second management node contained in the computer room, write operations are performed on the data to be written, and metadata stored in the second management node is updated, and corresponding update information is generated;

[0046] Based on the update information, the metadata stored in the first management node included in each computer room is updated.

[0047] In an optional embodiment, the device further includes a monitoring module, which is configured to:

[0048] Obtain the amount of synchronized data between the first management node in each computer room and the second management node in the main computer room, and obtain the total number of remote procedure call (RPC) requests;

[0049] Display the amount of synchronized data and total requests through a visual interface;

[0050] When the amount of synchronized data exceeds the traffic threshold, or the total number of requests exceeds the RPC threshold, an alarm is triggered.

[0051] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0052] processor; and

[0053] Memory for storing programs,

[0054] The program includes instructions, and when the instructions are executed by the processor, the processor executes the data reading method as described in the first aspect.

[0055] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the data reading method as described in the first aspect.

[0056] In a fifth aspect, the present application provides a computer program product, which, when called by a computer, enables the computer to execute the steps of the data reading method as described in the first aspect.

[0057] The beneficial effects of this application are as follows:

[0058] In the data reading method provided in the embodiment of the present application, the location information corresponding to the target data is determined through the target first management node, wherein the location information is determined based on the metadata stored in the target first management node, and then the location information is returned to the client through the target first management node, so that the client reads the target data from the target data node based on the location information, wherein the data stored in the target data node is obtained by performing a write operation on the second management node included in the main computer room, and the metadata stored by the first management node included in each computer room is consistent with the metadata stored by the second management node included in the main computer room. In this way, in a disaster recovery scenario, a first management node is added to each computer room, the first management node is used to perform read operations, and the original main management node (i.e., the second management node included in the main computer room) is used to perform write operations, thereby realizing read-write separation, reducing the pressure on the main management node, and improving the read performance of HDFS, especially in high-frequency read scenarios, and being able to support high-concurrency read access.

[0059] In addition, other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or may be understood by practicing the present application. The objectives and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described here are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0061] Figure 1 A schematic diagram of an HDFS architecture applicable to embodiments of the present application;

[0062] Figure 2 A schematic diagram of an implementation flow of a data reading method provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of another implementation flow of a data reading method provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of an implementation process of metadata synchronization provided in an embodiment of the present application;

[0065] Figure 5 A schematic structural diagram of a data reading device provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0068] It should be understood that the various steps described in the method embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0069] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0070] It should be noted that the modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0071] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0072] The following is a brief introduction to the design concept of the embodiment of this application:

[0073] HDFS is a highly fault-tolerant system that provides high-throughput data access and is suitable for deploying big data services.

[0074] The HDFS system consists of a main server room and a backup server room. The main server room includes a primary management node and several data nodes, while the backup server room includes a backup management node and several data nodes. Data nodes are responsible for storing files, which are divided into blocks. Each block is backed up as a replica and stored on the data nodes. The primary management node manages metadata within the HDFS system, including file and directory names, block locations, and the number of replicas.

[0075] In the prior art, when performing a read operation or a write operation, a client usually sends a read request or a write request to a primary management node, and the primary management node performs the read operation or the write operation.

[0076] However, because the master management node must process both read and write requests, HDFS has low read and write performance across data centers, and the throughput of read operations is low. In particular, in high-frequency read scenarios, HDFS may not be able to process read requests normally.

[0077] In view of this, in an embodiment of the present application, a data reading method, device, electronic device and storage medium are provided, in which HDFS responds to a client's read request for target data, determines the target first management node based on the client's identification information and the working status of the first management node contained in each computer room, and then determines the location information corresponding to the target data through the target first management node, wherein the location information is determined based on the metadata stored in the target first management node, and then returns the location information to the client through the target first management node, so that the client reads the target data from the target data node based on the location information, wherein the data stored in the target data node is obtained by performing a write operation on the second management node contained in the main computer room, and the metadata stored by the first management node contained in each computer room is consistent with the metadata stored by the second management node contained in the main computer room. In this way, in a disaster recovery scenario, a new first management node is added to each computer room, the first management node is used to perform read operations, and the original second management node is used to perform write operations, thereby realizing read-write separation, reducing the pressure on the main management node, and improving the read performance of HDFS, especially in high-frequency read scenarios, and being able to support high-concurrency read access.

[0078] In particular, the preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0079] See Figure 1 As shown, it is a schematic diagram of the architecture of an HDFS provided in an embodiment of the present application. The system architecture may include: a main computer room and a backup computer room. Each computer room includes: a first management node 101, a second management node 102, and a data node 103. The first management node 101 is used to perform read operations, the second management node 102 is used to perform write operations, and the data node 103 is used to store files. The second management node 102 in the main computer room is the main management node, and the second management node 102 in the backup computer room is the backup node. The metadata stored in the first management node 101 contained in each computer room is consistent with the metadata stored in the second management node 102 contained in the main computer room.

[0080] In the embodiment of the present application, the files stored in the data nodes 103 in each computer room are consistent.

[0081] Each computer room has a corresponding client 104. The client 104 sends a read request for target data to the first management node 101 in the corresponding computer room. For example, the client 104 corresponding to the host side sends a write request for target data to the first management node 101 in the main computer room. The client 104 sends the write request for target data to the primary management node.

[0082] Among them, when configuring the client 104 to read data, the address of the first management node 101 in the local computer room is preferred. This configuration can be completed by modifying the core-site.xml file of the client without changing the code. Specifically, it includes: setting fs.defaultFS to point to the address of the local first management node 101 to ensure local processing of read requests.

[0083] In addition, it is worth noting that in the embodiment of the present application, the first management node 101 is lightweight. The first management node 101 does not receive the heartbeat information of the data node 103. It only needs to process the metadata synchronization from the main management node to reduce unnecessary resource usage. This shielding mechanism may reduce the first management node 101's perception of the overall health status of the cluster, and it is necessary to use other monitoring means (such as collecting monitoring indicators (remote procedure call RPC average response, I / O throughput) through timed triggering of monitoring scripts, HDFS's own Java management extension monitoring indicators, etc.) to supplement the comprehensive grasp of the data node 103 status to ensure the controllability of the overall system; there is no write-related logic module in the first management node 101, and it has efficient read performance.

[0084] The embodiment of the present application does not impose any restrictions on the number of backup computer rooms involved in the above system architecture and the number of data nodes in each computer room.

[0085] The data reading method provided by the exemplary embodiment of the present application is described below in combination with the above-mentioned system architecture and with reference to the accompanying drawings. It should be noted that the above-mentioned system architecture is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0086] See Figure 2 As shown, it is a schematic diagram of an implementation process of a data reading method provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0087] S20: In response to a read request for target data from the client, a target first management node is determined based on identification information of the client and a working status of a first management node included in each computer room.

[0088] The read request includes: the file name of the target data and the starting position of the file block, the identification information is the client name or client ID, etc., the working status is the normal working status or the abnormal working status, and the abnormal status indicates that the first management node fails.

[0089] In an embodiment of the present application, the client sends a read request for target data to HDFS. In response to the read request, HDFS determines the target first management node based on the client's identification information and the working status of the first management node contained in each computer room.

[0090] Optionally, in an embodiment of the present application, a possible embodiment is provided for determining a target first management node based on identification information of the client and the working status of the first management node contained in each computer room. Specifically, the following operations are performed:

[0091] S200: Determine a target computer room corresponding to the client based on the identification information of the client.

[0092] The target computer room is the local computer room of the client, and the target computer room can be a main computer room or a backup computer room.

[0093] For example, assuming the client ID is 00023, it is determined that the target computer room corresponding to the client is the main computer room A.

[0094] S201: Determine whether the working status of the first management node included in the target computer room is normal. If so, execute S202; otherwise, execute S203.

[0095] S202: Determine that the target first management node is the first management node included in the target computer room.

[0096] In an embodiment of the present application, when the working status of the first management node included in the target computer room is normal, the target first management node is determined to be the first management node included in the target computer room.

[0097] S203: Determine that the target first management node is a first management node included in another computer room.

[0098] In an embodiment of the present application, when the working status of the first management node included in the target computer room is abnormal, the target first management node is determined to be a first management node included in another computer room.

[0099] For example, when the computer room corresponding to the client is the main computer room A and the first management node in the main computer room A can be used normally, the target first management node is determined to be the first management node in the main computer room A; when the computer room corresponding to the client is the main computer room A and the first management node in the main computer room A cannot be used normally, the target first management node is determined to be the first management node in other computer rooms (backup computer room B); when the computer room corresponding to the client is the backup computer room B and the first management node in the backup computer room B can be used normally, the target first management node is determined to be the first management node in the backup computer room B; when the computer room corresponding to the client is the backup computer room B and the first management node in the backup computer room B cannot be used normally, the target first management node is determined to be the first management node in other computer rooms (main computer room A).

[0100] This allows the client to prioritize read operations on the primary management node in its target data center, localizing read operations and achieving efficient read performance while avoiding unnecessary cross-data center traffic. Furthermore, if the primary management node in the target data center experiences an abnormality, the primary management node in another data center can be selected for read operations, ensuring service stability.

[0101] S21: Determine location information corresponding to target data through the target first management node.

[0102] The location information is determined based on metadata stored in the target first management node.

[0103] In the embodiment of the present application, the target first management node is only used to perform read operations and does not perform write operations.

[0104] Optionally, in an embodiment of the present application, two possible methods are provided for determining the location information corresponding to the target data through the target first management node, as follows:

[0105] Method 1: When it is determined that the metadata corresponding to the target data is in the cache of the target first management node, the location information corresponding to the target data is determined based on the metadata corresponding to the target data contained in the cache.

[0106] Among them, the location information is used to indicate the location of the data block corresponding to the target data, and the cache is used to store metadata whose access frequency meets the preset high-frequency access condition. The metadata whose access frequency meets the preset high-frequency access condition can be metadata whose access frequency is greater than a preset frequency threshold, or it can be the top N metadata with the highest access frequency, where N is an integer greater than 0.

[0107] In an embodiment of the present application, a read cache mechanism is introduced in the first management node of each computer room, and metadata whose access frequencies meet preset high-frequency access conditions are stored in the caches of the first management nodes of each computer room.

[0108] For example, assuming that the metadata corresponding to the target data is metadata A, the cache of the target first management node stores 10 metadata, including metadata A. Then, the location information corresponding to the target data is determined directly based on metadata A in the cache.

[0109] This can reduce dependence on disks and networks and improve the efficiency of read operations.

[0110] Furthermore, in an embodiment of the present application, after determining the location information corresponding to the target data, the access frequency of the metadata corresponding to the target data is updated, and based on the updated access frequency, the metadata stored in the cache of the target first management node is updated.

[0111] Method 2: Query the local metadata of the target first management node to determine the location information corresponding to the target data.

[0112] Among them, local metadata can store all metadata, or it can store the remaining metadata after removing the metadata in the cache, that is, metadata whose access frequency does not meet the preset high-frequency access condition. This is not limited in the embodiments of the present application.

[0113] In an embodiment of the present application, when it is determined that the metadata corresponding to the target data is in the cache of the target first management node, the local metadata of the target first management node can be queried to determine the location information corresponding to the target data. Alternatively, the local metadata of the target first management node can be directly queried to determine the location information corresponding to the target data. This is not limited in the embodiment of the present application.

[0114] For example, assuming that the metadata corresponding to the target data is metadata B, there are 10 metadata stored in the cache of the target first management node, and metadata B is not included in the 10 metadata. Then, the local metadata of the target first management node is queried to determine the location information corresponding to the target data.

[0115] S22: Returning the location information to the client through the target first management node, so that the client reads the target data from the target data node based on the location information.

[0116] Among them, the data stored in the target data node is obtained by performing a write operation on the second management node contained in the main computer room, that is, the second management node contained in the main computer room responds to the client's write request for the data to be written, and returns write information to the client, so that the client writes the data to be written to the target data node. The metadata stored by the first management node contained in each computer room is consistent with the metadata stored by the second management node contained in the main computer room.

[0117] In an embodiment of the present application, the data node contained in the target computer room is preferentially selected as the target data node. When the working status of the data node contained in the target computer room is normal, the target data node is the data node contained in the target computer room. When the working status of the data node contained in the target computer room is abnormal, the target data node is the data node contained in other computer rooms.

[0118] For example, when the computer room corresponding to the client is the main computer room A and the data nodes in the main computer room A can be used normally, the target data node is determined to be the data node in the main computer room A; when the computer room corresponding to the client is the main computer room A and the data nodes in the main computer room A cannot be used normally, the target data node is determined to be the data node in other computer rooms (backup computer room B); when the computer room corresponding to the client is the backup computer room B and the data nodes in the backup computer room B can be used normally, the target data node is determined to be the data node in the backup computer room B; when the computer room corresponding to the client is the backup computer room B and the data nodes in the backup computer room B cannot be used normally, the target data node is determined to be the data node in other computer rooms (main computer room A).

[0119] This allows the client to prioritize data nodes in its target data center to retrieve target data, enabling localized data processing and efficient read performance while avoiding unnecessary cross-data center traffic. Furthermore, if a data node in the target data center experiences an abnormality, data nodes in other data centers can be selected to retrieve the target data, ensuring business stability.

[0120] Based on the above embodiments, see Figure 3 FIG. 1 is another implementation flow diagram of a data reading method provided in an embodiment of the present application. The specific implementation flow is as follows:

[0121] S30: The client sends a read request for target data to the first management node included in its corresponding computer room.

[0122] In addition, when the working status of the first management node included in the computer room corresponding to the client is abnormal, the client sends a read request for target data to the first management node included in other computer rooms.

[0123] S31: The first management node determines the location information of the target data based on the metadata stored therein.

[0124] S32: The first management node returns the location information to the client.

[0125] S33: Based on the location information, the client reads the target data from the data node in the corresponding computer room.

[0126] In addition, when the working status of the data node contained in the computer room corresponding to the client is abnormal, the client reads the target data from the data node contained in other computer rooms.

[0127] S34: The data node returns the target data to the client.

[0128] Furthermore, in the embodiment of the present application, the metadata stored by the first management node contained in each computer room is consistent with the metadata stored by the second management node contained in the main computer room. This is achieved by performing metadata synchronization after the second management node performs a write operation. Specifically, the following operations are performed:

[0129] SA1: Responding to a write request from a client for data to be written.

[0130] In an embodiment of the present application, the client sends a write request for the data to be written to HDFS, and HDFS responds to the write request.

[0131] SA2: Through the second management node contained in the main computer room, write operations are performed on the data to be written, and the metadata stored in the second management node is updated, and corresponding update information is generated.

[0132] In an embodiment of the present application, write information corresponding to the data to be written is generated by the second management node contained in the main computer room, and the write information is returned to the client, so that the client writes the data to be written to the data nodes contained in the main computer room and the data nodes contained in the backup computer room based on the write location information. Then, the metadata stored therein is updated by the second management node in the main computer room, and the updated information is written to the journal node (JournalNode).

[0133] The write information is used to indicate the location of the data block corresponding to the data to be written, and the write information includes: a data node list of the main computer room and a data node list of the standby computer room.

[0134] SA3: Based on the update information, the metadata stored in the first management node included in each computer room is updated.

[0135] In an embodiment of the present application, the first management node contained in each computer room obtains real-time update information of the metadata of the second management node in the main computer room through the log node, and updates the metadata stored by the first management node contained in each computer room, while relying on the Quorum Journal Manager (QJM) to ensure the atomicity and reliability of the update.

[0136] In this way, metadata synchronization is performed so that the metadata stored by the first management node in each computer room is consistent with the metadata stored by the second management node in the main computer room, so that the first management node can accurately perform read operations without relying on the second management node in the main computer room to perform write and read operations at the same time, thereby improving the performance of HDFS.

[0137] Based on the above embodiments, see Figure 4 FIG. 1 is a schematic diagram of an implementation process of metadata synchronization provided in an embodiment of the present application. The specific implementation process is as follows:

[0138] S40: The client sends a write request for the data to be written to the primary second management node.

[0139] The main second management node is the second management node contained in the main computer room.

[0140] S41: The master second management node returns the write information to the client.

[0141] S42: The client writes the data to be written to the data node based on the location information.

[0142] The client writes the data to be written to the data node of its corresponding computer room, and writes the data to be written to the data nodes of other computer rooms.

[0143] S43: The master second management node updates its stored metadata.

[0144] S44: The master second management node writes the updated information of the metadata into the log node.

[0145] S45: The first management node included in each computer room obtains metadata update information from the log node.

[0146] Among them, the first management node in the main computer room obtains metadata update information from the log node; the second management node in the standby computer room, that is, the standby second management node, obtains metadata update information from the log node; the first management node in the standby computer room obtains metadata update information from the log node, or obtains metadata update information from the standby second management node.

[0147] S46: The first management node included in each computer room updates its stored metadata based on the update information.

[0148] Furthermore, in the embodiment of the present application, traffic across computer rooms is monitored to ensure business stability, specifically by performing the following operations:

[0149] SB1: respectively obtain the amount of synchronized data between the first management node in each computer room and the second management node in the main computer room, and obtain the total amount of remote procedure call (RPC) requests.

[0150] In an embodiment of the present application, a monitoring tool is used to collect in real time the amount of synchronized data between the first management node contained in each computer room and the second management node contained in the main computer room, as well as the total amount of RPC requests across computer rooms.

[0151] Among them, the monitoring tool can be Prometheus, which is not limited in the embodiments of the present application.

[0152] SB2: Displays the amount of synchronized data and total requests through a visual interface.

[0153] In an embodiment of the present application, a visualization tool is used to display the amount of synchronized data between the first management node contained in each computer room and the second management node contained in the main computer room in the visualization interface, and the total amount of RPC requests is displayed in the visualization interface.

[0154] Among them, the visualization tool can be Grafana, which is not limited in the embodiments of the present application.

[0155] In this way, users can intuitively observe real-time traffic changes.

[0156] SB3: When the amount of synchronized data exceeds the traffic threshold, or the total number of requests exceeds the RPC threshold, an alarm is triggered.

[0157] In an embodiment of the present application, when the amount of synchronized data between the first management node contained in any computer room and the second management node contained in the main computer room is greater than the traffic threshold, a synchronized data volume alarm is triggered so that timely optimization can be performed; when the total number of RPC requests is greater than the RPC threshold, an RPC alarm is triggered so that timely optimization can be performed.

[0158] In this way, cross-data center traffic can be monitored in real time to ensure the business stability of HDFS.

[0159] Furthermore, based on the same technical concept, the embodiment of the present application provides a data reading device, which is used to implement the above method flow of the embodiment of the present application. Figure 5 As shown, the data reading device 500 may include: a response module 501, a first processing module 502, a second processing module 503, a synchronization module 504 and a monitoring module 505, wherein:

[0160] A response module 501 is configured to respond to a read request from a client for target data and determine a target first management node based on identification information of the client and the working status of the first management node in each computer room;

[0161] A first processing module 502 is configured to determine location information corresponding to target data through a target first management node, wherein the location information is determined based on metadata stored by the target first management node;

[0162] The second processing module 503 is used to return location information to the client through the target first management node, so that the client can read the target data from the target data node based on the location information; wherein, the data stored in the target data node is obtained by performing a write operation on the second management node contained in the main computer room, and the metadata stored in the first management node contained in each computer room is consistent with the metadata stored in the second management node contained in the main computer room.

[0163] In an optional embodiment, when determining the target first management node based on the identification information of the client and the working status of the first management node included in each computer room, the response module 501 is further configured to:

[0164] Based on the identification information of the client, determine the target computer room corresponding to the client, the target computer room being the main computer room or the backup computer room;

[0165] When the working status of the first management node included in the target computer room is normal, determining the target first management node as the first management node included in the target computer room;

[0166] When the working status of the first management node included in the target computer room is abnormal, the target first management node is determined to be a first management node included in another computer room.

[0167] In an optional embodiment, when the working status of the data node contained in the target computer room is normal, the target data node is the data node contained in the target computer room;

[0168] When the working status of the data node included in the target computer room is abnormal, the target data node is a data node included in another computer room.

[0169] In an optional embodiment, when determining the location information corresponding to the target data through the target first management node, the first processing module 502 is further configured to:

[0170] When it is determined that the metadata corresponding to the target data is in the cache of the target first management node, the location information corresponding to the target data is determined based on the metadata corresponding to the target data contained in the cache, wherein the cache is used to store metadata whose access frequency meets the preset high-frequency access condition.

[0171] In an optional embodiment, when determining the location information corresponding to the target data through the target first management node, the first processing module 502 is further configured to:

[0172] Query the local metadata of the target first management node to determine the location information corresponding to the target data.

[0173] In an optional embodiment, the apparatus further includes a synchronization module 504, and the synchronization module 504 is configured to:

[0174] Responding to a write request from a client for data to be written;

[0175] Through the second management node contained in the computer room, write operations are performed on the data to be written, and metadata stored in the second management node is updated, and corresponding update information is generated;

[0176] Based on the update information, the metadata stored in the first management node included in each computer room is updated.

[0177] In an optional embodiment, the apparatus further includes a monitoring module 505, which is configured to:

[0178] Obtain the amount of synchronized data between the first management node in each computer room and the second management node in the main computer room, and obtain the total number of remote procedure call (RPC) requests;

[0179] Display the amount of synchronized data and total requests through a visual interface;

[0180] When the amount of synchronized data exceeds the traffic threshold, or the total number of requests exceeds the RPC threshold, an alarm is triggered.

[0181] Based on the description of the above method embodiment and apparatus embodiment, the exemplary embodiments of the present invention further provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, and when executed by the at least one processor, the computer program causes the electronic device to perform a method according to an embodiment of the present invention.

[0182] An embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0183] An embodiment of the present application further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0184] See Figure 6As shown, the structural block diagram of the electronic device 600 that can be used as the server or client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0185] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0186] Multiple components within electronic device 600 are connected to I / O interface 605, including an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. Input unit 606 can be any type of device capable of inputting information into electronic device 600. Input unit 606 can receive input numeric or character information and generate key input signals related to user settings and / or function control of the electronic device. Output unit 607 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 608 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 609 allows electronic device 600 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a Worldwide Interoperability for Microwave Access (WiMax) device, a cellular communication device, and / or the like.

[0187] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned data reading method may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the above-mentioned data reading method by any other appropriate means (e.g., by means of firmware).

[0188] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0189] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0190] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0191] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0192] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0193] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0194] Furthermore, it should be understood that what is disclosed above is merely a preferred embodiment of the present application and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present application.

Claims

1. A data reading method, characterized in that: Applied to Hadoop Distributed File System HDFS, including: In response to a read request for target data from a client, determining a target first management node based on identification information of the client and a working status of a first management node included in each computer room; Determining, through the target first management node, location information corresponding to the target data, wherein the location information is determined based on metadata stored by the target first management node; The location information is returned to the client through the target first management node, so that the client reads the target data from the target data node based on the location information; wherein, the data stored in the target data node is obtained by performing a write operation on the second management node contained in the main computer room, and the metadata stored in the first management node contained in each of the computer rooms is consistent with the metadata stored in the second management node contained in the main computer room.

2. The method according to claim 1, wherein The determining of the target first management node based on the identification information of the client and the working status of the first management node respectively included in each computer room includes: Determining a target computer room corresponding to the client based on the identification information of the client, wherein the target computer room is a main computer room or a backup computer room; When the working status of the first management node included in the target computer room is normal, determining the target first management node to be the first management node included in the target computer room; When the working status of the first management node included in the target computer room is abnormal, the target first management node is determined to be a first management node included in another computer room.

3. The method according to claim 2, wherein When the working status of the data node contained in the target computer room is normal, the target data node is the data node contained in the target computer room; When the working status of the data node included in the target computer room is abnormal, the target data node is a data node included in the other computer room.

4. The method according to claim 1, wherein The determining, by the target first management node, location information corresponding to the target data includes: When it is determined that the metadata corresponding to the target data is in the cache of the target first management node, the location information corresponding to the target data is determined based on the metadata corresponding to the target data contained in the cache, wherein the cache is used to store metadata whose access frequency meets a preset high-frequency access condition.

5. The method according to claim 1, wherein The determining, by the target first management node, location information corresponding to the target data includes: The local metadata of the target first management node is queried to determine the location information corresponding to the target data.

6. The method according to claim 1, wherein The method further comprises: Responding to a write request from a client for data to be written; Performing a write operation on the data to be written through a second management node included in the computer room, updating metadata stored in the second management node, and generating corresponding update information; Based on the update information, the metadata stored in the first management node included in each of the computer rooms is updated.

7. The method according to claim 1, wherein The method further comprises: respectively obtaining the amount of synchronized data between the first management node contained in each of the computer rooms and the second management node contained in the main computer room, and obtaining the total amount of remote procedure call (RPC) requests; Display the amount of synchronized data and the total number of requests through a visual interface; When the amount of synchronized data exceeds the traffic threshold, or the total number of requests exceeds the RPC threshold, an alarm is triggered.

8. A data reading device, characterized in that: include: a response module, configured to respond to a read request for target data from a client and determine a target first management node based on identification information of the client and a working status of the first management node included in each computer room; a first processing module, configured to determine, through the target first management node, location information corresponding to the target data, wherein the location information is determined based on metadata stored by the target first management node; The second processing module is used to return the location information to the client through the target first management node, so that the client reads the target data from the target data node based on the location information; wherein, the data stored in the target data node is obtained by performing a write operation on the second management node included in the main computer room, and the metadata stored in the first management node included in each of the computer rooms is consistent with the metadata stored in the second management node included in the main computer room.

9. An electronic device comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images