A client performance statistics method, system, device and storage medium

Abstract

The application discloses a client performance statistics method, system, device and storage medium, and applies to the storage technical field. The method is applied to a server of HDFS and comprises the following steps: periodically reading a data operation total byte accumulation amount of a target client stored in a buffer according to a preset first time interval; determining a difference between the current read data operation total byte accumulation amount of the target client and a last read data operation total byte accumulation amount of the target client, and dividing the difference by the first time interval to obtain bandwidth data of the target client; and when a bandwidth viewing instruction for the target client sent by a user is received, the obtained bandwidth data of the target client is fed back to the user. According to the scheme, the performance statistics of the target client can be effectively realized, and the bandwidth change of the target client can be conveniently determined.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a client-side performance statistics method, system, device, and storage medium. Background Technology

[0002] HDFS (Hadoop Distributed File System) is a distributed file system designed to run on general-purpose hardware. It shares many similarities with other distributed file systems, but its differences are also significant. HDFS is a highly fault-tolerant system, suitable for deployment on inexpensive machines. HDFS provides high-throughput data access, making it ideal for applications with large-scale datasets. HDFS relaxes some of the constraints of POSIX (Portable Operating System Interface for Unix).

[0003] HDFS is fault-tolerant and designed to be deployed on low-cost hardware, while providing high throughput access to application data. Therefore, HDFS is suitable for applications with extremely large datasets. HDFS relaxes some POSIX requirements, allowing for streaming access to data within the file system.

[0004] Currently, HDFS supports performance statistics for the entire service node, but it does not perform performance statistics for different clients. In other words, it does not differentiate between client-specific performance statistics. For example, during HDFS performance testing, if a slowdown occurs, staff cannot clearly identify which client is causing the problem on the HDFS server, increasing the difficulty of locating and resolving the issue.

[0005] In conclusion, for HDFS, how to effectively differentiate client performance statistics is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a client performance statistics method, system, device, and storage medium, for HDFS, to effectively differentiate client performance statistics.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] A client-side performance statistics method, applied to an HDFS server, includes:

[0009] According to a preset first time interval, periodically read the total number of bytes of data operations from the target client stored in the cache.

[0010] The difference between the total number of bytes accumulated from the current data operations of the target client and the total number of bytes accumulated from the previous data operations of the target client is determined, and the difference is divided by the first time interval to obtain the bandwidth data of the target client.

[0011] Upon receiving a bandwidth viewing command from a user for the target client, the system will return all bandwidth data obtained from the target client to the user.

[0012] Preferably, the step of feeding back the bandwidth data of the target client obtained in each iteration to the user includes:

[0013] The bandwidth data of the target client obtained in each iteration is fed back to the user in the form of a bandwidth change diagram, and the maximum value of the bandwidth data of the target client obtained in each iteration is displayed.

[0014] Preferred options also include:

[0015] When a read data request is received from the target client, the maximum latency of the read operation, the total number of bytes of the read operation, the total duration of the read operation, and the average latency of the read operation stored in the cache are updated according to the current read operation of the target client.

[0016] Upon receiving a write data request from the target client, the cache is updated with the maximum write operation latency, the total number of bytes in the write operation, the total duration of the write operation, and the average write operation latency of the target client, all stored in the cache according to the current write operation of the target client.

[0017] Preferred options also include:

[0018] Upon receiving a performance viewing command from a user for the target client, the system feeds back to the user all the performance parameters corresponding to the target client stored in the cache.

[0019] Preferred options also include:

[0020] Record the maximum value of the maximum read operation latency of the target client determined in each iteration, the maximum value of the average read operation latency of the target client determined in each iteration, the maximum value of the maximum write operation latency of the target client determined in each iteration, and the maximum value of the average write operation latency of the target client determined in each iteration, and provide feedback to the user when a performance viewing command for the target client is received from the user.

[0021] Preferred options also include:

[0022] In the data stored on the HDFS server, the operation files belonging to the target client are counted;

[0023] Specifically, for any one block of data stored on the HDFS server, if the last data write operation on the block was initiated by the target client, the block belongs to the target client's operation file; or, for any one block of data stored on the HDFS server, if the block has been written to by the target client, the block belongs to the target client's operation file.

[0024] Preferred options also include:

[0025] Upon receiving a write data request from the target client, determine whether the target client has operation permissions for the write object;

[0026] If not, the write data request from the target client is ignored, and a message indicating write failure is sent back to the target client.

[0027] A client-side performance statistics system, applied to an HDFS server, includes:

[0028] The total data operation byte accumulation statistics module is used to periodically read the total data operation byte accumulation of the target client stored in the buffer according to a preset first time interval;

[0029] The bandwidth data update module is used to determine the difference between the current total number of bytes accumulated from the data operations of the target client and the previous total number of bytes accumulated from the data operations of the target client, and divide the difference by the first time interval to obtain the bandwidth data of the target client;

[0030] The bandwidth data feedback module is used to feed back the bandwidth data of the target client obtained in each instance to the user when the user sends a bandwidth viewing command for the target client.

[0031] A client-side performance statistics device, comprising:

[0032] Memory, used to store computer programs;

[0033] A processor for executing the computer program to implement the client performance statistics method as described above.

[0034] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the client performance statistics method described above.

[0035] By applying the technical solution provided in this invention, the HDFS server can periodically read the total accumulated data operation bytes of the target client stored in the cache according to a preset first time interval. Therefore, after reading the total accumulated data operation bytes of the target client stored in the cache, the difference between the currently read total accumulated data operation bytes and the previously read total accumulated data operation bytes is calculated. This difference is then divided by the first time interval to obtain the bandwidth data for the target client, effectively reflecting the bandwidth of the target client and thus realizing performance statistics for the target client. Furthermore, when receiving a bandwidth viewing command for the target client sent by the user, this application does not feed back the most recently calculated bandwidth data, but rather feeds back the bandwidth data of the target client obtained in each iteration. This allows the user to easily determine the bandwidth changes of the target client, enabling the user to have a more comprehensive understanding of the bandwidth performance of the target client and facilitating related work.

[0036] In summary, for HDFS, the solution proposed in this application can effectively achieve performance statistics for the target client, and it feeds back the bandwidth data of the target client obtained in each iteration to the user. This allows the user to easily determine the bandwidth changes of the target client, thereby gaining a more comprehensive understanding of the bandwidth performance of the target client and facilitating related work for the user. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the implementation of a client-side performance statistics method according to the present invention.

[0039] Figure 2 This is a schematic diagram of the structure of a client-side performance statistics system according to the present invention;

[0040] Figure 3 This is a schematic diagram of the structure of a client-side performance statistics device according to the present invention. Detailed Implementation

[0041] The core of this invention is to provide a client performance statistics method that can effectively perform performance statistics on the target client and feed back the bandwidth data of the target client obtained in each iteration to the user. This allows the user to easily determine the bandwidth changes of the target client, thereby gaining a more comprehensive understanding of the target client's bandwidth performance and facilitating related work.

[0042] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the implementation of a client-side performance statistics method according to the present invention. This client-side performance statistics method can be applied to HDFS servers and includes the following steps:

[0044] Step S101: Periodically read the total number of bytes of data operations of the target client stored in the buffer according to the preset first time interval.

[0045] Specifically, HDFS clients can take many forms, such as web-based or command-line interfaces. There are no restrictions on where the HDFS client runs, as long as the machine running the client can communicate with the HDFS server.

[0046] The target client of this application can be any one of the various HDFS clients. In other words, the performance statistics of any one client can be achieved using the scheme of this application. This application uses one of these clients, namely the target client, as an example to illustrate the scheme.

[0047] For the target client, the cache stores the total number of bytes accumulated from data operations performed by the target client. Understandably, after each HDFS data node powers on and starts up, when the target client has not performed any read / write operations, the total number of bytes accumulated from data operations performed by the target client in the cache is 0. As the target client begins to perform read / write operations, the total number of bytes accumulated from data operations performed by the target client will continuously increase.

[0048] HDFS has multiple servers, and any server can be used to execute the scheme in this application.

[0049] The specific value of the first time interval can be set and adjusted as needed, without affecting the implementation of the present invention.

[0050] Step S102: Determine the difference between the total number of bytes accumulated from the current data operation of the target client and the total number of bytes accumulated from the previous data operation of the target client, and divide the difference by the first time interval to obtain the bandwidth data of the target client.

[0051] Understandably, when the total number of bytes accumulated from the target client's data operation is read for the first time, bandwidth calculation is not required this time since there is no data from the previous operation.

[0052] For example, in a specific scenario, after the HDFS data nodes power on and start up, if the target client does not perform any read / write operations, the first time the total number of bytes accumulated from the target client's data operations stored in the cache is read, the value read will be 0. For example, 0 can be stored in the first interval checker. Then, for example, if the target client performs one read operation that requires reading 5 blocks of data, and this read operation is completed within the first time interval, the total number of bytes accumulated from the target client's data operations stored in the cache will become 5. That is, the second time the total number of bytes accumulated from the target client's data operations stored in the cache is read, the value read will be 5. For example, 5 can be stored in the second interval checker. Then, subtracting the data 0 from the data in the first interval checker from the data 5 in the second interval checker, and dividing the difference by the first time interval, gives the target client's bandwidth data. For example, the locally obtained bandwidth data can be recorded as x1.

[0053] After obtaining the bandwidth data x1, the data in the first interval value checker can be replaced with the data in the second interval value checker so that the bandwidth data can be calculated next time. For example, in the above example, after obtaining the bandwidth data x1, 5 is stored in the first interval value checker.

[0054] After obtaining the bandwidth data x1, for example, if the target client performs a write operation that requires writing 15 blocks of data, and this write operation is completed within the first time interval, then after the write operation is completed, the total number of bytes of data operations of the target client stored in the buffer becomes 5 + 15 = 20. That is to say, after reading the total number of bytes of data operations of the target client stored in the buffer for the third time, the value read is 20. This value can be stored in the second interval value checker. Then, the data 20 in the second interval value checker is subtracted from the data 5 in the first interval value checker, and the difference 15 is divided by the first time interval to obtain the bandwidth data of the target client. For example, the bandwidth data obtained locally can be recorded as x2.

[0055] Similarly, after obtaining the bandwidth data x2, the data in the first interval value checker can be replaced with the data in the second interval value checker so that the bandwidth data can be calculated next time. For example, in the above example, after obtaining the bandwidth data x2, 20 is stored in the first interval value checker.

[0056] After obtaining the bandwidth data x2, for example, if the target client performs another read operation, this read operation requires reading 200 blocks of data, and for example, after the first time interval, only 150 blocks of data are read, then after the first time interval, the total number of bytes of data operations of the target client stored in the buffer becomes 5 + 15 + 150 = 170. That is to say, after the fourth read of the total number of bytes of data operations of the target client stored in the buffer, the value read is 170. This value can be stored in the second interval value checker. Then, the data 170 in the second interval value checker is subtracted from the data 20 in the first interval value checker, and the difference 150 is divided by the first time interval to obtain the bandwidth data of the target client. For example, the bandwidth data obtained locally can be recorded as x3.

[0057] Similarly, after obtaining the bandwidth data x3, the data in the first interval value checker can be replaced with the data in the second interval value checker so that the bandwidth data can be calculated next time. For example, in the above example, after obtaining the bandwidth data x3, 170 is stored in the first interval value checker.

[0058] Step S103: Upon receiving a bandwidth viewing command from a user for the target client, the bandwidth data of the target client obtained in each instance is fed back to the user.

[0059] As described in step S102, the solution of this application can determine the bandwidth data of the target client within the first time interval each time interval has elapsed. Therefore, upon receiving a bandwidth viewing instruction for the target client sent by the user, the bandwidth data of the target client obtained in each instance can be fed back to the user.

[0060] Since this application does not provide feedback on the most recently calculated bandwidth data, but instead provides feedback on the bandwidth data of the target client obtained in each calculation, it allows users to easily determine the bandwidth changes of the target client, thereby gaining a more comprehensive understanding of the target client's bandwidth performance and facilitating related work for the user.

[0061] For example, after reviewing the bandwidth data of the target client obtained each time, if a user finds that the bandwidth is very low during a certain period, and that the user did send read / write data requests for a large amount of data during that period, the user can then conduct targeted troubleshooting for that period to determine the cause of the problem.

[0062] In one specific embodiment of the present invention, step S103, which describes feeding back the bandwidth data of the target client obtained in each iteration to the user, may specifically include:

[0063] The bandwidth data of the target client obtained in each iteration is fed back to the user in the form of a bandwidth change diagram, and the maximum value of the bandwidth data of the target client obtained in each iteration is displayed.

[0064] In this implementation, to facilitate users in viewing bandwidth data, when a bandwidth viewing command for a target client is received from a user, the bandwidth data of the target client obtained at each time is fed back to the user in the form of a bandwidth change diagram. For example, the bandwidth change diagram can be a simple line graph or a more complex fitted curve diagram.

[0065] In addition, this implementation also displays the maximum value of the bandwidth data of the target client obtained in each iteration, so that users can quickly know the approximate maximum bandwidth of the client, that is, the approximate bandwidth limit of the target client, which facilitates the user's work.

[0066] Of course, in other specific implementations, the bandwidth data of the target client obtained each time can be processed in other ways before being displayed to the user to assist the user's related work. Of course, the specific data processing method can be set and adjusted according to actual needs, without affecting the implementation of the present invention.

[0067] In one specific embodiment of the present invention, it further includes:

[0068] When a read data request is received from the target client, the maximum read operation latency, the total number of bytes read by the target client, the total read operation duration, and the average read operation latency of the target client stored in the cache are updated according to the current read operation of the target client.

[0069] When a write data request is received from the target client, the maximum write operation latency, the total number of bytes in the write operation, the total write operation duration, and the average write operation latency of the target client stored in the cache are updated according to the write operation of the target client.

[0070] This implementation takes into account that the client's bandwidth is an important performance indicator of the client. However, in some cases, in addition to the client's bandwidth, users may also want to know other performance indicators of the client.

[0071] In this implementation, the maximum read operation latency, total number of bytes read, total read operation duration, and average read operation latency of the target client can be calculated, so that users can obtain more comprehensive performance information of the target client, further facilitating their related work.

[0072] For example, when a data request is received from a target client, it can first determine whether the data request is a read data request or a write data request, and then update the corresponding data stored in the cache according to the determination result.

[0073] Taking the receipt of a read data request from the target client as an example, the maximum read operation latency of the target client stored in the cache can be updated according to the current read operation of the target client. When determining the maximum read operation latency, it is usually based on a single block, that is, the latency of each block read by the target client is calculated, and the maximum value is taken as the maximum read operation latency of the target client and stored in the cache.

[0074] The total number of bytes read indicates the total number of bytes read in this read operation. The total read operation duration indicates the total time taken for this read operation. The average read operation latency indicates the average latency of reading each block.

[0075] Furthermore, in this implementation, upon receiving a read data request from the target client, the maximum read operation latency, total number of bytes read, total read operation duration, and average read operation latency of the target client are updated according to the target client's current read operation. It is understood that, unlike the implementation described above which only updates the total number of bytes accumulated during the target client's data operations, this update is performed every first time interval, rather than waiting until the read operation is completed.

[0076] Similarly, when a read data request is received from the target client, a write data request is received from the target client, and the various performance parameters related to the write operation of the target client stored in the cache are updated according to the write operation of the target client.

[0077] In one specific embodiment of the present invention, it may further include:

[0078] When a user sends a command to view the performance of a target client, the system will send back all the performance parameters corresponding to the target client stored in the cache to the user.

[0079] In this implementation, when a user sends a performance viewing command for the target client, all the content corresponding to the target client stored in the cache can be fed back to the user. That is, all the performance parameters corresponding to the target client can be fed back to the user so that the user can know a more comprehensive performance information of the target client, which facilitates the user's related work.

[0080] For example, the cache can store the maximum read latency, total number of bytes in read operations, total read operation duration, average read operation latency, maximum write latency, total number of bytes in write operations, total write operation duration, average write operation latency, and the total accumulated data operation bytes corresponding to the target client, all of which can be fed back to the user. Furthermore, it is understood that in addition to the performance parameters of the target client described herein, other implementations may store other types of performance parameters of the target client in the cache, without affecting the implementation of the present invention, such as the number of write operations per second and the number of read operations per second of the target client, etc.

[0081] In one specific embodiment of the present invention, it may further include:

[0082] Record the maximum value of the maximum read latency, the maximum value of the average read latency, the maximum value of the maximum write latency, and the maximum value of the average write latency of the target client for each identified target client. Provide feedback to the user when a performance monitoring command for the target client is received.

[0083] This implementation takes into account that, in the example above, after each read operation by the target client, the maximum read operation latency and the average read operation latency stored in the cache will be updated. In other words, the maximum read operation latency and the average read operation latency stored in the cache are values ​​calculated based on the latest read request.

[0084] In some cases, users may want to know the maximum latency of a single block read operation in each read request of the target client, as well as the maximum average latency of read operations in each read request. Therefore, in this implementation, the maximum value of the maximum latency of read operations of the target client determined in each instance and the maximum value of the average latency of read operations of the target client determined in each instance are recorded. These two values ​​are then fed back to the user so that the user can have a more comprehensive and clear understanding of the performance of the target client, which facilitates the user's related work.

[0085] Similarly, in some situations, users may want to know the maximum latency of a single block write operation in each write request of the target client, as well as the maximum average latency of write operations in each write request. Therefore, in this implementation, the maximum value of the maximum latency of each determined write operation of the target client and the maximum value of the average latency of each determined write operation of the target client are recorded, and these two values ​​are also fed back to the user so that the user can have a more comprehensive and clear understanding of the performance of the target client, which facilitates the user's related work.

[0086] In one specific embodiment of the present invention, it may further include:

[0087] Within the data stored on the HDFS server, identify the operation files belonging to the target client;

[0088] Specifically, for any block of data stored on the HDFS server, if the last data write operation on the block was initiated by the target client, the block belongs to the target client's operation file; or, for any block of data stored on the HDFS server, if the block has been written to by the target client, the block belongs to the target client's operation file.

[0089] This implementation takes into account that different clients may be interested in different data in the HDFS file system. In order to make it easier for users to understand which part of the HDFS file system the target client is interested in, this implementation will count the operation files belonging to the target client.

[0090] Since different clients may perform read and write operations on data in the same storage location, in this implementation, for any one block, if the initiator of the last data write operation of that block is the target client, then that block can be regarded as an operation file belonging to the target client.

[0091] Of course, in some cases, a block can be defined as an operation file belonging to the target client as long as it has been written to. This approach is beneficial for displaying the target client's historical write operations, but its specificity is relatively weak. The former implementation, on the other hand, can more accurately reflect the data that the target client has recently been paying attention to. In practical applications, the choice can be made according to actual needs.

[0092] In one specific embodiment of the present invention, it may further include:

[0093] When a write data request is received from the target client, determine whether the target client has the permission to operate on the write object;

[0094] If not, ignore the write request from the target client and send a message indicating that the write failed to the target client.

[0095] As described above, HDFS clients can take many forms, such as web-based or command-line interfaces. There are no restrictions on where the HDFS client runs, as long as the machine running the client can communicate with the HDFS server.

[0096] This implementation takes into account the diverse forms of HDFS clients. To ensure HDFS data security, when a write request is received from a target client, a certain degree of write restriction is imposed. Specifically, it determines whether the target client has the necessary permissions to write the object. If not, the write request is ignored, and a write failure message is sent to the target client, such as: "Write failed, write operation permission not granted."

[0097] For example, in some situations, for certain critical data, it can be configured to prevent any client from writing data by sending write requests. For some relatively important data, it can be configured to allow only specific clients to write data by sending write requests. Of course, the specific data importance classification and the permission classification for different clients can be set and adjusted according to the actual situation without affecting the implementation of this invention.

[0098] By applying the technical solution provided in this invention, the HDFS server can periodically read the total accumulated data operation bytes of the target client stored in the cache according to a preset first time interval. Therefore, after reading the total accumulated data operation bytes of the target client stored in the cache, the difference between the currently read total accumulated data operation bytes and the previously read total accumulated data operation bytes is calculated. This difference is then divided by the first time interval to obtain the bandwidth data for the target client, effectively reflecting the bandwidth of the target client and thus realizing performance statistics for the target client. Furthermore, when receiving a bandwidth viewing command for the target client sent by the user, this application does not feed back the most recently calculated bandwidth data, but rather feeds back the bandwidth data of the target client obtained in each iteration. This allows the user to easily determine the bandwidth changes of the target client, enabling the user to have a more comprehensive understanding of the bandwidth performance of the target client and facilitating related work.

[0099] In summary, for HDFS, the solution proposed in this application can effectively achieve performance statistics for the target client, and it feeds back the bandwidth data of the target client obtained in each iteration to the user. This allows the user to easily determine the bandwidth changes of the target client, thereby gaining a more comprehensive understanding of the bandwidth performance of the target client and facilitating related work for the user.

[0100] Corresponding to the above method embodiments, this invention also provides a client-side performance statistics system, which can be referred to in conjunction with the above.

[0101] See Figure 2 The diagram shown is a structural schematic of a client-side performance statistics system according to the present invention, applied in an HDFS server, including:

[0102] The total data operation byte accumulation statistics module 201 is used to periodically read the total data operation byte accumulation of the target client stored in the buffer according to a preset first time interval;

[0103] The bandwidth data update module 202 is used to determine the difference between the total number of bytes accumulated from the current data operation of the target client and the total number of bytes accumulated from the previous data operation of the target client, and divide the difference by the first time interval to obtain the bandwidth data of the target client;

[0104] The bandwidth data feedback module 203 is used to feed back the bandwidth data of the target client obtained in each instance to the user when it receives a bandwidth viewing command sent by the user for the target client.

[0105] In one specific embodiment of the present invention, the bandwidth data feedback module 203 feeds back the bandwidth data of the target client obtained in each iteration to the user, including:

[0106] The bandwidth data of the target client obtained in each iteration is fed back to the user in the form of a bandwidth change diagram, and the maximum value of the bandwidth data of the target client obtained in each iteration is displayed.

[0107] In one specific embodiment of the present invention, it further includes:

[0108] The read data metrics update module is used to update the maximum read operation latency, total number of bytes read by the target client, total read operation duration, and average read operation latency of the target client stored in the cache when a read data request is received from the target client.

[0109] The write data metrics update module is used to update the cache with the maximum write operation latency, total number of bytes written by the target client, total write operation duration, and average write operation latency of the target client when a write data request is received from the target client.

[0110] In one specific embodiment of the present invention, it further includes:

[0111] The metrics feedback module is used to send back all the performance parameters corresponding to the target client stored in the cache to the user when it receives a performance viewing command sent by the user for the target client.

[0112] In one specific embodiment of the present invention, the indicator feedback module is further configured to:

[0113] Record the maximum value of the maximum read latency, the maximum value of the average read latency, the maximum value of the maximum write latency, and the maximum value of the average write latency of the target client for each identified target client. Provide feedback to the user when a performance monitoring command for the target client is received.

[0114] In one specific embodiment of the present invention, it further includes: a target client-related operation file statistics module, used for:

[0115] Within the data stored on the HDFS server, identify the operation files belonging to the target client;

[0116] Specifically, for any block of data stored on the HDFS server, if the last data write operation on the block was initiated by the target client, the block belongs to the target client's operation file; or, for any block of data stored on the HDFS server, if the block has been written to by the target client, the block belongs to the target client's operation file.

[0117] In one specific embodiment of the present invention, an authorization verification module is further included, used for:

[0118] When a write data request is received from the target client, determine whether the target client has the permission to operate on the write object;

[0119] If not, ignore the write request from the target client and send a message indicating that the write failed to the target client.

[0120] Corresponding to the methods and system embodiments described above, this invention also provides a client performance statistics device and a computer-readable storage medium, which can be referred to in conjunction with the above description. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the client performance statistics method as described in any of the above embodiments. The computer-readable storage medium referred to herein includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0121] See Figure 3 The diagram shown is a structural schematic of a client-side performance statistics device according to the present invention, applied in an HDFS server, comprising:

[0122] Memory 301 is used to store computer programs;

[0123] Processor 302 is configured to execute a computer program to implement the steps of the client performance statistics method as described in any of the above embodiments.

[0124] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0125] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0126] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for performance statistics of a client, characterized in that, Servers used in HDFS include: According to a preset first time interval, periodically read the total number of bytes of data operations from the target client stored in the cache. The difference between the total number of bytes accumulated from the current data operations of the target client and the total number of bytes accumulated from the previous data operations of the target client is determined, and the difference is divided by the first time interval to obtain the bandwidth data of the target client. Upon receiving a bandwidth viewing command from a user for the target client, the system will return all bandwidth data of the target client obtained in each instance to the user. The step of feeding back the bandwidth data of the target client obtained in each iteration to the user includes: The bandwidth data of the target client obtained in each iteration is fed back to the user in the form of a bandwidth change diagram, and the maximum value of the bandwidth data of the target client obtained in each iteration is displayed. Also includes: When a read data request is received from the target client, the maximum latency of the read operation, the total number of bytes of the read operation, the total duration of the read operation, and the average latency of the read operation stored in the cache are updated according to the current read operation of the target client. When a write data request is received from the target client, the maximum latency of the write operation of the target client, the total number of bytes of the write operation of the target client, the total duration of the write operation of the target client, and the average latency of the write operation of the target client stored in the cache are updated according to the write operation of the target client in this operation. Also includes: In the data stored on the HDFS server, the operation files belonging to the target client are counted; Specifically, for any one block of data stored on the HDFS server, if the last data write operation on the block was initiated by the target client, the block belongs to the target client's operation file; or, for any one block of data stored on the HDFS server, if the block has been written to by the target client, the block belongs to the target client's operation file. Also includes: The maximum read operation latency, average read operation latency, maximum write operation latency, and average write operation latency stored in the cache are updated. The maximum value of the maximum read operation latency, the maximum value of the average read operation latency, the maximum write operation latency, and the maximum write operation latency of the target client determined in each iteration are recorded. These values ​​are fed back to the user when a performance viewing command for the target client is received.

2. The client performance statistics method according to claim 1, characterized in that, Also includes: Upon receiving a performance viewing command from a user for the target client, the system feeds back to the user all the performance parameters corresponding to the target client stored in the cache.

3. The client performance statistics method according to claim 1 or 2, characterized in that, Also includes: When a write data request is received from the target client, it is determined whether the target client has operation permissions for the write object; If not, the write data request from the target client is ignored, and a message indicating write failure is sent back to the target client.

4. A client-side performance statistics system, characterized in that, Servers used in HDFS include: The total data operation byte accumulation statistics module is used to periodically read the total data operation byte accumulation of the target client stored in the buffer according to a preset first time interval; The bandwidth data update module is used to determine the difference between the current total number of bytes accumulated from the data operations of the target client and the previous total number of bytes accumulated from the data operations of the target client, and divide the difference by the first time interval to obtain the bandwidth data of the target client; The bandwidth data feedback module is used to feed back the bandwidth data of the target client obtained in each instance to the user when the user sends a bandwidth viewing instruction for the target client. The bandwidth data of the target client obtained in each iteration is fed back to the user, including: The bandwidth data of the target client obtained in each iteration is fed back to the user in the form of a bandwidth change diagram, and the maximum value of the bandwidth data of the target client obtained in each iteration is displayed. Also includes: The read data metric update module is used to update the maximum read operation latency, total number of bytes read by the target client, total read operation duration, and average read operation latency of the target client stored in the cache when a read data request is received from the target client. The write data metric update module is used to update the maximum write operation latency, total number of bytes of the write operation, total duration of the write operation, and average write operation latency of the target client stored in the cache when a write data request is received from the target client. It also includes a target client operation file statistics module, which is used to count the operation files belonging to the target client in the data stored on the HDFS server; Specifically, for any one block of data stored on the HDFS server, if the last data write operation on the block was initiated by the target client, the block belongs to the target client's operation file; or, for any one block of data stored on the HDFS server, if the block has been written to by the target client, the block belongs to the target client's operation file. Also includes: The maximum read operation latency, average read operation latency, maximum write operation latency, and average write operation latency stored in the cache are updated. The maximum value of the maximum read operation latency, the maximum value of the average read operation latency, the maximum write operation latency, and the maximum write operation latency of the target client determined in each iteration are recorded. These values ​​are fed back to the user when a performance viewing command for the target client is received.

5. A client-side performance statistics device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the client performance statistics method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the client performance statistics method as described in any one of claims 1 to 3.

Images