Data statistics method and device in distributed system, equipment and storage medium

Abstract

The invention discloses a data statistics method, device and equipment in a distributed system and a storage medium, the distributed system comprises a master node and a plurality of slave nodes, the method is applied to any slave node and comprises the steps that a target sub-task sent by the master node is received, and the target sub-task is obtained by performing sub-task division on an application task by the master node; executing the target subtask, and obtaining an execution log generated in the execution process of the target subtask; locking the local log file; if the locking is successful, storing the temporary log and the execution log before the locking time to a local log file; if the locking fails, taking the execution log as a temporary log to generate a temporary log file; and uploading the local log file to the master node according to a log pulling request issued by the master node, so that the master node performs merging processing on the local log file of each slave node to obtain the log file of the application task. According to the scheme, performance data statistics of any application software can be realized in a distributed system, and the method has relatively high application value.

Description

Technical Field

[0001] The present application relates to the technical field of distributed systems, and in particular to a data statistics method in a distributed system, and also to a data statistics device, an electronic device and a computer-readable storage medium in a distributed system. Background Art

[0002] In today's Internet era, distributed systems have become the key infrastructure supporting various large-scale applications. With the rapid growth of the number of users and the increasing complexity of business, the requirements for system performance are also getting higher and higher. As the endpoint that directly interacts with users, the performance of the client directly affects the user experience and the success of the business. Accurate and comprehensive statistics of client performance data are of vital importance for understanding the operation status of the system, discovering potential problems, optimizing system performance, and ensuring service quality.

[0003] When submitting tasks for execution, some distributed system computing applications will divide the application tasks from a master node into multiple subtasks. These subtasks are distributed to each slave node in the cluster for running and computing, and the final running results are fed back to the master node. In this process, some software has performance statistics functions, while others do not. In the process of displaying the final results, or in the process of technical personnel locating and analyzing problems, the performance data during the task operation is a very important part of the data. However, due to the isolation between the distributed cluster client nodes, this part of the report is difficult to generate. The existing statistical reports are nothing more than storing the performance data of the subtasks running on this node in their own nodes. Due to the huge differences between the various application software, there is no unified management mechanism in the distributed system, so each application software has to maintain its own set of performance statistics codes, which is very inconvenient and not conducive to the development of distributed computing systems.

[0004] Therefore, how to implement performance data statistics of any application software in a distributed system, ensure the system performance of the distributed system, and improve its application value is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0005] The purpose of the present application is to provide a data statistics method in a distributed system, which can realize performance data statistics of any application software in the distributed system, ensure the system performance of the distributed system, and improve its application value; another purpose of the present application is to provide a data statistics device, electronic device, computer-readable storage medium and computer program product in a distributed system, all of which have the above-mentioned beneficial effects.

[0006] In a first aspect, the present application provides a data statistics method in a distributed system, the distributed system comprising a master node and a plurality of slave nodes, the master node being communicatively connected with each of the slave nodes, the method being applied to any slave node in the distributed system, comprising:

[0007] Receiving a target subtask sent by the master node; wherein the target subtask is obtained by the master node dividing the application task into subtasks;

[0008] Execute the target subtask and obtain the execution log generated during the execution of the target subtask;

[0009] Lock the local log file and determine the locking result;

[0010] If the locking result is successful, the locking time is determined, and the temporary log before the locking time and the execution log are saved to a local log file;

[0011] If the locking result is a locking failure, the execution log is used as a temporary log to generate a corresponding temporary log file;

[0012] When receiving the log pull request sent by the master node, the local log file is uploaded to the master node according to the log pull request, so that the master node merges the local log files uploaded by each slave node to obtain the log file of the application task.

[0013] Optionally, if the locking result is a locking failure, the execution log is used as a temporary log to generate a corresponding temporary log file, including:

[0014] If the locking result is a locking failure, determining the generation time of the execution log;

[0015] Using the execution log as the temporary log to generate a corresponding temporary log file;

[0016] Determine a unique identifier corresponding to the temporary log file, and name the temporary log file using the generation time and the unique identifier;

[0017] Accordingly, if the temporary log before the lock time and the execution log are saved to a local log file, it includes:

[0018] Determine the target temporary log file before the lock time according to the time information in the file name of each temporary log file;

[0019] Saving the temporary log and the execution log in each of the target temporary log files to the local log file;

[0020] Correspondingly, if the temporary log before the lock time and the execution log are saved to the local log file, it also includes:

[0021] Each of the target temporary log files is cleared and deleted.

[0022] Optionally, the data statistics method in the distributed system further includes:

[0023] Performing heartbeat monitoring on the master node to determine the operating status of the master node;

[0024] When the running state of the master node is abnormal, other slave nodes in the distributed system are united to elect a new master node.

[0025] Optionally, performing heartbeat monitoring on the master node to determine the running status of the master node includes:

[0026] Send a heartbeat signal to the master node, and determine whether a heartbeat response signal fed back by the master node is received;

[0027] If the heartbeat response signal fed back by the master node is received, it is determined that the operating state of the master node is normal;

[0028] If the heartbeat response signal fed back by the master node is not received, the current heartbeat monitoring times are counted;

[0029] If the current heartbeat monitoring times do not reach the preset times, return to the step of sending a heartbeat signal to the master node to determine whether a heartbeat response signal fed back by the master node is received;

[0030] If the current number of heartbeat monitoring times reaches the preset number, it is determined that the operating state of the master node is the abnormal state.

[0031] Optionally, when the running state of the master node is abnormal, electing a new master node in conjunction with other slave nodes in the distributed system includes:

[0032] When the running state of the master node is abnormal, sending a master node election notification to each of the other slave nodes in the distributed system;

[0033] Receive the election results fed back by the other slave nodes, and determine the slave node with the highest number of votes as the new master node according to the election results.

[0034] Optionally, the data statistics method in the distributed system further includes:

[0035] Receiving the log file of the application task sent by the master node;

[0036] The log file is used to synchronize data of the local log file.

[0037] Optionally, the data statistics method in the distributed system further includes:

[0038] When receiving a log cleaning notification sent by the master node, clearing the local log file according to the log cleaning notification;

[0039] The log cleaning notification is sent by the master node to each of the slave nodes after receiving the local log files uploaded by all the slave nodes.

[0040] In a second aspect, the present application further discloses a data statistics device in a distributed system, the distributed system comprising a master node and a plurality of slave nodes, the master node being communicatively connected with each of the slave nodes, the device being applied to any slave node in the distributed system, comprising:

[0041] A receiving module, configured to receive a target subtask sent by the master node; wherein the target subtask is obtained by the master node dividing the application task into subtasks;

[0042] An execution module, used to execute the target subtask and obtain the execution log generated during the execution of the target subtask;

[0043] The locking module is used to lock the local log file and determine the locking result;

[0044] A saving module, for determining the locking time if the locking result is successful, and saving the temporary log before the locking time and the execution log to a local log file;

[0045] A generation module, configured to generate a corresponding temporary log file by using the execution log as a temporary log if the locking result is a locking failure;

[0046] The upload module is used to upload the local log file to the master node according to the log pull request when receiving the log pull request sent by the master node, so that the master node merges the local log files uploaded by each slave node to obtain the log file of the application task.

[0047] In a third aspect, the present application further discloses an electronic device, comprising:

[0048] Memory for storing computer programs;

[0049] A processor is used to implement the steps of any of the data statistics methods in the distributed system as described above when executing the computer program.

[0050] In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data statistics method in any distributed system as described above are implemented.

[0051] In a fifth aspect, the present invention further discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the data statistics methods in a distributed system as described above.

[0052] The present application provides a data statistics method in a distributed system, the distributed system comprising a master node and multiple slave nodes, the master node being communicatively connected with each of the slave nodes, the method being applied to any slave node in the distributed system, comprising: receiving a target subtask sent by the master node; wherein the target subtask is obtained by the master node dividing an application task into subtasks; executing the target subtask and obtaining an execution log generated by the target subtask during the execution process; locking a local log file and determining a locking result; if the locking result is a successful locking, determining a locking time, and saving a temporary log before the locking time and the execution log to a local log file; if the locking result is a failed locking, using the execution log as a temporary log to generate a corresponding temporary log file; when receiving a log pull request sent by the master node, uploading the local log file to the master node according to the log pull request, so that the master node merges the local log files uploaded by each of the slave nodes to obtain the log file of the application task.

[0053] Applying the technical solution provided by the present application, after the master node divides the application task into multiple target subtasks and sends them to different slave nodes, each slave node can record the log information generated by it to the local log file during the execution of the corresponding target subtask, and feed it back to the master node, so that the master node can obtain the log file of the application task by integrating the local log files fed back by each slave node, so as to realize the performance data statistics of the application task. Among them, in the process of log recording, in order to avoid the problem that the local log file is being read and written, resulting in log recording failure and log loss, the slave node can create a temporary log file for temporary recording of log information, and merge it into the local log file when the local log file is idle, and, in order to avoid the problem of concurrent duplication of temporary log files, the log merging operation is also realized with reference to the time information, so as to ensure the accuracy of the performance data statistics results. It can be seen that the technical solution can realize the performance data statistics of any application software in the distributed system, effectively ensure the system performance of the distributed system, and further improve its application value.

[0054] In one embodiment of the present application, since there is only one master node in the distributed system, once the master node fails, it will cause the interruption of the application tasks in the distributed system. To avoid this problem, a master node election mechanism is proposed. Any slave node in the distributed system can perform heartbeat monitoring on the master node to determine the running status of the master node. Once the running status of the master node is abnormal, the master node election mechanism is immediately triggered to re-elect a new master node from the remaining slave nodes to take over the management tasks of the old master node. It can be seen that this technical solution can effectively ensure that the master node is online in real time and avoid the problem of task interruption in the system.

[0055] The data statistics device, electronic device, computer-readable storage medium and computer program product in the distributed system provided by this application also have the above-mentioned technical effects, and this application will not elaborate on them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings required for use in the description of the prior art and the embodiments of the present application are briefly introduced below. Of course, the drawings described below in relation to the embodiments of the present application are only part of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present application.

[0057] Figure 1 A schematic diagram of the structure of a distributed system provided by this application;

[0058] Figure 2 A schematic diagram of a data statistics method in a distributed system provided by the present application;

[0059] Figure 3 A workflow diagram of a master node in a distributed system provided by this application;

[0060] Figure 4 A workflow diagram of a slave node in a distributed system provided by this application;

[0061] Figure 5 A schematic diagram of the structure of a data statistics device in a distributed system provided by the present application;

[0062] Figure 6 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0063] The core of this application is to provide a data statistics method in a distributed system, which can realize performance data statistics of any application software in the distributed system, ensure the system performance of the distributed system, and improve its application value; another core of this application is to provide a data statistics device, electronic device, computer-readable storage medium and computer program product in a distributed system, all of which have the above-mentioned beneficial effects.

[0064] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0065] An embodiment of the present application provides a data statistics method in a distributed system.

[0066] First, please refer to Figure 1 , Figure 1 This is a structural diagram of a distributed system provided by the present application, the distributed system includes a master node and a plurality of slave nodes, the master node and each slave node can be communicatively connected, and each slave node can be communicatively connected. The data statistics method in the distributed system provided by the present application is implemented based on the distributed system.

[0067] For further information, please refer to Figure 2 , Figure 2 A schematic diagram of a data statistics method in a distributed system provided by the present application, wherein the data statistics method in the distributed system is applied to Figure 1 Any slave node in the distributed system shown may include the following S101 to S106.

[0068] S101: Receive a target subtask sent by a master node; wherein the target subtask is obtained by dividing an application task into subtasks by the master node.

[0069] This step aims to achieve the acquisition of the target subtask, which is issued by the master node. Specifically, when the master node receives any application task, it will divide it into subtasks, obtain multiple target subtasks, and distribute them to multiple slave nodes (here refers to some or all slave nodes in the distributed system) to achieve parallel processing of all target subtasks based on multiple slave nodes to ensure the processing efficiency of application tasks.

[0070] In one embodiment of the present application, receiving a target subtask sent by a master node may include:

[0071] Receive the status collection request sent by the master node;

[0072] Determine the local current load based on the status collection request;

[0073] When the local current load exceeds the preset threshold, the feedback task rejection notification is sent to the master node to refuse to receive the target subtask sent by the master node;

[0074] When the local current load does not exceed the preset threshold, the feedback task receives a notification to the master node to receive the target subtask sent by the master node.

[0075] The embodiment of the present application provides a method for implementing a target subtask sent by a master node from a slave node, that is, determining whether to receive the target subtask sent by the master node based on its own actual load, so as to effectively ensure the stable operation of the distributed system. Specifically, before the master node sends the target subtask to the slave node, it can first send a status acquisition request to the slave node to determine whether the slave node receives the target subtask; for the target subtask, after receiving the status acquisition request sent by the master node, the local current load can be obtained. If the local current load is high, the target subtask sent by the master node can be refused; if the local current load is low, the target subtask sent by the master node can be received. Thus, each slave node determines whether to obtain and execute the target subtask based on its own actual load capacity, which can effectively ensure the stability of the distributed system.

[0076] It is understandable that the value of the above-mentioned preset threshold does not affect the implementation of the present technical solution, and can be set by technical personnel according to actual conditions, and this application does not impose any limitation on this.

[0077] S102: Execute the target subtask, and obtain the execution log generated during the execution of the target subtask.

[0078] This step aims to realize the execution of the target subtask and the recording of its execution log. Specifically, after receiving one or more target subtasks, the slave node can process them, and in the process of processing, the execution log corresponding to each target subtask can be recorded in real time. Among them, the execution log mainly includes various performance data of the corresponding target subtask, for example, it can include but not limited to CPU utilization, memory utilization, task interface call information, etc.

[0079] S103: Lock the local log file and determine the locking result.

[0080] S104: If the locking result is successful, determine the locking time, and save the temporary log and execution log before the locking time to the local log file.

[0081] S105: If the locking result is a locking failure, the execution log is used as a temporary log to generate a corresponding temporary log file.

[0082] The above steps are intended to achieve the preservation of the execution log of the target subtask. Specifically, the execution log generated by the execution of the target subtask from the node can be saved in the local log file. However, in a scenario with large concurrency, the local log file may be being read and written, resulting in the failure of writing the execution log to the local log file. Based on this, the local log file can be locked by locking the local log file to facilitate writing the execution log to the local log file. In the implementation process, after obtaining the execution log of the target subtask, the local log file can be locked. If the locking is successful, it means that the local log file is successfully locked, and the execution log of the target subtask can be saved to the local log file; if the locking fails, it means that the local log file has failed to lock, and the execution log of the target subtask cannot be saved to the local log file.

[0083] Furthermore, when the execution log of the target subtask cannot be saved to the local log file, a temporary log file can be created to temporarily store the execution log, and when the local log file is idle (when it can be successfully locked), it can be saved to the local log file together with the newly generated execution log. Based on this, when the local log file is successfully locked, the execution log of the target subtask can be saved to the local log file, and the temporary log in the temporary log file generated before the current lock time can also be saved to the local log file, so as to effectively ensure the integrity of the log information.

[0084] In one embodiment of the present application, if the locking result is a locking failure, the execution log is used as a temporary log to generate a corresponding temporary log file, which may include: if the locking result is a locking failure, determining the generation time of the execution log; using the execution log as a temporary log to generate a corresponding temporary log file; determining a unique identifier corresponding to the temporary log file, and naming the temporary log file using the generation time and the unique identifier;

[0085] Accordingly, if the temporary log and execution log before the lock time are saved to the local log file, it may include: determining the target temporary log file before the lock time according to the time information in the file name of each temporary log file; saving the temporary log and execution log in each target temporary log file to the local log file;

[0086] Correspondingly, after the temporary log and the execution log before the lock time are saved in the local log file, the method may further include: clearing each target temporary log file and deleting each target temporary log file.

[0087] It is understandable that in actual application scenarios, there may be a situation where multiple temporary log files are created. In order to effectively avoid concurrent duplication problems, the generation time and unique identifier of the execution log can be used to name the corresponding temporary log file to effectively ensure the uniqueness of each temporary log file. Thus, when the temporary log in the temporary log file is saved to the local log file, it can be implemented with reference to the time information in the file name of the temporary log file to effectively ensure the accuracy and uniqueness of the log data in the local log file. Furthermore, after the temporary log in the temporary log file is saved to the local log file, the temporary log file can also be emptied and deleted to effectively save storage resources and avoid unnecessary waste of storage resources.

[0088] S106: When receiving a log pull request sent by the master node, the local log file is uploaded to the master node according to the log pull request, so that the master node merges the local log files uploaded by each slave node to obtain the log file of the application task.

[0089] This step aims to achieve the upload processing of local log files. Specifically, the master node can send log pull requests to each slave node (here refers to the slave node that receives the target subtask) to obtain the execution logs generated by each slave node executing the corresponding target subtask, and merge them, thereby obtaining the complete log file of the application task.

[0090] In one embodiment of the present application, the data statistics method in the distributed system may further include: receiving a log file of an application task sent by a master node; and synchronizing data of a local log file using the log file.

[0091] The data statistics method in the distributed system provided by the embodiment of the present application can further realize data synchronization between the master and slave nodes, so that each slave node can obtain the processing status of the application task in a timely manner. Specifically, after the master node obtains the log file of the application task, it can distribute it to each slave node (here refers to the slave node that receives the target subtask), so that each slave node can refer to the log file to realize the data synchronization of the local log file and ensure the data consistency between multiple nodes.

[0092] In one embodiment of the present application, the data statistics method in the distributed system may also include: when a log cleaning notification sent by the master node is received, clearing the local log file according to the log cleaning notification; wherein, the log cleaning notification is sent by the master node to each slave node after receiving the local log files uploaded by all slave nodes.

[0093] The data statistics method in the distributed system provided by the embodiment of the present application can further realize the clearing of local log files. Specifically, when the application task is completed, the master node can send a log clearing notification to each slave node (here refers to the slave node that receives the target subtask), so that each slave node can respond to the log clearing notification and clear the local log file corresponding to the application task, so as to effectively save storage resources and avoid unnecessary waste of storage resources.

[0094] It can be seen that in the data statistics method in the distributed system provided by the embodiment of the present application, after the master node divides the application task into multiple target subtasks and sends them to different slave nodes, each slave node can record the log information generated by it to the local log file in the process of executing the corresponding target subtask, and feed it back to the master node, so that the master node can obtain the log file of the application task by integrating the local log files fed back by each slave node, so as to realize the performance data statistics of the application task. Among them, in the process of logging, in order to avoid the problem that the local log file is being read and written, causing the log record to fail and then causing the log loss, the slave node can create a temporary log file to temporarily record the log information, and merge it into the local log file when the local log file is idle, and, in order to avoid the problem of concurrent duplication of temporary log files, the log merging operation is also realized with reference to the time information, so as to ensure the accuracy of the performance data statistics results. It can be seen that the technical scheme can realize the performance data statistics of any application software in the distributed system, effectively ensure the system performance of the distributed system, and further improve its application value.

[0095] Based on the above embodiments:

[0096] In one embodiment of the present application, the data statistics method in the distributed system may further include:

[0097] Perform heartbeat monitoring on the master node to determine the running status of the master node;

[0098] When the running status of the master node is abnormal, other slave nodes in the joint distributed system elect a new master node.

[0099] The data statistics method in the distributed system provided by the embodiment of the present application can further implement a re-election scheme when the master node is abnormal. That is, the slave node can perform heartbeat monitoring on the master node to obtain the running status of the master node. If the master node is running normally, it can return to the step of performing heartbeat monitoring on the master node. If the master node cannot run normally, it can trigger the master node election mechanism to re-elect a new master node in conjunction with other slave nodes in the distributed system.

[0100] Obviously, in the embodiment of the present application, since there is only one master node in the distributed system, once the master node fails, it will cause the interruption of the application tasks in the distributed system. To avoid this problem, a master node election mechanism is proposed. Any slave node in the distributed system can perform heartbeat monitoring on the master node to determine the running status of the master node. Once the running status of the master node is abnormal, the master node election mechanism is immediately triggered to re-elect a new master node from the remaining slave nodes to take over the management tasks of the old master node. It can be seen that this technical solution can effectively ensure that the master node is online in real time and avoid the problem of task interruption in the system.

[0101] Among them, heartbeat monitoring of the master node to determine the operating status of the master node may include: sending a heartbeat signal to the master node to determine whether a heartbeat response signal fed back by the master node is received; if a heartbeat response signal fed back by the master node is received, determining that the operating status of the master node is normal; if no heartbeat response signal fed back by the master node is received, counting the current number of heartbeat monitoring times; if the current number of heartbeat monitoring times does not reach a preset number, returning to the step of sending a heartbeat signal to the master node to determine whether a heartbeat response signal fed back by the master node is received; if the current number of heartbeat monitoring times reaches a preset number, determining that the operating status of the master node is abnormal.

[0102] In order to effectively ensure the accuracy of heartbeat monitoring, the number of re-triggering times when the heartbeat is unresponsive can be set. That is to say, when the heartbeat monitoring is unresponsive, the heartbeat monitoring can be triggered multiple times. Only when there is no response in multiple heartbeat monitorings, can the main node be determined to be abnormal. In this way, the problem of misjudgment caused by unstable communication network can be effectively avoided. Among them, the specific value of the preset number of times does not affect the implementation of this technical solution, and can be set by technical personnel according to actual conditions. For example, it can be set to 3 times, and this application does not limit this.

[0103] Among them, when the running state of the master node is abnormal, the other slave nodes in the joint distributed system elect a new master node, which may include: when the running state of the master node is abnormal, sending a master node election notification to other slave nodes in the distributed system; receiving election results fed back by other slave nodes, and determining the slave node with the highest number of votes as the new master node based on the election results.

[0104] The embodiment of the present application provides a method for electing a new master node, which is based on a voting mechanism. That is, when the master node runs abnormally, all slave nodes can be triggered to vote, so that the slave node with the highest number of votes is used as the new master node.

[0105] Based on the above embodiments, an embodiment of the present application provides another data statistics method in a distributed system.

[0106] First, the basic idea of ​​the data statistics method in the distributed system provided by the embodiment of the present application mainly includes the following aspects:

[0107] (1) Performance data statistics:

[0108] CPU resource utilization: This can be measured by monitoring the ratio of the CPU's busy time per unit time to the total time. For example, you can use the performance monitoring tool provided by the operating system to obtain relevant data and calculate the CPU utilization.

[0109] Memory resource utilization: This can be determined by the ratio of used memory to total memory. The use of memory cache and buffers should also be considered to more accurately assess memory utilization efficiency.

[0110] Time statistics of distributed task interface calls: You can set up an interface monitoring function. You only need to enter the full path name of the corresponding function to monitor the number of calls of the function, the maximum, minimum and total call time. The average time of the function call can be calculated from the number of calls and the total time. The calculation method of each indicator is as follows:

[0111] Number of calls: Ops = ops_last_time + 1;

[0112] Call the minimum time: Min = Math.min(Min, thisOne);

[0113] The maximum time taken to call: Max = Math.max(Max, thisOne);

[0114] Total call time: Sum = Sum + thisOne.

[0115] (2) For a single slave node, each distributed subtask will generate its own statistical data. At this time, the data of a single node needs to be merged. The specific idea of ​​merging is: write the performance statistics log (execution log) to a local log file. In a scenario with large concurrency, the file may be being written or read. At this time, it can be written to a temporary file. To avoid concurrent duplication of temporary files, the temporary file naming rule is: timestamp + uuid format, which can ensure that the file name will not be repeated. When the subtask is executed here, if the local log file is not in use, it will be merged and deleted. This will not cause performance degradation due to the need to wait for the file lock to be released.

[0116] (3) Please refer to Figure 3 and Figure 4 , Figure 3 A workflow diagram of a master node in a distributed system provided by this application, Figure 4 A workflow diagram of slave nodes in a distributed system provided for this application. After the operation of step (2), only one performance statistics report file is generated for a distributed application on a single client node. Next, the local log files of each slave node need to be aggregated to the master node for summary output. The idea is: start a process on each slave node, and pull and aggregate local log files every minute. The first node in the etc / hosts file of the distributed system serves as the master node, and the other nodes serve as slave nodes. The master node is responsible for monitoring the status of each slave node. In addition, once the slave node detects that the master node has failed, it will re-elect and select a new master node from the remaining slave nodes. The master node is responsible for pulling the performance statistics logs of each distributed application of other slave nodes. After pulling the logs, it will merge the pulled log files with the node's own logs. When the merger is completed, the master node sends a request to each slave node to clear the current performance statistics log of the slave node. After one round of traversal, the master node will pull all the performance statistics information in the entire distributed system into a log file and send the log file to all slave nodes to ensure the data consistency of the distributed system. In addition, it can ensure that when the master node fails, the performance statistics work can still proceed normally after other slave nodes are elected as master nodes.

[0117] (4) Provide configuration parameters in advance and fill in the full path name of the function monitored by the distributed application in order to generate time-consuming statistical logs for these functions. Similarly, each step provides an external interface and a default implementation method. The provided external interface can be used by other developers for secondary development to meet customized individual needs.

[0118] Furthermore, the implementation process of the data statistics method in the distributed system is as follows:

[0119] (1) Write code to implement the performance data statistics function of distributed application subtasks, including CPU resource utilization, memory resource utilization, interface time consumption information statistics, etc.

[0120] (2) The performance data collected above is stored in a log file. This process requires locking the local file. If the file lock is seized successfully, the current data is merged with the previous data and written to the local file. If the lock is not seized successfully, the current data is written to a temporary file and merged with the subsequent subtasks.

[0121] (3) Code writing is complete. The master node monitors the running status of the slave nodes and pulls the statistical data of the slave nodes. After the merge is completed, it sends a log clearing signal to each slave node. In addition, the complete statistical data is sent to each slave node to ensure data consistency. The slave node receives the regular heartbeat information from the master node and responds to the information. When receiving the log clearing signal, it performs the cleanup operation and reports the processing results to the master node.

[0122] (4) Provide a unified framework and interface to the outside world, allowing for expansion, transformation and secondary development.

[0123] It can be seen that the data statistics method in the distributed system provided in the embodiment of the present application has the following advantages:

[0124] (1) In terms of performance optimization, accurate node data collection and cluster aggregation enable the system to fully and meticulously understand its own operating status. Each node collects client performance data separately, providing an accurate basis for locating performance bottlenecks. Whether it is an anomaly of a single node or a specific client connection problem, it can be quickly identified. At the same time, performance statistics of various distributed tasks help to deeply understand their operating characteristics, and then adjust resource allocation and task strategies in a targeted manner, greatly improving the overall performance of the system.

[0125] (2) For system stability, it provides early warning of potential risks. By continuously collecting data to establish performance baselines and trend models, it can promptly detect abnormal indicator changes, such as memory approaching the upper limit or task response time mutations, so that measures can be taken in advance to avoid failures. Moreover, after a failure occurs, detailed performance data can help quickly locate the root cause of the problem, greatly enhancing the maintainability of the system.

[0126] (3) From a business perspective, the business response speed has been significantly improved. Real-time performance data collection and analysis can quickly solve the problem of client request delay. Whether it is a problem with the network or the internal processing logic of the system, it can be accurately optimized to meet the user's demand for fast response, reduce user waiting time, and improve user satisfaction.

[0127] (4) In terms of business development support, comprehensive performance data provides a solid foundation for business expansion and innovation. By evaluating the system's carrying capacity based on performance data, we can rationally plan capacity expansion and upgrades to ensure smooth business growth. At the same time, we provide a set of architecture methods suitable for distributed applications, which can meet the requirements of distributed application performance statistics with minimal development.

[0128] An embodiment of the present application provides a data statistics device in a distributed system.

[0129] Please refer to Figure 5 , Figure 5This is a structural schematic diagram of a data statistics device in a distributed system provided by the present application, wherein the distributed system includes a master node and multiple slave nodes, the master node is communicatively connected with each slave node, and the data statistics device in the distributed system can be applied to any slave node in the distributed system, including:

[0130] Receiving module 1, used for receiving the target subtask sent by the master node; wherein the target subtask is obtained by the master node dividing the application task into subtasks;

[0131] Execution module 2, used to execute the target subtask and obtain the execution log generated during the execution of the target subtask;

[0132] Locking module 3, used to lock the local log file and determine the locking result;

[0133] The saving module 4 is used to determine the locking time if the locking result is successful, and save the temporary log and execution log before the locking time to the local log file;

[0134] Generating module 5, for generating a corresponding temporary log file by using the execution log as a temporary log if the locking result is a locking failure;

[0135] The upload module 6 is used to upload the local log file to the master node according to the log pull request when receiving the log pull request sent by the master node, so that the master node merges the local log files uploaded by each slave node to obtain the log file of the application task.

[0136] It can be seen that in the data statistics device in the distributed system provided by the embodiment of the present application, after the master node divides the application task into multiple target subtasks and sends them to different slave nodes, each slave node can record the log information generated by it to the local log file in the process of executing the corresponding target subtask, and feed it back to the master node, so that the master node can obtain the log file of the application task by integrating the local log files fed back by each slave node, so as to realize the performance data statistics of the application task. Among them, in the process of log recording, in order to avoid the problem that the local log file is being read and written, resulting in the failure of log recording and the loss of logs, a temporary log file can be created to temporarily record the log information, and the local log file can be merged into the local log file when the local log file is idle, and in order to avoid the problem of concurrent duplication of temporary log files, the log merging operation is also realized with reference to the time information, so as to ensure the accuracy of the performance data statistics results. It can be seen that the technical scheme can realize the performance data statistics of any application software in the distributed system, effectively ensure the system performance of the distributed system, and further improve its application value.

[0137] In one embodiment of the present application, the above-mentioned generation module 5 can be specifically used to determine the generation time of the execution log if the locking result is a locking failure; use the execution log as a temporary log to generate a corresponding temporary log file; determine the unique identifier corresponding to the temporary log file, and use the generation time and the unique identifier to name the temporary log file;

[0138] Accordingly, the above-mentioned saving module 4 can be specifically used to determine the target temporary log file before the lock time according to the time information in the file name of each temporary log file; save the temporary log and execution log in each target temporary log file to the local log file;

[0139] Correspondingly, the data statistics device in the distributed system may further include a clearing module for clearing and deleting each target temporary log file after saving the temporary log and execution log before the locking time to the local log file.

[0140] In one embodiment of the present application, the data statistics device in the distributed system may further include:

[0141] The monitoring module is used to perform heartbeat monitoring on the master node to determine the running status of the master node;

[0142] The election module is used to elect a new master node in conjunction with other slave nodes in the distributed system when the running status of the master node is abnormal.

[0143] In one embodiment of the present application, the above-mentioned monitoring module can be specifically used to send a heartbeat signal to the master node to determine whether a heartbeat response signal fed back by the master node is received; if a heartbeat response signal fed back by the master node is received, it is determined that the operating status of the master node is normal; if the heartbeat response signal fed back by the master node is not received, the current heartbeat monitoring times are counted; if the current heartbeat monitoring times do not reach the preset times, it returns to the step of sending a heartbeat signal to the master node to determine whether a heartbeat response signal fed back by the master node is received; if the current heartbeat monitoring times reaches the preset times, it is determined that the operating status of the master node is abnormal.

[0144] In one embodiment of the present application, the above-mentioned election module can be specifically used to send a master node election notification to other slave nodes in the distributed system when the operating status of the master node is abnormal; receive election results fed back by other slave nodes, and determine the slave node with the highest number of votes as the new master node based on the election results.

[0145] In one embodiment of the present application, the data statistics device in the distributed system may further include a synchronization module for receiving the log file of the application task sent by the master node; and synchronizing the data of the local log file using the log file.

[0146] In one embodiment of the present application, the data statistics device in the distributed system may also include a cleaning module, which is used to clear the local log file according to the log cleaning notification when it receives the log cleaning notification sent by the master node; wherein, the log cleaning notification is sent by the master node to each slave node after receiving the local log files uploaded by all slave nodes.

[0147] For an introduction to the device provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0148] An embodiment of the present application provides an electronic device.

[0149] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application, and the electronic device may include:

[0150] A memory 11, used for storing computer programs;

[0151] The processor 10 can implement the steps of any of the above-mentioned data statistics methods in a distributed system when executing a computer program.

[0152] like Figure 6 , which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0153] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0154] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of a data statistics method in a distributed system.

[0155] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0156] Receive the target subtask sent by the master node; wherein the target subtask is obtained by dividing the application task into subtasks by the master node;

[0157] Execute the target subtask and obtain the execution log generated during the execution of the target subtask;

[0158] Lock the local log file and determine the locking result;

[0159] If the locking result is successful, the locking time is determined, and the temporary log and execution log before the locking time are saved to the local log file;

[0160] If the locking result is a lock failure, the execution log is used as a temporary log to generate a corresponding temporary log file;

[0161] When a log pull request is received from the master node, the local log file is uploaded to the master node according to the log pull request, so that the master node merges the local log files uploaded by each slave node to obtain the log file of the application task.

[0162] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0163] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0164] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0165] Of course, it should be noted that Figure 6 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 6 More or fewer components than shown, or combinations of certain components.

[0166] An embodiment of the present application provides a computer-readable storage medium.

[0167] The computer-readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the steps of the data statistics method in any of the above-mentioned distributed systems can be implemented.

[0168] Among them, the computer-readable storage medium can be any available medium that can be stored in a computer or a data storage device such as a server or data center that includes one or more available media integrated. For example, it can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape, etc.), an optical medium (such as a DVD) or a semiconductor medium (such as a solid-state hard disk) and other media that can store computer program codes.

[0169] For an introduction to the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0170] An embodiment of the present application provides a computer program product.

[0171] The computer program product provided in the embodiments of the present application includes a computer program / instruction, which, when executed by a processor, can implement the steps of any of the above-mentioned data statistics methods in a distributed system.

[0172] Specifically, in the above embodiments, all or part of them can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of them can be implemented in the form of a computer program product.

[0173] Among them, the computer program product may include one or more computer programs / instructions, and when the computer program / instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application may be generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line, etc.) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0174] For an introduction to the computer program product provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0175] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0176] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0177] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0178] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples herein, and the description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A data statistics method in a distributed system, characterized in that: The distributed system includes a master node and multiple slave nodes, the master node is in communication connection with each of the slave nodes, and the method is applied to any slave node in the distributed system, including: Receiving a target subtask sent by the master node; wherein the target subtask is obtained by the master node dividing the application task into subtasks; Execute the target subtask and obtain the execution log generated during the execution of the target subtask; Lock the local log file and determine the locking result; If the locking result is successful, the locking time is determined, and the temporary log before the locking time and the execution log are saved to a local log file; If the locking result is a locking failure, the execution log is used as a temporary log to generate a corresponding temporary log file; When receiving the log pull request sent by the master node, the local log file is uploaded to the master node according to the log pull request, so that the master node merges the local log files uploaded by each slave node to obtain the log file of the application task.

2. The data statistics method in a distributed system according to claim 1, characterized in that: If the locking result is a locking failure, the execution log is used as a temporary log to generate a corresponding temporary log file, including: If the locking result is a locking failure, determining the generation time of the execution log; Using the execution log as the temporary log to generate a corresponding temporary log file; Determine a unique identifier corresponding to the temporary log file, and name the temporary log file using the generation time and the unique identifier; Accordingly, if the temporary log before the lock time and the execution log are saved to a local log file, it includes: Determine the target temporary log file before the lock time according to the time information in the file name of each temporary log file; Saving the temporary log and the execution log in each of the target temporary log files to the local log file; Correspondingly, if the temporary log before the lock time and the execution log are saved to the local log file, it also includes: Each of the target temporary log files is cleared and deleted.

3. The data statistics method in a distributed system according to claim 1 or 2, characterized in that: Also includes: Performing heartbeat monitoring on the master node to determine the operating status of the master node; When the running state of the master node is abnormal, other slave nodes in the distributed system are united to elect a new master node.

4. The data statistics method in a distributed system according to claim 3, characterized in that: Performing heartbeat monitoring on the master node to determine the operating status of the master node includes: Send a heartbeat signal to the master node, and determine whether a heartbeat response signal fed back by the master node is received; If the heartbeat response signal fed back by the master node is received, it is determined that the operating state of the master node is normal; If the heartbeat response signal fed back by the master node is not received, the current heartbeat monitoring times are counted; If the current heartbeat monitoring times do not reach the preset times, return to the step of sending a heartbeat signal to the master node to determine whether a heartbeat response signal fed back by the master node is received; If the current number of heartbeat monitoring times reaches the preset number, it is determined that the operating state of the master node is the abnormal state.

5. The data statistics method in a distributed system according to claim 3, characterized in that: When the running state of the master node is abnormal, the other slave nodes in the distributed system are combined to elect a new master node, including: When the running state of the master node is abnormal, sending a master node election notification to each of the other slave nodes in the distributed system; Receive the election results fed back by the other slave nodes, and determine the slave node with the highest number of votes as the new master node according to the election results.

6. The data statistics method in a distributed system according to claim 1, characterized in that: Also includes: Receiving the log file of the application task sent by the master node; The log file is used to synchronize data of the local log file.

7. The data statistics method in a distributed system according to claim 1, characterized in that: Also includes: When receiving a log cleaning notification sent by the master node, clearing the local log file according to the log cleaning notification; The log cleaning notification is sent by the master node to each of the slave nodes after receiving the local log files uploaded by all the slave nodes.

8. A data statistics device in a distributed system, characterized in that: The distributed system includes a master node and multiple slave nodes, the master node is in communication connection with each of the slave nodes, and the device is applied to any slave node in the distributed system, including: A receiving module, configured to receive a target subtask sent by the master node; wherein the target subtask is obtained by the master node dividing the application task into subtasks; An execution module, used to execute the target subtask and obtain the execution log generated during the execution of the target subtask; The locking module is used to lock the local log file and determine the locking result; A saving module, for determining the locking time if the locking result is successful, and saving the temporary log before the locking time and the execution log to a local log file; A generation module, configured to generate a corresponding temporary log file by using the execution log as a temporary log if the locking result is a locking failure; The upload module is used to upload the local log file to the master node according to the log pull request when receiving the log pull request sent by the master node, so that the master node merges the local log files uploaded by each slave node to obtain the log file of the application task.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data statistics method in a distributed system as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data statistics method in a distributed system as described in any one of claims 1 to 7 are implemented.

Images