Management method and device of distributed collection system for supercomputing internet and storage medium

Abstract

The application relates to a management method and device of a distributed collection system for a supercomputer Internet and a storage medium, the method obtains heartbeat information of all collection nodes in the distributed collection system from a first database, obtains management authority according to the heartbeat information of all the collection nodes, and in the case of determining that the management authority is obtained, obtains an instance list corresponding to the distributed collection system from a second database, and manages collection tasks on all the collection nodes in the distributed collection system according to the heartbeat information of all the collection nodes and the instance list. In the method, the decentralized effect can be achieved by adopting the mutual monitoring mode of all the collection nodes in the process of managing the collection tasks in the distributed collection system, and the management efficiency and stability of the whole system are improved. The collection nodes can avoid management conflicts by managing tasks in the mode of obtaining management authority.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a management method, apparatus, and storage medium for a distributed acquisition system for a supercomputing internet. Background Technology

[0002] In today's digital age, the supercomputing internet, with its powerful computing capabilities and extensive resource integration, has brought unprecedented development opportunities to numerous fields such as scientific research, industrial manufacturing, weather forecasting, and financial analysis. As various industries become increasingly reliant on the supercomputing internet, the demands on its underlying data acquisition system are becoming increasingly stringent. The supercomputing internet encompasses a large number of computing nodes, storage devices, and various sensors distributed across different geographical locations with varying performance and functions. These diverse components continuously generate massive amounts of data, and the distributed acquisition system bears the heavy responsibility of efficiently and accurately collecting this data for subsequent analysis, processing, and application. The widespread application of microservice architecture in the supercomputing internet further exacerbates the complexity of data acquisition. While microservice architecture, with its modularity and loose coupling, allows the system to more flexibly respond to complex and ever-changing business needs, it also requires the distributed acquisition system to coordinate a larger number and more diverse range of services to complete data acquisition tasks. In this distributed environment, the distributed acquisition system of the supercomputing internet faces many severe challenges. On the one hand, various unforeseen problems such as network failures, node failures, and software anomalies occur frequently. These failures may lead to the interruption of some data acquisition tasks, data loss, or incomplete data acquisition, seriously affecting the accuracy and integrity of supercomputing internet data processing. On the other hand, because supercomputing internet application scenarios have extremely high requirements for data real-time performance, such as in real-time weather monitoring and high-frequency financial trading, delays or interruptions in data acquisition may lead to decision-making errors and cause huge losses. Therefore, how to ensure that the distributed acquisition system has high availability and can operate continuously, stably, and efficiently in the complex supercomputing internet environment has become a critical issue that urgently needs to be addressed.

[0003] Existing data acquisition methods are insufficient to meet the stringent requirements of high availability for data acquisition systems in supercomputing internet, necessitating an innovative high-availability method for distributed acquisition systems in supercomputing internet to overcome this technical bottleneck. Summary of the Invention

[0004] Therefore, it is necessary to provide a management method, device, and storage medium for a distributed acquisition system oriented towards the supercomputing internet, which can improve the management efficiency of the distributed acquisition system and address the aforementioned technical problems.

[0005] Firstly, this application provides a management method for a distributed acquisition system for a supercomputing internet, which is applied to any acquisition node in the distributed acquisition system. The method includes:

[0006] Retrieve heartbeat information of all acquisition nodes in the distributed acquisition system from the first database, and obtain management permissions based on the heartbeat information of all acquisition nodes;

[0007] Once management privileges are obtained, the system retrieves the instance list corresponding to the distributed acquisition system from the second database, and manages the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the heartbeat information and instance list of all acquisition nodes.

[0008] The management method for a distributed acquisition system for a supercomputing internet provided in this application embodiment obtains heartbeat information of all acquisition nodes in the distributed acquisition system from a first database, acquires management permissions based on the heartbeat information of all acquisition nodes, and, upon obtaining management permissions, obtains an instance list corresponding to the distributed acquisition system from a second database. Based on the heartbeat information and instance list of all acquisition nodes, the acquisition tasks on all acquisition nodes in the distributed acquisition system are managed. Firstly, in the process of managing acquisition tasks in the distributed acquisition system, the method of mutual monitoring among all acquisition nodes achieves a decentralized effect, effectively avoiding the drawbacks of traditional middleware management methods, thereby improving the management efficiency and stability of the entire system to a certain extent. For example, the method avoids the problem of low management efficiency caused by the complexity of traditional management logic, and reduces the risk of system management crashes due to single points of failure in traditional middleware. Secondly, in the process of mutual monitoring among multiple acquisition nodes, each acquisition node manages tasks by acquiring management permissions, and ultimately, the acquisition node that successfully acquires management permissions manages the tasks. This avoids the chaos caused by multiple nodes simultaneously intervening in management, effectively preventing management conflicts and data inconsistencies. In addition, the data collection node can effectively manage the data collection task by using the heartbeat information in the first database and the instance list in the second database. This solves the problem of complex management logic caused by the coupling between nodes in traditional middleware management methods. The above method does not rely on complex management logic, which can further optimize the management process and improve management efficiency.

[0009] In some embodiments, the collection tasks on all collection nodes in the distributed collection system are managed based on the heartbeat information and instance list of all collection nodes, including:

[0010] The heartbeat information collection node is compared with the collection nodes included in the instance list to obtain the comparison results;

[0011] Based on the comparison results, the acquisition tasks on all acquisition nodes in the distributed acquisition system are managed.

[0012] The method described in this application embodiment can quickly identify whether there are faulty collection nodes in the distributed collection system by comparing the number of collection nodes to which the heartbeat information belongs with the number of collection nodes in the instance list, thereby improving the management efficiency of the distributed collection system.

[0013] In some embodiments, the acquisition tasks on all acquisition nodes in the distributed acquisition system are managed based on comparison results, including:

[0014] If the comparison results indicate that the number of collection nodes to which the heartbeat information belongs is inconsistent with the number of collection nodes included in the instance list, then it is determined that there is a faulty collection node in the distributed collection system, and the collection tasks on the faulty collection node and all non-faulty collection nodes are managed.

[0015] If the comparison results indicate that the number of collection nodes to which the heartbeat information belongs is consistent with the number of collection nodes included in the instance list, then it is determined that there are no faulty collection nodes in the distributed collection system, and the collection tasks in the distributed collection system are managed based on the heartbeat information of all collection nodes.

[0016] The method described in this application embodiment can reallocate the remaining data acquisition tasks on faulty nodes to ensure the continuity of task processing. For non-faulty nodes, the allocation of data acquisition tasks can be reasonably adjusted according to their load and processing capacity to ensure that the overall data acquisition efficiency of the system is not significantly affected. In the absence of faulty data acquisition nodes, the system manages the data acquisition tasks in the distributed data acquisition system based on the heartbeat information of all data acquisition nodes, prioritizing the allocation of data acquisition tasks to nodes with better performance and lower load, thereby improving the overall data acquisition efficiency and resource utilization of the system.

[0017] In some embodiments, the acquisition tasks on faulty acquisition nodes and all non-faulty acquisition nodes are managed, including:

[0018] Based on the previous heartbeat information of the faulty acquisition node and the heartbeat information of each non-faulty acquisition node, determine the first transfer task assigned to each non-faulty acquisition node.

[0019] The task table in the heartbeat information of each non-faulty acquisition node in the first database is updated according to the first transfer task. Each task table is used to instruct each non-faulty acquisition node to perform task processing according to the task in the corresponding task table.

[0020] The method described in this application determines the first transfer task by comprehensively considering the heartbeat information of the previous faulty node and the heartbeat information of each non-faulty node. This allows for the reasonable allocation of tasks to each non-faulty node, avoiding the situation where arbitrarily assigning tasks to a non-faulty node would lead to excessive load on that node while other nodes remain idle, thus improving the overall system's resource utilization efficiency. Furthermore, the rapid transfer of tasks from the faulty node to non-faulty nodes eliminates the need to wait for the faulty node to be repaired before resuming data acquisition. Instead, the task transfer mechanism immediately allows non-faulty nodes to take over the work, ensuring that the system's data acquisition is not interrupted by the failure of individual nodes. This maintains the continuity of data acquisition throughout the system. For applications that rely on real-time data, this avoids accidents caused by data loss, improving the system's availability and reliability.

[0021] In some embodiments, after determining that a faulty acquisition node exists in the distributed acquisition system, the method further includes:

[0022] Remove the fault acquisition node from the instance list to update the instance list.

[0023] The method described in this application embodiment, after removing faulty nodes, allows the monitored data to more accurately reflect the operating status of normal nodes in the system when monitoring other nodes, thus avoiding interference from abnormal data of faulty nodes on the monitoring results.

[0024] In some embodiments, the collection tasks in the distributed collection system are managed based on the heartbeat information of all collection nodes, including:

[0025] Based on the heartbeat information of all data acquisition nodes, determine whether there are any overloaded data acquisition nodes among all data acquisition nodes;

[0026] If present, manage the acquisition tasks on overloaded acquisition nodes and other acquisition nodes;

[0027] If it does not exist, then release the management privileges.

[0028] In some embodiments, managing acquisition tasks on overloaded acquisition nodes and other acquisition nodes includes:

[0029] Based on the heartbeat information of the overloaded acquisition node and the heartbeat information of other acquisition nodes, determine the second transfer task to be assigned to other acquisition nodes;

[0030] Update the task tables corresponding to all collection nodes in the first database according to the second transfer task.

[0031] The method described in this application addresses the issue that overloaded acquisition nodes are prone to performance degradation, response delays, and even crashes under high load. By promptly identifying overloaded nodes based on heartbeat information and transferring excess tasks to other nodes, the load pressure on those nodes can be effectively reduced, preventing them from collapsing due to overload and ensuring the stable operation of the entire distributed acquisition system. Furthermore, the crash of a node in the system can trigger a chain reaction, further increasing the load on other nodes and even causing a system-wide failure. Timely handling of overloaded nodes can control the risk of failure locally, preventing its spread and maintaining overall system stability. Additionally, allocating the second transfer task based on the heartbeat information of overloaded acquisition nodes and other acquisition nodes allows for a more even distribution of tasks across the nodes. Transferring some tasks from overloaded nodes to less loaded nodes fully utilizes idle resources, improving overall system resource utilization and preventing situations where some nodes are overloaded while others are idle. Finally, task transfer and task table updates based on real-time heartbeat information enable dynamic task management in the distributed acquisition system.

[0032] In some embodiments, management permissions are obtained based on the heartbeat information of all data collection nodes, including:

[0033] Based on the status information of the management lock in the heartbeat information of each acquisition node, determine the status of each acquisition node holding the management lock;

[0034] If all the management locks of the collection nodes are in an unheld state, then update the state of its own management lock to a held state in order to obtain management privileges;

[0035] If there is a target acquisition node among all acquisition nodes that holds the management lock, then management privileges are obtained based on the heartbeat information of the target acquisition node.

[0036] In a distributed acquisition system, if multiple nodes attempt to acquire management privileges simultaneously, it may lead to problems such as conflicting management commands and inconsistent data. By checking the status of the management lock, a node can only update its own management lock status to a held state and acquire management privileges when all nodes' management lock statuses are in an unheld state. This ensures that only one node has management privileges at any given time, avoiding the chaos caused by multiple nodes performing management operations simultaneously and guaranteeing the orderliness of system management.

[0037] In some embodiments, management permissions are obtained based on the heartbeat information of the target acquisition node, including:

[0038] Extract the lock holding time from the heartbeat information of the target acquisition node;

[0039] If the lock holding time exceeds the preset time threshold, the status of the target acquisition node's management lock will be updated to the non-held state, and its own management lock status will be updated to the held state to obtain management privileges.

[0040] If the lock holding time is not greater than the preset time threshold, it is determined that management privileges have not been obtained.

[0041] The method described in this application embodiment can cause system management to stagnate if the target acquisition node is unable to perform its management duties normally due to a fault, network interruption, or other abnormal situation, but the management lock is still held. By checking the lock holding time, if the lock holding time of the target acquisition node exceeds a certain threshold, it indicates that the target acquisition node may be abnormal and is not suitable for task management. Therefore, timely revocation and reallocation of management authority can effectively avoid this situation and ensure the continuity and stability of system management.

[0042] Secondly, this application also provides a management system for a distributed acquisition system for a supercomputing internet. The management system includes: a distributed acquisition node cluster, a first database engine, and a second database engine; each acquisition node in the distributed acquisition node cluster is connected to the first database engine and the second database engine respectively; a first database is deployed on the first database engine, and a second database is deployed on the second database engine.

[0043] Any one of the acquisition nodes in the distributed acquisition node cluster is used to execute the method as described in any of the embodiments of the first aspect above.

[0044] Thirdly, this application also provides a management device for a distributed acquisition system for a supercomputing internet, the device comprising:

[0045] The acquisition module is used to obtain the heartbeat information of all acquisition nodes in the distributed acquisition system from the first database, and to obtain management permissions based on the heartbeat information of all acquisition nodes.

[0046] The management module, once management privileges are obtained, retrieves the instance list corresponding to the distributed acquisition system from the second database and manages the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the heartbeat information and instance list of all acquisition nodes.

[0047] Fourthly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0048] Retrieve heartbeat information of all acquisition nodes in the distributed acquisition system from the first database, and obtain management permissions based on the heartbeat information of all acquisition nodes;

[0049] Once management privileges are obtained, the system retrieves the instance list corresponding to the distributed acquisition system from the second database, and manages the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the heartbeat information and instance list of all acquisition nodes.

[0050] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0051] Retrieve heartbeat information of all acquisition nodes in the distributed acquisition system from the first database, and obtain management permissions based on the heartbeat information of all acquisition nodes;

[0052] Once management privileges are obtained, the system retrieves the instance list corresponding to the distributed acquisition system from the second database, and manages the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the heartbeat information and instance list of all acquisition nodes.

[0053] Sixthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0054] Retrieve heartbeat information of all acquisition nodes in the distributed acquisition system from the first database, and obtain management permissions based on the heartbeat information of all acquisition nodes;

[0055] Once management privileges are obtained, the system retrieves the instance list corresponding to the distributed acquisition system from the second database, and manages the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the heartbeat information and instance list of all acquisition nodes.

[0056] The aforementioned management method, device, and storage medium for a distributed acquisition system for supercomputing internet involve obtaining heartbeat information from a first database for all acquisition nodes in the distributed acquisition system, acquiring management permissions based on this information, and then retrieving a list of instances corresponding to the distributed acquisition system from a second database. Based on the heartbeat information and instance list, the acquisition tasks on all acquisition nodes in the distributed acquisition system are managed. Firstly, the method achieves decentralization by employing mutual monitoring among all acquisition nodes during task management, effectively avoiding the drawbacks of traditional middleware management methods and thus improving the overall system's management efficiency and stability. For example, it avoids the low management efficiency caused by complex traditional management logic and reduces the risk of system management crashes due to single points of failure in traditional middleware. Secondly, during mutual monitoring among multiple acquisition nodes, each node manages tasks by acquiring management permissions, with the node that successfully acquires management permissions ultimately managing the tasks. This avoids the chaos caused by multiple nodes simultaneously intervening in management, effectively preventing management conflicts and data inconsistencies. In addition, the data collection node can effectively manage the data collection task by using the heartbeat information in the first database and the instance list in the second database. This solves the problem of complex management logic caused by the coupling between nodes in traditional middleware management methods. The above method does not rely on complex management logic, which can further optimize the management process and improve management efficiency. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the management system architecture for a distributed acquisition system for a supercomputing internet in some embodiments;

[0058] Figure 2 This is one of the flowcharts illustrating a management method for a distributed acquisition system for a supercomputing internet in some embodiments;

[0059] Figure 3 This is the second flowchart illustrating a management method for a distributed acquisition system for a supercomputing internet in some embodiments.

[0060] Figure 4 This is the third flowchart illustrating the management method of a distributed acquisition system for a supercomputing internet in some embodiments;

[0061] Figure 5 This is the fourth flowchart illustrating the management method of a distributed acquisition system for a supercomputing internet in some embodiments;

[0062] Figure 6 This is the fifth flowchart illustrating the management method of a distributed acquisition system for a supercomputing internet in some embodiments;

[0063] Figure 7 This is the sixth flowchart illustrating the management method of a distributed acquisition system for a supercomputing internet in some embodiments;

[0064] Figure 8 This is the seventh flowchart illustrating the management method of a distributed acquisition system for a supercomputing internet in some embodiments;

[0065] Figure 9 This is the eighth flowchart illustrating a management method for a distributed acquisition system for a supercomputing internet in some embodiments.

[0066] Figure 10 This is the ninth flowchart illustrating a management method for a distributed acquisition system for a supercomputing internet in some embodiments.

[0067] Figure 11 This is a structural block diagram of the management device of a distributed acquisition system for a supercomputing internet in some embodiments;

[0068] Figure 12 This is a diagram showing the internal structure of a computer device in some embodiments. Detailed Implementation

[0069] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0070] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0071] In the embodiments of this application, the term "at least one" means one or more. For example, at least one of A, B and C can represent six situations: A exists alone, B exists alone, C exists alone, A and B exist simultaneously, A and C exist simultaneously, B and C exist simultaneously, and A, B and C exist simultaneously.

[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0073] In today's digital age, the supercomputing internet, with its powerful computing capabilities and extensive resource integration, has brought unprecedented development opportunities to numerous fields such as scientific research, industrial manufacturing, weather forecasting, and financial analysis. As various industries become increasingly reliant on the supercomputing internet, the demands on its underlying data acquisition system are becoming increasingly stringent. The supercomputing internet encompasses a large number of computing nodes, storage devices, and various sensors distributed across different geographical locations with varying performance and functions. These diverse components continuously generate massive amounts of data, and the distributed acquisition system bears the heavy responsibility of efficiently and accurately collecting this data for subsequent analysis, processing, and application. The widespread application of microservice architecture in the supercomputing internet further exacerbates the complexity of data acquisition. While microservice architecture, with its modularity and loose coupling, allows the system to more flexibly respond to complex and ever-changing business needs, it also requires the distributed acquisition system to coordinate a larger number and more diverse range of services to complete data acquisition tasks. The distributed acquisition system of the supercomputing internet, relying on the supercomputing internet architecture, can efficiently collect, integrate, and preprocess various types of data across multiple geographically dispersed nodes. This system breaks through the limitations of traditional centralized data acquisition architectures, fully utilizing computing and data resources distributed across different regions and institutions to achieve comprehensive acquisition of large-scale, multi-source, and heterogeneous data. However, in this distributed environment, the distributed acquisition system of the supercomputing internet faces numerous severe challenges. On the one hand, various unforeseen problems such as network failures, node failures, and software anomalies occur frequently. These failures may lead to the interruption of some acquisition tasks, data loss, or incomplete acquisition, seriously affecting the accuracy and integrity of supercomputing internet data processing. On the other hand, because supercomputing internet application scenarios have extremely high requirements for data real-time performance, such as in real-time weather monitoring and high-frequency financial trading, delays or interruptions in data acquisition may lead to decision-making errors and huge losses. Therefore, how to ensure the high availability of the distributed acquisition system in the complex supercomputing internet environment, enabling it to operate continuously, stably, and efficiently, has become a critical issue that urgently needs to be addressed. Existing data acquisition methods are insufficient to meet the stringent high availability requirements of the supercomputing internet for data acquisition systems, necessitating an innovative high-availability method for distributed acquisition systems in the supercomputing internet to overcome this technical bottleneck.

[0074] In view of this, embodiments of this application propose a management method, apparatus and storage medium for a distributed acquisition system for supercomputing internet. By decentralizing the management of the acquisition node cluster in the distributed acquisition system, the management efficiency of the distributed acquisition system can be improved.

[0075] It should be noted that the beneficial effects or technical problems solved by the embodiments of this application are not limited to this one, but may also be other implicit or related problems. For details, please refer to the description of the embodiments below.

[0076] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0077] In some embodiments, the management method for a distributed acquisition system for a supercomputing internet provided in this application can be applied to, for example... Figure 1 The management system of the distributed acquisition system for the supercomputing internet shown includes a distributed acquisition node cluster 101, a first database engine 102, and a second database engine 103. The distributed acquisition node cluster 101 includes multiple acquisition nodes 1011. Each acquisition node 1011 in the distributed acquisition node cluster 101 is connected to the first database engine 102 and the second database engine 103, respectively. Each acquisition node 1011 in the distributed acquisition node cluster 101 can communicate with the first database engine 102 and the second database engine 103 via a network. Each acquisition node can be deployed on a computer device, which can be a server or a server cluster consisting of multiple servers. The first database engine 102 has a first database 1021 deployed on it, and the second database engine 103 has a second database 1031 deployed on it. The multiple acquisition nodes 1011 in the distributed acquisition node cluster 101 are used to collect working status information from the computing cluster, specifically executing the management method of the distributed acquisition system for the supercomputing internet described in any of the following embodiments.

[0078] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the management system of the distributed acquisition system for the supercomputing internet applied thereto. The specific management system of the distributed acquisition system for the supercomputing internet may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0079] In some embodiments, such as Figure 2 As shown, a management method for a distributed acquisition system for a supercomputing internet is provided, which is then applied to... Figure 1 Taking the computer device where any acquisition node in the distributed acquisition system is located as an example, the following steps are included:

[0080] S201: Obtain heartbeat information of all acquisition nodes in the distributed acquisition system from the first database, and obtain management permissions based on the heartbeat information of all acquisition nodes.

[0081] The first database is a non-relational database, such as a Redis cluster. It stores heartbeat information for all data acquisition nodes in the distributed data acquisition system. Heartbeat information includes at least one of the following: node load information, node acquisition performance information, node status of acquiring management permissions (whether management permissions have been acquired), the time the node acquired management permissions, and the node's task table (including the node's total task load and current task processing progress). Heartbeat information states include those with and without heartbeat information. Management permissions are the permissions to identify nodes and reallocate tasks in the distributed data acquisition system, and are used to lock the data acquisition process. For example, management permissions can be distributed locks. It is important to note that the timeliness of task processing in a distributed data acquisition system is crucial, directly affecting data timeliness and the overall system responsiveness. The latest status of each data acquisition node plays a key role in ensuring timely task processing. Only by accurately grasping the real-time status of each node can tasks be allocated rationally and potential problems be identified and resolved promptly. Heartbeat information, as an effective way to reflect the latest status of each node, clearly shows whether the node is operating normally and has the ability to process tasks. To efficiently store and manage this heartbeat information, Redis is selected as the primary database in this embodiment. Redis boasts excellent read / write performance and high concurrency capabilities, enabling rapid storage and retrieval of heartbeat information, ensuring the system can acquire node status in real time. Simultaneously, Redis's in-memory storage characteristics result in extremely fast data access, further enhancing the system's response speed to changes in node status. By storing heartbeat information in Redis, real-time monitoring and rapid response to the status of each acquisition node can be achieved. Furthermore, task allocation can be dynamically adjusted based on node status, effectively ensuring the timeliness of task processing by the acquisition system and improving the overall system's operational efficiency and stability.

[0082] In this embodiment, a first database can be pre-deployed on a first database engine, and the first database engine can be connected to all nodes in the distributed acquisition system for communication. When any acquisition node in the distributed system manages an acquisition task, the server where that acquisition node is located can periodically (e.g., every few seconds or minutes) obtain the heartbeat information of all acquisition nodes in the distributed acquisition system from the first database through the first database engine using a monitoring tool or a written script. Then, it extracts the management permissions recorded by each acquisition node from the heartbeat information, and determines whether the current management permissions of the acquisition system are idle based on the management permission status of other nodes. If the current management permissions of the acquisition system are idle, the server can obtain management permissions and record them in its own acquisition node's heartbeat information so that other nodes can view the management permission status in real time. If the current management permissions of the acquisition system are not idle, it means that a certain acquisition node in the acquisition system has already obtained management permissions and is managing the acquisition tasks of the distributed acquisition system. Therefore, the server has not obtained management permissions this time. In this case, the server can repeat the steps of obtaining the heartbeat information of all acquisition nodes in the distributed acquisition system from the first database and obtaining management permissions based on the heartbeat information of all acquisition nodes after a preset interval. Optionally, to ensure that the data collection nodes that have obtained management privileges can perform task management normally, the server can further determine whether the data collection node is abnormal based on the heartbeat information of the data collection node when it determines that a data collection node has obtained management privileges. If the data collection node is abnormal, the server can take over the management privileges on the abnormal data collection node, obtain management privileges, undo the management privileges of the abnormal data collection node, and update the status of management privileges in the heartbeat information of the abnormal data collection node.

[0083] S202, after confirming that management privileges have been obtained, retrieve the instance list corresponding to the distributed acquisition system from the second database, and manage the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the heartbeat information and instance list of all acquisition nodes.

[0084] The second database is a relational database, such as MySQL or PostgreSQL. It stores a list of instances corresponding to the distributed data acquisition system. This list contains the identifiers of all currently active data acquisition nodes. The list is updated when a new node joins the system. For example, it might store the IDs of nodes like node 1, 2, ..., n. It's important to note that the instance list is essentially stored relational information. This embodiment uses a relational database as the second database because it stores data in a structured manner on disk. While its read / write speed might be slower than Redis in some scenarios, it's perfectly adequate for data like the instance list that doesn't require real-time updates. Furthermore, disk storage is far less expensive than memory storage. Using a relational database to store the instance list significantly reduces storage costs while maintaining basic system functionality, allowing for more efficient resource allocation and utilization.

[0085] In this embodiment, a second database can be pre-deployed on a second database engine, and the second database engine can be connected to all nodes in the distributed acquisition system. Once the server containing a certain acquisition node in the distributed acquisition system obtains management privileges based on the above steps, it can begin identifying and failover operations for the acquisition nodes in the distributed acquisition system. Specifically, the server can obtain the instance list corresponding to the distributed acquisition system from the second database through the second database engine. Then, based on the consistency between the acquisition nodes in the heartbeat information of all acquisition nodes and the acquisition nodes in the instance list, it determines whether there is a faulty node in the distributed acquisition system. If it is determined that there is no faulty node in the distributed acquisition system, the server's monitoring ends, and management privileges can be returned. If it is determined that there is a faulty node in the distributed acquisition system, the faulty node can be identified, and the acquisition tasks on the faulty node can be evenly distributed to other acquisition nodes. When evenly distributing the acquisition tasks on the faulty node to other acquisition nodes, the acquisition tasks on the faulty node can be evenly distributed to acquisition nodes near the faulty node. Optionally, the acquisition tasks on the faulty node can be distributed to other acquisition nodes with good acquisition performance. Optionally, the acquisition tasks on the faulty node can be distributed to other acquisition nodes with low acquisition load. Optionally, multiple nodes with good acquisition performance and / or low load can be selected from all normal nodes. Then, from these nodes, acquisition nodes near the faulty node can be selected to inherit the acquisition tasks from the faulty node for management. Nodes with acquisition rates greater than a rate threshold can be considered as nodes with good acquisition performance, and nodes with loads less than a load threshold can be considered as nodes with low loads.

[0086] The management method for a distributed acquisition system for a supercomputing internet provided in this application embodiment obtains heartbeat information of all acquisition nodes in the distributed acquisition system from a first database, acquires management permissions based on the heartbeat information of all acquisition nodes, and, upon obtaining management permissions, obtains an instance list corresponding to the distributed acquisition system from a second database. Based on the heartbeat information and instance list of all acquisition nodes, the acquisition tasks on all acquisition nodes in the distributed acquisition system are managed. Firstly, in the process of managing acquisition tasks in the distributed acquisition system, the method of mutual monitoring among all acquisition nodes achieves a decentralized effect, effectively avoiding the drawbacks of traditional middleware management methods, thereby improving the management efficiency and stability of the entire system to a certain extent. For example, the method avoids the problem of low management efficiency caused by the complexity of traditional management logic, and reduces the risk of system management crashes due to single points of failure in traditional middleware. Secondly, in the process of mutual monitoring among multiple acquisition nodes, each acquisition node manages tasks by acquiring management permissions, and ultimately, the acquisition node that successfully acquires management permissions manages the tasks. This avoids the chaos caused by multiple nodes simultaneously intervening in management, effectively preventing management conflicts and data inconsistencies. In addition, the data collection node can effectively manage the data collection task by using the heartbeat information in the first database and the instance list in the second database. This solves the problem of complex management logic caused by the coupling between nodes in traditional middleware management methods. The above method does not rely on complex management logic, which can further optimize the management process and improve management efficiency.

[0087] In some embodiments, a specific method for managing acquisition tasks on all acquisition nodes in a distributed acquisition system is also provided, such as... Figure 3 As shown, the phrase "managing the collection tasks on all collection nodes in the distributed collection system based on the heartbeat information and instance list of all collection nodes" in S202 above includes:

[0088] S301, compare the collection node to which the heartbeat information belongs with the collection nodes included in the instance list to obtain the comparison result.

[0089] The comparison results include either the number of collection nodes to which the heartbeat information belongs being the same as the number of collection nodes in the instance list, or the comparison results include the number of collection nodes to which the heartbeat information belongs being different from the number of collection nodes in the instance list.

[0090] In this embodiment of the application, after the server obtains the heartbeat information and instance list of all collection nodes, it can further determine whether the number of collection nodes to which the heartbeat information belongs is consistent with the number of collection nodes included in the instance list, and determine whether there are collection nodes in the instance list but without heartbeat information.

[0091] S302 manages the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the comparison results.

[0092] In this embodiment of the application, after the server obtains the comparison results based on the above steps, it can determine whether there are faulty collection nodes based on the comparison results. If the number of collection nodes to which the heartbeat information belongs is consistent with the number of collection nodes contained in the instance list, it means that all collection nodes contained in the instance list have heartbeat information. If the number of collection nodes to which the heartbeat information belongs is inconsistent with the number of collection nodes contained in the instance list, it means that there are nodes in the instance list that do not have heartbeat information.

[0093] The method described in this application embodiment can quickly identify whether there are faulty collection nodes in the distributed collection system by comparing the number of collection nodes to which the heartbeat information belongs with the number of collection nodes in the instance list, thereby improving the management efficiency of the distributed collection system.

[0094] In some embodiments, a specific implementation method for managing acquisition tasks on all acquisition nodes in a distributed acquisition system is also provided, such as... Figure 4 As shown, the "management of acquisition tasks on all acquisition nodes in the distributed acquisition system based on the comparison results" in S302 above includes:

[0095] S401, if the comparison result indicates that the number of collection nodes to which the heartbeat information belongs is inconsistent with the number of collection nodes included in the instance list, then it is determined that there is a faulty collection node in the distributed collection system, and the collection tasks on the faulty collection node and all non-faulty collection nodes are managed.

[0096] Among them, the faulty acquisition node refers to the acquisition node that does not have heartbeat information.

[0097] In this embodiment of the application, after the server obtains the comparison result based on the above steps, and determines that the number of collection nodes to which the heartbeat information belongs is inconsistent with the number of collection nodes included in the instance list, it determines that there is a faulty collection node in the distributed collection system. Then the server can reassign the unfinished collection tasks on the faulty collection node to the non-faulty collection node, and manage all collection tasks (including previous collection tasks and newly assigned collection tasks) on the non-faulty collection node.

[0098] S402, if the comparison result indicates that the number of collection nodes to which the heartbeat information belongs is consistent with the number of collection nodes included in the instance list, then it is determined that there are no faulty collection nodes in the distributed collection system, and the collection tasks in the distributed collection system are managed according to the heartbeat information of all collection nodes.

[0099] In this embodiment of the application, when the server obtains the comparison result based on the above steps, and determines that the number of collection nodes to which the heartbeat information belongs is consistent with the number of collection nodes included in the instance list, it is determined that there are no faulty collection nodes in the distributed collection system. Then, it can be considered that the server's supervision has ended, and the management authority is returned. After a preset interval, the server returns to the execution of step S201 to manage the collection tasks in the distributed collection system according to the heartbeat information of all collection nodes.

[0100] The method described in this application embodiment can reallocate the remaining data acquisition tasks on faulty nodes to ensure the continuity of task processing. For non-faulty nodes, the allocation of data acquisition tasks can be reasonably adjusted according to their load and processing capacity to ensure that the overall data acquisition efficiency of the system is not significantly affected. In the absence of faulty data acquisition nodes, the system manages the data acquisition tasks in the distributed data acquisition system based on the heartbeat information of all data acquisition nodes, prioritizing the allocation of data acquisition tasks to nodes with better performance and lower load, thereby improving the overall data acquisition efficiency and resource utilization of the system.

[0101] In some embodiments, a specific implementation method for managing data collection tasks is also provided when the number of data collection nodes to which the heartbeat information belongs is inconsistent with the number of data collection nodes included in the instance list, such as... Figure 5 As shown, the "determine the first transfer task assigned to each non-faulty acquisition node based on the previous heartbeat information corresponding to the faulty acquisition node and the heartbeat information of each non-faulty acquisition node" in S401 above includes:

[0102] S501, based on the previous heartbeat information of the faulty acquisition node and the heartbeat information of each non-faulty acquisition node, determine the first transfer task to be assigned to each non-faulty acquisition node.

[0103] The previous heartbeat information refers to the last heartbeat information before the faulty acquisition node failed. The first transfer task is the task that the non-faulty acquisition node needs to take over from the faulty acquisition node.

[0104] In this embodiment, when the server determines that a faulty acquisition node exists in the acquisition node cluster of the distributed acquisition system, it can obtain the last heartbeat information of the faulty acquisition node before the failure from the first database as the previous heartbeat information, and obtain the heartbeat information of all non-faulty acquisition nodes. Then, it extracts the unprocessed acquisition tasks of the faulty acquisition node from the previous heartbeat information. Finally, based on at least one of the node load information, acquisition performance information, etc., carried in the heartbeat information of each non-faulty acquisition node, it assigns the acquisition tasks on the faulty acquisition node to suitable non-faulty acquisition nodes, thus obtaining the first transfer task for the non-faulty acquisition nodes. Specifically, based on the node load information carried in the heartbeat information of each non-faulty acquisition node, the acquisition tasks on the faulty acquisition node can be assigned to other non-faulty acquisition nodes with low acquisition load. Optionally, based on the acquisition performance information carried in the heartbeat information of each non-faulty acquisition node, the acquisition tasks on the faulty acquisition node can be assigned to other non-faulty acquisition nodes with better acquisition performance. Optionally, based on the acquisition performance information of each non-faulty acquisition node, the acquisition tasks on the faulty acquisition node can be assigned to non-faulty acquisition nodes near the faulty acquisition node. Nodes with a sampling rate greater than the rate threshold can be considered as nodes with better sampling performance, while nodes with a load less than the load threshold can be considered as nodes with lower load.

[0105] S502, update the task table in the heartbeat information corresponding to each non-faulty acquisition node in the first database according to the first transfer task. Each task table is used to instruct each non-faulty acquisition node to perform task processing according to the task in the corresponding task table.

[0106] The task table records the pending data collection tasks and their processing progress.

[0107] In this embodiment, after the server obtains the first transfer task for each non-faulty acquisition node based on the above steps, it can update the task table of each non-faulty acquisition node from the heartbeat information of each non-faulty acquisition node in the first database, which is used to instruct each non-faulty acquisition node to process tasks according to the tasks in the corresponding task table. For example, the first transfer task can be added to the pending acquisition tasks in the original task table of each non-faulty acquisition node.

[0108] The method described in this application determines the first transfer task by comprehensively considering the heartbeat information of the previous faulty node and the heartbeat information of each non-faulty node. This allows for the reasonable allocation of tasks to each non-faulty node, avoiding the situation where arbitrarily assigning tasks to a non-faulty node would lead to excessive load on that node while other nodes remain idle, thus improving the overall system's resource utilization efficiency. Furthermore, the rapid transfer of tasks from the faulty node to non-faulty nodes eliminates the need to wait for the faulty node to be repaired before resuming data acquisition. Instead, the task transfer mechanism immediately allows non-faulty nodes to take over the work, ensuring that the system's data acquisition is not interrupted by the failure of individual nodes. This maintains the continuity of data acquisition throughout the system. For applications that rely on real-time data, this avoids accidents caused by data loss, improving the system's availability and reliability.

[0109] In some embodiments, after determining that there is a faulty acquisition node in the distributed acquisition system, the management method of the distributed acquisition system further includes: removing the faulty acquisition node from the instance list to update the instance list.

[0110] In this embodiment, after determining that a faulty acquisition node exists in the distributed acquisition system, the server can delete the identification information of the faulty acquisition node from the instance list or mark it as faulty to update the instance list. Subsequently, after the faulty acquisition node recovers, it can be added to the instance list as a new acquisition node to update the instance list. Optionally, after the faulty acquisition node recovers, the faulty status can be removed from the instance list, or it can be marked as normal.

[0111] The method described in this application embodiment, after removing faulty nodes, allows the monitored data to more accurately reflect the operating status of normal nodes in the system when monitoring other nodes, thus avoiding interference from abnormal data of faulty nodes on the monitoring results.

[0112] In some embodiments, a specific implementation method for managing data collection tasks is also provided when the number of data collection nodes to which the heartbeat information belongs is consistent with the number of data collection nodes included in the instance list, such as... Figure 6 As shown, the "management of acquisition tasks on all acquisition nodes in the distributed acquisition system based on the heartbeat information and instance list of all acquisition nodes" in S402 above includes:

[0113] S601, based on the heartbeat information of all acquisition nodes, determines whether there are any overloaded acquisition nodes among all acquisition nodes.

[0114] Among them, overloaded acquisition nodes are those whose load is less than the load threshold. The load threshold can be determined based on the actual acquisition task requirements and the performance of the acquisition nodes. The load threshold for each acquisition node can be the same or different.

[0115] In this embodiment, after obtaining the heartbeat information of all data acquisition nodes, the server can extract the load information of all data acquisition nodes from the heartbeat information, and compare the load of each data acquisition node with a preset load threshold to determine whether the load of each data acquisition node is less than the preset load threshold. If there are data acquisition nodes with loads less than the load threshold, it is determined that there are overloaded data acquisition nodes among all data acquisition nodes, and these nodes are identified as overloaded data acquisition nodes. If there are no data acquisition nodes with loads less than the load threshold, it is determined that there are no overloaded data acquisition nodes among all data acquisition nodes.

[0116] S602, if present, manages the acquisition tasks on overloaded acquisition nodes and other acquisition nodes.

[0117] In this embodiment of the application, if the server determines that there are overloaded acquisition nodes based on the above steps, it can further redistribute the overloaded acquisition tasks on the overloaded acquisition nodes to other acquisition nodes, and manage the acquisition tasks on the overloaded acquisition nodes and other acquisition nodes.

[0118] Optional, such as Figure 7 As shown, the "managing the acquisition tasks in the distributed acquisition system based on the heartbeat information of all acquisition nodes" in S602 above includes:

[0119] S6021, based on the heartbeat information of the overloaded acquisition node and the heartbeat information of other acquisition nodes, determine the second transfer task to be assigned to other acquisition nodes.

[0120] The second transfer task is the task that other acquisition nodes need to take over from the overloaded acquisition nodes.

[0121] In this embodiment, when the server determines that an overloaded data collection node exists, it can determine the amount of overloaded data collection tasks based on the current load information and load threshold in the heartbeat information of the overloaded data collection node. Specifically, it determines the overload percentage based on the current load information and load threshold, and then determines the corresponding percentage of data collection tasks as the amount of overloaded data collection tasks. For example, if the load threshold of the overloaded data collection node is 80% and the current load information of the overloaded data collection node is 95%, then the overload percentage is 15%, and 15% of the total data collection tasks on the overloaded data collection node can be determined as the amount of overloaded data collection tasks. After determining the amount of overloaded data collection tasks on the overloaded data collection node, the server can allocate the data collection tasks on the overloaded data collection node to suitable other data collection nodes based on at least one of the node load information, data collection performance information, etc., carried in the heartbeat information of other data collection nodes, resulting in the reduction of data collection tasks on the overloaded data collection node and the addition of second transfer tasks on other data collection nodes. Specifically, it can allocate the data collection tasks on the overloaded data collection node to other data collection nodes with low data collection load based on the node load information carried in the heartbeat information of other data collection nodes. Optionally, based on the acquisition performance information carried in the heartbeat information of other acquisition nodes, the acquisition tasks on the overloaded acquisition node can be reassigned to other acquisition nodes with better acquisition performance. Alternatively, based on the acquisition performance information of other acquisition nodes, the acquisition tasks on the overloaded acquisition node can be reassigned to other acquisition nodes near the overloaded acquisition node. Nodes with acquisition rates greater than a rate threshold can be considered as nodes with better acquisition performance, and nodes with loads less than a load threshold can be considered as nodes with lower loads.

[0122] S6022, Update the task table corresponding to all collection nodes in the first database according to the second transfer task.

[0123] In this embodiment, after the server obtains the reduced collection tasks of overloaded collection nodes and the newly added second transfer tasks of other collection nodes based on the above steps, it can update the task tables corresponding to other collection nodes in the first database according to the second transfer tasks, and update the task tables corresponding to overloaded collection nodes in the first database according to the reduced collection tasks of overloaded collection nodes. After the first transfer tasks of each non-faulty collection node, the task tables of each non-faulty collection node can be updated from the heartbeat information of each non-faulty collection node in the first database, to instruct each non-faulty collection node to process tasks according to the tasks in the corresponding task tables. For example, the first transfer task can be added to the pending collection tasks in the original task tables of each non-faulty collection node.

[0124] If S603 does not exist, then release management privileges.

[0125] In this embodiment of the application, if the server determines that there are no overloaded collection nodes based on the above steps, then the server's supervision can be considered to be over, and the management authority is returned. After a preset interval, the server returns to the execution of step S201 to manage the collection tasks in the distributed collection system according to the heartbeat information of all collection nodes.

[0126] The method described in this application addresses the issue that overloaded acquisition nodes are prone to performance degradation, response delays, and even crashes under high load. By promptly identifying overloaded nodes based on heartbeat information and transferring excess tasks to other nodes, the load pressure on those nodes can be effectively reduced, preventing them from collapsing due to overload and ensuring the stable operation of the entire distributed acquisition system. Furthermore, the crash of a node in the system can trigger a chain reaction, further increasing the load on other nodes and even causing a system-wide failure. Timely handling of overloaded nodes can control the risk of failure locally, preventing its spread and maintaining overall system stability. Additionally, allocating the second transfer task based on the heartbeat information of overloaded acquisition nodes and other acquisition nodes allows for a more even distribution of tasks across the nodes. Transferring some tasks from overloaded nodes to less loaded nodes fully utilizes idle resources, improving overall system resource utilization and preventing situations where some nodes are overloaded while others are idle. Finally, task transfer and task table updates based on real-time heartbeat information enable dynamic task management in the distributed acquisition system.

[0127] In some embodiments, a method for obtaining administrative privileges is also provided, such as Figure 8 As shown, the "obtain management permissions based on the heartbeat information of all data collection nodes" in S201 above includes:

[0128] S701 determines the status of the management lock held by each acquisition node based on the status information of the management lock in the heartbeat information of each acquisition node.

[0129] The status information of the management lock includes the holding status of the management lock and the holding time. The holding status of the management lock includes the holding status and the non-holding status.

[0130] In this embodiment of the application, after the server obtains the heartbeat information of all collection nodes in the distributed collection system from the first database through the first database engine, it extracts the status information of the management lock in each collection node from the heartbeat information, and then determines the status of each collection node holding the management lock based on the status information of the management lock in the heartbeat information of each collection node.

[0131] S702, if the management lock status of all acquisition nodes is in an unheld state, then update the status of its own management lock to a held state in order to obtain management privileges.

[0132] In this embodiment of the application, if the server determines that the management locks of all collection nodes are in an unheld state, it means that the current management permissions of the collection system are idle. Then the server can acquire management permissions and update the status of its own management lock to a held state, so that other nodes can view the management permission status in real time.

[0133] S703, if there is a target acquisition node among all acquisition nodes that holds the management lock, then acquire management privileges based on the heartbeat information of the target acquisition node.

[0134] Among them, the target acquisition node is the acquisition node whose management lock is in the holding state.

[0135] In this embodiment of the application, if the server determines that there is a target acquisition node among all acquisition nodes in the state of holding the management lock, it means that the current management authority of the acquisition system is not idle, that is, a certain acquisition node in the acquisition system has obtained management authority and is performing acquisition task management of the distributed acquisition system. Then, the server can further obtain the heartbeat information of the target acquisition node, and then obtain management authority based on the heartbeat information of the target acquisition node.

[0136] In a distributed acquisition system, if multiple nodes attempt to acquire management privileges simultaneously, it may lead to problems such as conflicting management commands and inconsistent data. By checking the status of the management lock, a node can only update its own management lock status to a held state and acquire management privileges when all nodes' management lock statuses are in an unheld state. This ensures that only one node has management privileges at any given time, avoiding the chaos caused by multiple nodes performing management operations simultaneously and guaranteeing the orderliness of system management.

[0137] In some embodiments, a specific implementation method for obtaining management permissions is also provided, such as... Figure 9 As shown, the "obtain management permissions based on the heartbeat information of the target acquisition node" in S703 above includes:

[0138] S801 extracts the lock holding time from the heartbeat information of the target acquisition node.

[0139] Among them, the lock holding time refers to the holding time of the management lock, that is, the time of obtaining management rights.

[0140] In this embodiment of the application, after the server obtains the heartbeat information of the target acquisition node, it can extract the lock holding time from the heartbeat information of the target acquisition node, and then compare the lock holding time with a preset time threshold to determine whether the lock holding time of the target acquisition node is greater than the preset time threshold.

[0141] S802, if the lock holding time is greater than the preset time threshold, then the state of the management lock of the target acquisition node is updated to the non-held state, and the state of its own management lock is updated to the held state, so as to obtain management rights.

[0142] The preset time threshold can be determined based on the actual data acquisition task requirements and the performance of the acquisition nodes. The preset time threshold for each acquisition node can be the same or different.

[0143] In this embodiment of the application, if the server determines that the lock holding time of the target acquisition node is greater than a preset time threshold based on the above steps, it indicates that the target acquisition node has low processing efficiency or other abnormalities. In this case, the server can take over the management authority on the target acquisition node, update the status of the management lock of the target acquisition node to the unheld state, thereby unlocking the management authority of the target acquisition node, update the status of its own management lock to the held state, and update the status of the management lock in the heartbeat information of the target acquisition node.

[0144] S803 If the lock holding time is not greater than the preset time threshold, it is determined that management privileges have not been obtained.

[0145] In this embodiment of the application, if the server determines that the lock holding time of the target acquisition node is greater than the preset time threshold based on the above steps, it means that the target acquisition node is managing the acquisition task normally. In this case, it means that the management permission has not been obtained. The server can return to the execution of step S201 after a preset time interval.

[0146] The method described in this application embodiment can cause system management to stagnate if the target acquisition node is unable to perform its management duties normally due to a fault, network interruption, or other abnormal situation, but the management lock is still held. By checking the lock holding time, if the lock holding time of the target acquisition node exceeds a certain threshold, it indicates that the target acquisition node may be abnormal and is not suitable for task management. Therefore, timely revocation and reallocation of management authority can effectively avoid this situation and ensure the continuity and stability of system management.

[0147] In summary, based on all the above embodiments, a management method for a distributed acquisition system for a supercomputing internet is also provided, such as... Figure 10 As shown, the method includes:

[0148] S901: Obtain the heartbeat information of all collection nodes in the distributed collection system from the first database, and determine the status of each collection node holding the management lock based on the management lock status information in the heartbeat information of each collection node.

[0149] S902, if the management lock status of all acquisition nodes is in an unheld state, then update the status of its own management lock to a held state in order to obtain management privileges.

[0150] S903, if there is a target acquisition node among all acquisition nodes whose management lock is in a holding state, then extract the lock holding time from the heartbeat information of the target acquisition node.

[0151] S904 If the lock holding time is greater than the preset time threshold, the state of the target acquisition node's management lock is updated to the unheld state, and the state of its own management lock is updated to the held state, so as to obtain management privileges.

[0152] S905, if the lock holding time is not greater than the preset time threshold, it is determined that management privileges have not been obtained, and after the preset time interval, the process returns to the execution of step S901.

[0153] S906, after confirming that management privileges have been obtained, retrieves the instance list corresponding to the distributed acquisition system from the second database, and compares the acquisition node to which the heartbeat information belongs with the acquisition nodes included in the instance list to obtain the comparison result.

[0154] S907 If the number of collection nodes to which the heartbeat information belongs is inconsistent with the number of collection nodes in the instance list, it is determined that there is a faulty collection node in the distributed collection system, and the faulty collection node is removed from the instance list to update the instance list.

[0155] S908 determines the first transfer task to be assigned to each non-faulty acquisition node based on the previous heartbeat information corresponding to the faulty acquisition node and the heartbeat information of each non-faulty acquisition node.

[0156] S909, update the task table in the heartbeat information corresponding to each non-faulty acquisition node in the first database according to the first transfer task. Each task table is used to instruct each non-faulty acquisition node to perform task processing according to the task in the corresponding task table.

[0157] S910, if the number of collection nodes to which the heartbeat information belongs is consistent with the number of collection nodes included in the instance list, then it is determined that there are no faulty collection nodes in the distributed collection system, and based on the heartbeat information of all collection nodes, it is determined whether there are overloaded collection nodes among all collection nodes.

[0158] S911, if it exists, then based on the heartbeat information of the overloaded acquisition node and the heartbeat information of other acquisition nodes, determine the second transfer task to be assigned to other acquisition nodes, and update the task table corresponding to all acquisition nodes in the first database according to the second transfer task.

[0159] If S912 does not exist, release the management privileges and return to the execution of step S901 after a preset time interval.

[0160] The method described in this application embodiment enables the distributed data acquisition system to self-heal and failover if one of the multiple data acquisition service instances fails. This ensures that users outside the distributed system are unaware of the problem, all data acquisition tasks run normally, and the acquired data is normal, continuous, and stable.

[0161] The methods described in each of the above steps have been described in the foregoing embodiments. For details, please refer to the foregoing descriptions. They will not be repeated here.

[0162] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0163] Based on the same inventive concept, this application also provides a management device for a distributed acquisition system for a supercomputing internet, which implements the management method for the distributed acquisition system for a supercomputing internet described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the management device for a distributed acquisition system for a supercomputing internet provided below can be found in the limitations of the management method for a distributed acquisition system for a supercomputing internet described above, and will not be repeated here.

[0164] In some embodiments, such as Figure 11 As shown, a management device for a distributed acquisition system for a supercomputing internet is provided, comprising:

[0165] The acquisition module 11 is used to obtain the heartbeat information of all acquisition nodes in the distributed acquisition system from the first database, and to obtain management permissions based on the heartbeat information of all acquisition nodes.

[0166] The management module 12 is used to retrieve the instance list corresponding to the distributed acquisition system from the second database when it is determined that management permissions have been obtained, and to manage the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the heartbeat information of all acquisition nodes and the instance list.

[0167] In some embodiments, the above-mentioned management module includes:

[0168] The comparison unit is used to compare the acquisition node to which the heartbeat information belongs with the acquisition nodes included in the instance list to obtain the comparison result.

[0169] The management unit is used to manage the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the comparison results.

[0170] In some embodiments, the above-mentioned management unit includes:

[0171] The first management subunit is used to determine if there is a faulty acquisition node in the distributed acquisition system if the comparison result shows that the number of acquisition nodes to which the heartbeat information belongs is inconsistent with the number of acquisition nodes included in the instance list, and to manage the acquisition tasks on the faulty acquisition node and all non-faulty acquisition nodes.

[0172] The second management subunit is used to determine that there are no faulty acquisition nodes in the distributed acquisition system if the comparison result shows that the number of acquisition nodes to which the heartbeat information belongs is consistent with the number of acquisition nodes included in the instance list, and to manage the acquisition tasks in the distributed acquisition system based on the heartbeat information of all acquisition nodes.

[0173] In some embodiments, the first management subunit is specifically configured to determine a first transfer task to be assigned to each non-faulty acquisition node based on the previous heartbeat information corresponding to the faulty acquisition node and the heartbeat information of each non-faulty acquisition node; and update the task table in the heartbeat information corresponding to each non-faulty acquisition node in the first database according to the first transfer task, wherein each task table is used to instruct each non-faulty acquisition node to perform task processing according to the task in the corresponding task table.

[0174] In some embodiments, the above-mentioned management unit further includes:

[0175] The removal unit is used to remove faulty data acquisition nodes from the instance list in order to update the instance list.

[0176] In some embodiments, the second management subunit is specifically used to determine whether there is an overloaded acquisition node among all acquisition nodes based on the heartbeat information of all acquisition nodes; if so, to manage the acquisition tasks on the overloaded acquisition node and other acquisition nodes; if not, to release management permissions.

[0177] In some embodiments, the second management subunit is specifically used to determine a second transfer task to be assigned to other acquisition nodes based on the heartbeat information of the overloaded acquisition node and the heartbeat information of other acquisition nodes; and to update the task table corresponding to all acquisition nodes in the first database according to the second transfer task.

[0178] In some embodiments, the above-mentioned acquisition module includes:

[0179] The determining unit is used to determine the status of the management lock held by each acquisition node based on the status information of the management lock in the heartbeat information of each acquisition node.

[0180] The first acquisition unit is used to update its own management lock status to a held state if all acquisition nodes are in an unheld state, thereby acquiring management privileges.

[0181] The second acquisition unit is used to acquire management permissions based on the heartbeat information of the target acquisition node if there is a target acquisition node among all acquisition nodes in the state of holding the management lock.

[0182] In some embodiments, the second acquisition unit includes:

[0183] The extraction sub-unit is used to extract the lock holding time from the heartbeat information of the target acquisition node.

[0184] The update sub-unit is used to update the state of the target acquisition node's management lock to the non-held state and update its own management lock to the held state if the lock holding time is greater than a preset time threshold, so as to obtain management rights.

[0185] The determination sub-unit is used to determine that management privileges have not been obtained if the lock holding time is not greater than a preset time threshold.

[0186] The modules in the management device of the aforementioned distributed acquisition system for the supercomputing internet can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the memory of the computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0187] In some embodiments, a computer device is provided, which may be a terminal or a server, and its internal structure diagram may be as follows. Figure 12 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a management method for a distributed acquisition system oriented towards a supercomputing internet. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0188] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0189] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the management method of the distributed acquisition system for the supercomputing internet described above.

[0190] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the management method for the distributed acquisition system for the supercomputing internet described above.

[0191] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the management method for the distributed acquisition system for the supercomputing internet described above.

[0192] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0193] The technical features of the above embodiments can be combined in any way. For the sake of brevity, as long as there is no contradiction in the combination of these technical features, they should all be considered to be within the scope of this specification.

[0194] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A management method for a distributed data acquisition system for a supercomputing internet, characterized in that, The management method is applied to any acquisition node in the distributed acquisition system, and each acquisition node communicates with a first database and a second database. The method includes: The heartbeat information of all collection nodes in the distributed collection system is obtained from the first database, and the state of the management lock in the collection node is determined according to the heartbeat information of all collection nodes. If the state of the management lock determined by the heartbeat information is unheld, or if there is a target collection node with the management lock in the held state and the lock holding time in the heartbeat information of the target collection node is greater than a preset time threshold, then management permission is obtained. Once the management privileges are obtained, a list of instances corresponding to the distributed acquisition system is retrieved from the second database. The list of instances is used to record all acquisition nodes that are currently working in the distributed acquisition system. Based on the consistency between the acquisition node to which the heartbeat information belongs and the acquisition node recorded in the instance list, the acquisition tasks on all acquisition nodes in the distributed acquisition system are managed.

2. The method according to claim 1, characterized in that, The management of collection tasks on all collection nodes in the distributed collection system based on the consistency between the collection node to which the heartbeat information belongs and the collection nodes recorded in the instance list includes: The heartbeat information is collected by comparing it with the collection nodes in the instance list to obtain the comparison results. Based on the comparison results, the acquisition tasks on all acquisition nodes in the distributed acquisition system are managed.

3. The method according to claim 2, characterized in that, The management of acquisition tasks on all acquisition nodes in the distributed acquisition system based on the comparison results includes: If the comparison result indicates that the number of collection nodes to which the heartbeat information belongs is inconsistent with the number of collection nodes included in the instance list, then it is determined that there is a faulty collection node in the distributed collection system, and the collection tasks on the faulty collection node and all non-faulty collection nodes are managed. If the comparison result indicates that the number of acquisition nodes to which the heartbeat information belongs is consistent with the number of acquisition nodes included in the instance list, then it is determined that there are no faulty acquisition nodes in the distributed acquisition system, and the acquisition tasks in the distributed acquisition system are managed based on the heartbeat information of all acquisition nodes.

4. The method according to claim 3, characterized in that, The management of acquisition tasks on the faulty acquisition node and all non-faulty acquisition nodes includes: Based on the previous heartbeat information corresponding to the faulty acquisition node and the heartbeat information of each non-faulty acquisition node, a first transfer task is determined and assigned to each non-faulty acquisition node. The task table in the heartbeat information corresponding to each of the non-faulty acquisition nodes in the first database is updated according to the first transfer task. Each task table is used to instruct each of the non-faulty acquisition nodes to perform task processing according to the tasks in the corresponding task table.

5. The method according to claim 3, characterized in that, After determining that a faulty acquisition node exists in the distributed acquisition system, the method further includes: Remove the fault acquisition node from the instance list to update the instance list.

6. The method according to any one of claims 3-5, characterized in that, The management of acquisition tasks in the distributed acquisition system based on the heartbeat information of all acquisition nodes includes: Based on the heartbeat information of all the acquisition nodes, determine whether there are any overloaded acquisition nodes among all the acquisition nodes; If present, the acquisition tasks on the overloaded acquisition node and other acquisition nodes will be managed. If it does not exist, then release the aforementioned management privileges.

7. The method according to claim 6, characterized in that, The management of data acquisition tasks on the overloaded acquisition node and other acquisition nodes includes: Based on the heartbeat information of the overloaded acquisition node and the heartbeat information of other acquisition nodes, a second transfer task is determined and assigned to the other acquisition nodes; Update the task table corresponding to all collection nodes in the first database according to the second transfer task.

8. The method according to claim 1, characterized in that, The step of obtaining management permissions and determining the status of the management lock in the acquisition node based on the heartbeat information of all acquisition nodes includes: Based on the status information of the management lock in the heartbeat information of each of the acquisition nodes, determine the status of each acquisition node holding the management lock; If the management lock status determined by each of the heartbeat information is in an unheld state, or if there is a target acquisition node with a management lock in a held state and the lock holding time in the heartbeat information of the target acquisition node is greater than a preset time threshold, then management permission is obtained, including: If all the management locks of the collection nodes are in an unheld state, then the management lock of the node itself is updated to a held state in order to obtain the management permission; If there is a target acquisition node among all acquisition nodes that is in a holding state of the management lock, and if the lock holding time in the heartbeat information of the target acquisition node is greater than a preset time threshold, then the management permission is obtained.

9. The method according to claim 8, characterized in that, If the lock holding time in the heartbeat information of the target acquisition node is greater than a preset time threshold, then the management permission is obtained, including: Extract the lock holding time from the heartbeat information of the target acquisition node; If the lock holding time is greater than a preset time threshold, the state of the management lock of the target acquisition node is updated to an unheld state, and the state of its own management lock is updated to a held state, so as to obtain the management permission; If the lock holding time is not greater than a preset time threshold, it is determined that the management permission has not been obtained.

10. A management system for a distributed acquisition system for a supercomputing internet, characterized in that, The management system includes: a distributed data acquisition node cluster, a first database engine, and a second database engine; each data acquisition node in the distributed data acquisition node cluster is connected to the first database engine and the second database engine respectively; a first database is deployed on the first database engine, and a second database is deployed on the second database engine; Any one of the acquisition nodes in the distributed acquisition node cluster is used to execute the method as described in any one of claims 1-9.

11. A management device for a distributed acquisition system for a supercomputing internet, belonging to any acquisition node in the distributed acquisition system, wherein each acquisition node communicates with a first database and a second database, characterized in that, The device includes: The acquisition module is used to acquire the heartbeat information of all acquisition nodes in the distributed acquisition system from the first database, and determine the state of the management lock in the acquisition node based on the heartbeat information of all acquisition nodes. If the state of the management lock determined by the heartbeat information is unheld, or if there is a target acquisition node with the management lock in the held state and the lock holding time in the heartbeat information of the target acquisition node is greater than a preset time threshold, then the management permission is acquired. The management module is used to retrieve the instance list corresponding to the distributed acquisition system from the second database when it is determined that the management permission has been obtained. The instance list is used to record all acquisition nodes that are currently working in the distributed acquisition system, and to manage the acquisition tasks on all acquisition nodes in the distributed acquisition system based on the consistency between the acquisition node to which the heartbeat information belongs and the acquisition node recorded in the instance list.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images