Data balanced collection method and device, collection server and collection service cluster of data center

Abstract

The application is suitable for the technical field of data collection, and provides a data balanced collection method and device, a collection server and a collection service cluster of a data center. The method is applied to the collection server in the collection service cluster, and includes: when the local machine is used as a collection server with a scheduling function, each collection task is balancedly scheduled to each collection server according to the measuring point information of each collection task to be scheduled and the capacity information of each collection server, so that each collection server collects measuring point data based on the corresponding collection task; when the collection task changes, the measuring point information of the changed collection task is used to determine whether to apply a new server as a collection server when the server is expanded, and the changed collection task is balancedly scheduled to each collection server. The application can effectively meet the data collection demand in the massive measuring point scene, and can timely perform dynamic expansion of the collection server when needed, thereby improving the data collection efficiency and stability.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, specifically to a data balanced acquisition method, device, acquisition server, and data center acquisition service cluster. Background Technology

[0002] As the core carrier of various information systems and data resources, data centers are crucial for ensuring the stable operation of these systems and supporting continuous business operations. Therefore, real-time monitoring and analysis of the data center's operational status and environment are essential, requiring the collection of relevant data from data center equipment.

[0003] In data acquisition, the acquisition server typically collects measurement data from each device. Each device, acting as a field physical terminal, generally corresponds to multiple measurement points, and the measurement data is uploaded to the corresponding acquisition server via its respective acquisition channel.

[0004] In related technologies, data acquisition servers typically employ a master-slave deployment architecture, with all acquisition channels connected to a single master server at any given time. However, as data centers continue to expand and the scale of related equipment and measurement points continues to grow, the aforementioned data acquisition methods are no longer sufficient to meet the demand for efficient and stable data acquisition from massive numbers of measurement points. Summary of the Invention

[0005] In view of this, embodiments of this application provide a data balanced acquisition method, apparatus, acquisition server, and data center acquisition service cluster to solve the technical problem that related technologies cannot meet the acquisition needs of massive measurement points.

[0006] In a first aspect, embodiments of this application provide a data balanced acquisition method, applied to an acquisition server in an acquisition service cluster, the method comprising: When the local machine acts as a data acquisition server with scheduling function, it will evenly schedule each data acquisition task to each data acquisition server according to the measurement point information of each data acquisition task to be scheduled and the capacity information of each data acquisition server in the data acquisition service cluster, so that each data acquisition server can perform measurement point data acquisition based on the corresponding data acquisition task. When the data collection task changes, the system determines whether to expand the server capacity based on the measurement point information of the changed data collection task. If the system determines that server expansion is necessary, a new server is requested as the data collection server, and the changed data collection task is evenly distributed to each data collection server.

[0007] In one possible implementation of the first aspect, the measuring point information includes the number of measuring points, and the capacity information includes the remaining capacity; The step of evenly distributing each acquisition task to each acquisition server based on the measurement point information of each acquisition task to be scheduled and the capacity information of each acquisition server in the acquisition service cluster includes: Based on the number of measurement points in each data acquisition task to be scheduled, sort the data acquisition tasks in descending order of the number of measurement points to obtain the sorting order; According to the sorting order, for each acquisition task, the acquisition task is scheduled to the acquisition server with the largest remaining capacity, which is greater than or equal to the number of measurement points of the acquisition task.

[0008] In one possible implementation of the first aspect, the changes in the data acquisition task include adding new data acquisition tasks and increasing the number of measurement points for the data acquisition tasks; When the data acquisition task changes, the system determines whether to expand server capacity based on the measurement point information of the changed task, including: If a new data collection task is added, it is determined whether the number of measurement points in the new data collection task is greater than the remaining capacity of each data collection server. If the number of measurement points in the new data collection task is greater than the remaining capacity of each data collection server, it is determined to expand the server capacity. If the number of measurement points in a data acquisition task increases, it is determined whether the number of new measurement points in the target data acquisition task exceeds the remaining capacity of the corresponding data acquisition server, and whether the number of measurement points in the target data acquisition task exceeds the remaining capacity of each data acquisition server. If the number of new measurement points in the target data acquisition task exceeds the remaining capacity of the corresponding data acquisition server, and the number of measurement points in the target data acquisition task exceeds the remaining capacity of each data acquisition server, it is determined to expand the server capacity. Here, the target data acquisition task is the data acquisition task with the increased number of measurement points.

[0009] In one possible implementation of the first aspect, the method further includes: Perform heartbeat detection on data acquisition servers other than the local machine and obtain the detection results; Based on the detection results, determine whether the data acquisition servers other than the local machine are abnormal; If an abnormal data collection server is found, the data collection tasks corresponding to the abnormal data collection server will be evenly distributed to the other data collection servers.

[0010] In one possible implementation of the first aspect, the method further includes: When the data acquisition task changes, the system determines whether to scale down the server capacity based on the measurement point information of the changed data acquisition task and the capacity information of each data acquisition server. If server scaling down is determined, the data collection server to be scaled down is identified, and the data collection tasks corresponding to the data collection server to be scaled down are evenly distributed to the data collection servers other than the data collection server to be scaled down.

[0011] In one possible implementation of the first aspect, the method further includes: When this machine acts as any data acquisition server, it will preempt the distributed lock every preset time period. If the distributed lock is acquired, it will act as a data acquisition server with scheduling capabilities. If the distributed lock is not acquired, the server will function as a collection server without scheduling capabilities.

[0012] In one possible implementation of the first aspect, the method further includes: When the local machine acts as any acquisition server, it establishes a connection with the corresponding acquisition channel according to the acquisition channel identifier in the acquisition task assigned to it. Data from corresponding measurement points is collected based on the established acquisition channels.

[0013] Secondly, embodiments of this application provide a data equalization acquisition device, comprising: The scheduling module is used to distribute the data acquisition tasks evenly to each data acquisition server when the local machine is used as a data acquisition server with scheduling function, based on the measurement point information of each data acquisition task to be scheduled and the capacity information of each data acquisition server in the data acquisition service cluster, so that each data acquisition server can perform measurement point data acquisition based on the corresponding data acquisition task.

[0014] The expansion module is used to determine whether to expand the server capacity when the data acquisition task changes, based on the measurement point information of the changed data acquisition task. If the server expansion is determined, a new server is requested as the data acquisition server, and the changed data acquisition task is evenly scheduled to each data acquisition server.

[0015] Thirdly, embodiments of this application provide a data acquisition server, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the data equalization acquisition method as described in any of the first aspects.

[0016] Fourthly, embodiments of this application provide a data center acquisition service cluster, including at least one acquisition server as described in the third aspect.

[0017] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data equalization acquisition method as described in any of the first aspects.

[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0019] The data balancing acquisition method, apparatus, acquisition server, and data center acquisition service cluster provided in this application embodiment, when the local machine acts as an acquisition server with scheduling function in the acquisition service cluster, balances the acquisition tasks to each acquisition server according to the measurement point information of each acquisition task to be scheduled and the capacity information of each acquisition server in the acquisition service cluster. This allows each acquisition server to collect measurement point data based on the corresponding acquisition task. In this way, by setting up multiple acquisition servers and balancing the acquisition tasks, the data acquisition needs in scenarios with massive measurement points can be effectively met, while avoiding excessive load on a single server, improving data acquisition efficiency and stability. When the acquisition task changes, the server capacity is determined based on the measurement point information of the changed acquisition task, so that the acquisition server can be dynamically expanded in a timely manner when needed, further adapting to the acquisition needs of the measurement points, realizing elastic scaling of the cluster's operating capacity, and improving the cluster's scalability and resource utilization.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application; Figure 2 This is a schematic flowchart of a data equalization acquisition method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a data equalization acquisition device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a data acquisition server provided in one embodiment of this application. Detailed Implementation

[0023] The present application will be described more clearly below with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the function of the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0024] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0025] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] In the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] Furthermore, the term "multiple" mentioned in the embodiments of this application should be interpreted as two or more.

[0029] Real-time monitoring and analysis of data center operation status and server room environment are crucial; therefore, it is necessary to collect relevant data from data center equipment. Data acquisition typically involves a data acquisition server collecting measurement data from each device. The acquisition channel serves as the data transmission link between the acquisition server and the devices, with one end connected to the acquisition server and the other end connected to at least one device. Each device, as a physical terminal in the field, usually corresponds to multiple measurement points, and the measurement point data is uploaded to the corresponding acquisition server through its assigned acquisition channel.

[0030] In related technologies, data acquisition servers typically employ a master-slave deployment architecture, with all acquisition channels connected to a single master server at any given time. However, as data centers continue to expand and the scale of related equipment and measurement points continues to grow, the aforementioned data acquisition methods are no longer sufficient to meet the demand for efficient and stable data acquisition from massive numbers of measurement points.

[0031] To address the aforementioned issues, in the embodiments of this application, multiple acquisition servers can be configured within the acquisition service cluster. When the local machine acts as an acquisition server with scheduling capabilities within the acquisition service cluster, the acquisition tasks are evenly distributed to each acquisition server based on the measurement point information of each acquisition task to be scheduled and the capacity information of each acquisition server. This allows each acquisition server to collect measurement point data based on the corresponding acquisition task, effectively meeting the data acquisition needs in scenarios with massive measurement points. Simultaneously, it avoids excessive load on a single server, improving data acquisition efficiency and stability. When acquisition tasks change, the server capacity is determined based on the measurement point information of the changed acquisition tasks. This allows for timely dynamic expansion of the acquisition servers when needed, further adapting to the acquisition requirements of the measurement points, achieving elastic scaling of the cluster's operating capacity, and improving the cluster's scalability and resource utilization.

[0032] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application. The application scenario includes a data acquisition service cluster and devices to be acquired. The data acquisition service cluster includes at least one data acquisition server, and one of the data acquisition servers serves as a scheduling center with scheduling functions. The devices to be acquired refer to devices whose data is to be acquired, such as uninterruptible power supplies (UPS), batteries, air conditioners, etc., related to data centers.

[0033] The dispatch center, based on the measurement point information of each data acquisition task and the capacity information of each data acquisition server, evenly distributes the tasks to the respective servers. This allows each server to collect measurement point data based on its assigned task. When a task changes, the center, based on the updated measurement point information, determines whether server expansion is needed, requests a new server as the new data acquisition server, and then evenly distributes the updated tasks to all servers. There can be multiple devices and multiple data acquisition tasks.

[0034] Figure 2 This is a schematic flowchart of a data equalization acquisition method provided in an embodiment of this application. Figure 2 As shown in the embodiments of this application, the method is applied to the collection server in the collection service cluster, and the above method may include: S201. When the local machine acts as a data acquisition server with scheduling function, it will evenly schedule each data acquisition task to each data acquisition server based on the measurement point information of each data acquisition task to be scheduled and the capacity information of each data acquisition server in the data acquisition service cluster, so that each data acquisition server can perform measurement point data acquisition based on the corresponding data acquisition task.

[0035] In the data acquisition service cluster, only one data acquisition server with scheduling capabilities, namely the scheduling center, exists at any given time to avoid concurrent conflicts in the scheduling logic. The scheduling center is used to uniformly schedule all data acquisition tasks to the various data acquisition servers within the cluster (including the scheduling center itself). Furthermore, each data acquisition server in the data acquisition service cluster is a stateless server. Stateless servers do not store the runtime state of the data acquisition tasks themselves; the runtime state is uniformly stored in shared storage.

[0036] There can be multiple data acquisition tasks, stored in a pre-defined database. As the smallest scheduling unit, a data acquisition task includes information about the corresponding device to be acquired, information about the corresponding measurement points of the device, information about the corresponding acquisition channel, and a set of overall measurement point information. For example, if the device to be acquired is a UPS, the corresponding measurement points include output voltage, current, power, etc.; if the device to be acquired is a dedicated air conditioner for a computer room, the corresponding measurement points include return air temperature and humidity, supply air temperature and humidity, etc.

[0037] The data acquisition task's measurement point information includes the number of measurement points, and the data acquisition server's capacity information includes the remaining capacity, which is the number of measurement points the data acquisition server can still support.

[0038] In some embodiments, when allocating all acquisition tasks to all acquisition servers in a balanced manner, the acquisition tasks can be sorted according to the number of measurement points of each acquisition task to be scheduled, in descending order of the number of measurement points, to obtain a sorting order. Then, according to the sorting order, for each acquisition task, the acquisition task is scheduled to the acquisition server with the largest remaining capacity and a number of measurement points greater than or equal to that acquisition task.

[0039] Here, following the principle of balanced scheduling, the acquisition tasks to be scheduled are sorted in descending order of the number of measurement points. Then, in that order, each acquisition task is sequentially scheduled to the acquisition server with the largest remaining capacity that meets the measurement point requirement for that task. Specifically, an acquisition server that meets the measurement point requirement can be either a server with remaining capacity greater than or equal to the number of measurement points for that task, or a server with remaining capacity greater than the number of measurement points for that task, where the difference between the two is greater than a preset difference.

[0040] Thus, since the data acquisition service cluster includes multiple acquisition servers, it can meet the data acquisition needs of a massive number of measurement points. On this basis, balanced scheduling of acquisition tasks can achieve a reasonable match between acquisition tasks and the processing capacity of acquisition servers, avoid uneven load on acquisition servers, improve the overall utilization rate of the data acquisition service cluster, and ensure the stability and efficiency of data acquisition.

[0041] Redis is a distributed database that is stored in memory and has a fast read and write speed. In some embodiments, after scheduling each collection task to each collection server, the scheduling information can be stored in the Redis database, thereby realizing configuration sharing and state synchronization among the collection servers in the cluster. The scheduling information may include the correspondence between the label of the collection task and the label of the collection server to which the collection task is scheduled.

[0042] S202. When the data collection task changes, determine whether to expand the server capacity based on the measurement point information of the changed data collection task. If it is determined that the server capacity needs to be expanded, apply for a new server as the data collection server and evenly schedule the changed data collection task to each data collection server.

[0043] In real-world applications, data collection tasks may change. For example, changes could include adding new tasks or increasing the number of measurement points in existing tasks. Furthermore, the scheduling center can subscribe to a Kafka message queue to receive notifications of changes to the data collection configuration. Upon receiving messages indicating the addition, deletion, or modification of a data collection task, a change is identified. When a change is detected, the system can determine whether server expansion is necessary based on the number of measurement points in the changed task and the remaining capacity of the data collection server. This allows for timely requests for new servers to join the data collection service cluster when expansion is required, enabling dynamic scaling of the data collection server and elastic scaling of the cluster's operating capacity. This further adapts to the data collection needs of the measurement points while preventing issues such as excessive load, data collection delays, or decreased service stability.

[0044] Furthermore, when a new data collection server joins the cluster (for example, when an administrator manually starts a new server or when a new server automatically applies for service), the new server will register with the data collection server (scheduling center) that has scheduling capabilities. After receiving the registration request, the scheduling center adds the new server to the scheduling pool and can distribute some data collection tasks evenly to the new server based on the current load of each data collection server.

[0045] In some embodiments, when determining whether to expand server capacity, it is possible to A1. If a new data collection task is added, determine whether the number of measurement points in the new task is greater than the remaining capacity of each data collection server. If the number of measurement points in the new task is greater than the remaining capacity of each data collection server, determine whether to expand the server capacity.

[0046] A2. If the number of measurement points in the data acquisition task increases, determine whether the number of new measurement points in the target data acquisition task is greater than the remaining capacity of the corresponding data acquisition server, and whether the number of measurement points in the target data acquisition task is greater than the remaining capacity of each data acquisition server. If the number of new measurement points in the target data acquisition task is greater than the remaining capacity of the corresponding data acquisition server, and the number of measurement points in the target data acquisition task is greater than the remaining capacity of each data acquisition server, then determine whether to expand the server capacity. Here, the target data acquisition task is the data acquisition task with the increased number of measurement points.

[0047] In this embodiment, if a new data acquisition task is added, and the number of measurement points for the new task exceeds the remaining capacity of each data acquisition server in the cluster, it is determined that the cluster cannot meet the data acquisition requirements and server expansion is necessary. Otherwise, the new data acquisition task is scheduled to the data acquisition server with the largest remaining capacity.

[0048] If the number of measurement points for a data acquisition task increases, and if the number of new measurement points for the target data acquisition task exceeds the remaining capacity of the currently scheduled data acquisition server, and the number of measurement points for the target data acquisition task exceeds the remaining capacity of all data acquisition servers in the cluster, then it is determined that the cluster cannot meet the data acquisition requirements, and server expansion is necessary. Otherwise, server expansion is not considered necessary. If the number of new measurement points for the target data acquisition task is less than or equal to the remaining capacity of the currently scheduled data acquisition server, the target data acquisition task does not need to be rescheduled. If the number of measurement points for the target data acquisition task is less than or equal to the remaining capacity of any data acquisition server in the cluster, the target data acquisition task is scheduled to the data acquisition server with the largest remaining capacity.

[0049] In this embodiment, different situations of changes in data acquisition tasks are distinguished, and a differentiated expansion judgment strategy is adopted. This strategy can accurately identify whether the cluster's carrying capacity is sufficient in different scenarios. Thus, when the cluster's carrying capacity is insufficient, the data acquisition server can be flexibly expanded in a timely manner, realizing the elastic scaling of the cluster's operating capacity and further adapting to the data acquisition needs of the measurement points.

[0050] In this embodiment, the information of each collection server in the collection service cluster, such as registration information, can be stored in a Redis database, thereby realizing configuration sharing and status synchronization among the collection servers in the cluster.

[0051] When a new data acquisition server joins the cluster, the new data acquisition server first registers with the scheduling center. After receiving the registration request, the scheduling center will evenly distribute the data acquisition tasks to the new data acquisition server.

[0052] In some embodiments, changes to the acquisition task may also include deleting acquisition tasks and reducing the number of measurement points in existing acquisition tasks. When an acquisition task changes, based on the measurement point information of the changed acquisition task and the capacity information of each acquisition server, it is determined whether to perform server downsizing. If server downsizing is determined, the acquisition server to be downsized is identified, and the acquisition tasks corresponding to the acquisition server to be downsized are evenly distributed to the acquisition servers other than the acquisition server to be downsized.

[0053] The capacity information for the data acquisition server also includes the occupied capacity, which is the number of measurement points currently being handled by the server. For each data acquisition server, the occupied capacity plus the remaining capacity equals the total capacity.

[0054] For example, if the change in the data acquisition task involves deleting acquisition tasks and / or reducing the number of measurement points in the acquisition task, then after the change, the acquisition server in the cluster whose occupied capacity is less than a preset capacity threshold is identified as the target acquisition server. If the occupied capacity of the target acquisition server is 0 and the duration is greater than a first preset duration, then the target acquisition server is identified as an acquisition server to be scaled down. If the occupied capacity of the target acquisition server is greater than 0 and the duration is greater than the first preset duration, and there are acquisition servers in the cluster with remaining capacity greater than or equal to the occupied capacity of the target acquisition server, then the target acquisition server is identified as a server to be scaled down, and based on the principle of balanced scheduling, the acquisition tasks corresponding to the target acquisition server are scheduled to the aforementioned acquisition servers with remaining capacity greater than or equal to the occupied capacity of the target acquisition server. Otherwise, the target acquisition server is not determined to be an acquisition server to be scaled down.

[0055] After identifying the data acquisition server to be scaled down and distributing the data acquisition tasks corresponding to that server evenly to other data acquisition servers, the data acquisition server to be scaled down is removed from the scheduling pool.

[0056] In this way, by determining whether to scale down servers when data collection tasks change, idle data collection servers can be released promptly when cluster resources are redundant, avoiding long-term idle waste and improving resource utilization. Simultaneously, by prioritizing data collection tasks and then removing data collection servers to be scaled down, smooth scheduling of data collection tasks is ensured.

[0057] In some embodiments, heartbeat detection can also be performed on data acquisition servers other than the local machine to obtain detection results. Based on the detection results, it can be determined whether the data acquisition servers other than the local machine are abnormal. If there are abnormal data acquisition servers, the data acquisition tasks corresponding to the abnormal data acquisition servers are evenly scheduled to the data acquisition servers other than the abnormal data acquisition servers.

[0058] For example, the detection results can include whether the data acquisition server is normal or abnormal. For instance, during heartbeat detection, if a data acquisition server does not respond or its health status is unhealthy, the corresponding detection result is that the data acquisition server is abnormal. If the data acquisition server can provide a normal response, the corresponding detection result is that the data acquisition server is normal. For abnormal data acquisition servers, the data acquisition tasks already scheduled to that server are evenly distributed to other data acquisition servers, and the abnormal server is removed from the scheduling pool. The above-described even distribution process can be referred to the relevant descriptions in the preceding embodiments, and will not be repeated here.

[0059] Here, heartbeat detection can promptly identify abnormal data acquisition servers, thereby evenly distributing the acquisition tasks corresponding to the abnormal servers to the normal ones. This avoids data acquisition interruptions caused by the failure of a single server, improving the fault tolerance and overall reliability of the data acquisition service cluster.

[0060] Optionally, in real-world scenarios, there may also be situations where a data acquisition server with scheduling functions, i.e., a scheduling center, is malfunctioning. To avoid the above situation from having a significant impact on data acquisition, this embodiment can also implement scheduling center election and switching through a distributed lock mechanism.

[0061] In some embodiments, when the local machine acts as any acquisition server in the acquisition service cluster, it attempts to acquire the distributed lock every preset time period. If it acquires the distributed lock, it acts as an acquisition server with scheduling function; if it fails to acquire the distributed lock, it acts as an acquisition server without scheduling function.

[0062] Optionally, the distributed lock mentioned above can be a Redis distributed lock. Each collection server in the cluster periodically competes for the distributed lock. If the current scheduling center is normal, the scheduling center will continuously renew the distributed lock to maintain its holding of the distributed lock. Although other collection servers periodically initiate lock acquisition, they fail to acquire the distributed lock because it has already been occupied.

[0063] If the scheduling center malfunctions and cannot renew the distributed lock, the distributed lock will be automatically released. In the new round of lock acquisition, the acquisition server that acquires the distributed lock will become the new scheduling center, while the remaining acquisition servers will become ordinary acquisition servers without scheduling functions.

[0064] The new scheduling center can continue to balance the scheduling of data collection tasks based on the preset collection tasks in the database and the information of each collection server in the Redis database, so as to ensure the stability of data collection.

[0065] In this embodiment, a distributed lock is used to elect the scheduling center, ensuring that only one data acquisition server exercises scheduling authority at any given time. This effectively avoids concurrent conflicts caused by multiple data acquisition servers executing scheduling logic simultaneously. Furthermore, each data acquisition server periodically participates in lock contention, enabling rapid switching of the scheduling center in the event of an anomaly in the current one. This prevents overall scheduling interruption due to the failure of a single data acquisition server, significantly improving the high availability and fault tolerance of the data acquisition service cluster and ensuring continuous and stable data acquisition operation.

[0066] In some embodiments, when the local machine is any acquisition server in the acquisition service cluster, it can establish a connection with the corresponding acquisition channel according to the identifier of the acquisition channel in the acquisition task assigned to it, and acquire the corresponding measurement point data based on the established acquisition channel.

[0067] As mentioned above, the data acquisition task includes information about the corresponding acquisition channel, such as the channel identifier, which can specifically be an IP address. Here, after the scheduling center evenly distributes each acquisition task to the acquisition servers in the cluster, for each acquisition server, it establishes a connection with the acquisition channel (such as a network connection or communication connection) based on the IP address of the acquisition channel in the assigned acquisition task, and completes connection verification. After the connection is successfully established, the server acquires the corresponding measurement point data based on the established acquisition channel to complete the assigned acquisition task.

[0068] In this way, each acquisition server can be accurately matched with the corresponding acquisition channel, avoiding channel confusion or duplicate acquisition, improving the accuracy and reliability of data acquisition, and ensuring that each acquisition server efficiently executes the assigned acquisition tasks.

[0069] In some embodiments, the scheduling status of a data acquisition task includes unscheduled, in scheduling, and successfully scheduled. After the scheduling center evenly distributes each data acquisition task to the data acquisition servers in the cluster, it updates the scheduling status of each data acquisition task in the scheduling information to in scheduling.

[0070] After the connection between each acquisition server and the corresponding acquisition channel is successfully established, a scheduling success message is sent to the scheduling center. Based on the aforementioned scheduling success message, the scheduling center updates the scheduling status of the corresponding acquisition task in the scheduling information to "scheduling successful".

[0071] If a data collection task is scheduled for a duration longer than the second preset duration, the task is considered to have failed to be scheduled. The scheduling center will update the scheduling status of the task to "not scheduled" and remove the data collection server from the scheduling information so that the scheduling center can reschedule the task in the future.

[0072] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0073] Figure 3 This is a schematic diagram of the structure of a data equalization acquisition device provided in an embodiment of this application. Figure 3 As shown, the data balancing acquisition device provided in this embodiment may include a scheduling module 301 and an expansion module 302.

[0074] The scheduling module 301 is used to, when the local machine is used as a data acquisition server with scheduling function, allocate each data acquisition task to each data acquisition server in a balanced manner according to the measurement point information of each data acquisition task to be scheduled and the capacity information of each data acquisition server in the data acquisition service cluster, so that each data acquisition server can perform measurement point data acquisition based on the corresponding data acquisition task.

[0075] The expansion module 302 is used to determine whether to expand the server capacity based on the measurement point information of the changed acquisition task when the acquisition task changes. When it is determined that the server capacity needs to be expanded, a new server is requested as the acquisition server, and the changed acquisition task is evenly scheduled to each acquisition server.

[0076] Optionally, the measurement point information includes the number of measurement points, and the capacity information includes the remaining capacity; the scheduling module 301 is further configured to: Based on the number of measurement points in each data acquisition task to be scheduled, sort the data acquisition tasks in descending order of the number of measurement points to obtain the sorting order; According to the sorting order, for each acquisition task, the acquisition task is scheduled to the acquisition server with the largest remaining capacity, which is greater than or equal to the number of measurement points of the acquisition task.

[0077] Optionally, changes to the data acquisition task include adding new data acquisition tasks and increasing the number of measurement points for each task; the expansion module 302 is also used for: If a new data collection task is added, it is determined whether the number of measurement points in the new data collection task is greater than the remaining capacity of each data collection server. If the number of measurement points in the new data collection task is greater than the remaining capacity of each data collection server, it is determined to expand the server capacity. If the number of measurement points in a data acquisition task increases, it is determined whether the number of new measurement points in the target data acquisition task exceeds the remaining capacity of the corresponding data acquisition server, and whether the number of measurement points in the target data acquisition task exceeds the remaining capacity of each data acquisition server. If the number of new measurement points in the target data acquisition task exceeds the remaining capacity of the corresponding data acquisition server, and the number of measurement points in the target data acquisition task exceeds the remaining capacity of each data acquisition server, it is determined to expand the server capacity. Here, the target data acquisition task is the data acquisition task with the increased number of measurement points.

[0078] Optionally, the scheduling module 301 is also used for: Perform heartbeat detection on data acquisition servers other than the local machine and obtain the detection results; Based on the detection results, determine whether the data acquisition servers other than the local machine are abnormal; If an abnormal data collection server is found, the data collection tasks corresponding to the abnormal data collection server will be evenly distributed to the other data collection servers.

[0079] Optionally, the expansion module 302 is also used for: When the data acquisition task changes, the system determines whether to scale down the server capacity based on the measurement point information of the changed data acquisition task and the capacity information of each data acquisition server. If server scaling down is determined, the data collection server to be scaled down is identified, and the data collection tasks corresponding to the data collection server to be scaled down are evenly distributed to the data collection servers other than the data collection server to be scaled down.

[0080] Optionally, the scheduling module 301 is also used for: When this machine acts as any data acquisition server, it will preempt the distributed lock every preset time period. If the distributed lock is acquired, it will act as a data acquisition server with scheduling capabilities. If the distributed lock is not acquired, the server will function as a collection server without scheduling capabilities.

[0081] Optionally, the scheduling module 301 is also used for: When the local machine acts as any acquisition server, it establishes a connection with the corresponding acquisition channel according to the acquisition channel identifier in the acquisition task assigned to it. Data from corresponding measurement points is collected based on the established acquisition channels.

[0082] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0083] Figure 4 This is a schematic diagram of the structure of a data acquisition server provided in one embodiment of this application. Figure 4 As shown, the data acquisition server 400 in this embodiment includes a processor 410 and a memory 420. The memory 420 stores a computer program 421 that can run on the processor 410. When the processor 410 executes the computer program 421, it implements the steps in any of the above method embodiments, for example... Figure 2The steps S201 to S202 are shown. Alternatively, when the processor 410 executes the computer program 421, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 3 The functions of modules 301 to 302 are shown.

[0084] For example, computer program 421 may be divided into one or more modules / units, one or more of which are stored in memory 420 and executed by processor 410 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 421 in acquisition server 400.

[0085] Those skilled in the art will understand that Figure 4 This is merely an example of a data acquisition server and does not constitute a limitation on the data acquisition server. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0086] The processor 410 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0087] The memory 420 can be an internal storage unit of the acquisition server, such as the hard drive or memory of the acquisition server, or an external storage device of the acquisition server, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the acquisition server. The memory 420 can also include both internal and external storage units of the acquisition server. The memory 420 is used to store computer programs and other programs and data required by the acquisition server. The memory 420 can also be used to temporarily store data that has been output or will be output.

[0088] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0089] One embodiment of this application also provides a data center acquisition service cluster, including at least one acquisition server as described above.

[0090] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0091] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0092] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / acquisition server and method can be implemented in other ways. For example, the apparatus / acquisition server embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0096] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data balanced acquisition method, characterized in that, The method, applied to a data acquisition server in a data acquisition service cluster, includes: When the local machine acts as a data acquisition server with scheduling function, it will evenly schedule each data acquisition task to each data acquisition server according to the measurement point information of each data acquisition task to be scheduled and the capacity information of each data acquisition server in the data acquisition service cluster, so that each data acquisition server can perform measurement point data acquisition based on the corresponding data acquisition task. When the data collection task changes, the system determines whether to expand the server capacity based on the measurement point information of the changed data collection task. If the system determines that server expansion is necessary, a new server is requested as the data collection server, and the changed data collection task is evenly distributed to each data collection server.

2. The data equalization acquisition method according to claim 1, characterized in that, The measuring point information includes the number of measuring points, and the capacity information includes the remaining capacity; The step of evenly distributing each acquisition task to each acquisition server based on the measurement point information of each acquisition task to be scheduled and the capacity information of each acquisition server in the acquisition service cluster includes: Based on the number of measurement points in each data acquisition task to be scheduled, sort the data acquisition tasks in descending order of the number of measurement points to obtain the sorting order; According to the sorting order, for each acquisition task, the acquisition task is scheduled to the acquisition server with the largest remaining capacity, which is greater than or equal to the number of measurement points of the acquisition task.

3. The data equalization acquisition method according to claim 2, characterized in that, Changes in data acquisition tasks include the addition of new data acquisition tasks and an increase in the number of measurement points for each task. When the data acquisition task changes, the system determines whether to expand server capacity based on the measurement point information of the changed task, including: If a new data collection task is added, it is determined whether the number of measurement points in the new data collection task is greater than the remaining capacity of each data collection server. If the number of measurement points in the new data collection task is greater than the remaining capacity of each data collection server, it is determined to expand the server capacity. If the number of measurement points in a data acquisition task increases, it is determined whether the number of new measurement points in the target data acquisition task exceeds the remaining capacity of the corresponding data acquisition server, and whether the number of measurement points in the target data acquisition task exceeds the remaining capacity of each data acquisition server. If the number of new measurement points in the target data acquisition task exceeds the remaining capacity of the corresponding data acquisition server, and the number of measurement points in the target data acquisition task exceeds the remaining capacity of each data acquisition server, it is determined to expand the server capacity. Here, the target data acquisition task is the data acquisition task with the increased number of measurement points.

4. The data equalization acquisition method according to any one of claims 1 to 3, characterized in that, The method further includes: Perform heartbeat detection on data acquisition servers other than the local machine and obtain the detection results; Based on the detection results, determine whether the data acquisition servers other than the local machine are abnormal; If an abnormal data collection server is found, the data collection tasks corresponding to the abnormal data collection server will be evenly distributed to the other data collection servers.

5. The data equalization acquisition method according to any one of claims 1 to 3, characterized in that, The method further includes: When the data acquisition task changes, the system determines whether to scale down the server capacity based on the measurement point information of the changed data acquisition task and the capacity information of each data acquisition server. If server scaling down is determined, the data collection server to be scaled down is identified, and the data collection tasks corresponding to the data collection server to be scaled down are evenly distributed to the data collection servers other than the data collection server to be scaled down.

6. The data equalization acquisition method according to any one of claims 1 to 3, characterized in that, The method further includes: When this machine acts as any data acquisition server, it will preempt the distributed lock every preset time period. If the distributed lock is acquired, it will act as a data acquisition server with scheduling capabilities. If the distributed lock is not acquired, the server will function as a collection server without scheduling capabilities.

7. The data equalization acquisition method according to any one of claims 1 to 3, characterized in that, The method further includes: When the local machine acts as any acquisition server, it establishes a connection with the corresponding acquisition channel according to the acquisition channel identifier in the acquisition task assigned to it. Data from corresponding measurement points is collected based on the established acquisition channels.

8. A data equalization acquisition device, characterized in that, include: The scheduling module is used to evenly schedule each acquisition task to each acquisition server when the local machine is used as an acquisition server with scheduling function, based on the measurement point information of each acquisition task to be scheduled and the capacity information of each acquisition server in the acquisition service cluster, so that each acquisition server can perform measurement point data acquisition based on the corresponding acquisition task. The expansion module is used to determine whether to expand the server capacity when the data acquisition task changes, based on the measurement point information of the changed data acquisition task. If the server expansion is determined, a new server is requested as the data acquisition server, and the changed data acquisition task is evenly scheduled to each data acquisition server.

9. A data acquisition server, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the data equalization acquisition method as described in any one of claims 1 to 7.

10. A data acquisition service cluster for a data center, characterized in that, It includes at least one acquisition server as described in claim 9.

Images