Data collection method, apparatus, device, medium, and program product

By acquiring the full cache and hot data of the supercomputing cluster in the data center and using a preset frequency and thread pool mechanism for synchronous processing, the resource pressure and consistency issues in supercomputing cluster data acquisition are resolved, achieving efficient and accurate data acquisition.

CN118643043BActive Publication Date: 2026-02-06DAWNING INFORMATION IND (BEIJING) CO LTD +2
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202310235600.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-13
Publication Date
2026-02-06
Estimated Expiration
2043-03-13

AI Technical Summary

Technical Problem

Existing technologies cannot simultaneously address the system's resource pressure and data consistency across different supercomputing clusters in cross-regional supercomputing cluster data acquisition.

Method used

By acquiring full cache data and hot data from various supercomputing clusters in the data center, and using the hot data to synchronously process the full cache data, combined with a preset frequency and thread pool mechanism, real-time data acquisition and reasonable load management are achieved.

Benefits of technology

While reducing system load, it ensures the consistency and real-time performance of data across supercomputing clusters, thereby improving the efficiency and accuracy of data collection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118643043B_ABST
    Figure CN118643043B_ABST
Patent Text Reader

Abstract

The application relates to a data collection method, device, equipment, medium and program product. The method comprises the following steps: obtaining full cache data and hotspot data of each supercomputing cluster in a data center, and performing synchronous processing on the corresponding full cache data according to the hotspot data to obtain full data of each supercomputing cluster. The full cache data is collected from each supercomputing cluster according to a first preset frequency, and the hotspot data is collected from each supercomputing cluster in real time. The method can balance the collection resource pressure of the system and the consistency of the data of each supercomputing cluster.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data collection method and device, equipment, medium and program product. BACKGROUND

[0002] With the rapid development of computer technology, there are often multiple regional supercomputer clusters (referred to as supercomputer clusters) performing cross-regional jobs. In this case, the data of multiple supercomputer clusters can be collected into a data center for unified query of cross-cluster data.

[0003] In related technologies, the data of each supercomputer cluster is summarized to a data center, which can be collected by distributed caching, timed remote calling interface, message middleware, and other methods.

[0004] However, the collection method in the related art cannot balance the collection resource pressure of the system and the consistency of the data of each supercomputer cluster. SUMMARY

[0005] Therefore, it is necessary to provide a data collection method, device, equipment, medium and program product that balances the collection resource pressure of the system and the consistency of the data of each supercomputer cluster.

[0006] In a first aspect, the present application provides a data collection method, which comprises:

[0007] Obtaining full cache data and hot data of each supercomputer cluster in the data center; the full cache data is collected from each supercomputer cluster according to a first preset frequency, and the hot data is collected from each supercomputer cluster in real time;

[0008] Synchronizing the corresponding full cache data according to the hot data to obtain full data of each supercomputer cluster.

[0009] In the technical solution of the embodiment, the full amount cache data and the hotspot data of each supercomputing cluster in the data center are acquired, and the full amount cache data is synchronously processed according to the hotspot data, to obtain the full amount data of each supercomputing cluster. The full amount cache data is collected from each supercomputing cluster according to a first preset frequency, and the hotspot data is collected from each supercomputing cluster in real time. Since the full amount cache data is collected according to the first preset frequency, the problem of excessive system load pressure caused by frequent data collection is avoided, and the hotspot data is collected in real time, so that the real-time hotspot data obtained is more accurate. Further, the full amount cache data is synchronously processed according to the hotspot data, which means that the system load pressure and the real-time performance of the data of each supercomputing cluster are considered in the process of obtaining the full amount data. The acquisition process can ensure the consistency of the data of each supercomputing cluster under the condition that the collection resource pressure of the system is reasonable.

[0010] In one of the embodiments, the full amount data of each supercomputing cluster is obtained by synchronously processing the corresponding full amount cache data according to the hotspot data, including:

[0011] The synchronously processing the corresponding full amount cache data according to the hotspot data is performed according to a second preset frequency to obtain the full amount data of each supercomputing cluster, and the second preset frequency is less than the first preset frequency.

[0012] In the technical solution of the embodiment, the synchronously processing the corresponding full amount cache data according to the hotspot data is performed according to a second preset frequency to obtain the full amount data of each supercomputing cluster, which reduces the calculation burden of the synchronous processing. In addition, the second preset frequency is less than the first preset frequency, so that each synchronous processing is performed on the basis of the updated full amount cache data, which improves the effectiveness of the synchronous processing.

[0013] In one of the embodiments, the hotspot data and the full amount cache data both carry the supercomputing cluster identifier and the timestamp, and the full amount data of each supercomputing cluster is obtained by synchronously processing the corresponding full amount cache data according to the hotspot data, including:

[0014] The supercomputing cluster identifier and the timestamp in the hotspot data and the full amount cache data are used to determine the hotspot data corresponding to each full amount cache data.

[0015] Each full amount cache data is synchronously processed according to the hotspot data corresponding to the full amount cache data, to obtain the full amount data of each supercomputing cluster.

[0016] The technical scheme of the embodiment of the application is characterized in that, based on the supercomputing cluster identifier and the timestamp, the full cache data is subjected to synchronous processing, the corresponding hot data and full cache data are quickly located based on the supercomputing cluster identifier, it is judged whether the full cache data needs to be synchronized according to the timestamp, and in the case that the full cache data needs to be synchronized, the hot data corresponding to the full cache data is screened out, thereby improving the efficiency of the synchronous processing mode.

[0017] In one of the embodiments, the hot data corresponding to each full cache data is determined according to the supercomputing cluster identifier and the timestamp in the hot data and the full cache data, and the method comprises the following steps.

[0018] For any full cache data, the hot data with the same supercomputing cluster identifier and the matching timestamp as the full cache data is determined as the hot data corresponding to the full cache data; wherein each full cache data corresponds to multiple hot data.

[0019] In the technical scheme of the embodiment of the application, the hot data with the same supercomputing cluster identifier as the full cache data and the matching timestamp is determined as the hot data corresponding to the full cache data by comparing the supercomputing cluster identifier and the timestamp of the hot data and the full cache data, which is logically clear and easy to implement, and improves the reliability of the hot data.

[0020] In one of the embodiments, each full cache data is subjected to synchronous processing according to the hot data corresponding to each full cache data, and the full data of each supercomputing cluster is obtained, and the method comprises the following steps.

[0021] The thread pool of each full cache data is determined according to the correspondence between the supercomputing cluster identifier and the thread pool.

[0022] In the thread pool of each full cache data, each full cache data is subjected to synchronous processing according to the hot data corresponding to each full cache data, and the full data of each supercomputing cluster is obtained.

[0023] In the technical scheme of the embodiment of the application, the process of synchronous processing of each supercomputing cluster is independent by means of the multi-supercomputing cluster and multi-thread pool, which improves the efficiency of multi-thread concurrent processing and avoids the probability of lock competition when each supercomputing cluster is subjected to synchronous processing, so that the synchronous processing mode is more reliable.

[0024] In one of the embodiments, before the full cache data and the hot data of each supercomputing cluster in the data center are obtained, the data acquisition method further comprises the following steps.

[0025] According to the preset data update strategy, the full cache data of each supercomputing cluster in the data center is subjected to data cleaning.

[0026] In the technical solution of the embodiment, the data cleaning is performed on the full cache data of each supercomputing cluster in the data center according to the preset data update strategy, the redundant full cache data is removed while the effectiveness of the full cache data is ensured, and the storage burden of the data collection process is reduced.

[0027] In one of the embodiments, the data update strategy includes at least one of the following:

[0028] According to the collection time of the full cache data of each supercomputing cluster in the data center, the data collected before the preset time is deleted; or,

[0029] If the total data amount of the full cache data of each supercomputing cluster in the data center is greater than the preset data amount threshold, the data is deleted according to the collection time until the total amount is less than the preset data amount threshold; or,

[0030] According to the preset time length, the cache time interval corresponding to the current time is determined, and the data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

[0031] In the technical solution of the embodiment, the corresponding full cache data is cleaned through multiple preset data update strategies, different data cleaning actions can be performed according to the attributes of multiple dimensions of the full cache data, and the flexibility of data cleaning is improved.

[0032] In a second aspect, the application further provides a data collection device, which comprises:

[0033] The data acquisition module is configured to acquire the full cache data and the hot data of each supercomputing cluster in the data center; the full cache data is collected from each supercomputing cluster according to a first preset frequency, and the hot data is collected from each supercomputing cluster in real time;

[0034] The synchronization processing module is configured to perform synchronization processing on the corresponding full cache data according to the hot data, and obtain the full data of each supercomputing cluster.

[0035] In a third aspect, the application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in any one of the embodiments of the first aspect when executing the computer program.

[0036] In a fourth aspect, the application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method in any one of the embodiments of the first aspect.

[0037] In a fifth aspect, the present application provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of the method in any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 An internal structure diagram of a computer device for a data collection method in an embodiment;

[0039] Figure 2 A flowchart of a data collection method in an embodiment;

[0040] Figure 3 A flowchart of a full-amount data acquisition step in an embodiment;

[0041] Figure 4 A flowchart of a data synchronization processing step in an embodiment;

[0042] Figure 5 A flowchart of a full-amount data acquisition step in another embodiment;

[0043] Figure 6 A flowchart of a data synchronization processing step in another embodiment;

[0044] Figure 7 A flowchart of a data synchronization processing step in another embodiment;

[0045] Figure 8 A structure block diagram of a data collection device in an embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0047] The data collection method provided by the embodiments of the present application can be applied to a computer device. The computer device can be a server, and its internal structure diagram can be as shown in FIG. 1. Figure 1As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data acquisition data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a data acquisition method. Those skilled in the art can understand, Figure 1 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0048] With the rapid development of computer technology, there are often multiple regional supercomputer clusters (supercomputer clusters for short) to perform cross-regional jobs. In the supercomputer operation management end (management end for short), data of each supercomputer cluster is collected to a data center through access to multiple regional supercomputer clusters, and unified query is performed.

[0049] In related technologies, common data collection methods include cross-cluster distributed caching, timed remote interface calling, message middleware collection and the like. However, in the case of large data volume, too frequent data collection frequency will cause excessive system collection resource pressure, resulting in collection backlog and even system crash and the like. In this case, reducing the collection frequency to reduce system pressure cannot guarantee the consistency of the data of the supercomputer clusters.

[0050] Based on this, the present application provides a data collection method, which synchronously processes the full-volume cached data collected by timing through the hot data collected in real time to obtain full-volume data. Such a data collection method can balance the collection resource pressure of the system and the consistency of the data of each supercomputer cluster.

[0051] In one embodiment, as shown in the figure, Figure 2 A data collection method is provided, including the following steps:

[0052] S201, obtain full cache data and hotspot data of each supercomputing cluster in the data center; the full cache data is collected from each supercomputing cluster according to a first preset frequency, and the hotspot data is collected from each supercomputing cluster in real time.

[0053] The supercomputing cluster refers to a supercomputer cluster with strong data processing and storage capabilities, which can undertake various large-scale scientific computing and engineering computing tasks. When a user needs to use one or more regional supercomputing clusters, the job information of multiple supercomputing clusters needs to be concerned, that is, the job data of the supercomputing cluster scheduling system. When the supercomputing cluster is in an idle state, it can be used by the user.

[0054] The supercomputing operation management terminal accesses multiple supercomputing clusters, collects the job information of each supercomputing cluster to a data center, and facilitates user query of the job information of each supercomputing cluster. In the data center, the job information of each supercomputing cluster is divided into full cache data and hotspot data.

[0055] The full cache data refers to all data in each supercomputing cluster, and the corresponding information is relatively comprehensive and the data volume is large. Therefore, in order to avoid excessive resource collection pressure on the system, the first preset frequency is determined according to the bottom load capacity, and the bottom scheduling system command is called to collect the full cache data of each supercomputing cluster from each supercomputing cluster according to the first preset frequency.

[0056] The hotspot data refers to job operation data in each supercomputing cluster, such as new, re-run, adjust run time, cancel, and other job operation data, and the corresponding information can reflect the job operation of each supercomputing cluster, and the data volume is small. Therefore, in order to accurately obtain the job operation of each supercomputing cluster, the hotspot data of each supercomputing cluster can be collected by real-time monitoring of each supercomputing cluster.

[0057] Exemplarily, in the data center, the full cache data is placed in a full cache folder, and the hotspot data is placed in a hotspot cache folder. Correspondingly, the full cache data and the hotspot data of each supercomputing cluster are obtained from the corresponding folder.

[0058] S202, synchronously process the corresponding full cache data according to the hotspot data to obtain full data of each supercomputing cluster.

[0059] It can be understood that the hotspot data refers to single job data of each supercomputing cluster, and the full cache data refers to all job data of each supercomputing cluster, that is, the full cache data and the hotspot data have a corresponding relationship, and the full cache data includes historical job data corresponding to the hotspot data.

[0060] The single job data corresponding to the hotspot data is obtained from the full cache data, the single job data corresponding to the hotspot data in the full cache data is processed according to the hotspot data, the synchronous processing of the full cache data is realized, and the full cache data after the synchronous processing is taken as the full data of each supercomputing cluster.

[0061] In the embodiment, the full cache data and the hotspot data of each supercomputing cluster in the data center are obtained, and the corresponding full cache data is synchronously processed according to the hotspot data to obtain the full data of each supercomputing cluster. The full cache data is collected from each supercomputing cluster according to a first preset frequency, and the hotspot data is collected from each supercomputing cluster in real time. Since the full cache data is collected according to the first preset frequency, the problem of excessive system load pressure caused by frequent data collection is avoided, and the hotspot data is collected in real time, so that the real-time hotspot data obtained is more accurate. Further, the full cache data is synchronously processed according to the hotspot data, which is equivalent to considering the system load pressure and the real-time of the data of each supercomputing cluster in the process of obtaining the full data. The acquisition process can ensure the consistency of the data of each supercomputing cluster under the condition that the system acquisition resource pressure is reasonable.

[0062] In the process of data collection, the corresponding full cache data is synchronously processed according to the hotspot data. Since the hotspot data is collected in real time and the full cache data is collected at a fixed time, the corresponding synchronous processing process can be real-time synchronization or fixed-time synchronization. Based on this, the process of synchronous processing is described below through an embodiment.

[0063] In one embodiment, the corresponding full cache data is synchronously processed according to the hotspot data to obtain the full data of each supercomputing cluster, including:

[0064] The corresponding full cache data is synchronously processed according to the hotspot data to obtain the full data of each supercomputing cluster according to a second preset frequency; the second preset frequency is less than the first preset frequency.

[0065] In the synchronous processing process, first, the second preset frequency is determined according to the load capacity in the synchronous processing, and then the corresponding full cache data is synchronously processed according to the hotspot data to obtain the full data of each supercomputing cluster according to the second preset frequency.

[0066] It can be understood that the second preset frequency is determined according to the load capacity of the cache middleware involved in the synchronization processing, and the first preset frequency is determined according to the load capacity of the system bottom layer. Moreover, the full cache data is collected and updated at the first preset frequency, and the synchronization processing is performed at the second preset frequency. Therefore, in the case that the second preset frequency is less than the first preset frequency, it can be ensured that each synchronization processing is performed on the basis of the updated full cache data.

[0067] In the embodiments of the present application, the synchronization processing of the corresponding full cache data according to the hot data is performed at the second preset frequency to obtain the full data of each supercomputing cluster, thereby reducing the calculation burden of the synchronization processing. In addition, the second preset frequency is less than the first preset frequency, so that each synchronization processing is performed on the basis of the updated full cache data, thereby improving the effectiveness of the synchronization processing.

[0068] In the process of obtaining the full data of each supercomputing cluster, the synchronization processing is usually performed based on the correspondence between the full cache data and the hot data to improve the accuracy and rapidity of the full data. Therefore, the following describes the acquisition method of the full data through an embodiment.

[0069] In one embodiment, as shown in Figure 3 The hot data and the full cache data each carry the supercomputing cluster identifier and the timestamp thereof, the synchronization processing of the corresponding full cache data according to the hot data is performed to obtain the full data of each supercomputing cluster, including:

[0070] S301, determining the hot data corresponding to each full cache data according to the supercomputing cluster identifier and the timestamp in the hot data and the full cache data.

[0071] Each supercomputing cluster corresponds to a different supercomputing cluster identifier, and the supercomputing cluster identifier is used to distinguish different supercomputing clusters. The hot data is a single job information obtained by listening to the job operation of each supercomputing cluster, and carries the supercomputing cluster identifier and the timestamp. The full cache data is obtained by collecting all the job information of each supercomputing cluster, and carries the supercomputing cluster identifier and the timestamp. That is, the same supercomputing cluster identifier corresponds to the full cache data and the hot data.

[0072] After obtaining the supercomputing cluster identifier and the timestamp in the hot data and the full cache data, the hot data corresponding to the same supercomputing cluster identifier is determined in the full cache data corresponding to the same supercomputing cluster identifier according to the correspondence between the supercomputing cluster identifier, the full cache data and the hot data. Then, the hot data corresponding to the latest timestamp is determined as the hot data corresponding to each full cache data by comparing the timestamp of the full cache data and the timestamp of the hot data.

[0073] S302, synchronizing each full cache data according to the hotspot data corresponding to each full cache data, to obtain full data of each supercomputing cluster.

[0074] After determining the hotspot data corresponding to each full cache data, the full cache data is processed synchronously according to the hotspot data, such as replacement, coverage, synthesis and other modifications. The full cache data after synchronous processing is the full data of each supercomputing cluster.

[0075] In the embodiments of the application, the full cache data is processed synchronously based on the supercomputing cluster identifier and the timestamp. The corresponding hotspot data and full cache data are quickly located through the supercomputing cluster identifier. Whether the full cache data needs to be synchronized is determined according to the timestamp. In the case of needing to be synchronized, the hotspot data corresponding to the full cache data is screened out, and the efficiency of the synchronous processing mode is improved.

[0076] The foregoing embodiments describe the basis for determining the hotspot data, which is the supercomputing cluster identifier and the timestamp. Based on this, the implementation of how to determine the hotspot data according to the supercomputing cluster identifier and the timestamp is described below through an embodiment.

[0077] In one embodiment, the hotspot data corresponding to each full cache data is determined according to the supercomputing cluster identifier and the timestamp in the hotspot data and the full cache data, comprising:

[0078] For any full cache data, the hotspot data with the same supercomputing cluster identifier as the full cache data and the matching timestamp is determined as the hotspot data corresponding to the full cache data; wherein each full cache data corresponds to multiple hotspot data.

[0079] For any full cache data, including all job data of multiple supercomputing clusters, if there is a job operation in each supercomputing cluster, the corresponding hotspot data will be generated. It can be understood that the number of job operations corresponds to the number of hotspot data, and each hotspot data also corresponds to a single supercomputing cluster.

[0080] By comparing the supercomputing cluster identifiers of the full cache data and the hotspot data, the hotspot data with the same supercomputing cluster identifier as the cache data is determined, and by comparing the timestamps of the full cache data and the hotspot data, the hotspot data closest to the timestamp of the full cache data is determined.

[0081] Optionally, in the plurality of hotspot data, the same supercomputing cluster identifier can be identified, and it is further determined whether the timestamps match to determine the hotspot data corresponding to the full cache data. In the case of matching timestamps, it is further determined whether the supercomputing cluster identifiers are consistent to determine the hotspot data corresponding to the full cache data. The order of the supercomputing cluster identifier determination and the timestamp matching determination is not limited in the present application.

[0082] Taking one supercomputing cluster as an example, as shown in Figure 4 The synchronization process of the supercomputing cluster includes the following steps:

[0083] S401, obtaining full cache data.

[0084] The underlying scheduling system command is called to obtain the job information data of each supercomputing cluster, and the job information data of each supercomputing cluster is cached as full cache data. The job information data is obtained in a cycle according to a first preset frequency as full cache data. The full cache data includes a timestamp, a cluster identifier, and full job information data.

[0085] S402, obtaining hotspot data.

[0086] In the process of real-time monitoring of the job operation of each supercomputing cluster, when there is a job change in the supercomputing cluster, the corresponding hotspot data is obtained and cached, the hotspot data carrying the timestamp and the supercomputing cluster identifier is assembled, and is sent to the message middleware. The message middleware is a basic software for completing message sending and receiving in a distributed system.

[0087] S403, full data timing synchronization.

[0088] After the full cache data and the hotspot data are queried, the full cache data is assembled according to the hotspot data and is sent to the message middleware again. The synchronization process is performed according to a second preset frequency, and the second preset frequency is determined based on the load capacity of the message middleware.

[0089] In the embodiment of the present application, by comparing the supercomputing cluster identifiers and the timestamps of the hotspot data and the full cache data, the hotspot data with the same supercomputing cluster identifier as the full cache data and the matching timestamp is determined as the hotspot data corresponding to the full cache data. The logic is clear and easy to implement, and the reliability of the hotspot data is improved.

[0090] The full cache data includes a plurality of different supercomputing cluster data, and then the full cache data can be synchronized through one channel or a plurality of synchronization channels. Based on this, the following embodiment describes the full data acquisition method.

[0091] In one embodiment, as shown inFigure 5 As shown, according to the hot spot data corresponding to each full cache data, the full cache data is synchronously processed to obtain the full data of each supercomputing cluster, including:

[0092] S501, according to the correspondence between the supercomputing cluster identifier and the thread pool, the thread pool of each full cache data is determined.

[0093] The thread pool refers to a set of threads created in the process of initializing a multi-threaded application. When executing new tasks of the application, these threads need to be restarted, without creating new threads.

[0094] In the embodiment of the application, the full cache data of the supercomputing cluster is synchronously processed by the thread pool, that is, one thread pool corresponds to one supercomputing cluster and one synchronization thread. For example, A supercomputing cluster corresponds to A thread pool and A synchronization thread.

[0095] In the scenario of synchronously processing multiple supercomputing clusters, the synchronization process is sequentially performed by a single thread, so after determining the full cache data and the hot spot data, the thread pool corresponding to the supercomputing cluster synchronization processing needs to be determined according to the correspondence between the supercomputing cluster identifier and the thread pool.

[0096] S502, in the thread pool of each full cache data, each full cache data is synchronously processed according to the hot spot data corresponding to each full cache data, to obtain the full data of each supercomputing cluster.

[0097] In the embodiment of the application, different thread pools correspond to different supercomputing clusters. After determining the thread pool of each full cache data, the full cache data of the supercomputing cluster is transmitted to the corresponding thread pool, and the thread pool transmits the received full cache data to the synchronization thread of the corresponding supercomputing cluster. In the thread, each full cache data is synchronously processed according to the hot spot data corresponding to each full cache data, to obtain the full data of each supercomputing cluster.

[0098] Taking three supercomputing clusters (A supercomputing cluster, B supercomputing cluster and C supercomputing cluster) as an example, the process of synchronous processing is described, as shown in Figure 6 It can be seen from Figure 6 that the full cache data is connected to each supercomputing cluster through A thread pool, B thread pool and C thread pool. According to the supercomputing cluster identifier in the full cache data, the full cache data of the A supercomputing cluster is submitted to the A synchronization thread through the A thread pool, the full cache data of the B supercomputing cluster is submitted to the B synchronization thread through the B thread pool, and the full cache data of the C supercomputing cluster is submitted to the C synchronization thread through the C thread pool.

[0099] In the updating process of the full cache data, the following steps are included:

[0100] S601, start. Start performing full cache data update.

[0101] S602, pull full cache data. Pull full cache data from the message middleware.

[0102] S603, determine whether the message is expired. Determine whether the full cache data is expired through the timestamp of the full cache data.

[0103] S604, if the message is not expired, submit a synchronization processing task according to the supercomputing cluster identifier.

[0104] S605, determine whether to end the loop. Since the full cache data is collected according to the first preset frequency, the full cache data of the message middleware is also cached at a fixed time. When there is no new full cache data received in the message middleware, it is not necessary to repeatedly pull the full cache data and determine, and the loop is ended at this time.

[0105] S606, end. End performing full cache data update.

[0106] In the synchronization thread of each supercomputing cluster, the following steps are included:

[0107] S607, start. Start performing the synchronization thread.

[0108] S608, delete full cache data before the preset timestamp. Delete full cache data before the preset timestamp according to the timestamp carried by the thread pool.

[0109] S609, write full data. Receive full cache data transmitted by the thread pool and write it into the corresponding supercomputing cluster.

[0110] S610, end. End performing the synchronization thread.

[0111] In the embodiment of the application, through the multi-supercomputing cluster multi-thread pool mode, the synchronization processing processes of each supercomputing cluster are independent of each other, which not only improves the efficiency of multi-thread concurrent processing, but also avoids the probability of lock competition when each supercomputing cluster is synchronized, and such a synchronization processing mode is more reliable.

[0112] Since the supercomputing cluster includes a large number of computer devices, the storage memory occupied by the full cache data is getting larger and larger in the process of collecting full cache data of each supercomputing cluster, and the more distant the collection time of the full cache data is from the current time, the less the full cache data means. Based on this, the following embodiment describes the processing process of the full cache data of each supercomputing cluster.

[0113] In an embodiment, before obtaining the full cache data and the hotspot data of each supercomputing cluster in the data center, the data collection method further comprises:

[0114] According to the preset data update strategy, the full cache data of each supercomputing cluster in the data center is cleaned up.

[0115] It should be noted that before obtaining the full cache data and the hotspot data of each supercomputing cluster in the data center, the data center already has full cache data at multiple collection time points, and the data center stores the full cache data at the collection time points in the data center according to the collection time points. Figure 6 Based on the updated full cache data, the storage process corresponding to the data center is as shown in Figure 7 The full cache data in each synchronization thread and the updated hotspot data are stored to the data center. The updating process of the hotspot data comprises the following steps:

[0116] S611, start. Start updating the hotspot data in real time.

[0117] S612, pull the hotspot data from the message middleware.

[0118] S613, judge whether the timestamp carried by the hotspot data is updated.

[0119] S614, update the hotspot data carrying the timestamp. If the hotspot data is updated, the updated hotspot data is stored to the data center.

[0120] S615, judge whether to end the loop. If the hotspot data is not updated, end the loop.

[0121] S616, end. End updating the hotspot data. It should be understood that the hotspot data is collected in real time, and the corresponding hotspot data updating process is also executed in real time. If the message middleware no longer collects the hotspot data, there is no updating of the hotspot data, and the updating of the hotspot data is ended.

[0122] Since the full cache data at each collection time point is the collection of the job information of each cluster, according to the preset data update strategy, the full cache data of each supercomputing cluster in the data center is cleaned up.

[0123] In the embodiment of the application, according to the preset data update strategy, the full cache data of each supercomputing cluster in the data center is cleaned up, which not only ensures the effectiveness of the full cache data, but also removes the redundant full cache data, thereby reducing the storage burden of the data collection process.

[0124] In the data cleaning process of the full cache data, a data update strategy can be formulated according to the time effectiveness of the full cache data, the storage amount of the cache, and other factors, and the data cleaning is performed to obtain effective full data. Based on this, the data cleaning process under different data update strategies is described below through an embodiment.

[0125] In one embodiment, the data update strategy includes at least one of the following:

[0126] According to the collection time of the full cache data of each supercomputing cluster in the data center, the data collected before the preset time is deleted; or,

[0127] If the total data amount of the full cache data of each supercomputing cluster in the data center is greater than a preset data amount threshold, data is deleted according to the collection time until the total amount is less than the preset data amount threshold; or,

[0128] According to a preset time length, a cache time interval corresponding to the current time is determined, and the data of the full cache data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

[0129] The preset data update strategy can be a single data cleaning strategy or a combination of multiple data cleaning strategies. According to the preset data update strategy, the full cache data of each supercomputing cluster can be cleaned by executing one data cleaning strategy or simultaneously executing multiple data cleaning strategies.

[0130] Optionally, considering the time effectiveness of the full cache data, the earlier the collection time of the full cache data, the lower the reference of the full cache data. Therefore, according to the collection time of the full cache data of each supercomputing cluster in the data center, the data collected before the preset time is deleted.

[0131] Optionally, considering the limited storage amount of the full cache data, when the total data amount of the full cache data of each supercomputing cluster in the data center is greater than a preset data amount threshold, it means that the full cache data cannot be stored at this time. Therefore, the collection time of each full cache data is sorted, and the full cache data with the earliest collection time is deleted until the total data amount of the full cache data of each supercomputing cluster in the data center is less than the preset data amount threshold.

[0132] Optionally, considering the relevance of the full cache data to the current time, the farther the collection time of the full cache data is from the current time, the smaller the reference value of the full cache data is. According to the preset time length combined with the current time, the cache time interval corresponding to the current time is determined, and then it is judged whether the collection time of the full cache data is within the cache time interval, and the full cache data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

[0133] In the embodiments of the application, the corresponding full cache data is cleaned up through a plurality of preset data update strategies, different data cleaning actions can be performed according to the attributes of a plurality of dimensions of the full cache data, and the whole data cleaning operation is more flexible.

[0134] In one embodiment, the data collection method comprises the following steps:

[0135] (1) According to the preset data update strategy, the full cache data of each supercomputing cluster in the data center is cleaned up.

[0136] Among them, the data cleaning includes at least one of the following: ① according to the collection time of the full cache data of each supercomputing cluster in the data center, the data collected before the preset time is deleted; ② if the total data amount of the full cache data of each supercomputing cluster in the data center is greater than the preset data amount threshold, the data is deleted according to the collection time until the total amount is less than the preset data amount threshold; ③ according to the preset time length, the cache time interval corresponding to the current time is determined, and the data of the full cache data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

[0137] (2) Obtain the full cache data and hot data of each supercomputing cluster in the data center; the full cache data is collected from each supercomputing cluster according to the first preset frequency, and the hot data is collected from each supercomputing cluster in real time.

[0138] (3) According to the second preset frequency, the corresponding full cache data is processed according to the hot data to obtain the full data of each supercomputing cluster; the second preset frequency is less than the first preset frequency.

[0139] (4) For any full cache data, the hot data with the same supercomputing cluster identifier and matching timestamp as the full cache data is determined as the corresponding hot data of the full cache data. Wherein, each full cache data corresponds to a plurality of hot data.

[0140] (5) According to the supercomputing cluster identifier and the timestamp in the hot data and the full cache data, the hot data corresponding to each full cache data is determined.

[0141] (6), according to the correspondence between the super calculation cluster and the thread pool, determine the thread pool of each full cache data.

[0142] (7), in each full cache data thread pool, each full cache data is synchronized according to the corresponding hot data of each full cache data, and the full data of each super calculation cluster is obtained.

[0143] In the embodiment of the application, the full cache data and hot data of each super calculation cluster in the data center are obtained, and the corresponding full cache data is synchronized according to the hot data, and the full data of each super calculation cluster is obtained. The full cache data is collected from each super calculation cluster according to the first preset frequency, and the hot data is collected from each super calculation cluster in real time. Since the full cache data is collected according to the first preset frequency, the problem of excessive system load caused by frequent data collection is avoided, and the real-time hot data is collected in real time, so that the real-time hot data obtained is more accurate. Further, the full cache data is synchronized by the hot data, which is equivalent to considering the system load pressure and the real-time of the data of each super calculation cluster in the process of obtaining the full data. The acquisition process can ensure the consistency of the data of each super calculation cluster under the condition that the system acquisition resource pressure is reasonable.

[0144] It should be understood that, although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated herein, the execution of these steps has no strict order limitation, and these steps can be executed in other order. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0145] Based on the same inventive concept, the embodiments of the present application also provide a data acquisition device for implementing the above-mentioned data acquisition method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data acquisition device embodiments provided below can refer to the limitations of the data acquisition method in the above text, and will not be repeated here.

[0146] In one embodiment, as shown in Figure 8 a data acquisition device 800 is provided, comprising a data acquisition module 820 and a synchronization processing module 840, wherein:

[0147] The data acquisition module 820 is configured to acquire full cache data and hotspot data of each supercomputing cluster in the data center. The full cache data is collected from each supercomputing cluster according to a first preset frequency, and the hotspot data is collected from each supercomputing cluster in real time.

[0148] The synchronization processing module 840 is configured to perform synchronization processing on the corresponding full cache data according to the hotspot data, to obtain full data of each supercomputing cluster.

[0149] In an embodiment, the synchronization processing module 840 includes an execution unit configured to perform synchronization processing on the corresponding full cache data according to the hotspot data at a second preset frequency, to obtain full data of each supercomputing cluster. The second preset frequency is less than the first preset frequency.

[0150] In an embodiment, the synchronization processing module 840 further includes a data determination unit and a data synchronization unit, wherein:

[0151] The data determination unit is configured to determine the hotspot data corresponding to each full cache data according to the supercomputing cluster identifier and the timestamp in the hotspot data and the full cache data.

[0152] The data synchronization unit is configured to perform synchronization processing on each full cache data according to the hotspot data corresponding to each full cache data, to obtain full data of each supercomputing cluster.

[0153] In an embodiment, the data determination unit is further configured to, for any full cache data, determine the hotspot data having the same supercomputing cluster identifier as the full cache data and matching the timestamp as the hotspot data corresponding to the full cache data. Each full cache data corresponds to multiple hotspot data.

[0154] In an embodiment, the data synchronization unit includes a thread pool determination subunit and a data processing subunit, wherein:

[0155] The thread pool determination subunit is configured to determine the thread pool of each full cache data according to the correspondence between the supercomputing cluster identifier and the thread pool.

[0156] The data processing subunit is configured to perform synchronization processing on each full cache data in the thread pool of each full cache data with the hotspot data corresponding to each full cache data, to obtain full data of each supercomputing cluster.

[0157] In an embodiment, the data acquisition apparatus 800 further includes a data cleaning module configured to perform data cleaning on the full cache data of each supercomputing cluster in the data center according to a preset data updating strategy.

[0158] In one embodiment, the data update strategy comprises at least one of the following:

[0159] According to the collection time of the full cache data of each supercomputing cluster in the data center, data collected before a preset time is deleted; or,

[0160] If the total data quantity of the full cache data of each supercomputing cluster in the data center is greater than a preset data quantity threshold, data is deleted according to the collection time until the total quantity is less than the preset data quantity threshold; or,

[0161] According to a preset time length, a cache time interval corresponding to the current time is determined, and data of the full cache data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

[0162] Each module in the data collection device can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0163] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0164] Obtaining full cache data and hot data of each supercomputing cluster in a data center; the full cache data is collected from each supercomputing cluster according to a first preset frequency, and the hot data is collected from each supercomputing cluster in real time;

[0165] Synchronously processing the corresponding full cache data according to the hot data to obtain full data of each supercomputing cluster.

[0166] In one embodiment, the processor further implements the following steps when executing the computer program:

[0167] Synchronously processing the corresponding full cache data according to the hot data to obtain full data of each supercomputing cluster according to a second preset frequency; the second preset frequency is less than the first preset frequency.

[0168] In one embodiment, the processor further implements the following steps when executing the computer program:

[0169] Determining the corresponding hot data of each full cache data according to the supercomputing cluster identifier and the timestamp in the hot data and the full cache data;

[0170] Synchronously processing each full cache data according to the corresponding hot data of each full cache data to obtain full data of each supercomputing cluster.

[0171] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0172] For any full cache data, the hotspot data with the same supercomputing cluster identification and matching timestamp of the full cache data is determined as the hotspot data corresponding to the full cache data; wherein each full cache data corresponds to multiple hotspot data.

[0173] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0174] According to the correspondence between the supercomputing cluster identification and the thread pool, the thread pool of each full cache data is determined;

[0175] In the thread pool of each full cache data, each full cache data is synchronously processed with the hotspot data corresponding to each full cache data, and the full data of each supercomputing cluster is obtained.

[0176] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0177] According to the preset data update strategy, the full cache data of each supercomputing cluster in the data center is cleaned.

[0178] In one embodiment, the data update strategy includes at least one of the following:

[0179] According to the collection time of the full cache data of each supercomputing cluster in the data center, the data collected before the preset time is deleted; or,

[0180] If the total data amount of the full cache data of each supercomputing cluster in the data center is greater than the preset data amount threshold, the data is deleted according to the collection time until the total amount is less than the preset data amount threshold; or,

[0181] According to the preset time length, the cache time interval corresponding to the current time is determined, and the data of the full cache data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

[0182] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0183] Obtain the full cache data and hotspot data of each supercomputing cluster in the data center; the full cache data is collected from each supercomputing cluster according to a first preset frequency, and the hotspot data is collected from each supercomputing cluster in real time;

[0184] Synchronize the corresponding full cache data according to the hotspot data to obtain the full data of each supercomputing cluster.

[0185] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0186] Synchronize the corresponding full cache data according to the hotspot data to obtain the full data of each supercomputing cluster according to the second preset frequency; the second preset frequency is less than the first preset frequency.

[0187] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0188] According to the supercomputing cluster identifier and the timestamp in the hotspot data and the full cache data, determine the corresponding hotspot data of each full cache data;

[0189] Synchronize each full cache data according to the corresponding hotspot data of each full cache data to obtain the full data of each supercomputing cluster.

[0190] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0191] For any full cache data, the hotspot data with the same supercomputing cluster identifier and the matching timestamp of the full cache data is determined as the corresponding hotspot data of the full cache data; wherein each full cache data corresponds to multiple hotspot data.

[0192] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0193] According to the correspondence between the supercomputing cluster identifier and the thread pool, determine the thread pool of each full cache data;

[0194] In the thread pool of each full cache data, synchronize each full cache data according to the corresponding hotspot data of each full cache data to obtain the full data of each supercomputing cluster.

[0195] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0196] According to the preset data update strategy, clean up the full cache data of each supercomputing cluster in the data center.

[0197] In one embodiment, the data update strategy includes at least one of the following:

[0198] According to the collection time of the full cache data of each supercomputing cluster in the data center, delete the data collected before the preset time; or,

[0199] If the total data quantity of the full quantity cache data of each supercomputing cluster in the data center is greater than a preset data quantity threshold, data is deleted according to collection time until the total quantity is less than the preset data quantity threshold; or

[0200] According to a preset time length, a cache time interval corresponding to a current time is determined, and data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

[0201] In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the following steps:

[0202] Full quantity cache data and hot data of each supercomputing cluster in the data center are acquired; the full quantity cache data is collected from each supercomputing cluster according to a first preset frequency, and the hot data is collected from each supercomputing cluster in real time;

[0203] The corresponding full quantity cache data is synchronously processed according to the hot data, to obtain full quantity data of each supercomputing cluster.

[0204] In an embodiment, the computer program, when executed by the processor, further implements the following steps:

[0205] The corresponding full quantity cache data is synchronously processed according to the hot data, to obtain full quantity data of each supercomputing cluster, according to a second preset frequency; the second preset frequency is less than the first preset frequency.

[0206] In an embodiment, the computer program, when executed by the processor, further implements the following steps:

[0207] The corresponding hot data of each full quantity cache data is determined according to the supercomputing cluster identifier and the time stamp in the hot data and the full quantity cache data;

[0208] Each full quantity cache data is synchronously processed according to the corresponding hot data of each full quantity cache data, to obtain full quantity data of each supercomputing cluster.

[0209] In an embodiment, the computer program, when executed by the processor, further implements the following steps:

[0210] For any full quantity cache data, hot data with the same supercomputing cluster identifier and matching time stamp as the full quantity cache data is determined as the corresponding hot data of the full quantity cache data; wherein each full quantity cache data corresponds to multiple hot data.

[0211] In an embodiment, the computer program, when executed by the processor, further implements the following steps:

[0212] The thread pool of each full quantity cache data is determined according to the correspondence between the supercomputing cluster identifier and the thread pool.

[0213] In each thread pool of the full cache data, each full cache data is synchronized with the corresponding hotspot data to obtain the full data of each supercomputing cluster.

[0214] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0215] According to the preset data update strategy, the full cache data of each supercomputing cluster in the data center is cleaned.

[0216] In one embodiment, the data update strategy includes at least one of the following:

[0217] According to the collection time of the full cache data of each supercomputing cluster in the data center, the data collected before the preset time is deleted; or,

[0218] If the total data amount of the full cache data of each supercomputing cluster in the data center is greater than the preset data amount threshold, the data is deleted according to the collection time until the total amount is less than the preset data amount threshold; or,

[0219] According to the preset time length, the cache time interval corresponding to the current time is determined, and the data of the full cache data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

[0220] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0221] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0222] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data acquisition method, characterized by, The method comprises: acquiring full cache data and hotspot data of each supercomputing cluster in a data center; the full cache data is collected from each supercomputing cluster according to a first preset frequency, and the hotspot data is collected from each supercomputing cluster by real-time monitoring of job operation data of each supercomputing cluster; the first preset frequency is determined according to a bottom load capacity; the hotspot data and the full cache data each carry a supercomputing cluster identifier and a timestamp of the supercomputing cluster; determining, according to a correspondence between a supercomputing cluster identifier, full cache data and hotspot data, hotspot data corresponding to the same supercomputing cluster identifier in full cache data corresponding to the same supercomputing cluster identifier; comparing, for full cache data and hotspot data corresponding to the same supercomputing cluster identifier, a timestamp of the full cache data and a timestamp of the hotspot data, determining hotspot data corresponding to the latest timestamp as hotspot data corresponding to the full cache data, and synchronously processing the corresponding full cache data according to the hotspot data to obtain full data of each supercomputing cluster; the full cache data includes historical job data corresponding to the hotspot data; the synchronous processing includes replacing, covering or synthetically modifying a single piece of job data corresponding to the hotspot data in the full cache data according to the hotspot data.

2. The method of claim 1, wherein, The synchronous processing of the corresponding full cache data according to the hotspot data to obtain full data of each supercomputing cluster comprises: performing synchronous processing of the corresponding full cache data according to the hotspot data to obtain full data of each supercomputing cluster according to a second preset frequency; the second preset frequency is less than the first preset frequency.

3. The method according to claim 1 or 2, characterized in that, The determination of hotspot data corresponding to the same supercomputing cluster identifier in full cache data corresponding to the same supercomputing cluster identifier according to a correspondence between a supercomputing cluster identifier, full cache data and hotspot data comprises: for any full cache data, determining hotspot data corresponding to the same supercomputing cluster identifier and matching the timestamp of the full cache data as hotspot data corresponding to the full cache data; each full cache data corresponds to multiple pieces of hotspot data.

4. The method according to claim 1 or 2, characterized in that, The synchronous processing of each full cache data according to hotspot data corresponding to each full cache data to obtain full data of each supercomputing cluster comprises: determining a thread pool of each full cache data according to a correspondence between a supercomputing cluster identifier and the thread pool; synchronously processing each full cache data according to hotspot data corresponding to each full cache data in the thread pool of each full cache data to obtain full data of each supercomputing cluster.

5. The method according to claim 1 or 2, characterized in that, Before acquiring full cache data and hotspot data of each supercomputing cluster in a data center, the method further comprises: performing data cleaning on full cache data of each supercomputing cluster in the data center according to a preset data updating strategy.

6. The method of claim 5, wherein, The data updating strategy comprises at least one of the following: deleting data collected before a preset time according to a collection time of full cache data of each supercomputing cluster in the data center; or If the total data amount of the full-amount cache data of each supercomputing cluster in the data center is greater than a preset data amount threshold, data is deleted according to collection time until the total amount is less than the preset data amount threshold. Or, According to a preset time length, a cache time interval corresponding to a current time is determined, and data of the full-amount cache data of each supercomputing cluster in the data center that does not meet the cache time interval is deleted.

7. A data acquisition device, characterized by The apparatus comprises: The data acquisition module is configured to acquire full-amount cache data and hot data of each supercomputing cluster in a data center. The full-amount cache data is collected from each supercomputing cluster according to a first preset frequency, and the hot data is collected from each supercomputing cluster by real-time monitoring of job operation data of each supercomputing cluster. The first preset frequency is determined according to a bottom load capacity. The synchronization processing module is configured to determine, in the full-amount cache data corresponding to a same supercomputing cluster identifier, hot data corresponding to the same supercomputing cluster identifier according to a correspondence between supercomputing cluster identifiers, full-amount cache data, and hot data. For the full-amount cache data and the hot data corresponding to the same supercomputing cluster identifier, the time stamp of the full-amount cache data and the time stamp of the hot data are compared to determine hot data corresponding to a latest time stamp as hot data corresponding to the full-amount cache data, and the full-amount cache data is synchronized according to the hot data to obtain full-amount data of each supercomputing cluster. The full-amount cache data includes historical job data corresponding to the hot data. The synchronization processing includes replacing, covering, and synthetically modifying a single job data corresponding to the hot data in the full-amount cache data according to the hot data.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Citation Information

Patent Citations

  • Method and a device for realizing centralized caching of business system data

    CN109656956A