Distributed artificial intelligence training operation analysis system

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a distributed artificial intelligence training operation analysis system, which comprises a data collection system, a data pipeline, a data warehouse, a data caching service and an analysis client. According to the invention, the data collection system collects the analysis data of each node, and converts the analysis data through the data pipeline, so that mass analysis data can be efficiently and uniformly stored in the data warehouse; moreover, a data caching service is arranged between the data warehouse and the analysis client, the function of caching analysis data in advance can be realized by the data caching service, the analysis data required by the user can be quickly provided for the analysis client, and low-delay access is ensured; and the analysis client allows analysts to selectively obtain required analysis data, so that the situation that excessive data is obtained at a time is avoided, visualization can be realized, and intuitive analysis is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an analysis system for distributed artificial intelligence training operations. Background Art

[0002] With the advancement of AI (Artificial Intelligence) technology, large-scale distributed AI training jobs have become key to developing high-performance deep learning models. Efficiently analyzing these training jobs is crucial for optimizing performance, diagnosing inefficiencies, and ensuring efficient resource utilization.

[0003] However, in large-scale distributed AI training clusters, multiple nodes will collect a large amount of analytical data. It is difficult to push a large amount of analytical data directly to the analyst's client, and it is also difficult for analysts to extract the required information from a large amount of analytical data. Summary of the Invention

[0004] In view of this, the present invention provides an analysis system for distributed artificial intelligence training jobs to solve the problem of difficulty in analyzing large-scale distributed training jobs.

[0005] In a first aspect, the present invention provides an analysis system for distributed artificial intelligence training jobs, comprising: a data collection system, a data pipeline, a data warehouse, a data cache service, and an analysis client;

[0006] The data collection system is used to obtain the original analysis data of the training job in the distributed training cluster and push the original analysis data to the data pipeline;

[0007] The data pipeline is used to convert the original analysis data into first analysis data, and send the first analysis data to the data warehouse;

[0008] The data warehouse is used to write the first analysis data to the corresponding storage node;

[0009] The data cache service is used to cache the second analysis data in the data warehouse to a corresponding cache node; upon receiving an acquisition request initiated by the analysis client for acquiring target analysis data, if the cache node has cached the target analysis data, the cache node sends the target analysis data to the analysis client;

[0010] The analysis client is used to initiate the acquisition request to the data cache service, and receive and display the target analysis data returned by the data cache service.

[0011] In some optional embodiments, the data collection system includes: a data collector, a data streaming platform, and a data consumption service;

[0012] The data collector is used to obtain the original analysis data of the corresponding nodes in the distributed training cluster; the original analysis data includes log data, indicator data and tracking data of the training job;

[0013] The data flow platform is used to collect the raw analysis data acquired by the plurality of data collectors and distribute the raw analysis data to corresponding data consumption services;

[0014] The data consumption service is used to obtain the original analysis data from the data flow platform and push the original analysis data to the data pipeline.

[0015] In some optional implementations, the process of converting the raw analysis data into first analysis data by the data pipeline includes:

[0016] Get multiple threads from a pre-created thread pool;

[0017] The corresponding original analysis data are converted into first analysis data in parallel based on each thread.

[0018] In some optional implementations, the data pipeline is further used to:

[0019] Checking whether the data quality and data structure of the original analysis data meet the requirements;

[0020] When the data quality of the original analysis data does not meet the requirements, a first warning message of data quality abnormality is issued, and the original analysis data is written into the first data table of the data warehouse;

[0021] If the data structure of the original analysis data does not meet the requirements, a second warning message of data structure abnormality is issued, and the original analysis data is written into the second data table of the data warehouse;

[0022] When the data quality and data structure of the original analysis data meet the requirements, a step of converting the original analysis data into first analysis data is performed.

[0023] In some optional implementations, the data warehouse includes: a master node cluster and multiple slave node clusters; the master node cluster includes multiple master storage nodes, and the slave node cluster includes multiple slave storage nodes;

[0024] The master node cluster is configured to: in response to a write request for the first analysis data, determine a target master storage node corresponding to the first analysis data, and write the first analysis data to the target master storage node;

[0025] The target master storage node is used to: copy the first analysis data into multiple copies and distribute the copies to multiple slave node clusters;

[0026] The slave node cluster is used to: write the first analysis data distributed by the target master storage node to the corresponding target slave storage node;

[0027] The slave node cluster is further configured to: send the second analysis data to the data cache service in response to a read request initiated by the data cache service for reading the second analysis data.

[0028] In some optional implementations, the master node cluster further includes a master cluster cache, and the slave node cluster further includes a slave cluster cache;

[0029] Writing the first analysis data to the target primary storage node includes:

[0030] Writing the first analysis data to the primary cluster cache;

[0031] The target primary storage node reads the first analysis data from the primary cluster cache;

[0032] Writing the first analysis data distributed by the target master storage node to the corresponding target slave storage node includes:

[0033] Writing the first analysis data distributed by the target master storage node into the slave cluster cache;

[0034] The target slave storage node reads the first analysis data from the slave cluster cache.

[0035] In some optional implementations, the data cache service includes: a master cache node and a plurality of slave cache nodes;

[0036] The master cache node is used to: obtain the second analysis data from the data warehouse; copy the second analysis data into multiple copies, and distribute the copies to the multiple slave cache nodes;

[0037] The slave cache node is used to: cache the second analysis data distributed by the master cache node;

[0038] The master cache node is further configured to, upon receiving an acquisition request for target analysis data initiated by the analysis client, determine a target slave cache node that has cached the target analysis data, and instruct the target slave cache node to send the target analysis data to the analysis client.

[0039] In some optional implementations, the primary cache node is further configured to:

[0040] Determining adjacent analysis data that is adjacent in time to the target analysis data, and acquiring the adjacent analysis data from the data warehouse;

[0041] The neighboring analysis data is sent to the target slave cache node to instruct the target slave cache node to cache the neighboring analysis data.

[0042] In some optional implementations, the data cache service includes: a plurality of cache sites at different locations, the cache sites including a master cache node and a plurality of slave cache nodes;

[0043] The data cache service is also used to:

[0044] According to the location of the analysis client, a target cache site closest to the analysis client is determined; and the target cache site is instructed to send the target analysis data to the analysis client according to a master cache node and a corresponding slave cache node.

[0045] In some optional embodiments, the analysis system further comprises: a raw data storage service and / or a data retirement service;

[0046] The raw data storage service is used to back up the raw analysis data acquired by the data collection system;

[0047] The data retirement service is used to identify the analytical data to be retired in the data warehouse, and eliminate the analytical data to be retired, or archive the analytical data to be retired.

[0048] In some optional implementations, the process of the analysis client receiving and displaying the target analysis data returned by the data cache service includes:

[0049] receiving the target analysis data returned by the data cache service;

[0050] Filtering the target analysis data, performing bottleneck analysis based on the filtered target analysis data, and determining corresponding bottleneck information;

[0051] The filtered target analysis data and / or the bottleneck information are displayed.

[0052] The analysis system for distributed artificial intelligence training operations provided by the present invention collects analysis data from each node by a data collection system and converts the analysis data through a data pipeline, which can efficiently and uniformly store massive analysis data in a data warehouse; in addition, a data caching service is set between the data warehouse and the analysis client, and the data caching service can realize the function of caching analysis data in advance, and can quickly provide the analysis data required by the user to the analysis client, ensuring low-latency access; the analysis client allows analysts to selectively obtain the required analysis data, avoiding the acquisition of too much data at one time, and can realize visualization, facilitating intuitive analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in related technologies, the following briefly introduces the drawings required for use in the specific embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 is a schematic structural diagram of an analysis system according to an embodiment of the present invention;

[0055] Figure 2 is a schematic structural diagram of another analysis system according to an embodiment of the present invention;

[0056] Figure 3 is a schematic diagram of the structure of a data retirement service according to an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of a data warehouse according to an embodiment of the present invention;

[0058] Figure 5 is a schematic diagram of the working principle of a data warehouse according to an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of the structure of a data cache service according to an embodiment of the present invention;

[0060] Figure 7 This is a schematic diagram of the working principle of the data cache service according to an embodiment of the present invention;

[0061] Figure 8 2 is a schematic diagram of the structure of an analysis client according to an embodiment of the present invention.

[0062] Description of reference numerals:

[0063] 10. Data collection system; 20. Data pipeline; 30. Data warehouse; 40. Data caching service; 50. Analysis client; 60. Raw data storage service; 70. Data retirement service; 101. Data collector; 102. Data flow platform; 103. Data consumption service; 31. Master node cluster; 32. Slave node cluster; 401. Master cache node; 402. Slave cache node. DETAILED DESCRIPTION

[0064] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0065] In the autonomous driving industry, analyzing large-scale distributed AI training jobs can optimize the training process, improve performance, and accelerate the development cycle.

[0066] For example, training large-scale AI models, such as those used in autonomous driving, often requires distributed systems. Analysis can optimize resource allocation (e.g., GPUs, CPUs, memory, and storage), ensuring efficient use and reducing costs. It also helps identify bottlenecks and optimize scaling across distributed systems, which is crucial for processing the massive amounts of data required in autonomous driving scenarios.

[0067] Autonomous driving models, including perception, planning, and control systems, are highly complex. Analysis helps debug and fine-tune these models for optimal performance. It also provides insights into the training process, pinpointing inefficiencies such as communication overhead, slow data loading, or workload imbalance. By identifying inefficiencies, the framework helps reduce training time, enabling more iterations and faster development of new algorithms or updates. Furthermore, real-time insights during training enable immediate adjustments to hyperparameters or system configurations.

[0068] Large-scale distributed AI training operations involving heterogeneous hardware (GPUs, TPUs, CPUs, etc.) and multi-cloud environments complicate unified analytics. AI training generates terabytes of analytical data, making it difficult to effectively collect, store, process, analyze, and visualize. This analytical data must be easily interpretable by analysts such as AI engineers, data scientists, and system administrators.

[0069] There are currently some performance analysis tools, such as the Nsight system and PyTorch analyzer, but these performance analysis tools do not support distributed AI training-level data, and the information they collect is also limited, making it difficult to cope with analysis scenarios of massive data.

[0070] In this embodiment, a distributed artificial intelligence training job analysis system is provided for analyzing AI training jobs in a distributed training cluster. Figure 1 A structural diagram of the analysis system is shown in FIG. Figure 1 As shown, the analysis system includes: a data collection system 10 , a data pipeline 20 , a data warehouse 30 , a data cache service 40 and an analysis client 50 .

[0071] The data collection system 10 is used to obtain the original analysis data of the training job in the distributed training cluster and push the original analysis data to the data pipeline 20.

[0072] The data pipeline 20 is used to convert the original analysis data into first analysis data and send the first analysis data to the data warehouse 30 .

[0073] The data warehouse 30 is configured to write the first analysis data to a corresponding storage node.

[0074] The data cache service 40 is used to cache the second analysis data in the data warehouse 30 to the corresponding cache node; when receiving an acquisition request initiated by the analysis client 50 for obtaining target analysis data, if the cache node caches the target analysis data, the target analysis data is sent to the analysis client 50.

[0075] The analysis client 50 is used to initiate an acquisition request to the data cache service 40 , and receive and display the target analysis data returned by the data cache service 40 .

[0076] In this embodiment, when a training job is executed in a distributed training cluster, job data related to the training job is generated, including data that can be used to analyze the training job. The data collection system 10 collects the analysis data related to the training job. For ease of description, this analysis data is referred to as raw analysis data. Furthermore, the data collection system 10 pushes the collected raw analysis data to the data pipeline 20 for subsequent processing based on the data pipeline 20.

[0077] The data collection system 10 may only collect the original analysis data, or the data collection system 10 may also collect complete operation data including the original analysis data. This embodiment does not limit whether the data collection system 10 collects other data in addition to the original analysis data.

[0078] Raw analysis data is directly collected data, which may contain some useless data, such as duplicate data. Based on the data pipeline, this raw analysis data is processed, such as by removing duplicate data and correcting erroneous data, to convert it into first analysis data. Furthermore, this first analysis data is intended for storage in the data warehouse 30 . Therefore, the data pipeline 20 can also perform data type or data format conversion to ensure that the first analysis data meets the data warehouse 30's data type / data format requirements. That is, the converted data type and / or data format of the first analysis data meet the requirements of the data warehouse 30.

[0079] After being processed by the data pipeline 20 , not only can duplicate data in the massive data be deleted to reduce the data volume, but data cleaning can also be achieved to improve the quality of the analyzed data.

[0080] In this embodiment, the data pipeline 20 sends the converted first analysis data to the data warehouse 30 for data storage. The data warehouse includes multiple storage nodes, and the first analysis data is stored in corresponding storage nodes. The storage nodes cooperate with each other to store massive amounts of analysis data.

[0081] Furthermore, the first analysis data may also be provided with multiple copies, which are stored in different storage nodes respectively, to avoid data loss and achieve fault tolerance.

[0082] Among them, since the data warehouse 30 stores a massive amount of analytical data, and due to the large number of training jobs, there may be a large number of analysts with analysis needs, so an additional data cache service 40 is set between the data warehouse 30 and the analysis client 50 used by the analysts. The data cache service 40 can realize the data caching function to cache the analytical data required by the analysts in advance.

[0083] Specifically, the data cache service 40 includes multiple cache nodes. The data cache service 40 can cache a portion of the analysis data (ie, the second analysis data) in the data warehouse 30 to the corresponding cache nodes in advance, so as to provide the analysis client 50 with the required analysis data in a timely manner.

[0084] In this embodiment, analysts can use the analysis client 50 to query the required analysis data for analysis. When analysis is required, the analyst can initiate a request to the data cache service 40 to obtain the relevant analysis data based on the analysis client 50. For ease of description, the required analysis data is referred to as target analysis data.

[0085] For example, for a certain training job, if the analyst needs to analyze the training job, he or she can initiate an acquisition request for obtaining analysis data related to the training job (i.e., target analysis data) through the analysis client 50. It can be understood that the acquisition request is a read request for obtaining analysis data.

[0086] After receiving the acquisition request, data cache service 40 can determine whether the target analysis data exists in its own cache node. For example, if data cache service 40 has previously read the target analysis data from data warehouse 30 as the second analysis data and has not cleared the target analysis data, then the target analysis data is cached in the cache node of data cache service 40. Therefore, data cache service 40 can directly send the cached target analysis data to analysis client 50. In this case, there is no need to request the target analysis data from data warehouse 30, which can improve the efficiency of analysis client 50 in acquiring the target analysis data, especially when the target analysis data volume is large.

[0087] If the data cache service 40 does not currently cache the target analysis data, it is necessary to read the target analysis data from the data warehouse 30, that is, to use the target analysis data as the second analysis data, obtain the target analysis data from the data warehouse 30 side, and then return the obtained target analysis data to the analysis client 50 to ensure that the analysis client 50 can successfully obtain the required target analysis data.

[0088] After receiving the target analysis data returned by the data cache service 40 , the analysis client 50 can visualize the target analysis data and display relevant information to the analyst to facilitate the analyst's analysis.

[0089] Since the target analysis data has a large amount of data, the analysis client 50 generally does not support web pages. The analysis client 50 may specifically be an application program to process the target analysis data with a large amount of data.

[0090] The analysis system for distributed artificial intelligence training operations provided in this embodiment collects analysis data from each node by a data collection system 10 and converts the analysis data through a data pipeline 20, which can efficiently and uniformly store massive amounts of analysis data in a data warehouse 30; and a data cache service 40 is set between the data warehouse 30 and the analysis client 50. The data cache service 40 can realize the function of caching analysis data in advance, and can quickly provide the analysis data required by the user to the analysis client 50 to ensure low-latency access; the analysis client 50 allows analysts to selectively obtain the required analysis data to avoid obtaining too much data at one time, and can realize visualization to facilitate intuitive analysis.

[0091] In this embodiment, a structural diagram of another distributed artificial intelligence training job analysis system is provided, such as Figure 2 As shown, the analysis system includes Figure 1 The data collection system 10, data pipeline 20, data warehouse 30, data cache service 40, and analysis client 50 are shown. The data collection system 10 includes a data collector 101, a data flow platform 102, and a data consumption service 103. Furthermore, the analysis system may also include a raw data storage service 60 and / or a data retirement service 70.

[0092] Specifically, the data collector 101 is used to obtain the original analysis data of the corresponding nodes in the distributed training cluster; the original analysis data includes log data, indicator data and tracking data of the training job.

[0093] The data flow platform 102 is used to collect the original analysis data obtained by multiple data collectors 101 and distribute the original analysis data to corresponding data consumption services 103.

[0094] The data consumption service 103 is used to obtain raw analysis data from the data flow platform 102 and push the raw analysis data to the data pipeline 20 .

[0095] In this embodiment, the data collector 101 can be deployed on each node (such as a server) in a distributed training cluster. The data collector 101 on each node can collect various data types used to analyze AI training jobs in parallel, that is, raw analysis data. These raw analysis data specifically include log data, indicator data, and tracking data, so that the most comprehensive analysis and visualization results can be provided to users subsequently.

[0096] Log data is a log generated by capturing system and application related events, which can be used for debugging and error tracking.

[0097] Metrics are performance indicators related to node resources. They can be monitored to obtain real-time performance indicators of nodes, such as CPU / GPU utilization, memory usage, disk I / O, network bandwidth, and training statistics.

[0098] Tracing data is used to represent the trace of an AI training job. It provides a detailed execution path of the workload in the AI ​​training job, which can include timestamped events throughout the system.

[0099] Based on each data collector 101, raw analysis data is collected from distributed nodes and sent to the data flow platform, thereby achieving comprehensive performance monitoring, debugging, optimization and system-wide analysis, ensuring faster and more reliable AI training in a large-scale distributed environment.

[0100] A distributed AI training job involves multiple nodes and GPUs, and generates a large amount of logs, metrics, and other data. The raw analysis data may reach TB levels, making efficient processing and storage challenging.

[0101] First, for the raw data collected by each data collector 101, some valid data related to the analysis can be extracted from it; for example, valid data can be extracted from the raw data and invalid data can be filtered out based on keywords related to the analysis; for example, relevant information can be extracted from noisy logs to focus on performance indicators and related issues.

[0102] Secondly, the filtered valid data can be compressed using batch compression to reduce the data volume; and a batch list can be generated to reduce network queries. The compressed valid data can be used as raw analysis data and pushed to the back-end data flow platform 102.

[0103] In addition, the data collected by the data collectors 101 of different nodes may have different timestamps, and the data collected by different nodes can also be synchronized across nodes to ensure that the data (such as logs) collected by different nodes are synchronized in time and have accurate timestamps.

[0104] The data flow platform 102 is used to collect, process and distribute raw analysis data from the data collectors 101 of multiple nodes in real time, such as log data, indicator data and tracking data. The data flow platform 102 acts as a pipeline for real-time data.

[0105] The data flow platform 102 is a distributed data flow processing platform that includes multiple processing nodes, each of which can be a single server in the data flow platform 102. The processing nodes can efficiently extract and process data; distributed queues are used to cache data and control throughput, ensuring that the backend (data consumption service 103) can smoothly extract the required data; and as the scale of the distributed training system grows, the processing nodes can be expanded to process more and more data collected by the data collector 101.

[0106] Specifically, the data flow platform 102 can be a publish-subscribe messaging platform, such as Kafka. The data flow platform 102 categorizes messages by topic, and each message sent to the data flow platform 102 must specify a topic. The data collector 101, as a producer, is responsible for publishing raw analytical data in the form of messages to specific topics, while consumers are responsible for subscribing to and consuming topics of interest.

[0107] The topics of related training jobs can be defined based on the team name, job type, and job ID corresponding to the training job. Furthermore, each topic can be divided into multiple partitions to handle large data sets that cannot fit on a single disk. This makes it easier to organize, track, and use the data of a single training job. Furthermore, large amounts of job data (including raw analysis data) can be divided into corresponding partitions of the data flow platform 102, which facilitates highly parallel I / O processing.

[0108] In general, a partition is an ordered, immutable sequence of messages. This ensures the correct order of data streams from multiple nodes, especially when timestamps are involved.

[0109] Furthermore, data for a topic can be replicated across multiple processing nodes to form multiple redundant copies to ensure high availability. Furthermore, the data for these topics can be used by multiple clients for various analytical purposes.

[0110] The data consumption service 103 is a consumer of the data flow platform 102. There are multiple data consumption services 103. Each data consumption service 103 is deployed on a corresponding server node and is responsible for retrieving and forwarding relevant original analysis data (including indicators, logs and tracking data, etc.) from the data flow platform 102 to the downstream system. The downstream system can be, for example, a data pipeline 20, an original data storage service 60, etc.

[0111] Among them, although a topic can be subscribed by multiple consumers, in this embodiment, in order to ensure data processing efficiency and avoid interference, a corresponding consumer group is set for each topic. Each consumer group includes one or more consumers, that is, the data consumption service 103 corresponds to only one topic, and a topic can correspond to multiple data consumption services 103.

[0112] Specifically, consumers corresponding to each topic, ie, data consumption service 103, may be started asynchronously. Data consumption service 103 obtains original analysis data of the subscribed topic and pushes it to data pipeline 20, thereby realizing parallel data processing.

[0113] In addition, the data consumption service 103 can also push the original analysis data obtained from the topic to the original data storage service 60 for backup to solve the fault tolerance problem.

[0114] Optionally, as described above, the analysis system further includes: a raw data storage service 60 and / or a data retirement service 70 .

[0115] The original data storage service 60 is used to back up the original analysis data acquired by the data collection system 10 .

[0116] In this embodiment, the data collection system 10 is further configured to send the acquired raw analysis data to the raw data storage service 60 . For example, the data consumption service 103 may also push the raw analysis data to the raw data storage service 60 .

[0117] The raw data storage service 60 can be implemented based on a data lake and is specifically used to process and store high-frequency data streams from large-scale distributed systems and manage the massive amounts of data generated by distributed nodes. The raw data storage service 60 is primarily used for backup, ensuring data is not lost in the event of an analysis system failure, achieving fault tolerance, and ensuring data availability, integrity, and long-term accessibility for AI job analysis and optimization.

[0118] The raw data storage service 60 can implement storage tiering. For example, frequently accessed hot data can be stored in fast, expensive storage (e.g., solid-state drives), while less frequently accessed cold data can be stored in less expensive storage (e.g., object storage), achieving a balanced storage experience through tiered storage.

[0119] The data retirement service 70 is used to identify the analytical data to be retired in the data warehouse 30 and eliminate the analytical data to be retired or archive the analytical data to be retired.

[0120] In this embodiment, the data retirement service 70 ensures the proper management, retirement, and disposal of old or unused data, which is crucial for data storage and cost efficiency for each relevant work and its associated teams. For example, a reasonable expiration time can be set for the data in the data warehouse 30, and upon expiration, the corresponding data can be designated as data to be retired, such as analytical data to be retired.

[0121] Among them, for the analytical data in the data warehouse 30, the data retirement service 70 can classify the data; and based on it, the retirement strategy corresponding to the data of each data classification can be determined. Figure 3 As shown, the data retirement service 70 can obtain analytical data that needs to be retired from the data warehouse 30 (generally, only the metadata of the analytical data needs to be obtained), classify the analytical data based on the data classification component, and query the corresponding retirement policy based on the policy management component. The retirement policy can specifically use machine learning algorithms to intelligently identify and update these policies based on factors such as data relevance, frequency of use, importance to ongoing training work, and team-specific quotas.

[0122] The data retirement service 70 can more accurately determine whether the analytical data is to be retired based on the data classification to which it belongs and the corresponding retirement policy, thereby identifying the analytical data to be retired. For analytical data to be retired, it is no longer needed for operation, so it can be directly eliminated from the data warehouse 30; or, it can be archived to ensure data relevance and optimize storage utilization. Figure 3 As shown, the action planning component ensures that data is organized by time range, and each data segment contains analytical data (logs, traces, and indicators) for a given job collected from the distributed computing nodes of the data warehouse 30. For the analytical data to be retired, the action planning component generates corresponding retirement actions to the data warehouse 30 (such as eliminating these analytical data to be retired) to realize data retirement.

[0123] In addition, for each team's analytical data, the data retirement service 70 can also monitor the data retirement process, thereby providing progress updates and final reports for related training jobs, enabling effective collaboration between stakeholders. Figure 3 As shown, the progress monitoring component can monitor the progress of the decommissioning process in the data warehouse 30 in real time and synchronize the progress to the analysis team (such as the client used by the analysis team); in addition, after the decommissioning process is completed, the progress monitoring component can also trigger a report generation instruction to the audit tracking report component so that the audit tracking report component can provide the final report to the analysis team.

[0124] In some optional implementations, the process of the data pipeline 20 converting the original analysis data into the first analysis data includes: obtaining multiple threads from a pre-created thread pool; and converting the corresponding original analysis data into the first analysis data in parallel based on each thread.

[0125] In this embodiment, the data pipeline 20 pre-creates a thread pool containing multiple threads. When the original analysis data needs to be processed, multiple threads are started at the same time, and parallel operations are implemented based on the multiple threads in the thread pool to improve processing efficiency.

[0126] Specifically, the data pipeline 20 may be an ETL (Extract-Transform-Load) pipeline, which extracts original analysis data from stream data based on the ETL pipeline, converts the original analysis data into first analysis data, and finally loads the converted first analysis data into a data warehouse.

[0127] Using thread pools to parallelize ETL operations can efficiently extract, transform, and save large amounts of data for downstream analysis and decision-making.

[0128] Optionally, the data pipeline 20 is further used to detect whether the data quality and data structure of the original analysis data meet the requirements.

[0129] If the data quality of the original analysis data does not meet the requirements, a first warning message indicating a data quality anomaly is issued, and the original analysis data is written to the first data table of the data warehouse 30. If the data structure of the original analysis data does not meet the requirements, a second warning message indicating a data structure anomaly is issued, and the original analysis data is written to the second data table of the data warehouse 30. If both the data quality and data structure of the original analysis data meet the requirements, the step of converting the original analysis data into the first analysis data is executed.

[0130] Since the data quality and data structure of the analysis data will affect the downstream analysis effect, this embodiment uses the data pipeline 20 to implement data quality and data structure detection.

[0131] Specifically, after obtaining the original analysis data, for example, extracting the original analysis data from the data stream, the data pipeline 20 can detect whether the data quality of the original analysis data meets the requirements, such as detecting whether the original analysis data is missing, incomplete, or noisy, etc. These data quality issues may complicate downstream analysis.

[0132] If data pipeline 20 detects abnormal quality in the raw analysis data, it issues a first data quality anomaly warning message to notify operations and maintenance personnel or managers of the corresponding training operations. Furthermore, data warehouse 30 includes a table (namely, a first data table) for recording data with abnormal quality. Data pipeline 20 sends the abnormal quality raw analysis data to data warehouse 30, which stores it in this first data table for subsequent tracing.

[0133] Similarly, the analysis data is required to have a certain data structure, or the data structure that the data pipeline 20 can process is limited; the data pipeline 20 can detect whether the data structure of the original analysis data meets the requirements, for example, whether there are undefined indicators, fields, etc. in the data structure.

[0134] If the data structure of the raw analysis data does not meet the requirements, data pipeline 20 can generate a second alarm message indicating the data structure anomaly and provide it to operations and maintenance personnel or managers of the corresponding training operations. Furthermore, data warehouse 30 also includes a table for recording data with abnormal data structures, namely a second data table. Data pipeline 20 sends the raw analysis data with abnormal structures to data warehouse 30, which stores it in this second data table for subsequent tracing.

[0135] If the data quality and data structure of the original analysis data meet the requirements, the step of converting the original analysis data into the first analysis data can be performed, which will not be repeated here.

[0136] In this embodiment, by analyzing the data quality and data structure of the data, analysis data that does not meet the requirements can be eliminated to ensure the reliability of subsequent analysis.

[0137] In some optional implementations, since the data warehouse 30 is a key storage cluster, it needs to store a large amount of analytical data and support user query and analysis. Based on this, the data warehouse 30 is divided into a master cluster and multiple slave clusters. Specifically, see Figure 4 As shown, the data warehouse 30 includes: a master node cluster 31 and multiple slave node clusters 32; wherein, the master node cluster 31 includes multiple master storage nodes, and the slave node cluster 32 includes multiple slave storage nodes. Figure 4 Two slave node clusters 32 are taken as an example.

[0138] The master node cluster 31 is configured to: in response to a write request for first analysis data, determine a target master storage node corresponding to the first analysis data, and write the first analysis data to the target master storage node.

[0139] The target master storage node is used to copy the first analysis data into multiple copies and distribute the copies to multiple slave node clusters 32 .

[0140] The slave node cluster 32 is used to write the first analysis data distributed by the target master storage node to the corresponding target slave storage node.

[0141] The slave node cluster 32 is further configured to send the second analysis data to the data cache service 40 in response to a read request initiated by the data cache service 40 for reading the second analysis data.

[0142] like Figure 4 As shown, the master node cluster 31 includes M master storage nodes, and each slave node cluster 32 includes N slave storage nodes. The master node cluster 31 is mainly used to perform write operations, while because there may be a large number of analysts who need to obtain the required analysis data, multiple slave node clusters 32 are mainly used to perform read operations to ensure efficient data access.

[0143] Specifically, after generating the first analysis data, the data pipeline 20 submits the first analysis data to the data warehouse 30 and generates a write request for the first analysis data. The master node cluster 31 responds to the write request and determines which master storage node to store the first analysis data in. This master storage node is referred to as the target master storage node.

[0144] The target master storage node stores the first analysis data and replicates the first analysis data in multiple copies, thereby distributing each replicated copy of the first analysis data to a corresponding slave node cluster 32. The first analysis data may be distributed to all slave node clusters 32 or only to some slave node clusters 32, depending on actual circumstances.

[0145] For each slave node cluster 32, if it receives the first analysis data distributed by the target master storage node, it can determine the slave storage node that needs to store the first analysis data, that is, the target slave storage node, and then write the first analysis data to the target slave storage node.

[0146] like Figure 4 As shown, the first analysis data is written to the master storage node 1 (i.e., the target master storage node). The master storage node 1 replicates the first analysis data into two copies and sends them to two slave node clusters 32. If the first slave node cluster 32 stores the first analysis data in the slave storage node A1, the slave storage node A1 will be used as the target slave storage node for the first slave node cluster 32. Similarly, if the second slave node cluster 32 stores the first analysis data in the slave storage node B1, the slave storage node B1 will be used as the target slave storage node for the second slave node cluster 32.

[0147] For example, the storage nodes in the master node cluster 31 and each slave node cluster 32 may correspond one to one, for example, M=N. The target slave storage node of each slave node cluster 32 may be determined based on the target master storage node storing the first analysis data.

[0148] Furthermore, when data cache service 40 needs to obtain the second analysis data, it can initiate a read request to read the second analysis data. In this embodiment, the corresponding slave node cluster 32 responds to the request. For example, to ensure the efficiency of data cache service 40 in obtaining the second analysis data, the slave node cluster 32 closest to data cache service 40 can respond to the read request, thereby sending the second analysis data stored in the corresponding slave storage node to data cache service 40, completing the data read.

[0149] In this embodiment, write queries are directed to the master node cluster, while read queries are executed only on the slave node cluster. This read-write separation ensures efficient data access and optimizes the read and write performance of the data warehouse 30. Furthermore, data in the master storage node is replicated across multiple slave storage nodes, effectively preventing data loss and ensuring redundancy and high availability.

[0150] Alternatively, see Figure 5 As shown, the master node cluster 31 further includes a master cluster cache, and the slave node cluster 32 further includes a slave cluster cache.

[0151] The process of the master node cluster 31 “writing the first analysis data to the target master storage node” specifically includes: writing the first analysis data to the master cluster cache; and the target master storage node reading the first analysis data from the master cluster cache.

[0152] Furthermore, the process of the slave node cluster 32 "writing the first analysis data distributed by the target master storage node to the corresponding target slave storage node" specifically includes: writing the first analysis data distributed by the target master storage node to the slave cluster cache; and the target slave storage node reading the first analysis data from the slave cluster cache.

[0153] In this embodiment, after master node cluster 31 obtains the first analysis data, it first writes the first analysis data to the master cluster cache. The master cluster cache can use a high-performance storage device, such as a solid-state drive, to quickly respond to write requests for the first analysis data. Furthermore, each master storage node in master node cluster 31 can periodically retrieve the data it needs to store from the master cluster cache. The analysis data to be stored by each master storage node can be pre-agreed upon, or the master storage node corresponding to the analysis data can be determined based on consistent hashing or other methods, although this embodiment does not limit this.

[0154] The target master storage node can read the corresponding first analysis data from the master cluster cache and then store the first analysis data locally. Furthermore, when distributing the first analysis data to each slave node cluster 32, the target master storage node can perform the distribution itself. Alternatively, if the master cluster cache currently caches the first analysis data (i.e., the first analysis data has not been cleared from the master cluster cache), the master cluster cache can also perform data distribution. This embodiment is not limited to this.

[0155] Similar to the function of the master cluster cache, by setting a slave cluster cache for each slave node cluster 32, data reading and writing efficiency can also be improved.

[0156] Specifically, the slave node cluster 32 writes the first analysis data distributed by the target master storage node into the slave cluster cache; then the corresponding target slave storage node reads the first analysis data from the slave cluster cache to achieve persistent storage of the first analysis data.

[0157] Furthermore, when the data cache service 40 reads the second analysis data, the slave node cluster 32 may first determine whether the second analysis data exists in the slave cluster cache. If so, the slave cluster cache directly responds to the read request and returns the second analysis data in the slave cluster cache to the data cache service 40, without passing through the lower-level slave storage nodes, thereby achieving a fast response. If the second analysis data does not exist in the slave cluster cache, the second analysis data is sent to the data cache service 40 using the responding slave storage node.

[0158] like Figure 5 As shown, the data warehouse 30 can implement read and write scheduling using the first partition manager. Specifically, for the first analytical data sent by the data pipeline 20, the first partition manager can schedule the write request to the master node cluster 31 to complete the write request. For read requests initiated by the data cache service 40, the first partition manager accurately directs the read request to a specific slave node cluster 32 by searching the slave node cluster that stores the partition of the second analytical data. This minimizes latency, ensures real-time response, and supports near-real-time extraction and query.

[0159] In addition, the data warehouse 30 may also be provided with a monitoring system to monitor whether there are problems such as insufficient disk space, network congestion, full cache on the master node and / or slave node, etc., which can ensure efficient use of resources and prevent system bottlenecks.

[0160] In some alternative embodiments, see Figure 6 As shown, the data cache service 40 includes: a master cache node 401 and multiple slave cache nodes 402.

[0161] The master cache node 401 is used to: obtain the second analysis data from the data warehouse 30 ; copy the second analysis data into multiple copies, and distribute the copies to the multiple slave cache nodes 402 .

[0162] The slave cache node 402 is used to cache the second analysis data distributed by the master cache node 401 .

[0163] Furthermore, the master cache node 401 is further configured to: upon receiving an acquisition request for target analysis data initiated by the analysis client 50 , determine a target slave cache node that has cached the target analysis data, and instruct the target slave cache node to send the target analysis data to the analysis client 50 .

[0164] In this embodiment, Figure 6 As shown, when data cache service 40 needs to obtain the second analysis data, master cache node 401 can initiate a read request to the data warehouse 30 to read the second analysis data, thereby obtaining the second analysis data from the data warehouse 30, for example, from the corresponding slave storage node. Furthermore, master cache node 401 replicates the obtained second analysis data and distributes them to each slave cache node 402, thereby enabling distributed caching of the second analysis data.

[0165] When the analyst needs to obtain the target analysis data, he can initiate an acquisition request for obtaining the target analysis data based on the analysis client 50. After receiving the acquisition request, the master cache node 401 can determine the slave cache node 402 that has cached the target analysis data, that is, the target slave cache node; then, the target slave cache node can send the locally stored target analysis data to the analysis client 50.

[0166] In this embodiment, master cache node 401 is primarily used for writing data, while slave cache nodes 402 are primarily used for reading data. This ensures efficient caching while providing the required analysis data to analysis client 50 in a timely manner, improving read and write performance. Furthermore, when data in master cache node 401 is updated, it is synchronously cached in multiple slave cache nodes 402, ensuring the consistency and availability of the analysis data.

[0167] Furthermore, if the current master cache node 401 fails, a new master cache node can be automatically elected from the available slave cache nodes 402 to ensure that the write operation can continue.

[0168] Optionally, in order to quickly provide target analysis data required by analysts at different locations, in this embodiment, the data cache service 40 includes: multiple cache sites at different locations. Each cache site includes the above-mentioned master cache node 401 and multiple slave cache nodes 402.

[0169] The data cache service 40 is further configured to: determine the target cache site closest to the analysis client 50 according to the location of the analysis client 50 ; and instruct the target cache site to send the target analysis data to the analysis client 50 according to the primary cache node 401 and the corresponding secondary cache node 402 .

[0170] Figure 7 A schematic diagram showing the working principle of the data cache service 40 is shown. Figure 7 As shown, the data cache service 40 is provided with a DNS (Domain Name System) server, such as GeoDNS (Geographic DNS), which dynamically selects a DNS server according to the user's geographical location, thereby being able to route the user's request to the appropriate area where the user is working, thereby being able to determine the cache site closest to the analysis client 50, i.e., the target cache site, as shown in FIG. Figure 7 Cache site Si in. Returning target analysis data based on the target cache site can reduce latency and improve data access speed.

[0171] Among them, such as Figure 7As shown, each cache site can contain multiple cache partitions, each of which includes a master cache node 401 and multiple slave cache nodes 402, and each cache partition is managed by a second partition manager; each cache partition can only cache the work data of its own user, reducing unnecessary data duplication and optimizing memory usage. After locating the target cache site, the second partition manager of the target cache site determines the cache partition that will cache the target analysis data, which will be referred to as the target cache partition later. Figure 7 The cache partition Pj in is the target cache partition.

[0172] The master cache node 401 of the target cache partition determines the slave cache node 402 storing the target analysis data, that is, the target slave cache node, for example Figure 7 The target slave cache node k in the Cache Node 1 further instructs the target slave cache node to send the target analysis data to the analysis client 50 .

[0173] In this embodiment, each cache site in data caching service 40 only needs to cache data on demand. If relevant regional data is to be cached, data can be preloaded as needed to avoid unnecessary cross-region caching. Distributing data across cache partitions at different cache sites effectively pre-warms massive datasets and optimizes data accessibility. Furthermore, the slave cache nodes in each cache partition only handle read operations, achieving redundancy and load balancing.

[0174] Optionally, the master cache node 401 is further used to: determine adjacent analysis data that is adjacent in time to the target analysis data, and obtain the adjacent analysis data from the data warehouse 30; and send the adjacent analysis data to the target slave cache node to instruct the target slave cache node to cache the adjacent analysis data.

[0175] In this embodiment, in addition to being able to independently obtain and cache hotspot data (second analysis data) from the data warehouse 30, the main cache node 401 will also determine other analysis data that are adjacent to the target analysis data in time after receiving an acquisition request to read the target analysis data, that is, adjacent analysis data; for example, if the analyst currently obtains the target analysis data within the latest ten minutes, the analysis data within the previous ten to twenty minutes can be used as adjacent analysis data. In addition, if the adjacent analysis data is not currently cached in the target slave cache node, the main cache node 401 obtains the adjacent analysis data from the data warehouse 30 and sends the adjacent analysis data to the target slave cache node that responds to the acquisition request, so that the target slave cache node can cache the adjacent analysis data in advance. When the subsequent analyst needs to obtain the adjacent analysis data, the target slave cache node can push the adjacent analysis data in time to ensure the real-time data acquisition on the user side.

[0176] like Figure 7As shown, the process of the analysis client 50 acquiring target analysis data includes the following steps.

[0177] Step S601: The analysis client 50 initiates an acquisition request to the DNS server to obtain target analysis data.

[0178] Step S602: Access the nearest cache site storing the target analysis data according to the DNS server, that is, the target cache site, also known as cache site Si.

[0179] The DNS server may return the IP address of the target cache site to the analysis client 50 for access by the analysis client 50 .

[0180] Step S603: The second partition manager of the target cache site locates the cache partition that caches the target analysis data, namely, cache partition Pj.

[0181] Step S604: The master cache node determines a slave cache node k that stores the target analysis data.

[0182] Step S605: The target analysis data is sent from the cache node k to the analysis client for analysis.

[0183] Step S606: The primary cache node requests the data warehouse to read adjacent analysis data.

[0184] Step S607: The data warehouse returns the adjacent analysis data to the primary cache node.

[0185] In step S608 , the master cache node copies the adjacent analysis data to the slave cache node k for subsequent acquisition by the analysis client.

[0186] The cache sizes of master cache node 401 and slave cache node 402 are limited, and critical analysis data may be cleared, potentially impacting the analysis process. Therefore, when master cache node 401 and slave cache node 402 need to free up cache space, they prioritize the least recently used (LRU) cache space to accommodate new analysis data. Furthermore, they prioritize freeing up space for adjacent data, ensuring that a more comprehensive dataset remains accessible to users.

[0187] In this embodiment, the data cache service 40 is based on DNS routing, partitioned data replication, intelligent cache eviction, fault-tolerant master-slave architecture and regionalized caching strategy, which effectively alleviates challenges related to data volume, consistency, latency, fault tolerance and cold start, and ensures high-performance analysis of large-scale distributed AI jobs.

[0188] In some optional implementations, the process of the analysis client 50 receiving and displaying the target analysis data returned by the data caching service 40 includes: receiving the target analysis data returned by the data caching service; filtering the target analysis data, performing bottleneck analysis based on the filtered target analysis data, and determining corresponding bottleneck information; and displaying the filtered target analysis data and / or bottleneck information.

[0189] In this embodiment, Figure 8 As shown, the analysis client 50 can obtain the target analysis data returned by the data cache service 40 through data extraction; wherein, adjacent data can be obtained, that is, data within a neighboring time range can be pre-fetched to minimize repeated network calls; based on resumable download and multi-threaded download methods, different data partitions can be downloaded simultaneously for faster retrieval.

[0190] Through data decryption, data verification, data decompression and other data unpacking operations, the acquired data can be restored to its original form for faster processing and analysis. In addition, the unpacked data is cached in a local cache such as NVMe (Non-Volatile Memory Express, Non-Volatile Memory Host Controller Interface Specification) cache or main memory cache, so that the data can be loaded directly into the fastest available storage on the current system to optimize access speed. In the event of a network failure, interrupted downloads can be resumed to avoid data loss or duplication.

[0191] The target analysis data from the local cache is then loaded and filtered. Specifically, filtering can be performed based on the time range, data partition, data category, etc. of the analysis data. Hybrid search can also be performed to narrow the scope of the analysis data and significantly reduce the amount of visualized data.

[0192] For filtered target analysis data, bottleneck analysis can be performed based on it to identify bottleneck points in the analysis data and obtain corresponding bottleneck information. For example, bottlenecks can be classified by type (such as network, memory, and computing), bottleneck points can be detected in the analysis data, bottlenecks can be prioritized based on severity or impact, correlations between bottlenecks can be determined, and the analysis data of the current job can be compared with the baseline job to implement benchmark management, etc., making it easier to identify differences or anomalies.

[0193] Furthermore, when displaying to the user based on the analysis client 50 , the bottleneck of the training job can be displayed in a highlighted form to visually emphasize the bottleneck and improve user recognition and attention.

[0194] In this embodiment, as described above, data cache service 40 pre-fetches analytics data within a contiguous time range, minimizing duplicate network calls. Furthermore, analytics client 50 is allowed to download data from different cache partitions simultaneously, enabling faster data retrieval and acquisition. Furthermore, a fast rendering API is utilized to achieve high-performance, cross-platform rendering across programming languages ​​and operating systems. Detail rendering is dynamically adjusted based on user interaction (e.g., zooming in), avoiding unnecessary rendering of fine details.

[0195] In addition, for the analysis results of each training job, the generated visualization results can be saved and shared among participating team members for reuse by other team members or tools. Figure 8 As shown, the analysis client 50 can save the generated visualization results for reuse by other team members or tools, and based on the result sharing function, it can facilitate sharing of the saved visualization results among the team members involved in the work.

[0196] In this embodiment, the analysis client 50 can effectively address challenges related to network bottlenecks, fault tolerance, visualization performance, and collaboration. By combining caching, rendering, and sharing mechanisms with powerful fault tolerance and security protocols, it can provide a comprehensive and user-friendly solution for analyzing large-scale distributed AI jobs.

[0197] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations should all be included in the scope of protection of the present invention.

Claims

1. A distributed artificial intelligence training job analysis system, characterized in that: The analysis system includes: a data collection system (10), a data pipeline (20), a data warehouse (30), a data cache service (40) and an analysis client (50); The data collection system (10) is used to obtain original analysis data of training jobs in a distributed training cluster and push the original analysis data to the data pipeline (20); The data pipeline (20) is used to convert the original analysis data into first analysis data, and send the first analysis data to the data warehouse (30); The data warehouse (30) is used to write the first analysis data to the corresponding storage node; The data cache service (40) is used to cache the second analysis data in the data warehouse (30) to a corresponding cache node; upon receiving an acquisition request for obtaining target analysis data initiated by the analysis client (50), if the cache node has cached the target analysis data, the target analysis data is sent to the analysis client (50); The analysis client (50) is used to initiate the acquisition request to the data cache service (40), and receive and display the target analysis data returned by the data cache service (40).

2. The analysis system according to claim 1, characterized in that The data collection system (10) includes: a data collector (101), a data flow platform (102) and a data consumption service (103); The data collector (101) is used to obtain original analysis data of corresponding nodes in the distributed training cluster; the original analysis data includes log data, indicator data and tracking data of the training operation; The data flow platform (102) is used to collect the original analysis data acquired by the plurality of data collectors (101), and distribute the original analysis data to corresponding data consumption services (103); The data consumption service (103) is used to obtain the original analysis data from the data flow platform (102) and push the original analysis data to the data pipeline (20).

3. The analysis system according to claim 1, characterized in that The process of the data pipeline (20) converting the original analysis data into first analysis data includes: Get multiple threads from a pre-created thread pool; The corresponding original analysis data are converted into first analysis data in parallel based on each thread.

4. The analysis system according to claim 1 or 3, characterized in that The data pipeline (20) is also used to: Checking whether the data quality and data structure of the original analysis data meet the requirements; When the data quality of the original analysis data does not meet the requirements, a first warning message of data quality abnormality is issued, and the original analysis data is written into a first data table of the data warehouse (30); When the data structure of the original analysis data does not meet the requirements, a second warning message of data structure abnormality is issued, and the original analysis data is written into a second data table of the data warehouse (30); When the data quality and data structure of the original analysis data meet the requirements, a step of converting the original analysis data into first analysis data is performed.

5. The analysis system according to claim 1, wherein The data warehouse (30) includes: a master node cluster (31) and multiple slave node clusters (32); the master node cluster (31) includes multiple master storage nodes, and the slave node cluster (32) includes multiple slave storage nodes; The master node cluster (31) is used to: in response to a write request for the first analysis data, determine a target master storage node corresponding to the first analysis data, and write the first analysis data to the target master storage node; The target master storage node is used to: copy the first analysis data into multiple copies and distribute the copies to multiple slave node clusters (32); The slave node cluster (32) is used to: write the first analysis data distributed by the target master storage node to the corresponding target slave storage node; The slave node cluster (32) is further configured to send the second analysis data to the data cache service (40) in response to a read request initiated by the data cache service (40) for reading the second analysis data.

6. The analysis system according to claim 5, characterized in that The master node cluster (31) further includes a master cluster cache, and the slave node cluster (32) further includes a slave cluster cache; Writing the first analysis data to the target primary storage node includes: Writing the first analysis data to the primary cluster cache; The target primary storage node reads the first analysis data from the primary cluster cache; Writing the first analysis data distributed by the target master storage node to the corresponding target slave storage node includes: Writing the first analysis data distributed by the target master storage node into the slave cluster cache; The target slave storage node reads the first analysis data from the slave cluster cache.

7. The analysis system according to claim 1, wherein The data cache service (40) includes: a master cache node (401) and a plurality of slave cache nodes (402); The master cache node (401) is used to: obtain the second analysis data from the data warehouse (30); copy the second analysis data into multiple copies, and distribute them to multiple slave cache nodes (402); The slave cache node (402) is used to cache the second analysis data distributed by the master cache node (401); The master cache node (401) is further configured to: upon receiving an acquisition request for acquiring target analysis data initiated by the analysis client (50), determine a target slave cache node that has cached the target analysis data, and instruct the target slave cache node to send the target analysis data to the analysis client (50).

8. The analysis system according to claim 7, characterized in that The primary cache node (401) is further configured to: Determining adjacent analysis data that is adjacent in time to the target analysis data, and acquiring the adjacent analysis data from the data warehouse (30); The neighboring analysis data is sent to the target slave cache node to instruct the target slave cache node to cache the neighboring analysis data.

9. The analysis system according to claim 7, characterized in that The data cache service (40) includes: a plurality of cache sites at different locations, wherein the cache sites include a master cache node (401) and a plurality of slave cache nodes (402); The data cache service (40) is further used to: According to the location of the analysis client (50), a target cache site closest to the analysis client (50) is determined; and the target cache site is instructed to send the target analysis data to the analysis client (50) according to a master cache node (401) and a corresponding slave cache node (402).

10. The analysis system according to claim 1, wherein The analysis system further includes: a raw data storage service (60) and / or a data retirement service (70); The raw data storage service (60) is used to back up the raw analysis data acquired by the data collection system (10); The data retirement service (70) is used to identify the analytical data to be retired in the data warehouse (30), and eliminate the analytical data to be retired, or archive the analytical data to be retired.

11. The analysis system according to claim 1, wherein The process of the analysis client (50) receiving and displaying the target analysis data returned by the data cache service (40) includes: receiving the target analysis data returned by the data cache service (40); Filtering the target analysis data, performing bottleneck analysis based on the filtered target analysis data, and determining corresponding bottleneck information; The filtered target analysis data and / or the bottleneck information are displayed.

Images