Data collaboration system based on LSTM (Long Short Term Memory) prediction model and electronic equipment
By using a data collaboration system based on LSTM prediction models, and leveraging pre-warming requests and distributed caching to cache datasets in advance, the problem of time-consuming cross-regional dataset loading during large model training is solved, achieving efficient data loading services and improving training efficiency.
Patent Information
- Application Number
- CN202511077495.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-01
- Publication Date
- 2025-11-18
AI Technical Summary
During the training of large models, loading datasets across regions is time-consuming, resulting in low training efficiency. Existing technologies have failed to effectively solve the problem of dataset loading capacity for training clusters dynamically deployed in multiple regions.
A data collaboration system based on an LSTM prediction model is adopted. A preheating request is generated through a preheating device, and data is cached in advance in different regions using a distributed caching device. An LSTM model is trained based on historical traffic indicators to predict future data loading needs, thereby achieving proactive data preheating.
It improves data loading speed, reduces client waiting time, enhances the efficiency of large model training, and adapts to the needs of training clusters with dynamic deployment in multiple regions.
Smart Images
Figure CN120975128A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data collaborative processing, and in particular to a data collaboration system based on an LSTM prediction model and an electronic device. BACKGROUND
[0002] With the rapid evolution of large language models, multi-modal models and other large model technologies, the model parameter scale has jumped from tens of billions to hundreds of billions, and the training data volume has also increased exponentially. The efficiency of model training not only depends on the computing power of the computing cluster, but also is limited by the loading speed of the training data. If the data set cannot be quickly and stably transmitted from the storage system to the computing node, the computing resources will be in an idle state of "waiting for data" for a long time, directly slowing down the training process. The current architecture contradiction of "fixed data and mobile computing" will be very time-consuming if the training cluster is temporarily loaded across regions during model training, affecting the efficiency of large model training.
[0003] Therefore, how to provide efficient data set loading capability for the training cluster dynamically deployed in multiple regions has become a key problem to be solved for storage systems in the era of large models.
[0004] At present, there is no effective solution to the above problems. SUMMARY
[0005] The present application provides a data collaboration system based on an LSTM (Long Short Term Memory) prediction model and an electronic device to solve the technical problem of "how to provide efficient data set loading capability for the training cluster dynamically deployed in multiple regions".
[0006] According to one aspect of an embodiment of the present application, the present application provides a data collaboration system based on an LSTM prediction model, comprising: a preheating device, a data storage device and a distributed cache device, wherein the preheating device is configured to generate a preheating request according to the prediction result of the LSTM prediction model and send the preheating request to a target cluster corresponding to the prediction result, and the LSTM prediction model is configured to predict future data loading demand; the data storage device is configured to store data sets of multiple businesses; and the distributed cache device comprises multiple cache clusters deployed in different regions, and a target cluster in the cache cluster is configured to read a target data set corresponding to the preheating request from the data storage device and cache the target data set to a local storage to provide data loading service for a client in the case of receiving the preheating request.
[0007] Optionally, the system further comprises a monitoring device configured to monitor traffic indicators of each cache cluster in the distributed cache device and store the traffic indicators in a data collection module, wherein the traffic indicators include data download time, data belonging business name and data download region.
[0008] Optionally, the preheating device comprises an LSTM prediction module, configured to acquire a plurality of traffic indexes in the data collection module from the monitoring device, and perform model training on the plurality of traffic indexes as training data to obtain an LSTM prediction model.
[0009] Optionally, the preheating device comprises a cluster management module, configured to receive and record cluster information issued by a control node of each cache cluster, wherein the cluster information comprises an IP address of the control node, a port number, and a region name of a region to which the control node belongs.
[0010] Optionally, the preheating device comprises a preheating control module, configured to acquire a prediction result from the LSTM prediction module, extract a target service name and a target region name of a target data set to be loaded from the prediction result, determine a target IP address and a target port number of a control node corresponding to the target region name through the cluster management module, and generate a preheating request according to the target service name, the target region name, the target IP address, and the target port number.
[0011] Optionally, the preheating control module is further configured to extract a target loading time from the prediction result, and send the preheating request to the target IP address and the target port number before the target loading time.
[0012] Optionally, the data collection module performs data preprocessing on each traffic index after caching the traffic index of each cache cluster, for model training.
[0013] Optionally, the target cluster is configured to extract the target service name from the preheating request, read a target data set corresponding to the target service name from the data storage device, and cache the target data set to a local storage.
[0014] Optionally, the cluster management module is further configured to establish a corresponding relationship between the control node, the IP address, the port number, and the region name of each cache cluster and save the corresponding relationship after receiving the IP address, the port number, and the region name corresponding to the control node of each cache cluster.
[0015] According to another aspect of the embodiments of the present application, the present application provides an electronic device comprising the above-mentioned data collaboration system based on an LSTM prediction model.
[0016] The above-mentioned technical solutions provided by the embodiments of the present application have the following advantages compared with related technologies:
[0017] The application provides a data coordination system based on an LSTM prediction model, comprising a preheating device, a data storage device and a distributed cache device, wherein the preheating device is used for generating a preheating request according to a prediction result of the LSTM prediction model and sending the preheating request to a target cluster corresponding to the prediction result, and the LSTM prediction model is used for predicting future data loading demand; the data storage device is used for storing data sets of multiple businesses; the distributed cache device comprises multiple cache clusters deployed in different regions, and a target cluster in the cache cluster is used for reading a target data set corresponding to the preheating request from the data storage device and caching the target data set to a local storage to provide data loading service for a client in the case of receiving the preheating request. The future data loading demand is predicted by the LSTM model, the preheating device generates a request and sends it to the cache cluster in the corresponding region, the cluster reads the target data set from the data storage device and caches it locally to provide fast data loading service for the client nearby. The problem of how to provide efficient data set loading capability for the training cluster dynamically deployed in multiple regions is solved. BRIEF DESCRIPTION OF DRAWINGS
[0018] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the application or the related art, the accompanying drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.
[0020] Figure 1 An optional data coordination system based on an LSTM prediction model according to an embodiment of the application is shown in the schematic diagram.
[0021] Figure 2 Another optional data coordination system based on an LSTM prediction model according to an embodiment of the application is shown in the schematic diagram. DETAILED DESCRIPTION
[0022] In order to make the objects, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.
[0023] In the following description, the suffixes such as "module", "component", or "unit" used for an element are used only for convenience of explanation of the present application, and have no specific meaning by themselves. Thus, "module" and "component" can be used interchangeably.
[0024] With the rapid evolution of large language models, multi-modal models, and other large model technologies, the model parameter size has jumped from tens of billions to hundreds of billions, and the training data volume has also increased exponentially. The efficiency of model training not only depends on the computing power of the computing cluster, but also is limited by the loading speed of the training data. If the data set cannot be quickly and stably transmitted from the storage system to the computing node, the computing resources will be in an idle state for a long time, directly slowing down the training process.
[0025] In actual production environment, there is a significant "mismatch" feature in the storage and computing architecture of large model training, which is specifically manifested as:
[0026] Enterprise-level large model training data sets (such as text corpus, image library, multi-modal interaction data, etc.) are usually stored in a single type of large-capacity storage system (such as a distributed file system, object storage, etc.). Due to hardware cost and maintenance complexity considerations, such storage systems often do not use the highest performance media (such as all-flash arrays), and to avoid redundancy overhead, they are usually not deployed across multiple regions with multiple copies, and the data physical location is relatively fixed.
[0027] The deployment of the training cluster is highly flexible and deeply bound to the business scenario: training tasks of different business lines may be distributed in different geographic regions; even for the same business, to balance the load or for disaster recovery needs, redundant training clusters may be deployed in multiple regions. More importantly, the scheduling of training tasks is dynamic, and the cluster location may be adjusted in real time according to business needs (such as resource expansion, cost optimization).
[0028] The current "fixed data, mobile computing" architecture contradiction is that when performing model training, temporarily loading the training cluster across regions can be very time-consuming, affecting the efficiency of large model training.
[0029] Therefore, how to provide efficient data set loading capability for the training cluster deployed in multiple regions dynamically has become a key problem to be solved for storage systems in the era of large models.
[0030] In order to solve the problems mentioned in the background art, according to an aspect of an embodiment of the present application, a data coordination system based on an LSTM prediction model is provided, as shown in Figure 1 The data coordination system includes a preheating device 101, a data storage device 102, and a distributed cache device 103.
[0031] The preheating device 101 is configured to generate a preheating request according to a prediction result of an LSTM prediction model, and send the preheating request to a target cluster corresponding to the prediction result, the LSTM prediction model being configured to predict future data loading demand; the data storage device 102 is configured to store data sets of a plurality of businesses; and the distributed cache device 103 includes a plurality of cache clusters deployed in different regions, and a target cluster in the cache clusters is configured to read a target data set corresponding to the preheating request from the data storage device 102 and cache the target data set to a local storage to provide a data loading service for a client in a case where the preheating request is received.
[0032] The application is directed to a scenario in which a training cluster is fixed in position and a plurality of positions are dispersed in different regions, and a distributed cache and an intelligent data preheating strategy are used to improve the data set loading speed of a business.
[0033] The data storage device is deployed in a single region and centrally manages data sets of a plurality of businesses. The data sets are independent of each other and are stored in different data volumes of the storage system, and are isolated from each other to ensure that the data do not interfere with each other. For example, a storage system can include a data set of business A, a data set of business B, a data set of business C, and a data set of business D.
[0034] The distributed cache device includes a plurality of cache clusters, each cluster corresponding to a physical region, and each cluster including a plurality of nodes, wherein a control node is responsible for receiving a preheating request and managing a cluster state. After receiving the preheating request, a target data set of a specified business is read from the data storage device and cached to a local storage in advance to reduce client waiting time. When a client request for data is received, the target data set can be directly provided to the client.
[0035] Each cluster can include a plurality of machine nodes, one of which exposes a management port to provide an HTTP API (Hypertext Transfer Protocol Application Programming Interface) for preheating data.
[0036] The preheating device predicts future download demand based on historical data (i.e., traffic indicators of each cache cluster), and actively triggers a preheating operation of the distributed cache device to enable the required data to be cached to a local storage in advance. Through a "prediction-preheating" closed loop, passive responses to client requests are converted to active adaptation to demand, and data loading delay is reduced from the source.
[0037] The application predicts future data loading demand through an LSTM model, generates a request by a preheating device and sends the request to a cache cluster in a corresponding region, the cache cluster reads a target data set from a data storage device and locally caches the target data set to provide a client with a fast data loading service in proximity.
[0038] In one aspect, the application is directed to the problem of cross-region data loading, and a cache system is deployed in different regions to cache data loaded from a data storage device, so that a training cluster can read data in proximity to improve loading speed. In another aspect, the application is directed to the problem that different businesses have different loading times, and an LSTM is used to predict the training time of different businesses, and then targeted data preheating is performed in advance to preheat the required data to the local storage of the corresponding region, thereby maximizing the training efficiency.
[0039] As an optional embodiment, the system further comprises a monitoring device for monitoring traffic indicators of each cache cluster in the distributed cache device and storing the traffic indicators in the data collection module, wherein the traffic indicators include a data download time, a business name to which the data belongs and a data download region.
[0040] The monitoring device is in communication connection with the control nodes of each cache cluster.
[0041] At each data download, the control node of the cache cluster automatically records the data download time, the business name to which the data belongs and the data download region, and sends them to the monitoring device in real time to provide training data for the LSTM prediction model.
[0042] As an optional embodiment, the preheating device comprises an LSTM prediction module, which is configured to obtain a plurality of traffic indicators in the data collection module from the monitoring device and perform model training on the plurality of traffic indicators as training data to obtain an LSTM prediction model.
[0043] The LSTM prediction model can learn long-term dependencies in sequence data by introducing a "gating mechanism" and a "memory unit".
[0044] The application uses the historical traffic indicators (data download time, business name to which the data belongs and data download region) preprocessed by the monitoring device to train the model, so that the model learns to predict the possible download parameters (time point ts, region location, business business) next time.
[0045] As an optional embodiment, the preheating device comprises a cluster management module, which is configured to receive and record cluster information sent by the control node of each cache cluster, wherein the cluster information includes an IP address of the control node, a port number and a region name of the region to which the control node belongs.
[0046] For example, each cache cluster has its own control node, which sends its address information, including IP address and port number, to the cluster management module, which records the IP addresses, port numbers and region information of all cache control nodes.
[0047] The cluster management module can maintain the IP addresses, port numbers and corresponding region information of the control nodes of all cache clusters, which can be regarded as an address query library for preheating requests.
[0048] As an optional embodiment, the preheating device comprises a preheating control module, which is configured to obtain a prediction result from the LSTM prediction module, extract a target service name and a target region name of a target data set to be loaded from the prediction result, determine a target IP address and a target port number of a control node corresponding to the target region name through the cluster management module, and generate a preheating request according to the target service name, the target region name, the target IP address and the target port number.
[0049] For example, the preheating control module can obtain the target IP address and the target port number of the control node of the target region from the cluster management module before the next possible download time ts according to the prediction result, and send a preheating request to make the distributed cache device load data in advance.
[0050] The preheating request can be understood as a simulation request sent by the system or service before processing the user traffic formally, and the purpose is to make the system complete initialization, resource loading or state preparation in advance, so as to avoid performance fluctuation or delay when the first real request is sent.
[0051] The application provides an API format of the preheating request: http: / / <ip> : <port> / warmup?location= <location>&business= <business>wherein, ip and port represent the IP address and port number of the cache cluster management node, location is the area name where the cache is located, such as beijing, wuhan, etc., business represents the business name to which the data to be preheated belongs, and the business name corresponds to the name of the data volume of the business in the storage, which can be a subdirectory or a PV (Persistent Volume), depending on the actual design of the storage system.
[0052] The present application is described with respect to a target cluster, and in fact, any cache cluster will read the data of the volume corresponding to the business into its own cache after receiving the preheating request, and then wait for the business client to read.
[0053] As an optional embodiment, the preheating control module is further configured to extract a target loading time from the prediction result, and send a preheating request to the target IP address and the target port number before the target loading time.
[0054] The most basic requirement is to send a preheating request to the target IP address and the target port number at the target loading time. In fact, the transmission speed of information needs to be considered, so the optimal solution is to send a preheating request to the target IP address and the target port number at a time not less than a preset time length from the target loading time.
[0055] The preset time length can be set and adjusted according to the actual data transmission speed and other conditions.
[0056] As an optional embodiment, the data collection module caches the traffic indicators of each cache cluster, and the LSTM prediction module performs data preprocessing on each traffic indicator for model training.
[0057] When the LSTM prediction model is obtained through model training, first, the data of the data collection module is preprocessed to obtain a training set (cleaning data, removing outliers, filling missing values, data normalization, extracting useful features, and generating a training set), and then the LSTM prediction model is trained using the training set.
[0058] For example, the outliers can be processed by related detection methods (such as statistical methods (such as 3σ principle), visualization (box plot, scatter plot), and machine learning methods (isolated forest)).
[0059] For example, missing value filling can be performed by forward fill (FFill), backward fill (BFill), mean / median, interpolation (e.g., KNN imputation), or other models (e.g., random forest) to predict missing values and then fill them.
[0060] In addition, if the proportion of missing values of certain values is high (e.g., > 50%) or the data is missing completely at random (MCAR), the data can be directly deleted.
[0061] The training process of the LSTM prediction model has a relatively mature similar procedure in the prior art. Since the training process of the model is not the focus of the present application, the present application does not limit and specially describe it.
[0062] As an optional embodiment, the target cluster is configured to extract the target service name from the preheating request, read the target data set corresponding to the target service name from the data storage device, and cache the target data set to the local storage.
[0063] After receiving the preheating request, the target service name can be parsed directly from the preheating request, and then the target data set corresponding to the target service name can be read and cached from the data storage device.
[0064] Caching the downloaded target data set to the local storage can enable the client to quickly and directly obtain the target data set from the local cache when the target data set is needed, without temporarily calling across regions, thereby greatly improving the data acquisition efficiency.
[0065] As an optional embodiment, the cluster management module is further configured to establish a corresponding relationship between the control node, the IP address, the port number, and the region name of each cache cluster and save the corresponding relationship after receiving the IP address, the port number, and the region name corresponding to the control node of each cache cluster.
[0066] Each cache cluster has its own control node, which sends its address information, including the IP address and the port number, to the cluster management module. The cluster management module records the IP address, the port number, and the region information of the control node of each cache cluster.
[0067] The corresponding relationship between the control node, the IP address, the port number, and the region name of each cache cluster is established, so that the cluster management module can provide the corresponding IP address and port number as long as the preheating control module provides the region name.
[0068] Figure 2 Another schematic diagram provided in the present application is shown in the figure. The data collaboration system based on the LSTM prediction model comprises four devices, the names and functions of which are as follows:
[0069] 1. Data storage device: single-area centralized storage, bearing multiple independent data sets of services. Different service data (such as service A data, service B data, service C data, service D data, etc.) are isolated by data volumes (which may be subdirectories or PVs, depending on the storage design), independent of each other and not interfering with each other.
[0070] 2. Distributed cache device: cross-regional distributed cache cluster (including multiple clusters, cache cluster 1, cache cluster 2, and cache cluster 3, etc.), optimizing client data download speed. Read data from the data storage device and cache in the local area to provide "near download" services for clients. Deploy cache clusters by region, and each cluster contains multiple machine nodes, one of which is a management node that exposes HTTP API for preheating operations. After triggering preheating, the cache cluster will actively read the data of the specified service from the data storage device and cache, waiting for client access.
[0071] 3. Monitoring device: cache cluster traffic index collection and storage. Data read time, service name, read region, and data are finally stored in the "data collection module" to provide training data for subsequent model training.
[0072] 4. Intelligent preheating device (corresponding to the preheating device of the present application, containing three modules): LSTM prediction module: train the LSTM model based on the traffic data (i.e., traffic indicators) of the monitoring device to predict future data loading demand; cache cluster management module (corresponding to the cluster management module of the present application): maintain the meta information of all cache clusters; preheating control module (corresponding to the control module of the present application): trigger active preheating according to the LSTM prediction results.
[0073] The data collaboration system based on the LSTM prediction model of the present application adopts a closed-loop design, realizing the full-process intelligentization of "predicting demand based on historical data - actively preheating cache - improving access speed - data feedback optimizing model", and maximizing client download efficiency by pre-caching the required data.
[0074] The application provides a data coordination system based on an LSTM prediction model, comprising a preheating device, a data storage device and a distributed cache device, wherein the preheating device is configured to generate a preheating request according to a prediction result of the LSTM prediction model and send the preheating request to a target cluster corresponding to the prediction result, and the LSTM prediction model is configured to predict future data loading demand; the data storage device is configured to store data sets of a plurality of businesses; and the distributed cache device comprises a plurality of cache clusters deployed in different regions, and a target cluster in the cache clusters is configured to read a target data set corresponding to the preheating request from the data storage device and cache the target data set to a local storage to provide data loading service for a client in the case that the preheating request is received. The future data loading demand is predicted by the LSTM model, the preheating device generates a request and sends it to the cache cluster in the corresponding region, the cluster reads the target data set from the data storage device and caches it locally to provide fast data loading service for the client nearby. The problem of how to provide efficient data set loading capability for the training cluster deployed dynamically in multiple regions is solved.
[0075] According to another aspect of the embodiments of the application, the application provides an electronic device comprising the above-mentioned data coordination system based on an LSTM prediction model.
[0076] Optionally, specific examples in the embodiments can refer to the examples described in the above-mentioned embodiments, and the embodiments will not be described here again.
[0077] When the embodiments of the application are implemented, reference can be made to the above-mentioned embodiments, and the corresponding technical effects are achieved.
[0078] It can be understood that the embodiments described herein can be realized by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units for performing the functions described in the application or a combination thereof.
[0079] For software implementation, the technologies described herein can be realized by units performing the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.
[0080] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0081] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.
[0082] In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other ways. For example, the apparatus embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and an actual implementation can be another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.
[0083] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
[0084] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.
[0085] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various program codes that can be stored in the medium. It should be noted that, in this document, relationship terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the elements defined by the statement "including a" do not exclude the presence of other identical elements in the processes, methods, articles or devices that include the elements.
[0086] The above description is only a specific implementation of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.< / business> < / location> < / port> < / ip>
Claims
1. A data collaboration system based on LSTM prediction model, characterized in that, The system comprises a preheating device, a data storage device and a distributed cache device, wherein, The preheating device is configured to generate a preheating request according to a prediction result of an LSTM prediction model, and send the preheating request to a target cluster corresponding to the prediction result, wherein the LSTM prediction model is configured to predict future data loading demand; The data storage device is configured to store data sets of multiple businesses; The distributed cache device comprises multiple cache clusters deployed in different regions, and the target cluster in the cache clusters is configured to read a target data set corresponding to the preheating request from the data storage device and cache the target data set to a local storage to provide data loading service for a client when the preheating request is received.
2. The system of claim 1, wherein, The system further comprises a monitoring device configured to monitor traffic indicators of each cache cluster in the distributed cache device and store the traffic indicators in a data collection module, wherein the traffic indicators comprise data download time, business name to which the data belongs and data download region.
3. The system of claim 2, wherein, The preheating device comprises an LSTM prediction module configured to obtain multiple traffic indicators in the data collection module from the monitoring device, and perform model training by taking the multiple traffic indicators as training data to obtain the LSTM prediction model.
4. The system of claim 3, wherein, The preheating device comprises a cluster management module configured to receive and record cluster information issued by a control node of each cache cluster, wherein the cluster information comprises IP address of the control node, port number and region name of a region to which the control node belongs.
5. The system of claim 4, wherein, The preheating device comprises a preheating control module configured to obtain the prediction result from the LSTM prediction module, extract target business name and target region name of the target data set to be loaded from the prediction result; Determine target IP address and target port number of the control node corresponding to the target region name through the cluster management module, and generate the preheating request according to the target business name, the target region name, the target IP address and the target port number.
6. The system of claim 5, wherein, The preheating control module is further configured to extract a target loading time from the prediction result, and send the preheating request to the target IP address and the target port number before the target loading time.
7. The system of claim 2, wherein, The data collection module caches the traffic indicators of each cache cluster, and the LSTM prediction module performs data preprocessing on each traffic indicator for model training.
8. The system of claim 1, wherein, The target cluster is configured to extract target business name from the preheating request, read the target data set corresponding to the target business name from the data storage device, and cache the target data set to the local storage.
9. The system of claim 4, wherein, The cluster management module is further configured to, after receiving the IP address, the port number and the zone name corresponding to the control node of each cache cluster, establish a corresponding relationship between the control node, the IP address, the port number and the zone name of each cache cluster and save the corresponding relationship. 10.An electronic device comprising the data collaboration system based on the LSTM prediction model according to any one of claims 1 to 9.