A massive data caching method based on message queue and logical storage

Abstract

The present invention relates to the technical field of data storage, and specifically provides a method for caching massive data based on message queues and logical storage, comprising: pre-setting an index heap based on message queue arrangement, distributed middleware for logical storage, and a cloud storage cluster for data caching; wherein the cloud storage cluster is configured with a cache code, and the cache code corresponds to a unique data address; setting a matching mechanism between the index heap and the distributed middleware to form a dedicated transmission link; setting a request mechanism between the distributed middleware and the cloud storage cluster to form a cache application response; and caching data when massive data exists based on the cache application response and the dedicated transmission link.

Description

Technical Field

[0001] The present invention relates to the technical field of data caching, and in particular to a mass data caching method based on message queues and logical storage. Background Art

[0002] At present, in today's information society, most enterprises and institutions have their own information software systems. These units often need to display report data. These report data often have multiple data sources, complex data organization logic processes, and strong requirements for personalized display.

[0003] In software systems, the traditional approach involves logically processing unit data according to different query conditions, saving the processed data to a database table, and then generating reports based on user-specific display formats when users query the data. However, this approach has significant drawbacks: slow query speeds, high server pressure from multiple concurrent users, and the inability to achieve real-time query and instant display.

[0004] In patent document 202110627991.6, a storage method, apparatus, device, and readable medium for a distributed message queue is described, which aims to solve network partitioning, request interaction, and metadata consistency issues under a topic by establishing an interaction mechanism within the distributed message queue node, thereby achieving data storage.

[0005] However, during the storage process, point-to-point interaction can be achieved for relatively small amounts of data. If it is a massive amount of data, it needs to be stored. In this case, it needs to respond one by one according to the information type, and then perform point-to-point storage. In this way, the storage efficiency is low, and it will lead to technical problems such as slow caching speed and high pressure on multi-user concurrent servers. Summary of the Invention

[0006] The present invention provides a mass data caching method based on message queue and logic storage, which is used to solve the problem in the prior art that, for the storage of mass data, it is impossible to respond to requests one by one according to information types and realize point-to-point storage.

[0007] The present invention proposes a mass data caching method based on message queues and logical storage, comprising:

[0008] Pre-set the index heap based on message queue arrangement, distributed middleware for logical storage and cloud storage cluster for data cache;

[0009] The cloud storage cluster is configured with a cache code, which corresponds to a unique data address;

[0010] Set up a matching mechanism between the index heap and distributed middleware to form a dedicated transmission link;

[0011] Set up a request mechanism for distributed middleware and cloud storage clusters to form a cache request response;

[0012] Based on cache request responses and dedicated transmission links, data caching is performed when there is massive data.

[0013] Preferably, the message queue arrangement is determined based on a trigger mode; wherein,

[0014] Trigger modes include time mode, data source mode and event mode.

[0015] Preferably, the index stack is composed of index values ​​and cluster feature trees; wherein,

[0016] The index value includes a first index value, a second index value, and a third index value, wherein the first index value is an execution feature of the data to be stored, the second index value is a path feature of the data to be stored, and the third index value is a content feature of the data to be stored;

[0017] The index value and the clustering feature tree are fused to form an index heap;

[0018] The clustering feature tree includes the density and feature depth corresponding to different index values.

[0019] Preferably, the distributed middleware is composed of a distributed interface call library and a management component; wherein,

[0020] The distributed interface call library is associated with the index heap, and the corresponding data interface is associated with the index value of the index heap;

[0021] Management components include: process management component, storage management component, communication service component, process communication component and log component.

[0022] Preferably, the cloud storage cluster includes a virtual storage layer, a cache processing layer, a timing calculation layer and a data forwarding layer; wherein,

[0023] The virtual storage layer is configured with a data identification unit associated with the distributed middleware;

[0024] The cache processing layer is used to record data and divide the data storage area and storage space in the virtual storage layer;

[0025] The timing calculation layer is used to perform storage timing on the cache data in each storage space and determine the storage margin;

[0026] The data forwarding layer is used to call and forward data in the virtual storage layer when there is a data read request.

[0027] Preferably, the cache encoding includes the following steps:

[0028] Based on the message queue, the queue position of the cached data is marked to determine the amount of data to be cached;

[0029] When there is only one cache request in the message queue, an encoding macro is constructed to extract data features and encode the data.

[0030] When there are multiple cache requests in the cache queue, the data vector is calculated, and encoding is performed once according to the data vector. After the encoding, the data features are extracted and secondary encoding is performed.

[0031] Preferably, the setting of the matching mechanism between the index heap and the distributed middleware includes the following steps:

[0032] According to the index heap, construct the data feature matrix;

[0033] Build a data format matrix based on distributed middleware;

[0034] The vector analysis model is used to calculate the correlation between the data feature matrix and the data format matrix to determine the fusion representation vector;

[0035] The vector analysis model is trained using positive and negative sample pairs and adversarial samples;

[0036] The fusion representation vector is obtained by adjusting the model parameters based on the gradient descent algorithm of Nesterov momentum and the decoupled weight decay strategy;

[0037] According to the fused representation vector, a correlation matching mechanism based on the ANNOY index tree is constructed.

[0038] Preferably, the dedicated transmission link is constructed through the following steps:

[0039] Based on the matching mechanism between the index heap and the distributed middleware, a multi-channel transmission link is set up;

[0040] Set up encryption mechanism for each channel of multi-channel transmission link;

[0041] According to the encryption mechanism, when the data to be cached is transmitted, any transmission link is set in the multi-channel transmission link, and the link encryption is performed to generate a dedicated transmission link.

[0042] Preferably, the setting of the request mechanism of the distributed middleware and the cloud storage cluster includes the following steps:

[0043] Set up a response mechanism in the cloud storage cluster;

[0044] According to the response mechanism, set the version trigger mechanism for each distributed interface of the distributed middleware;

[0045] Through the version trigger mechanism, when there is data access in the distributed middleware, a data cache request is generated, and the cache address in the cloud storage cluster is automatically determined through the response mechanism.

[0046] Preferably, the data cache further includes:

[0047] Classify massive data and pre-process data according to data type;

[0048] Data preprocessing includes data decomposition, data classification and dividing each data into hierarchical categories;

[0049] Determine analytical information for massive data based on data preprocessing;

[0050] Analytical information includes: time series arrangement, element classification and data behavior analysis;

[0051] According to the analysis information, corresponding message queue response mechanisms are set up respectively, and the cache information of massive data is updated according to the message queue response mechanism.

[0052] The beneficial effects of the present invention are:

[0053] This application can achieve fast data storage. When massive amounts of data need to be cached, it can classify the data and then match the data index terms with the middleware interface based on the data index terms to generate a special transmission link and store the data in any storage area of ​​the cloud cache cluster.

[0054] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0055] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0057] Figure 1 A method for caching massive data based on message queues and logical storage in an embodiment of the present invention;

[0058] Figure 2 This is a flowchart of the matching mechanism between the index heap and the distributed middleware in an embodiment of the present invention;

[0059] Figure 34 is a flow chart of the construction of a dedicated transmission link in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0061] The present invention relates to the field of data storage technology, and specifically proposes a mass data caching method based on message queues and logical storage, comprising:

[0062] Pre-set the index heap based on message queue arrangement, distributed middleware for logical storage and cloud storage cluster for data cache;

[0063] The cloud storage cluster is configured with a cache code, which corresponds to a unique data address;

[0064] Set up a matching mechanism between the index heap and distributed middleware to form a dedicated transmission link;

[0065] Set up a request mechanism for distributed middleware and cloud storage clusters to form a cache request response;

[0066] Based on cache request responses and dedicated transmission links, data caching is performed when there is massive data.

[0067] The principle of the above technical solution is:

[0068] As attached Figure 1 As shown, in the process of data caching, the application first needs to quickly classify and parse the data, build a data classification index mechanism framework, and perform data processing;

[0069] Therefore, this application first sets up an index heap arranged by a message queue. When there is a message request for data storage, the characteristics of these messages can be extracted according to the message queue method, and an index heap of these messages can be constructed. The index heap can extract the index method and index words according to the message request. The index heap will be associated with the interface of the distributed middleware. When the data index queries the data characteristics of the cached data, the data selection of the data interface can be implemented based on the data characteristics. The technical data interface encodes the data or transforms the format of the data, so that it can be cached faster and more conveniently, and finally the data is stored in the cloud storage cluster.

[0070] The distributed middleware for logical storage is a data conversion middleware that is associated with data of different formats in the index heap and is used to convert data formats. Each format conversion middleware in the distributed middleware is a basic middleware node, and each individual middleware is a basic unit of data format conversion middleware. Multiple middlewares that process the same type but different formats are called a middleware integrator. For example, a middleware integrator for multiple image formats would integrate multiple image processing middlewares.

[0071] A cloud storage distribution cluster is a distributed data storage cluster. Each storage unit in the cluster corresponds to a dedicated transmission link. Therefore, each storage unit is set with a corresponding cache code, and the unique address of each storage unit is set through the cache code.

[0072] The dedicated transmission link can ensure the speed of data storage when caching data. Each middleware in the dedicated transmission link is equipped with a separate data transmission channel to ensure that the same type of data can be transmitted quickly.

[0073] When there is a cache requirement, the cache request response can generate a corresponding response to classify massive data and convert and transmit it through distributed middleware, and then build a special transmission link to store the corresponding data.

[0074] The beneficial effects of the above technical solution are:

[0075] This application can achieve fast data storage. When massive amounts of data need to be cached, it can classify the data and then match the data index terms with the middleware interface based on the data index terms to generate a special transmission link and store the data in any storage area of ​​the cloud cache cluster.

[0076] Preferably, the message queue arrangement is determined based on a trigger mode; wherein,

[0077] Trigger modes include time mode, data source mode and event mode.

[0078] The principle of the above technical solution is:

[0079] When arranging the message queue, this application can use different trigger modes such as time mode, data source mode and event mode according to the data characteristics of massive data to arrange the data message queue.

[0080] The beneficial effects of the above technical solution are:

[0081] The time model allows massive amounts of data to be stored sequentially along the timeline, preventing data order deviations. The data source model allows data to be stored according to the importance of the data source, ensuring smooth data traceability. The event model divides data according to event occurrence, process, and result to ensure data integrity.

[0082] Specifically, the index stack is composed of index values ​​and cluster feature trees; wherein,

[0083] The index value includes a first index value, a second index value, and a third index value, wherein the first index value is an execution feature of the data to be stored, the second index value is a path feature of the data to be stored, and the third index value is a content feature of the data to be stored;

[0084] The index value and the clustering feature tree are merged to form an index heap;

[0085] The clustering feature tree includes the density and feature depth corresponding to different index values.

[0086] The principle of the above technical solution is:

[0087] During the storage process, this application can perform refined data segmentation because the index heap is based on index values ​​and cluster feature trees. The cluster feature tree can divide massive amounts of data according to index values. According to the data hierarchy, each index value corresponds to a node, thus dividing related data into clusters and realizing index retrieval of data.

[0088] The index heap composed of index values ​​and feature trees can be used to determine the density of the data. The density is used to determine the total amount of data and is a cardinality function of the topological space. The feature depth represents the data profiling of each class of data in terms of format, data volume, and data content. The feature depth of the final data is determined based on the depth of the data profiling.

[0089] The beneficial effects of the above technical solution are:

[0090] This application can set up a data index heap according to different data types, then divide the data features through the data index heap, and then fuse the data features according to the clustering feature tree to realize the construction of the index heap and classify massive data.

[0091] Specifically, the distributed middleware is composed of a distributed interface call library and a management component; wherein,

[0092] The distributed interface call library is associated with the index heap, and the corresponding data interface is associated with the index value of the index heap;

[0093] Management components include: process management component, storage management component, communication service component, process communication component and log component.

[0094] The principle of the above technical solution is:

[0095] The distributed interface call library is an interface call database that stores many middlewares, and associates the corresponding data interfaces through index values ​​to implement interface calls; the process management component includes processes responsible for creation, loading, execution, suspension, resumption and termination; the storage management component includes memory allocation and release, memory protection, data illegal access maintenance, etc.; the communication service component includes loss, rereading, disorder, flow control, routing selection, etc.; the process communication component is mainly a protocol component, which is used for message sharing, message queues, message format conversion, message synchronization, etc.; the log component includes parameters, status, exceptions, errors, etc. during program runtime, which can be used for subsequent analysis, debugging, auditing, etc.

[0096] The beneficial effects of the above technical solution are:

[0097] This application can build a partial verification middleware, set up a distributed data interface based on different index values ​​in the index heap, and manage components through the distributed data interface, including process management components, storage management components, communication service components, process communication components and log components.

[0098] Specifically, the cloud storage cluster includes a virtual storage layer, a cache processing layer, a timing calculation layer and a data forwarding layer; wherein,

[0099] The virtual storage layer is configured with a data identification unit associated with the distributed middleware;

[0100] The cache processing layer is used to record data and divide the data storage area and storage space in the virtual storage layer;

[0101] The timing calculation layer is used to perform storage timing on the cache data in each storage space and determine the storage margin;

[0102] The data forwarding layer is used to call and forward data in the virtual storage layer when there is a data read request.

[0103] The principle of the above technical solution is:

[0104] The cloud storage cluster of this application includes a pseudo-storage layer, a cache processing layer, a timing calculation layer, and a data forwarding layer;

[0105] The virtual storage layer is a storage area for data caching;

[0106] The cache processing layer can divide the data storage area and configure the storage space;

[0107] The timing calculation layer can calculate the space margin of the entire virtual storage layer;

[0108] The data forwarding layer can determine the data cache address and forward the data based on the data reading request.

[0109] The beneficial effects of the above technical solution are:

[0110] By dividing the number of layers in the above cloud storage cluster, massive amounts of data can be quickly stored, calculated, and addressed.

[0111] Specifically, the cache encoding includes the following steps:

[0112] Based on the message queue, the queue position of the cached data is marked to determine the amount of data to be cached;

[0113] When there is only one cache request in the message queue, an encoding macro is constructed to extract data features and encode the data.

[0114] When there are multiple cache requests in the cache queue, the data vector is calculated, and encoding is performed once according to the data vector. After the encoding, the data features are extracted and secondary encoding is performed.

[0115] The principle of the above technical solution is:

[0116] During cache encoding, this application uses a message queue to perform queue calibration on cached data and calculate the specific amount of cached data. Based on the specific amount of cached data, multiple cached data and single cached data can be processed separately according to cache requests. Single data is encoded and stored using programming macros. When multiple cache requests are made, repeated encoding and caching can be performed by calculating data vectors.

[0117] The beneficial effects of the above technical solution are:

[0118] Through cache coding, data can be encoded and stored based on data features when storing data. Data vectors can also be calculated (data vectors are used to identify data in two-dimensional space or three-dimensional space) and then stored. The data vectors already occupy a very small data space, but the data vectors can still be encoded again through data features, so data encoding can be used to encode the data again and store the corresponding data.

[0119] Specifically, setting the matching mechanism between the index heap and the distributed middleware includes the following steps:

[0120] According to the index heap, construct the data feature matrix;

[0121] Build a data format matrix based on distributed middleware;

[0122] The vector analysis model is used to calculate the correlation between the data feature matrix and the data format matrix to determine the fusion representation vector;

[0123] The vector analysis model is trained using positive and negative sample pairs and adversarial samples;

[0124] The fusion representation vector is obtained by adjusting the model parameters based on the gradient descent algorithm of Nesterov momentum and the decoupled weight decay strategy;

[0125] According to the fused representation vector, a correlation matching mechanism based on the ANNOY index tree is constructed.

[0126] The principle of the above technical solution is:

[0127] As attached Figure 2 As shown, when setting up the matching mechanism, the distributed middleware and index heap both have a large number of data types. Therefore, in the process of data matching, by constructing the data feature matrix and the data format matrix, a vector analysis model is built to realize the calculation of the representation vector, and finally, the association matching is realized through the ANNOY index tree.

[0128] The data format matrix organizes and stores data in a matrix format. Within the two dimensions of rows and columns, one dimension represents data content features, and the other represents the data format. Vector analysis is then used to determine the correlation between the data feature matrix and the data format matrix, and the corresponding fused representation vector is determined. Nesterov momentum is an optimization algorithm used to reduce the gradient of the loss function for sample data, namely the data feature matrix and the data format matrix. The weight decay strategy adjustment model is an adjustment model based on policy-based techniques to suppress overfitting of massive data. Fusion during the gradient descent of the loss function causes model weights to tend toward smaller values. By fusing representation vectors, a more accurate correlation matching mechanism can be constructed. The ANNOY index tree is an open-source algorithm for quickly finding the top K vectors of similar distance from massive high-dimensional vectors. However, traditionally, this algorithm has only been used to partition data into multiple sub-controls to accelerate high-dimensional data searches.

[0129] The beneficial effects of the above technical solution are:

[0130] The correlation matching mechanism constructed in this application is not only more accurate when matching, but also associated with the index heap, both of which are index tree structures, which are faster and more convenient when matching middleware and data.

[0131] Specifically, the dedicated transmission link is constructed through the following steps:

[0132] Based on the matching mechanism between the index heap and the distributed middleware, a multi-channel transmission link is set up;

[0133] Set up encryption mechanism for each channel of multi-channel transmission link;

[0134] According to the encryption mechanism, when the data to be cached is transmitted, any transmission link is set in the multi-channel transmission link, and the link encryption is performed to generate a dedicated transmission link.

[0135] The principle of the above technical solution is:

[0136] As attached Figure 3 As shown, in the process of constructing a dedicated transmission link, the multi-channel transmission link can set different encryption mechanisms according to each different channel, and then when performing cache data transmission, any transmission link can be selected for transmission. If an encryption mechanism exists, this transmission channel is a data dedicated transmission link.

[0137] The beneficial effects of the above technical solution are:

[0138] By building an encryption mechanism, any transmission link is set up in multiple transmission links for encryption, and data is encrypted and transmitted after encryption.

[0139] Specifically, the request mechanism for setting up the distributed middleware and the cloud storage cluster includes the following steps:

[0140] Set up a response mechanism in the cloud storage cluster;

[0141] According to the response mechanism, set the version trigger mechanism for each distributed interface of the distributed middleware;

[0142] Through the version trigger mechanism, when there is data access in the distributed middleware, a data cache request is generated, and the cache address in the cloud storage cluster is automatically determined through the response mechanism.

[0143] The principle of the above technical solution is:

[0144] This application will set up a response mechanism, associate the response mechanism with the distributed interface, and trigger the corresponding version interface. Therefore, after being triggered, the distributed interface of this application will respond to the corresponding distributed middleware and the corresponding cache address and special transmission channel in the cloud storage cluster, thereby achieving rapid response and rapid caching of data.

[0145] The beneficial effects of the above technical solution are:

[0146] This application sets up a request mechanism, which can be associated with a version trigger mechanism. The version trigger mechanism locates and caches data in the cloud cache cluster through a cache request and response mechanism when data is accessed.

[0147] Specifically, the data cache further includes:

[0148] Classify massive data and pre-process data according to data type;

[0149] Data preprocessing includes data decomposition, data classification and dividing each data into hierarchical categories;

[0150] Determine analytical information for massive data based on data preprocessing;

[0151] Analytical information includes: time series arrangement, element classification and data behavior analysis;

[0152] According to the analysis information, corresponding message queue response mechanisms are set up respectively, and the cache information of massive data is updated according to the message queue response mechanism.

[0153] The principle of the above technical solution is:

[0154] In the process of caching and storing massive data, this application will also classify the data, perform data preprocessing through data classification, and then perform data decomposition, data classification and hierarchical analysis based on the data preprocessing method, and then analyze the massive data. A massive data update mechanism - a message queue response mechanism is set up. When the time series arrangement, element category division and data behavior analysis of massive data change, a corresponding data response is made, and the cached massive data is updated accordingly.

[0155] The beneficial effects of the above technical solution are:

[0156] Through corresponding updates to massive data, this application can not only analyze the specific arrangement information, data classification information and data behavior information of massive data through hierarchical analysis, data classification and data decomposition of massive data, but also perform corresponding data processing when massive data changes, thereby ensuring the iterative update capability of data storage and the storage supervision capability of massive data.

[0157] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for caching massive data based on message queues and logical storage, characterized in that: include: Pre-set the index heap based on message queue arrangement, distributed middleware for logical storage and cloud storage cluster for data cache; The cloud storage cluster is configured with a cache code, which corresponds to a unique data address; Setting a matching mechanism between the index heap and the distributed middleware to form a dedicated transmission link; wherein setting a matching mechanism between the index heap and the distributed middleware includes the following steps: According to the index heap, construct the data feature matrix; Build a data format matrix based on distributed middleware; The vector analysis model is used to calculate the correlation between the data feature matrix and the data format matrix to determine the fusion representation vector; The vector analysis model is trained using positive and negative sample pairs and adversarial samples; The fusion representation vector is obtained by adjusting the model parameters based on the gradient descent algorithm of Nesterov momentum and the decoupled weight decay strategy; According to the fusion representation vector, a correlation matching mechanism based on the ANNOY index tree is constructed; The dedicated transmission link is constructed through the following steps: Based on the matching mechanism between the index heap and the distributed middleware, a multi-channel transmission link is set up; Set up encryption mechanism for each channel of multi-channel transmission link; According to the encryption mechanism, when the cached data is transmitted, any transmission link is set in the multi-channel transmission link, and the link encryption is performed to generate a dedicated transmission link; Setting a request mechanism for the distributed middleware and the cloud storage cluster to form a cache request response; wherein said setting a request mechanism for the distributed middleware and the cloud storage cluster comprises the following steps: Set up a response mechanism in the cloud storage cluster; According to the response mechanism, set the version trigger mechanism for each distributed interface of the distributed middleware; Through the version trigger mechanism, when data is accessed in the distributed middleware, a data cache request is generated, and the cache address in the cloud storage cluster is automatically determined through the response mechanism; Based on cache request responses and dedicated transmission links, data caching is performed when there is massive data.

2. A method for caching massive data based on message queues and logical storage according to claim 1, characterized in that: The message queue arrangement is determined based on the trigger mode; wherein, Trigger modes include time mode, data source mode and event mode.

3. A method for caching massive data based on message queues and logical storage according to claim 1, characterized in that: The index heap is composed of index values ​​and cluster feature trees; wherein, The index value includes a first index value, a second index value, and a third index value, wherein the first index value is an execution feature of the data to be stored, the second index value is a path feature of the data to be stored, and the third index value is a content feature of the data to be stored; The index value and the clustering feature tree are merged to form an index heap; The clustering feature tree includes the density and feature depth corresponding to different index values.

4. A method for caching massive data based on message queues and logical storage according to claim 1, characterized in that: The distributed middleware is composed of a distributed interface call library and management components; wherein, The distributed interface call library is associated with the index heap, and the corresponding data interface is associated with the index value of the index heap; Management components include: process management component, storage management component, communication service component, process communication component and log component.

5. The method for caching massive data based on message queues and logical storage according to claim 1, wherein: The cloud storage cluster includes a virtual storage layer, a cache processing layer, a timing calculation layer and a data forwarding layer; wherein, The virtual storage layer is configured with a data identification unit associated with the distributed middleware; The cache processing layer is used to record data and divide the data storage area and storage space in the virtual storage layer; The timing calculation layer is used to perform storage timing on the cache data in each storage space and determine the storage margin; The data forwarding layer is used to call and forward data in the virtual storage layer when there is a data read request.

6. A method for caching massive data based on message queues and logical storage according to claim 1, characterized in that: The cache encoding comprises the following steps: Based on the message queue, the queue position of the cached data is marked to determine the amount of data to be cached; When there is only one cache request in the message queue, an encoding macro is constructed to extract data features and encode the data. When there are multiple cache requests in the cache queue, the data vector is calculated, and encoding is performed once according to the data vector. After the encoding, the data features are extracted and secondary encoding is performed.

7. A method for caching massive data based on message queues and logical storage according to claim 1, characterized in that: The data cache further includes: Classify massive data and pre-process data according to data type; Data preprocessing includes data decomposition, data classification and dividing each data into hierarchical categories; Determine analytical information for massive data based on data preprocessing; Analytical information includes: time series arrangement, element classification and data behavior analysis; According to the analysis information, corresponding message queue response mechanisms are set up respectively, and the cache information of massive data is updated according to the message queue response mechanism.

Images